Pre-trained Language Models Improve the Few-shot  Prompt Ability of Decision Transformer
Authors: Yu Yang
Pan Xu
Venue: Accepted by TMLR ()

Abstract: Decision Transformer (DT) has emerged as a promising class of algorithms in offline reinforcement learning (RL) tasks, leveraging pre-collected datasets and Transformer's capability to model long sequences. Recent works have demonstrated that using parts of trajectories from training tasks as prompts in DT enhances its performance on unseen tasks, giving rise to Prompt-DT methods. However, collecting data from specific environments can be both costly and unsafe in many scenarios, leading to suboptimal performance and limited few-shot prompt abilities due to the data-hungry nature of Transformer-based models. Additionally, the limited datasets used in pre-training make it challenging for Prompt-DT type of methods to distinguish between various RL tasks through prompts alone. To address these challenges, we introduce the Language model-initialized Prompt Decision Transformer (LPDT) framework, which leverages pretrained language models providing rich prior knowledge for RL tasks and fine-tunes the sequence model using Low-rank Adaptation (LoRA) for meta-RL problems. We further incorporate prompt regularization to effectively differentiate between tasks based on prompt feature representations. Comprehensive empirical studies demonstrate that initializing with a pre-trained language model provides the prior knowledge and achieves a similar performance with Prompt-DT under only $10\%$ data in some MuJoCo control tasks. We also provide a thorough ablation study to validate the effectiveness of each component, including sequence modeling, language models, prompt regularizations, and prompt strategies.

Published in Transactions on Machine Learning Research (11/2025)
Pre-traing Language Models Improve the Few-shot Prompt Ability of Decision Transformer
Yu Yang
yn.yyog@@duhe.edl
Deportment of Electrical and Computer Engineering
Dake University
Pan Xu
pen..zu@ddueee edu
Deportment Biostatistics and Bioinformatiics Department of Computer Science
Department of Electrical and Computer Engineering Dake Universily
Reviewed on OpenReview: https: // openreview. net/ formm? id= k52013XEMK
Abstract
Decision Transformer (DT) has emerged a promising class of algorithms in offline rein- forcement learning (RL) lasks, leveraging pre-collected datasets and Transformer's capabil- ity to model long sequences. Recent works have demonstrated that using parts of trajectories from training tasks prompts in DT enhances its performance on unseen tasks, giving risc to Prompt-DT methods. However, collecting data from specific environments can be both costly and unsafe in many scenarios, leading to suboptimal performance and limited few-shot prompt abilities due to the data Sata-hungry nature of Transformer- based models. Additionally, the limited datasets used in pre-training make it challenging for Prompt-DT type of methods to distinguish between various RT tasks through prompts alone. To address these challenges, introduce the Language model-initialized Prompt Decision Transformer (LPDT) frame- work, which leverages pretrained language models providing rich prior knowledge for RL tasks and fine-tunes the sequence model using Low-rank Adaptation (LoRA) for meta-RL problems. We further incorporate prompt regularization to effectively differentiate between tasks based on prompt feature representations. Comprehensive empirical studies demon- strate that initializing with pre-trained language model provides the prior knowledge and achieves a similar performance with Prompt-DT under only 10% data in some MuJoCo control tasks. We also provide thorough ablation study to validate the effectiveness of each component, including sequence modeling, language models, prompt regularizations, and prompt strategies.
1 Introduction
In many sequential decision-making applications such as robotic manipulation and autonomous driving (Sinha et al., 2022; Kumar al., 2022), can be expensive or even unsafe for agents learn through trial- and-error in the environment. Offline reinforcement learning (RL) methods (Levine et al., 2020) have emerged powerful paradigm for optimizing agent policies without extensive online interaction. These methods learn optimal policy by leveraging pre-collected datasets obtained from set of behavior policies. Among them, Decision Transformer (DT) (Chen et al., 2021) and its successors (Wu et al., 2024; Zhuang et al., 2024) have become popular due to their scalability and training stability. DT models relurn-conditionel policy using the powerful Transformer architecture, solving RL as a sequence prediction problem in a supervised learning manner. It models the states, actions, and return-to-go from trajectories as the tokens of an input sequence, and generates actions conditioned on the return-to-go. Compared with dynamic programming- based offline RL methods (Kumar et al., 2019; Fujimoto et al., 2019; Kumar et al., 2020) that heavily rely

---

Published in Transactions on Machine Learning Research (11/2025)
the Markov Decision Process (MDP) assumption, DT can utilize entire trajectory histories to predict the next action, making it more applicable in partially observable environments where all past information is crucial for decision-making (Kaelbling et al., 1998; Ni et al., 2024). Furthermore, the supervised learning nature of DTs enhances the stability and scalability in the training process compared to dynamic programming algorithms based on Bellman equations (Chen el al., 2021; Janner et al., 2021; Zheng el al., 2022; Wu et al., 2024; Zhuang et al., 2024).
Transformers have demonstrated remarkable few-shot generalization capabilities, most notably in large lan- guage models (LLMs) (Brown et al, 2020; Achiam et al., 2023). In the context of LLMs, prompt- based learning, where task-relevant information is provided as a textual prefix, has been proven highly effective for adapting to new tasks without fine-tuning (Brown et al., 2020; Li & Liang, 2021). Inspired by this, recent works (Xu et al., 2022; Hu et al., 2023; Wang et al., 2024b) have developed variants of DT to leverage trajectory-based prompting learning to enhance its few-shot generalization. For instance, Prompt-DT (Xu et al., 2022) leverages segments of trajectories from offline datasets as prompts to encode task-spocific infor- mation. Similar how large language models (LLLMs) conditioned on textual prompls to generate coherent responses, Prompt-DT conditions on trajectory-based prompts to generate actions. The model is trained on these prompt-trajectory pairs and is subsoquently evaluated on unseen tasks using few-shot demonstrations as prompts. However, existing Prompt-DT DDT methods (Xu et al., 2022; Hu et al., 2023; 2024; Wang et al., 2024b) inherit the data-hungry nature of Transformers (Brown et al., 2020; Achiam et al., 2023), requir- ing training on large trajectory datasets, while offline RL datasets are often too small to fully realize their few-shot prompting potential. In contrast, modern LLMs achieve strong generalization with limited task examples by leveraging massive unsupervised pre-training on text. Inspired by this, another line of works has investigated using pre-trained language models to initialize decision transformers and transfer their rich linguistic priors to RL domains (Li et al., 2022; Reid et al., 2022; Shi et al., 2024). While this inilialization provides valuable prior knowledge and helps alleviate the need for large datasets, existing studies have pri- marily focused on single-task RL problems. The potential for enhancing few-shot prompt learning of DTs in multi-task settings using pre-trained language models remains unexplored.
To overcome these limitations, we propose a novel framework named Language model-initialized Prompt Decision Transformer (LPDT). Our approach leverages pro-trained language model for initialization to significantly improve the few-shot prompting capabilities of Decision Transformers. This incorporates pre- existing knowledge that can benefit downstream RT, tasks. To efficiently combine this pre-trained knowledge with domain-specific knowledge from multi-task RL, we use Low-Rank Adaptation (LoRA) (Hu et al., 2021), a parameter-offiiien fine-tuning method. We further introduce prompt regularization methods to help the fine-tuned model distinguish different RL tasks, thereby guiding action generation based on new, unseen task- specific prompt representations. A more detailed illustration of our model structure and training paradigm is provided in Figure 1. Beyond Decision Transformer, we also extend our framework one of its most recent and advanced variants, Reinformer (Zhuang et al., 2024), demonstrating its flexibility. We conduct extensive experiments to assess the capability of our proposed framework in MuJoCo control environments (Fu et al., 2020) and Meta- World ML1 tasks (Yu et al., 2020). Our method outperforms basclines in terms of cumulative rewards on unseen lasks. Empirical studies show that our proposed framework achieves similar performance to Prompt-DT using only 10% of the dataset. We also provide a detailed ablation study to demonstrate the effectiveness of our LPDT framework. Specifically, we evaluate our proposed framework to validate cach component. Additionally, we report results on various language model initializations and different sequence modeling strategies. Furthermore, we design experiments to demonstrate that the improved performance stems from better initialization rather than from the language model's inherent understanding of RL tasks.
We propose LPDT, a novel framework that improves the few-shot prompt capabilities of Decision Trans- formers and other sequence modeling methods for offline RI. Our approach leverages a pre-trained lan- guage model for initialization and incorporates a novel prompt regularization method, demonstrating enhanced few-shot generalization in multi-task settings.
We introduce unique combination of Low-Rank Adaptation (ToRA) and prompt regularization met to effectively combine pre-trained knowledge with domain-specific RL task knowledge. LoRA allows elficient fine-tuning by adapting a small subset of parameters, while our regularization methods enhance the model's ability to distinguish task information within prompts.
2

---

Published in Transactions on Machine Learning Research (11/2025)
Frozen Weight
Waight
Discandel Weight
language model
correcting text Seoc>
%2
A E madr
Pretrained Language Model
E Rxx
LoRA
Rrxd
<hoc>
langwage model
correcting
Language Task
Task
Prompt Regularization
Figure 1: Overview of LPDT. We first initialize our algorithm using a pre-trained language model. The pre-trained language model is trained on large text corpus using the causal language modeling objective, ie., predicting the next token. Our method LPDT replaces the word embedding layers with linear layers, discarding the learned word features, to fully learn and capture the features RL trajectory tokens. Wo fine-lune our model using paraaneeer-fffcient methods like Low-Rank Adaplation (LoRA). Specifically, we freeze the initial weights of the language model and update only the LoRA weights. The input our approach consists of prompts accompanied by training trajectories from the same tasks. Unlike traditional models that predict word tokens, method predicts action tokens in the RL trajectories. Additionally, we incorporate prompt regularization on the input prompts. This is achieved by introducing an additional loss on the prompt embeddings, which helps LPDT distinguish between different environments. More technical details our method are presented in Section 3.
We extend our proposed framework, LPDT, to one of the most recent and advanced variants of DT, i.e., Reinformer, to demonstrate its gencralizability.
Through extensive experiments on MuJoCo control and World ML1, we show that LPDT outper- forms baselines unseen tasks and achieves similar performance to Prompt-DT with only 10% of the data in Cheetah-vel and Ant-dir. We also validate each component of our framework through detailed ablation studies.
2 Preliminary
2.1 Offline Reinforcement Learning
Reinforcement learning is usually formulated as solving a Markov Decision Process (MDP) defined by a tuple (S, A. do, R),,, where S represents the set of states S, A represents the set of actions E A, T is the transition distribution defined as [(s++1|$: at), dg is the distribution initial states sg. R: A + E is the reward function, rt R(st, at) is the reward at timestep and 7 E 10 is the discount factor. The objective to find a policy π that maximizes the expected cumulative rewards J(T):
J(T)= (aoo:t(())))aaa)))==)===))--)
Σ D ((s(s
In offline RL, the agent has access to a dataset D containing trajectories collected by a behavior pol- icy instcad of access the environment. The agent is expected to find the optimal policy using only the offline dataset D, without interacting with the environment itself. Decision Transformer (Chen et al., 2021) leverages the Transformer structure (Vaswani et. al., 2017) to predict the next action conditioned
3

---

Published in Transactions on Machine Learning Research (11/2025)
the past trajectory. To reduce the prediction error that could accumulate the trajectory length in- creases, DT reformulates the trajectories (so, d0:7o 81, "1,,,,.......................... :.............. 87, ar, rr) from the offline dataset 'D to {30, Go, Ro, $1, 01, Ri...., .T.....,,TT,,, ST, ar, Rt} f and uses the latter one in the loss function defined by the prediction error between the true actions and predicted actions, where RS Ei=rt Ti is the return-to-go at timestep t.
2.2 Prompt Decision Transformer
The goal of Prompt-DT is to enable single offline RL agent to solve multiple distinct tasks without requiring explicit task labels. The key idea is to prepend a prompt 7; to the standard DT input trajectory т. Both the prompt and the main trajectory are sampled from the same multi-task offline datasct D. A prompt T; for a lask T₃ is a short sequence of K* transitions denoted as T;- = (R) 6;,1 ",, .. K- , x-)))
This prompt provides the model with crucial context about the task's dynamics and reward structure. Consequently, the input vector ?'mput is the concatenation of the task prompt and the input trajectory:
input = |e nl = (RE, GI K S "KKK GiKK, Ra, 81,1, di.b. Ri, K, S,K, 0i,k). 7
(2.1)
In addition to this, we denote the trajectory truncated at timestep t E fl...., K} as Ticn: and consequently imput
the imput vector at timestep as Tj,Ker. Then the learning objective of Prompt-DT can be formulated as the following maximum likelihood estimation:
iuput ~T; [E log (o -, Ri,, s1,))], where Mo denotes Prompt-DT with the parameter 0 which is usually a transformer. In practical implementation, we often use the mean squared error loss instead, which aims to predict the future action âi, given the history trajectory and current state by minimizing the loss function LPDT EF 7, "epppp"> [ E as,,)]] The training procedure of Prompt-DIi DT is to autoregressively generate the action conditioned on the current state, return-to-go, past trajectory and sampled prompt.
3
The Proposed Framework
Building on the success of LLMs in prompt-based few-shot learning, we propose Language model-initialized Prompt Decision Transformer (LPDT), a novel and effective framework to incorporate powerful pre-trained language models into Decision Transformers to improve their few-shot learning abilities. We further introduce design is flexible and prompt regularization during fine-tuning to better identify tasks. Furthermore, (Zhuang el al., 2024). easy to adapt to other sequence modeling methods for oflline RL such as Reinformer Figure illustrates the overview of our method. At high level, LPDT incorporates several key components:
Language model initialization for Prompt-DT: We first use a pre-trained language model as the initialization for our Decision Transformer. This design ensures compatibility with the Prompt-DT paradigm, where prompts from various tasks are appended the input sequence. Parameter-officiett fino-tuning on RL tasks: We adopt Low-Rank Adaptation (LoRA) (Hu et al., 2021) to fine-tune low-rank residual matrix while keeping the original weight matrix of the language model fixed throughout the learning process. This approach significantly reduces the number of parame- ters compared to standard full fine-tuning of the large language models.
Prompt regularization through supervised and contrastive objectives: We incorporate addi- tional regularization over the prompt. embeddings to better identify tasks. Specifically, we employ loss functions derived from both supervised and contrastive learning techniques to fully utilize task-relatod information from prompts, thereby preventing the language model from overfitting specific tasks.
s
Flexiblity of LPDT We further extend our proposed framework to other sequence modeling methods for offline RI. to demonstrate its flexibility. In particular, we adapt it to Reinformer-based models, incorporating regression on actions and expectile regression on returns to enhance trajectory stitching with maximum-return estimation in oflline RI. We demonstrate that language-model initialization benefits Transformer- based prompt sequence modeling methods.
We discuss these techniques in detail in the rest of this section. At the end of this section, built on these components, we present our learning algorithm in Algorithm 1.
4

---

Published in Transactions on Machine Learning Research (11/2025)
3.1 Language Model Initialization
We first use pretrained large language model (LLM) with weights denoted by Ma- as the initialization of LPDT. Recent advances in large language models have demonstrated strong few-shot learning abilities. With task-spocific information such as prompts for translation or answering questions, language models can generate task-specific responses. We adapt these language models to RL domains such as MuJoCo control tasks, providing prior knowledge that may be relevant to downstream tasks. However, this not immediately straightforward since LLMs take word tokens as input, which differ from the inputs used in RL. The common next-token prediction objective for the language model (LM) can be formulated as
LLM = Ei=1 log(Mo- (1e41 :1,....,::)).
(3.1)
where Mg- is the language model, 0" is the set of parameters of the pretrained language model and x represents the word token. Concretely, we decompose the pretrained parameters as 0 = = (Froken: T Woocab) where Looeen is the token embedding layer, T is the causal Transformer backbone, and W Weeab is the language model output head. To make Mg- compatible with RL trajectories, we kcep the causal Transformer backbone and replace the word embedding layer and language model oulpul head with lask-specific linear layers: an input projection ERL. E Rd Xdiaput that maps each RL "token", i..,, 2 _ [Re, St. ozl, into vector in Rdhipus the input for the Transformer backbone, and output head W E Rdduede X that maps the hidden state with dimension dmodel to a probability distribution over the action space. Since the original token embedding and output head are trained on the text corpus, they cannot. be adapted to RL trajectories. We therefore train our own input projection layer ERE and output head Wact from scratch on the RL trajectories dataset. To be more specific, the input projection layer ERI. and output head Wact are randomly initialized with the frozen pretrained language model backbone. Then the input projection and output head are fully adapted to RL task trajectories and output the action for RL problems. We denote 0 = {ERL, T", Wact) as the adapted parameters, and Ma is the initialized model. Given a prompt-augmented input sequence T, ?mpptt = [T/ m] the model produces action predictions autoregressively under the same causal mask as language modeling:
0z, = MM((T.ne = Wact T" (ERL(T (r))ppt)) )).
(3.2)
For continuous control settings (e.g., MuJoCo), we replace the cross-entropy MLE objective in (3.1) with an MSE regression objective by replacing the language model softmax head with the linear regression action head Waet We then optimize the standard Prompt-DT regression loss
CPDT = imput
k Exillor, â., 113)
(3.3)
By leveraging the pretrained language model, we preserve broad sequence-modeling priors while adapting only lightweight RL-specific projections.
3.2 Parameter-efficien Fine-tuning on RL Tasks
To efficiently adapt the frozen pretrained language model to downstream RL tasks, rather than updating all weights, we apply LoRA (Hu et al., 2021) by introducing trainable low-rank matrices into each target weight matrix while keeping the original backbone fixed. This significantly reduces the number of trainable parameters and enables scalable adaptation to large language models. To be specific, assume the model has L layers. In each layer I E fl,..., , L}, we update the attention projections (W:(), w((), W:?) which are query, key, and value projections using LoRA. As example, for the query projection w:(") Rdmode where dmodel is the model dimension and dg is the query dimension, we reparameterize
( w + AW q
w + ag ag/rrAA B())
(3.4)
where 9.0 is the frozen weight inherited from the language model, ag is the scaling hyperparameter, rg is the rank in LoRA, and A!") Xroodax D²")) e RTodd are the trainable low-rank factor matrices with rg & min dz). We apply similar updates to w!!) and Wi!". For simplicity, we denote LoRA factor matrices in the query, key, and value projections as (A q,k,us !W!" our method avoids full fine-tuning of the Transformer backbone parameters, substantially reducing compute and memory costs while preserving the generalization ability of the pretrained backbone.
(1) B(i) J. By learning only AW("), AW("), and
5

---

Published in Transactions on Machine Learning Research (11/2025)
3.3 Prompt Regularization with Supervised and Contrastive Objectives
When different. lasks are close to each other, naively using the prompis can yield embeddings that are non- discriminative across tasks. Making prompts distinguishable at test time is costly, Previous works such as Prompt-Tuning DT (Hu ct al., 2023) and Prompt Diffuser (Hu et al., 2024) aim to tunc prompts at test time on unseen tasks. These methods perform additional gradient updates during test-time adaptation, which incurs extra computational cost when deploying policies. To avoid the computational overhead of test-time prompt tuning and improve generalization to unseen tasks, we introduce a task-aware training objective. Specifically, we incorporate regularization term on the prompt embeddings, termed prompt regularization, to encourage the model to learn discriminative prompt representations that reflect task identity during training. Let the prompt encoder be 9 and define the prompt embedding z; =
We use the loss function for Prompt-DT defined in (3.3) as the base loss function, and then incorporate a prompt regularization term. The final loss function of our method is given by
Fiotal H, Rnpiip ~7, K ER - az,,)² + ACo.
(3.5)
where Ls is the loss for the prompt regularization, which we will specify in the rest of this section, and X is the hyperparameter for prompt regularization. The encoder Jo is randomly initialized and jointly trained by (3.5) from scratch.
In particular, we propose two practical implementations of prompt regularization based on supervised learn- ing and contrastive learning methods respectively.
Supervised learning-based prompt regularization. In this approach, we add a classifier head to the output of the prompt encoder. We use the task ID from the dataset as the label to help the prompt encoder learn a more meaningful embedding that can easily distinguish different task environments. To be specific, we attach a classífier C()) to the prompt encoder output and train with known task IDs, encouraging z to be lincarly separable across environments. We adopt the following cross-cntropy loss as the regularizer
L Celassifier = yı log(de).
(3.6)
where y is the task label and 0: Clzz) is the predicted probability for task i based on the prompt rit.
Contrastive learning- based prompt regularization. Unlike the supervised classifier regularization, the contrastive objective relies on the positive or negative information rather than explicit task labels for each task during optimization. This is achieved by assuming that all data points collected within the same batch originate from the same underlying task. Furthermore, this design also allows the method to be adapted to settings where tasks are continuously collected during training without changing the model structure. From information theory perspective, the ideal prompt encoder should aim to maximize the mutual information between the prompt representation and the tasks. We use the InfoNCE objective (Oord et al., 2018; Yuan & Lu, 2022) to calculate the loss over the prompt. We formulate L. as:
exp(sim Z,, 2 /T) exp(sim x, /T) + E exp(sim 2TT)
C CInfONCE_ -E log
(3.7)
where 2, and 2,t form positive prompt pair corresponding to the same task, while z denotes negative prompt sampled from different tasks. N is the number of negative samples, and is the temperature hyperparameter that controls the sharpness of the distribution. The function sim(..-) computes the similarity between two prompt embeddings in which we use the cosine similarity.
3.4 Flexibility of LPDT
Our proposed framework is highly flexible and can be scamlessly integrated with other Transformer-basee sequence modeling methods. To demonstrate this adaptability, we exlend LPDT to the recently introduced Reinformer (Zhuang et al., 2024) as an illustrative example. Reinformer is a retum--onnitiioned sequence model optimized for maximum-return policy learning, which shows statc-of-tho-art performance on single- task offline RL benchmarks. We demonstrate that our proposed framework LPDT can be easily adapted to Reinformer with minor adjustments.
6

---

Published in Transactions on Machine Learning Research (11/2025)
Algorithm LPDT: Training and Inference
Require:
Pretrained LLM Mg. dataset D, regularization weight X, temperature 1: Training:
2: Freeze all Transformer layers in l"
3: Replace the word embedding with linear layer ERI and the output layer Wact
4: Insert LoRA adapters {-4(0) g. B. into weight matrices
((0)
5: for Iteration = ...... do
6: for all muni-batele C D do
Build input sequence T!th +- [TT; Tı] for each 7j E
0,11KK < (T
9:
LPDT + 1/ B E EE | âr,e
10: if supervised regularization then
11:
L. + --11//B En Vi log
>Equttio (3.6)
12: else
(xpssi(zz,,,.)))
13:
Cs - - Σ x,, log exp(sim(z,,2,)) Ex (xp(sim((;,27.)))
>Equation (3.7)
14:
end if
15: Lrotal + LPDT + ALo
16: Update LoRA matrix (A qukut k. and linear layers {ERL. Waeth via VoCrotal
B
17: end for
18: end for
19: Inference
unseen tasks:
20: Sample few-shot prompt Test
21: for t= to horizon do
22: a₂ f Mo T1:-11)
23: end for
First, Reinformer differs from DT in the ordering of its input sequence. Specifically, Reinformer structures each trajectory segment as a tuple of state, predicted return, and action. This format allows the model to first predict return and then condition action generation on both the current state and the predicted return. Accordingly, we define the input trajectory for task i as:
rein-input = = Tel = (52,1, RC, " Si R 0f 81,1, RI,1, 01,1," 8i,K, Ri,K, ai,k). (3.8)
where si,r, Ri,, and aie denote the state, return, and action at time step i, respectively. To better estimate high-reward trajectories across diverse tasks, employ the expectile loss for return prediction:
Creturn E roinstngot ~ 1 Σk m - 1(AR < - AR?],
(3.9)
where AR Ri, Rr, is the difference between the ground-truth return and the predicted return, and [0, is hyperparameter controlling the weight on overestimation underestimation errors. Tn practice, we favor overestination to encourage optimistic trajectory planning during testing. The training loss in Algorithm 1 is then augmented with an additional return loss term as follows:
Liotal LPDT ALo + T/return
(3.10)
where 7) is the hyperparameter weight for Cretum- During inference, the return-first sequence structure is preserved. The model begins by encoding the few-shot prompt from a new task and initializing with the current state. It then predicts a return, followed by generating the corresponding action. This autoregressive rollout continues stsepb with the prompt providing task-specific conditioning to ensure alignment with the environment dynamics.

---

Published in Transactions on Machine Learning Research (11/2025)
4 Experiments
In this section, we conduct experiments to evaluate the few-shot generalization ability of our proposed framework LPDT. We evaluate the performance of LPDT on MuJoCo control lasks (Fu el al., 2020) and Keeta-Worle (Yu et al., 2020) with the episode accumulated reward as the evaluation metric. We also evaluate the prompting ability of LPDT under limited data settings. Our experiments aim to answer the following questions: (1) Can LPDT with language model initialization achieve better performance compared with Prompt-DT and other baselines? (2) Does the manguage-initialinn Transformer model contain the knowledge of the unseen RL tasks and help improve the performance under limited data? (3) Does prompt regularization help the model distinguish different tasks and enhance the prompt learning capability of LPDT? (4) Does the improved performance slem from the language model's better understanding of RL, or from its ability to provide a generally better initialization for learning RL tasks? (5) Can LPDT be adapted to different pre-trained language models other than GPT-2? (6) Can LPDT be extended to sequence models beyond Decision Transformer and Reinformer? (7) Can prompt selection strategies further improve performance?
4.1 Implementation
In the empirical study, we first implement our LPDT method with GPT-2 as the language model initializa- tion. GPT-2 is pre-trained OpenWebtext (Puri & Catanzaro, 2019). During the fine-tuning stage, we follow the same hyperparameters for Prompt-DT (see Appendix D for details). We also leverage LoRA to significantly reduce the number of trainable parameters. For the prompt regularization, we use an MLP to further encode the prompt embedding. For the supervised version of prompt regularization defined in (3.6), we directly use the logits from the MLP to compule the cross-ent.ropy loss and refer to the method as LPDT- Classifier. For the contrastive version of prompt regularization defined (3.7), we calculate the similarity matrix through the cosine similarity based on the logits from the MLP and refer to it as LPDT-IIfoNCE, We also extend our methods to another sequence modeling method adapted from Reinformer (Zhuang et al., 2024) named HPDT-Rein-Classffeee Classifier and L.PDT-Rein-InfoNCRR
4.2 Datasets and Baselines
In this work, we evaluate the performance of our proposed approach on MuJoCo control tasks and Meta- World, which are commonly used in existing Prompi-DT lype of methods (Xu el al., 2022; Hu et al., 2023; 2024; Wang et al., 2024b), namely, Cheetah-dir, Cheetah-vel, Ant-dir, Point-robot, Meta- World reach-v2.
In Cheetah-dir, there are two tasks with goal directions as forward and backward, where the reward function promotes high velocity along the goal direction. The training and testing phases both include the two tasks. Similar to Cheetah-dir, Aut-dir also segments the tasks by directions. There are 50 tasks in Ant-dir with different goal directions uniformly sampled in 2D space. The tasks are split into 45 training tasks and 5 testing tasks. The ant is also rewarded with high velocity along the goal direction. Different from segmenting the tasks by direction, Cheetah-vel penalizes the agent through the l2 errors with the target velocities sampled from the velocity interval. There are 40 tasks with different goal velocities where 35 tasks are training tasks and 5 tasks are testing tasks. Point-robot is a point navigating to the given goal position. In addition to the MuJoCo control meta-RL, tasks, we also test our approach on Meta-World (Yu et al., 2020) which is an open benchmark for meta-RL and multi-task lcarning. In this work, we evaluate our approach on Meta-World reach-v2. The objective of reach-v2 is to control the robot to reach the target position in 3D positions. Each task has a different goal position.
Specifically, we collect the data by first training the Soft Actor-Critic (Haarnoja et al., 2018) in these different RL lask environments and then extracting the data from the replay buffer. More details about the datasets can be found in Appendix C. We compare the few-shot generalization ability of our proposed LPDT with bascline algorithms. For each method, we evaluate performance using the accumulated reward. The basclines we choose include Prompt-DT (Xu et al., 2022), Prompt-Rein, CORRO (Yuan & Lu, 2022), CSRO (Gao et al., 2024) and Meta-DTT (Wang et al., 2024b).
8

---

Published in Transactions on Machine Learning Research (11/2025)
4.3 Results of LPDT and Baselines
In this section, we conduct experiments proposed LPDT framework and baseline methods to evaluate their performance. Additionally, we compare variants of LPDT that use different prompt regularization strategies and sequence modeling techniques. The average accumulated reward per episode in the test task set is used as the evaluation metric for all methods. The results for Prompt-DT, CORRO, CSRO, and Meta-DT are taken from the Meta-DT paper (Wang et al., 2024b), while the results for MW Reach-v2 are provided by our reproduction. We compare our approaches including both the supervisod classifier version and the contrastive InfoNCE version with prior works. The model is tested with three different random seeds, and we report the average return across all testing tasks.
Table 1 illustrates that LPDT outperforms bascline algorithms (Prompt-DT, Prompt-Rein, CORRO, and CSRO) on MuJoCo Control lasks and MW Reach-v2 and is competitive with Meta-DT. Specifically, our LPDT approaches outperform Meta-DT in Cheetah-dir and MW Reach-v2. In all other environments, LPDT achieves the second-best performance among the baselines and is competitive compared with Meta- DT. We conclude that using our LPDT approach improves the performance compared most baselines and is competitive with Meta-DT. It is worth noting that Meta-DT introduces an additional prompt selection stage test time to select high-quality prompts, while our methods avoid this computation overhead. We use the prompt regularization module to distinguish the prompt instead of iterating over the demonstration set to choose the best prompt. Nevertheless, the prompt selection strategy is orthogonal to our contribution and can be applied at the cost of extra computation.
Table 1: Results for MuJoCo control tasks and MW tasks. The best mean scores are highlighted in green and the second mean scores are highlighted in blue. For each environment, the length of the prompt is K = 5. The dataset we utilized is the full dataset. We test all the results on unseen tasks with three random seeds. LPDT outperforms baselines on the Chectah-dir and MW Reach-v2 environment and are competitive remaining environments.
Method
Chectal-dir
Cheetahi-el
Ant-dir
Point-robot
MW Reach-v2
Prompt-DT
960.32 + 17.07
-133,78 + 18.24
678.07 + 68.74
-7.99 1: 0.46
2906. 67 + 33.16
Prompt-Rein
991.10 + 30.59
-114.06 + 22.51
702.24 + 36.89
-7.77 + 0.90
3008.25 + 77.81
CORRO
628.64 + 61.11
-111.47 = 36.97
381.42 = 13.83
-7.76 = 0.18
CSRO
641.05 + 129.54
-129.00 + 24.24
417.37 + 39.70
-19.42 + 2.10
Meta-DT
874.91 73.45
-52.42
8.11
961.27
18.07
-6.90
0.11
1418.39 6.01
OPDT-Classifiee
1121.68
.68 + 25.13
-55.15 + 15.82
782.25 + 30.39
-8.19 + 0.46
2903.43 + 55.09
LPDT-InfoNCC
1118.95 1 35.14
-57.56 1 9.55
775.79 1 35.28
-9.06 1 0.22
2046 88 1 92.18
COoT-Rein-Clnssifiee
1131.94
51.09
-95.02 = 10.58
769.52
69.40
-7.30
0.35
3023.38
178.49
1113.96 + 20.52
-96.07
5.33
10.95 759.54
-7.37 0.27
2614.41
27.57
Mcan Variants
1121.63
-75.95
771.78
-7.98
2872.03
4.4 Data Efficiency of LPDT
In this section, we demonstrate the data efficiency of LPDT. The data efficiency shows how well our ap- proaches and baselines can perform with access to only a certain number of unique trajectories after sufficient training iterations. The pretrained language model provides prior knowledge for reinforcement learning tasks, leading to increased data efficiency within our framework. To validate this hypothesis, we conduct extensive experiments on various datasots split by ratios (1.0, 0.5, 0.1, 0.05, 0.013. A ratio of 1.0 corresponds to the original full dataset, while ratio of 0.01 represents an extreme scenario where only 1% of the original full dataset is available in training. We train LPDT and compare it with Prompt-D's DT and Meta-DT, under these conditions and evaluate their performance on unseen tasks. The results are presented in Table 2.
As depicted in Table 2, the return score decreases as the dataset size diminishes. In lasks such as Cheetah- dir and Ant-dir, our LPDT algorithm mostly achieves better performance compared to Prompt-DT and Mcta-DT. Specifically, on Cheotah-dir, LPDT is consistently the best across all splits from 1.0 to 0.05. On Cheetah-vel, Ant-dir and Point-robot, with full datasct, the performance of LPDT is comparable to Meta- DT. But when it decreases to a small dataset with 0.5, 0.1, 0.05, our LPDT outperforms the baseline. With
9

---

Published in Transactions on Machine Learning Research (11/2025)
only 10% of the dataset, proposed methods achieve similar even better performance compared to Prompt-DT in Cheetah-vel and Ant-dir. Nevertheless, at ratio of 0.01, an extreme case, all met perform poorly on Cheetah-dir and the scores are mostly determined by noise, which is expected due to insufficient training data. For the Cheetah-vel and MW Reach-v2 tasks, results are inconsistent and exhibit significant. variance, likely due to inherent lask challenges and difficulties learning stable policies these environments. Mcta-DT requires learning context encoder based on training trajectories and particularly sensitive limited data availability. Despite these challenges, the overall trend indicates that LPDT models maintain performance advantage over other baselines, especially when dataset sizes are moderately reduced. This underscores the benefit of leveraging prior knowledge from language models to enhance data efficiency in downstream reinforcement learning tasks.
Table 2: Results for MuJoCo control tasks and MW tasks with the {0.01, 0.05, 0.1, 0.5, 1.03 ratio dataset.
The best mean scores are highlighted in green and the second mcan scores highlighted in blue. For cach environment, the length of the prompl is K = We lest all the results unseen lasks with three random seeds. We demonstrate that language initialization can improve the performance of LPDT.
Datuset Ratio Algoritlon
Chestah-dir
Cheetah-vel
Am--dir
Point-robot
MW Reach-v2
Prompe-DT
960.32 + 17.07
-133.78
18.24
678.07 + 68.74
-7.99 = 0.46
2906 67 = 33.16
Meta-DT
871.91 + 73.45
-52.42
8.11
961.27 = 18.07
-6.90
0.11
1418.39 = 6.01
1.0
LPDT-Classfäer
1121.68 25.13
-55.15
15.83
782.25
30.39
-8.19 = 0.16
2903. 55.09
LPDT-InfoNCE
9.35
2946.888
92.18
L.PDT-Rein- Reein-CCassiffer
1131.94 + 61.09
-95002 + 10.58
709.52 + 69.40
-7.30
0.35
3023.38
178.49
L.DTTRRela-nnoooe
1113.96 = 20.52
-96.07 = 5.33
759.54 + 10.95
-7.37 = 0.27
2614/11 = 27.57
Mean Variants
1121.63
2872.03
Prompe-DT
691.55 72.87
-69.93
7.59
663.19
17.06
-6.53
0.41
3062.35
15.89
Meta-DTT
202.82
33.91
-75.61 7.77
508.73 + 61.73
-14.07 0.26
2086.79 + 2.62
0.5
.PDD-Classfßeer
1074.20 + 27 27.26
-55.19 = 10.94
705. = 51.27
-6.04 = 0.56
2126.30 50 +: 125. 30
LPDT-InfoNCE
1102.18
43.18
-64.23
714.31
-6.97
2430.34
J.DDT-Reln-Classiee
949.85
26.08
-101.84
4.23
746.45
56.32
-7.832
0.94
2841.01
136.95 94
L.PDT-Rein RoT--o
890.25
32.85
-006.94 + 8.52
750.38
70.14
-7.468 + 0.44
2902.60
92.70
Moan Variants
1006.37
-82.05
729.28
-7.08
2652.37
Prompe-DT Meta-DT
193.69 = 9.06
-72.47
3.14
598.29 = 21.61
-8.94
0.43
3541.67 67 1: 75.41
77.42
11.90
-114.32
10.24
509.25
51.03
-16.88
1.64
2040.93
106.34
0.1
JPDTTClassiße
750.90
88.86
-65.23
19.16
688.21 :: 36.78
-8.46
0.34
2519.92 1: 152.28
LPDT-InfoNCE
666.13 13 = 56.85
-77.08 = 17.71
660 63 + 52.00
-8.19
0.91
2577.42 = 60.06
L.PDT-Rrin-C Class-Clessefer
622.95
-107.20
687.71
68.32
-9.242
0.54
2781.77
J.DDT-Rein-InfoNCE
614.52
86.21
-112.00
14.25
641.87 +: 45.96
-9.600
1.23
2827.77
154.02
Moan of Variants
663.63
-90.57
669.61
-8.88
2676.72
Prompt-DT
112.40 14.43
-97.29
9.90
491-320 = 19.94
-8.26 = 0.07
3320.85
77.59
Meta-DTT
37.87 = 19.15
-118.15 = 13.35
309 = 63.76
-18.27 = 1.67
2001.55
41.59
0.05
LPDTTClassifer
386.01
84.45
-103.43 + 5.45
488.54
43.80
-7.80
1.22
2510.56
120
L.DDT-I66NCC
320.28 = 79.21
-102.5
5.09
456.17
65.82
-8.32
0.15
2592.62
153.21
L.PDT-Rein-C Rein--lassiffie
252.33
69.15
-113.70 + 6.33
534.28
52.31
-10.22
1.23
2508.67 67 + 124.36
LPDT-RRcin InfoNCE
279.48
-112.68
1.44
Mean Variants
309.53
-108.08
484.57
-9.21
2476.77
Prompt-DT Meta-IVT
18.97 4.52
-99.89
2.85
183.53 + 11.39
-8.91
0.41
2551.17
122.52
30.88
21.07
-173.93
28.00
90.86
0.91
-17.28
2.01
1934.47
28.61
0.01
LPDITCClsssfßer
17.80 = 12.73
-121.84 = 4.12
175.82
18.61
-10.56
0.21
2132.60
165.87
-PDT-IInfoNCF.
10.54 6.75
-127.81
3.26
130.76
3.64
-11.52
1.80
2390.92 02
79.37
LPDT-Rein Rein-CClssiifee
25.42 = 14.50
-125.22 + 2.58
264.66
12.04
-11.15 = 0.47
2508.67
124.36
OPDDT-Rein-Inowlee
9.26 10.32
-133.44 4: 4.71
270.43
8.36
-10.93 4 1.12
2449.91 41999911 91.08
Mean Variants
15.755
-127.08
-11.04
2370.50
4.5 Ablation Studies
In this section, we provide ablation studies on our proposed framework LPDT from various aspects. We focus on the effectiveness of prompt regularization and the language model. We also extend our proposod framework to other sequence modeling methods decision-making, Finally, we demonstrate the different prompt selection strategies during the testing stage.
10

---

Published in Transactions on Machine Learning Research (11/2025)
Table 3: Results for MuJoCo control tasks and MW tasks with different regularization methods of our method including w/o regularization, classifier regularization and InfoNCE regularization. For each environment, the length of the prompt is K* The datasct we utilized is the full dataset. We demonstrate that the regularization on prompts can help distinguish the task and improve the performance compared with the method without regularization.
We test all the results on unseen tasks with three random seeds,
LPDT
LPDT-Rein
Tusk
w/ regularization
w/ regularization
w/o regulacization
w/o regulacization
Classifir
InfoNCE
Mean
Classißer
InfoNCE
Mean
Cheetah-dir
1109.54 = 20.04
1121.68 + 25.13
1118.95
35.14
1120.32
1123.51
= 9.67
1131
51.09
1113.96 + 20.52
1122.95
Chestah-vel
-56.38 7.53
-55.15 15.83
-57.56 + 9.55
-56.4
-95.30
23.80
-95.02 10.58
-96.07 + 55.33
-95.55
Ant-die
762.19 = 59.22
782.25
30.39
775.79
1 35.28
779.02
718.78
= 43.41
769.52 1: 69.40
789.54 1: 10.95
764.53
Point-robot
-6.37 = 0:11
-8.19 = 0.16
-9.06 0.22
-8.63
-6.12 = 0.26
-7.30 + 0.35
-7.37 + 0.27
-7.31
NIW readbov2
2801.11
189.82
2903.43
4: 53.09
29.8.888
92.18
2925.16
2252.68
196.39
3123.38
1 178.49
2614.41
+ 27.57
2818.90
The role of prompt regularization We first compare our proposed model with its variants that do not utilize prompt regularization, denoted as LPDT w/o regularization and LPDDT-Rein w/o regularization. Table 3 illustrates that our proposed LPDT outperforms LPDT w/o regularization and LPDT--Rein w/o regularization most of our proposed scttings. We further compare the two prompt regularization variants. Both variants improve performance over the variants without regularization across most environments, except for Point-robot. The Classifier variant slightly outperforms InfoNCE while InfoNCE variants adapt better to settings with continuously collected tasks. The InfoNCE variant also demonstrates consistent improvements in most settings compared with the versions without regularization. These results show that prompt regularization improves the model's ability to distinguish between tasks and demonstrates improvements in most scttings.
The benefits of using pretrained language models We aim to answer the question of how initializing DT with pre-trained language models benefils RL lasks. Specifically, we aim to determine whether the performance improvement arises from (1) sigher-quolity prompt embeddings produced by the language model, which enhance the contextual representation of RL environments, or (2) knowledge priors for decision-mak- ing lasks encoded in the language model's hidden layers, which facilitate better understanding of the underlying environment dynamics. To this end, we conduct experiments where use the rexteeeetrained model's prompt embedding as input to a freshly initialized Decision Transformer or Reinformer, denoted as respectively. In dctail, these two
PrommptDT Croomp--D--Tex--Embeeee Embedddng and variants adopt a language model to encode the trajectory prompt, and the resulting prompt embedding is then used train a newly initialized Prompt-DT.. Since the language model is only employed extract trajectory embeddings while the DT/Reinformer parameters are trained from scratch, this ablation study allows us to assess whether the language model merely provides better representations or also be directly fine-tuned into an effective decision-maker, as in our LPDT methods.
The experimental results are summarized in Table 4. We first observe that Prompt-DT- Prompt-DT-Rein Tein--exx--Emanne Teeen-TTxx---mnddiinng outperform vanilla Prompt-DT methods across all environments except Chectah-dir, suggesting that the language model can, to some extent, generate more informative prompt embeddings. Moreover, our LPDT algorithms, which use pre-trained language models not only to initialize the embeddings of prompts and trajectories but also to initialize the intermediate layers of DT DT/Reinformer, significantly outperform the Text-Embeddiig baselines. This demonstrates that initializing DTs with language models provides not only improved representations of RL environments but also strong priors for decision-making enabling effective adaptation with only minimal fine-tuning.
and
Flexibility in the language model We implemented various pre-trained language models to demonstrate that our propased approach can be adapted to different language models. In this section, we provide the results with small language models such as Qwen2.5-0.5B (Qwen et al., 2025). Table 5 presents the Ant-dir results under different pre-trained language model initializations, The results indicate that with Qwen2.5- 0.5B, the performance remains competitive in I.PDT-Rein and improved in LPDT, demonstrating the
11

---

Published in Transactions on Machine Learning Research (11/2025)
Table 4: Results for MuJoCo control tasks on Crompp--DT-Teet--Emedddiee and Prompt-DT-Rein- Embedding. For each environment, the length of the prompt is K* 5. We test all the results on unseen tasks with three random seeds. The dataset we utilized is the full dataset. We demonstrate that improved performance comes from the initialization of the language model.
DT-bweed meelhods
Reinfforrmer-bseee taetbodk
Task
Promapt-DT
LPDT
Prompt-DD
LPDT
Vanilla
Embedding
Classifier
In5oNCE
Vanilla
Text- Enn-Embeedingg
Classificr
InfoNCE
Chestab-dir
960.32 + 17.07
-23.74 + 18.94
1121.68 26.
1118.90 35.14
991.10
30.09
1015.307 + D0.47
1131.94 + 51.09
1113-96 20.02
Choetalavel 133.78 18.24
72.28 8.25
55.15 15.82
57.58 9.55
-114.06 22.51
95.02
96.07 5.33
Ant-dir
678.07 + 68.74
716.02 + 44.15
782.25 + 30.30
775.79 + 36.25
702-24 36.89
006000 + 46.00
700.52 69.40
718.54 + 10.86
flexibility of our approach to various language model initializations. We hypothesize that larger language models such as 7B models are likely to further enhance performance in unseen test settings.
Table 5: Results for Ant-Dir with different initialization of pre-trained language model the full dataset. The length of the prompt is === 5. We test all the results unscen tasks with three random seeds. We remonstrate that LPDT with Qwen2.5-0.5B has improved performance compared to GPT-2.
LPDT Classifeee GPT-2
L.DT-- GPT-2
LPDT Classifier Qwen2.5.0.50 800.22 55.16
IPfo--IffoNC Qwen2.5.0.5B 786.16 15.55
782.25 + 30.39
775.79 + 35.28
PeDD-Reii-Classiffee GPT-2 769.52
LPDT- InfoNCC GPT-2 LPDD-Rein-Chassfeer Qwen2.5-0.50
OPPT-Rein-InfoNCE Qwen2.5-0.50 719.69 35.33
750.54 10.95
735.97 58.33
Extension to other DT methods Our proposed method builds upon the high-level concept of Prompt- DT by leveraging prompts and contextual information to guide models in unseen tasks, but extends beyond this specific architecture. Our framework is versatile plug-in method based prompt strategies that can be integrated into various architectures. While our primary implementation follows the Prompt-DT paradigm, we also demonstrate our algorithm's flexibility by adapting it to Reinformer, a recent DT-based method achieving stato-of-tho-art performance in singlo-task offiine RL problems. To further validate this generalizability, we now provide an additional implementation based on EDT (Wu et al., 2024). Table 6 compares our LPDT-EDT approach to standard Prompt-EDT without language initialization, demonstrat- ing significant performance improvements. These results highlight our framework's potential and flexibility to enhance a wide range of architectural designs.
Table Results for MuJoCo control tasks on LPDT-EDT on the full dataset. The best mean scores are highlighted green and the second mean scores are highlighted in bluc. The length of the prompt is === 5. We test all the results on unseen tasks with three random seeds. We demonstrate that our proposed framework LPDT can be flexible to the sequence modeling methods in decision making.
Task
Prompt-EDT
SPDT-EDT-Classinee
LPDT-EDT-IInNNCH
Cheetah-dir
796.56 + 87.09
1040.087 34.834
1024.172:52.357
Choetah-vel
- 1.42
-254.301=1.213
-222.05+11.589 115.087=20.265
Ant-dir
240.794 + 24.34
733.435 16.969
Advanced prompt sampling strategies In our implementation, prompt trajectories are randomly sam- pled from the demonstration set during testing unseen tasks. This is simple and straightforward approach that allows us to better highlight the benefits of language initialization and low-rank fine-tuning. We additionally implement a prompt selection method based on online interaction returns, using these re- turns as a performance metric to identify effective prompts. Specifically, we select several prompts from the demonstration set and then interact with the environment several times obtain the prompt with the highest mean return. We compare the performance of such methods (with suffix '-select') LPDT. The results, illustrated in Table 7, demonstrate performance improvement over random selection. However,
12

---

Published in Transactions on Machine Learning Research (11/2025)
the improvement over the current performance of our LPDT is not significant. These additional prompt selection strategies introduce additional model complexity with a little improvement. Thus, we use a simpler way which selects the prompts randomly from the demonstration set to implement our LPDT. Nonetheless, this selection strategy is orthogonal to our core LPDT framework and represents an interesting direction for future exploration.
Table 7: Results for MuJoCo control tasks on LPDT and prompt-selection variants. The best mean scores are highlighted in green and the second mean scores are highlighted in blue. For each environment, the length of the prompt is K = 5. We test all the results on unseen tasks with three random seeds. The dataset we utilized the full dataset. We demonstrate that our proposed framework LPDTcan be orthogonal to prompt selection strategics and can be improved with advancod prompt selection methods.
Task
Classifie
LPDT-
ClassifierPPrompl select
LPDT InfoNnC- InEoN select
Cheetah-dir
1121.68 25.13
1118.95 + 35.14
24.1.33422.36
1139.11+16.86
Cheetah-vel
-55.15-11..8
-57.56 + 9.55
16.00
- 56.38 + 14.21
Aut-dir
782.25 30.39 775.79 35.28
824.79:146.04
782.72138.68
Computational analysis To assess the computation cost, we design specific experiments to measure the number of trainable Transformer parameters, training time, and inference time for our proposed LPDT compared to the baselines. We select the Ant-dir as the instance to analyze. All experiments are conducted single NVIDIA A6000 GPU with Intel Xeon Ice Lake Gold 5317 processors. For the training time, compare the methods by achieving the same performance on unseen test tasks. Since these methods achieve different performance levels on Ant-dir, we choose a target score of approximately 650 for a fair comparison, which corresponds to the performance of Prompt-D'T. As shown in Table 8, our proposed LPDT achioves similar performance (around 650) to Prompt-DT, with only slightly higher memory and time costs. Compared to Meta-DT, our LPDT variants achieve faster inference because they avoid the additional overhead introduced by prompt selection. It is also worth noting that Meta-DT needs to train a world model which has longer training time. In addition, LPDT-Rein requires roughly twice the inference time of LPDT. This is reasonable as the LPDT-rein needs to predict the return and action during the inference.
Table 8: Comparison of computational costs on the Ant-dir environment. Training Time refers to the duration required for each method to achieve comparable performance. Inference Time is the overall time taken to generale one trajectory over five test lasks. The number of trainable parameters for the Transformer component of each model is also reported.
Method
Transformer Trainable Params (M)
Training Time
Inference Time
Prompt-D'T
0.60
3.8h
4.96s
Meta-D'T
0.79
~ 14.7h
10.41s
LPDD-CClssifier
0.91
55.6
7.33s
L.DDTInfoNCE
0.91
55.7h
7.82s
LPDDT-Rein-laassifeer
0.91
<..7h
16.96s
JeDD--ReinInffNCE
0.91
4..7
18.02s
5 Conclusion
In this work, we proposed novel flexible framework for improving the few-shot prompt ability of decision transformers and other sequence modeling methods in oflline reinforcement learning. This framework, termed Language model-initiallied Prompt Decision Transformer (LPDT), leverages pre-trained language models integrated with domain-specific RL datasets to enhance few-shot prompt capabilitics. LPDT demonstrates superior or competitive performance compared to existing baselines in terms of cunulative rewards on unseen tasks. Our approach holds significant potential for reducing data requirements in offline RL tasks, thereby
13

---

Published in Transactions on Machine Learning Research (11/2025)
increasing applicability in real-world scenarios where large-scale collection of RL trajectories is challenging. Moreover, our results underscore the importance of utilizing pre-trained language models as foundation for deeision-making tasks and demonstrate the effectiveness of our prompt regularization methods in enhancing task-spocific information discermment within prompts. Additionally, our LPDT framework cxhibits fiexibility to accommodate various pre-trained language model initializations and sequence modeling methodologies, including Reinformer and other recent DT variants.
While LPDT has shown promising results, there are several limitations to approach. Current comput- ing resource constraints limit use of language models to GPT-2 and Qwen2.5-0.5B. To fully harness the potential of pre-trained language models in decision-making tasks, future efforts will aim at extending our framework to encompass more exlensive open-source language models and implementing efficient fine- tuning techniques. Exploration into alternative architectures or the incorporation of multi-task learning could further enhance L.DT'ss performance. Future research will focus on addressing these limitations and expanding LPDT's application across a broader range of pre-trained language models and lasks, offering promising direction for advancements in offline reinforcement learning.
Acknowledgments
We would like to thank the anonymous reviewers and the editor for their helpful comments. YY and PX were supported in part by the National Science Foundation (DMS-2323112) and the Whitehead Scholars Program at the Duke University School of Medicine. The views and conclusions in this paper are those of the authors and should not be interpreted as representing any funding agency.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint ar arXim:2303.0877 087744 2023.
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-conditioned supervised learning work for offline reinforcement learning? Advances in Neural Infor- mation Processing Systems, 35:1542 1553, 2022. 17
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Newral Information Processing Systems, 33:1877-1901, 2020. 2
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Ar- avind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in Newral Information Processing Systems, 34:15084-15097 2021. 1, 2, 3, 17
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning. arXio preprint srXiv:2004.07211 2020. 2,
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019. 1
Yunkai Gao, Rui Zhang, Jiaming Guo, Fan Wu, Qi Yi, Shaohui Peng, Siming Lan, Ruizhi Chen, Zidong Du, Xing Hu, et al. Context shift reduction for offline meta-reeinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic; Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. oixiv preprint arXiv:2106. 096855 2021. 2, 4, 5, 17
14

---

Published in Transactions on Machine Learning Research (11/2025)
Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Prompt-tuning decision transformer with preference ranking. arXiv preprinl or Xiv:23005.0000
2023. 2, 6, 8
Shengehao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Prompt tuning with diffusion for few-shot pre-trained policy generalization, 2024. URL https://openreview wtt../toooomm.d//reelll
2, 6, 8, 17
Michael Janner, Qiyang Li, and Sergey Levine. Offlin reinforcement learning as one big sequence modeling
problem. Advances in neural information processing systems, 34:1273 1286, 2021. 2, 17
Leslie Pack Kaelbling, Michael I. Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):00-134 1998. 2
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019. 1
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-lcarning for offline reinforce- ment. learning. Advances in Newral Information Processing Systems, 33:1179 1199--199,,1111 2020.
Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model- free robotic reinforcement learming. In Conference on Robot Learning, pp. 417-428, PMLR, 2022.
Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. Multi gamte decision transformers. Advances in Neural Information Processing Systems, 35:27921-27936, 2022. 17
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.016: 2020. 1
Shuang Li, Xavier Puig, Chris Paxton, Yihun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems, 35:31199 09121222, 2022. 2, 17
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint wxXiv22101.01190 2021. 2
Zhishuai Liu, Yu Yang, Ruhan Wang, Pan Xu, and Dongruo Zhou. How to provably improve return condi- tioned supervised learning? arXiv preprint ar arXiv:2506.08400 2025. 17
Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Lierre-Luc Bacon. When do transformers shine in rI? decoupling memory from credit assignment. Advances in Neural Information Processing Systems, 36, 2024. 2
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arx:::1807.0011 2018. 6
Raul Puri and Bryan Catanzaro. Zero-shot text classification with generative language models. arXiv preprint XrX1::1912 10165, 2019. 8
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: / //axxvivvvaa .rgg wrg..bsg///.o.....ooooo........a abs/2441..55
11
Alec Radford, Jeffrey WI, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019. 17
Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement learning?
Xrviv preprint or Xic: 12122, 2022. 2, 17
15

---

Published in Transactions on Machine Learning Research (11/2025)
Ruizhe Shi, Yuyao Liu, Yanjie Ze, Simon Shaolei Du, and Huazhe Xu. Unleashing the power of pre-trained
language models for offline reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview..rt wwwp:/eee/fooonoe wttp://mm.oooooo///aaaaaa....// 2, 17
Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4rl: Surprisingly simple self-supervision for offline reinforcement learning in robotics. In Conference Robot Learning, pp. 907-917. PMLR, 2022. 1
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Ruhan Wang, Yu Yang, Zhishuai Liu, Dongruo Zhou, and Pan Xu. Return augmented decision transformerr for off-dynamics reinforcement learning. arXiv preprint arXiv: 12,450,, 2024a. 17
Zhi Wang, Li Zhang, Wenhao Wu, Yuanheng Zhu, Dongbin Zhao, and Chunlin Chen. Mela-dt: Offline meta- rl as conditional sequence modeling with world model disentanglement. Advances in Neural Information Processing Systems, 37:44845 44870, 2024b. 2, 8, 9, 17
Yuch-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. Advances in Neural Information Processing Systems, 36, 2024. 1, 2, 12, 17
Zhihui Xie, Zichuan Lin, Deheng Ye, Qiang Fu, Yang Wei, and Shuai Li. Future-conditioned unsupervised pretraining for decision transformer. In International Conference on Machine Learning, pp. 38187-38203. PMLR, 2023. 17
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In international conference on machine learning, pp. 24631-24645. PMLR, 2022. 2, 8,
Mengdi Xu, Yuchen Lu, Yikang Shen, Shun Zhang, Ding Zhao, and Chuang Gan. Hyper-decision transformer for efficient online policy adaptation. In The Eleventh International Conference on Learning Representa- tions, 2023. URL
AirruuuuC-A 17
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100. PMLR, 2020. 2, 8
Haoqi Yuan and Zongqing Lu. Robust task reprosentations for offline meta-reinforcement learning via contrastive learning. In International Conference on Machine Learning, pp. -275999 PMLR, 2022.
6, 8
Xiangyuan Zhang, Weichao Mao, Haoran Qiu, and Tamer Başar Decision transformer as foundation model for partially observable continuous control. arXie preprint arXiv:2444.02401 2024. 17
Hongling Zheng, Li Shen, Yong Luo, Tongliang Liu, Jialie Shen, and Dacheng Tao. Decomposed prompt docision transformer for efficient unsoen task generalization. In The Tighth- Annual Conference on Newral Information Processing Systems, 2024. 17
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In international conference on machine learning, pp. 27059. PMLR, 2022. 2
Zifeng Zhuang, Dengyun Peng, Jinxin Liu, Ziqi Zhang, and Donglin Wang, Reinformer: Max-relurn sequence modeling for offline RL. In Forty-first International Conference on Machine Learning, 2024. URL https: / //openreview wttpiiiwwwwwteeeeeee mer/forrm? d=nBc3Ps 1, 2, 4, 6, 8, 17
16

---

Published in Transactions on Machine Learning Research (11/2025)
A
Code Availability
All source code and experiment configurations are available at
B Related Work
Decision Transformer. Decision Transformer (DT) (Chen et al., 2021) emerges as a type of algorithm for oflline RL by using the powerful Transformer architecture for decision-making. DT models RL trajectories as a sequence generation problem and utilizes the next-token generation paradigm for training. Thus, DT takes the history tokens such as the return, state, and action to predict the next action, which formulates the decision-making fully utilize the whole trajectories and is easy to train compared with dynamic programming-based offline RL, many of the following works improved the performance under different settings. For example, Lee et al. (2022) proposes to use the Multi-game Decision Transformer which is trained on part of the Atari games as the multi-tasks training and fine-tuned on the remaining games to achieve efficient adaption. Hyper Decision Transformer (Xu et al., 2023) adds an adapter into DT and is fine-tuned on unseen tasks through the demonstration without expert actions. Xie et al. (2023) proposes predict the action conditioned on the future trajectory embedding as the hindsight instead of conditioned on the return. Trajectory Transformer (Janner et al., 2021) is another research line, which is trained on sequences of state, action, and rewards and generated with the beam scarch. Elastic DT (Wu et al., 2024) introduces the expectile regression over the returns and uses the different lengths of context to predict the next action. Reinformer (Zhuang et al., 2024) utilizes the expectile regression to max the return to improve the stitching ability during the sequence modeling. DT can be considered as special instance of return-conditioned supervised learning, and the theoretical capability has been discussed in Brandfonbrener et al. (2022) and Liu et al. (2025). It has also been applied to off-dynamics RL settings to address policy transfer problems (Wang et al., 2024a).
an action prediction sequence generation in a supervised fashion. Since DT can
Prompt-DT.
Prompt DT (Xu et al., 2022) utilizes the prompt-based framework to do the meta-RL. It is trained on multi-RL tasks with offline datasets. During the training, the prompts or demonstrations which are a small part of the trajectory are combined with trajectories. During the testing on unseen tasks, the prompt can be a guide for indicating the tasks and help the model predict the action to interact with the enviromments. Following Prompt-DT, several works are adopting the prompt tuning method to achieve a high-quality prompt. Prompt-Tunie
DT (Hu et al., 2021) uses the preference ranking function and black- box tuning method to tune the prompt when testing on unseen tasks to achieve high-quality prompt. Moreover, Prompt Diffuser (Hu et al., 2024) leverages the diffusion model to generate high-quality prompts leading to improved performance in downstream RL tasks. Different from these works, we adopt the prompt regularization which aims to learn a high-quality prompt embedding to distinguish the different but similar RL tasks. Our method adopts this rogularization during the training procodure in the prompt dataset. Mela-DT (Wang et al., 2024b) proposes to pre-train a context encoder to encode the context in trajectories. It uses the trajectory context as the input to incorporate the task representation into Prompt-DTT.
Language model based DT. Large language models have achieved many surprising effects in various tasks in recent years. Pre-trained on large datasets such as the corpus of the Internet, LLMs such GPTs (Radford et al., 2019) demonstrate prompt ability which can generate the text with the guide of the task information. The success of the large language models motivates the increasing use of pre-trained language models in improving Decision Transformer to solve RL tasks (Chen et al., 2021). Several works utilize the powerful representation generalization ability of language models as policies to do the decision- making. Li et al. (2022) proposed to adopt the pre-trained language models for interactive decision-making to convert the policies to sequence data. Wik-RL (Reid et al., 2022) uses pre-trained language model from the next-token generation paradigm as the initialization of DT for offline RL tasks. However, it suffers from inferior performance than directly using DT. To overcome these challenges and unlcash the power of language models, Shi et al. (2024) proposed the LaMo algorithm which uses a pre-trained language model and parameter-efit efffciient fine-tuning methods to improve the original DT. Zhang et al. (2024) also proposed to use LaMo partially obscrvable continuous control problems which demonstrates a strong generalization ability. All these methods are designed for single tasks. DPDT (Zheng et al., 2024) uses the pre-trained language model to provide the prior knowledge to train the prompt and utilizes the test time adaptation to
17

---

Published in Transactions on Machine Learning Research (11/2025)
align the cross-task prompts in unseen tasks. However, it still needs test time adaption phase to achieve the ideal performance. Our approach is fine-tuned for learning to identify different prompts for various RT tasks. And during the testing phase, just small part of the trajectories is used in our method as the prompt without updating the model.
C Details on the Experiment Environments
We evaluate our approach over MuJoCo control lasks and Meta-World ML1 tasks. We split the lasks in these environments into the training set and the testing set. The tasks in Cheetah-dir and Ant-dir are split by directions. The tasks in Chootah-vel are split by the goal velocitics. The tasks in Point-robot are split by the goal positions which are uniformly distributed in unit square. In Meta- World, the tasks are defined by different goal positions. The detailed task indexes be found in Table The experiments we conducted are all followed to this setting which guarantees consistency during the evaluation.
Table 9: Training and testing task indexes when lesting the generalization ability. We follow the tasks split between Prompt-DT and previous works to guarantee direct comparison with baselines.
Environment
Number of tasks
Tasks indexes
Cheetah-dir
Training set: 2
[0,1]
Testing set: 2
[0,1]
Training set: 45
[0-44]
Cheetah-vel
Testing set: 5
[45-49]
Training set: 45
[0-44]
Ant-dir
Testing set: 5
[45-49]
Training set: 45
[0-44]
Point-robot
Testing set: 5
[45-49)
Training set: 15
[0-14]
Meta- World reach-v2
Testing set: 5
[15-19]
D Hyperparameters
In this section, we show the hyperparameters of our LPDT conducted in Table 1. The hyperparameters have two parts which are the hyperparameters around the transformer and prompt regularization. We list these hyperparameters in Table 10.
E Training Return Curves
Figure 2 presents the evaluation of Prompi-DT and four LPDT approaches. All the plotted methods are tested through 50 episode returns the unseen tasks. The tasks Cheetah-dir, Cheetah-vel, and Ant-dir have prompts of length K = 5. Figure shows that they need fewer sample data compared with Prompt- DT to achieve superior performance. The benefit of introducing the language model is to improve the data efficiency which is important in real-world application.
18

---

Published in Transactions on Machine Learning Research (11/2025)
Table 10: Detail on hyperparameters used in our experiments in Table We show that the hyperparameters in two parts which are parameters for model backbone and prompt regularization respectively.
Hyperparameters
Value
Length of training т
20
Length of prompt K
Training batch size for each task
Number of evaluation episodes for each task
Learning rate
le-4
Learning rate decay weight
lc-4
Language inilialization
GPT-2
Embedding dimension Activation
128
ReLU
Classifier hyperparameter
0.1
Classifier layers
2
Classifier MLP dimension
128
InfoNCE hyperparameter
0.1
InfoNCE temperature
InfoNCE MLP dimension
128
Charahov
38
(a) Chectah-dir
(b) Cheetah-vel
(c) Aut-dir
Figure 2: Training curves on MuJoCo controls with three tasks: Cheetah-dir, Cheetah-vel and Ant-dir for Prompt-DTT and our four methods LPDT- Cllsssias,,,,,,,, LPDT-InfoNNCE. LPDT-Rein- Clann--esssiferr and LPDT- Rein-InforrCE.. The dataset we utilized is the full dataset. We plot the figures on one unseen task with the average returns over evaluation episodes. The figures demonstrate that our LPDT needs less sample data to achieve good performance and be more stable during the training.
19


Reviews:
[{"review_id": "nWc8BWtetZ", "reviewer": "Reviewer_dc7i", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper builds off of recent literature applying transformers to RL problems. Specifically, it aims to make Prompt-Decision Transformers better and more data efficient by initializing with pre-trained language models. Variants of this method are compared to pre-existing techniques, including under varying amounts of training data. Ablation studies are used to identify aspects of the method that are most important for performance.", "claims_and_evidence_yn": "No", "claims_explanation": "I think there is a problem with the way results are analyzed and interpreted here. Specifically, the authors test 4 variants of their LPDT method. When they report whether LPDT beats other methods, they seem to consider LPDT a success if any one of these variants is better than existing methods. This is not a fair comparison; the other methods aren't allowed many variants, so they don't have as many chances to perform best. And indeed it is not the same LPDT variant that performs best across different tests. To make a claim about LPDT in total, it may be more appropriate to report the mean performance across all LPDT variants.", "audience_yn": "Yes", "audience_explanation": "Exploring how pre-trained LLMs may help prompt-based DTs is a reasonable thing to attempt, and so this paper would be of use to anyone interested in the outcome of that.", "requested_changes": "I don't think the manuscript mentions how the encoder and action weights are trained. Please include that. \n\nIt seems like the contrastive learning for the prompt encoder still requires task labels (to know if two samples are from same or different tasks), yet it is described as being relevant for when tasks aren't labeled. Please reconcile this. \n\n\nThe manuscript makes reference to LPDT possibly being more computationally efficient than other methods (for example, when contextualizing Meta-DTs performance by saying \"It is worth noting that Meta-DT introduces an additional prompt selection stage at test time to select high-quality prompts, while our methods avoid this computation overhead.\"). Can the authors include the relative sizes of the models to understand computational costs? \n\n\nIt's not clear to me that these results represent significant differences, so this claim should be toned down: \"showing that prompt regularization improves the model’s ability to distinguish between tasks and enhances overall performance.\" In general the authors also need to address how results around the four different variants of LPDT are being handled, as mentioned above.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iI8JIvq0fQ", "reviewer": "Reviewer_vxMK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes the Language model-initialized Prompt Decision Transformer (LPDT), a novel framework for improving few-shot generalization in offline reinforcement learning (RL). The method leverages pretrained language models to provide strong prior knowledge, integrates Low-Rank Adaptation (LoRA) for efficient fine-tuning, and introduces prompt regularization to better distinguish between tasks. The approach is evaluated on MuJoCo and Meta-World ML1 tasks, and is further extended to Reinformer, showing strong performance with less training data compared to baselines. Extensive ablation studies validate the contributions of each component.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Extensive experiments verify the advantage of the proposed framework.", "audience_yn": "Yes", "audience_explanation": "A significant portion of the audience is likely to be interested in offline RL and in how large language models (LLMs) can be leveraged to improve the performance of offline RL algorithms.", "requested_changes": "The writing and presentation of the paper are clear and well-structured. I mainly have the following concerns:\n\nThe method integrates LLM initialization, LoRA, and prompt regularization. To better evaluate the practical trade-offs of the proposed approach, it would be better to report the associated computational costs relative to existing baselines.\n\nThe paper claims that LPDT achieves performance comparable to Prompt-DT with only 10% of the data. However, this outcome appears only in limited cases, which makes the claim potentially overstated. Moreover, in Section 4.4, the reported results in the MW Reach-v2 task suggest that performance sometimes improves as the amount of training data decreases—a counterintuitive trend that warrants further discussion.", "broader_impact_concerns": "NA", "additional_comments": "NA", "extra_scores": {}, "extra_text": {}}, {"review_id": "ymVMcrJujB", "reviewer": "Reviewer_6SHM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Contributions:\n\nThe authors propose Language model-initialized Prompt Decision Transformer (LPDT), a RL framework that incorporates pre-trained language models.\nThe authors use Low-Rank Adaptation (LoRA) and prompt regularization methods to efficiently fine-tune the model and enhance the model's ability to distinguish tasks.\nThe authors also demonstrate the generalizability by extending the framework to a recent DT variant Reinformer.\n\nStrength:\n\n1. The authors have conducted comprehensive experiments and ablations to show the effectiveness of the proposed method.\n2. The paper is well-written and easy to follow.\n\nWeakness:\n\n1. The improvement over baselines (Meta-DT) is not obvious (See Table 1). Meta-DT remains more competitive in three environment.\n2. Tables are not well-organized. The color and bold criterion are not consistent. Different LPDT variants are shown but no specific discussion over different variants.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors conduct comprehensive experiments and ablation studies to validate the proposed claims.", "audience_yn": "Yes", "audience_explanation": "This submission is valuable to the RL community for the people working on decision transformer. It brings a new direction to incorporate pre-trained language models into prompt decision transformer and introduces insights to further research.", "requested_changes": "1. As LPDT-Rein variants results are reported in Table 1, it is better to report the results of Reinformer to compare with.\n2. Table 3 mainly shows the ablation results, so it is unnecessary to show Prompt-DT results. It's better to re-organize the table so that the comparison between w/ and w/o regularization is more obvious.\n3. As mentioned in the weakness, the authors should consider make the color and bold criterion the same across all tables. Currently in Table 1 the best two results are colored; in table 2 the best results are bold while in other tables no values are colored or bolded.\n4. The authors report the values for different variants (Classifier/InfoNCE). It would be better if the authors could provide additional analysis on the performance of these two variants.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)