Decomposed Prompt Decision Transformer for Efficient Unseen Task Generalization

Authors: hongling  zheng
li  shen
yong  luo
tongliang  liu
jialie  shen
dacheng  tao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:17:58

Abstract: Multi-task offline reinforcement learning aims to develop a unified policy for diverse tasks without requiring real-time interaction with the environment. Recent work explores sequence modeling, leveraging the scalability of the transformer architecture as a foundation for multi-task learning. Given the variations in task content and complexity, formulating policies becomes a challenging endeavor, requiring careful parameter sharing and adept management of conflicting gradients to extract rich cross-task knowledge from multiple tasks and transfer it to unseen tasks. In this paper, we propose the Decomposed Prompt Decision Transformer (DPDT) that adopts a two-stage paradigm to efficiently learn prompts for unseen tasks in a parameter-efficient manner. We incorporate parameters from pre-trained language models (PLMs) to initialize DPDT, thereby providing rich prior knowledge encoded in language models. During the decomposed prompt tuning phase, we learn both cross-task and task-specific prompts on training tasks to achieve prompt decomposition. In the test time adaptation phase, the cross-task prompt, serving as a good initialization, were further optimized on unseen tasks through test time adaptation, enhancing the model's performance on these tasks. Empirical evaluation on a series of Meta-RL benchmarks demonstrates the superiority of our approach. The project is available at https://github.com/ruthless-man/DPDT.

Main paper: 
 introduction: 
 The purpose of offline reinforcement learning (Offline RL) [1] is to develop a reward-maximizing RL strategy using offline data. This approach is of highly valuable in real-world scenarios where online data collection is expensive, time-consuming, or impractical. Existing offline RL algorithms typically perform well in single task but often struggle for multiple tasks with similar conditions and objectives, as they lack the ability to separate common knowledge from conflicting task gradients. In contrast, humans can leverage knowledge from existing tasks to excel in new ones, which has led to increasing research interest in multi-task reinforcement learning (MTRL) [2, 3, 4] . The goal of MTRL is to develop a universal strategy applicable to tasks with certain similarities, thereby enhancing adaptability and performance in multi-task environments.Decision transformer (DT) [5] and Prompt-DT [6] introduce the transformer architecture to the field of RL, demonstrating the powerful data modeling capability of sequence offline RL. Additionally, they provide possibilities for integrating advancements [7] from language modeling into MTRL methodologies. Prompt-tuning DT [8] uses a gradient-free method to introduce prompts, retaining context-specific information and catering to specific preferences. These existing sequence-based offline RL methods are primarily trained and tested on the same task or perform fine-tuning using a † Corresponding authors.38th Conference on Neural Information Processing Systems (NeurIPS 2024).small portion of labeled test data [9] . As a result, these algorithms often perform poorly when faced with testing tasks that are unseen and unlabeled in a Meta-RL setting [10] .Leveraging MTRL to extract general knowledge offers a promising approach for facilitating crosstask knowledge transfer in Meta-RL scenarios. However, as the number of tasks increases, gradient conflicts become more pronounced, hindering MTRL performance due to unregulated parameter sharing. Additionally, while transformer architectures can capture extensive relationships in offline sequential data, their data-hungry nature means that insufficient training data for RL tasks significantly diminishes model performance. A feasible solution might be to enhance the model's prior knowledge.To remedy these drawbacks, we propose a prompt-based MTRL method named Decomposed Prompt Decision Transformer (DPDT), inspired by some works in natural language processing where knowledge transfer in multi-task learning scenarios is achieved through prompting strategies [11, 12] . We first employ pre-trained parameters from GPT to initialize a DPDT architecture. Incorporating the parameters of PLMs is motivated by several studies in RL [13, 14, 15] . Leveraging the rich prior knowledge encoded in Pre-trained Language Models (PLMs) effectively addresses the data hunger challenge of transformer architectures, providing ample semantic information for reinforcement learning tasks. Then, we design a two-stage training and testing framework for DPDT. (1) Decomposed prompt tuning phase: At this stage, we use prompt decomposition to avoid gradient conflicts between different tasks and to extract common knowledge. Specifically, we decompose the task prompt for each task into a cross-task prompt and a task-specific prompt. The cross-task prompt remains consistent across all training tasks, while the task-specific prompt is tailored to each task's unique characteristics. By isolating the cross-task prompt from the task-specific prompts, the model ensures that updates related to general knowledge do not conflict with those related to specific tasks. Compared to [12] , our structured decomposition enables more regulated and harmonious parameter updates, thereby enhancing parameter efficiency and facilitating the extraction of general knowledge more effectively. (2) Test time adaptation phase: The cross-task prompt, serving as a strong initialization, is further optimized on unlabeled unseen tasks by incorporating the Test Time Adaptation (TTA) [16] to our model. TTA dynamically adjusts the cross-task prompts during the testing phase based on task characteristics, enhancing the model's adaptability to unseen tasks features.Our DPDT has been empirically validated in the Meta-RL setting, and the results demonstrate its superiority compared with many recent and competitive counterparts [5, 6, 17] . Furthermore, we conducted ablation experiments covering aspects such as prompt length, scalability, and model variants to establish the superiority of the model. The main contributions of this work are as follows:• We reconsider the problem of knowledge extraction in MTRL and propose the method of Decomposed Prompt Decision Transformer. • We propose a two-stage paradigm, which includes decomposing the task prompts for training tasks into cross-task prompts and task-specific prompts, and aligning the cross-task prompts in unlabeled unseen tasks. • We demonstrate the effectiveness of DPDT through intensive experiments on a broad spectrum of benchmarks, highlighting its competitive performance in Meta-RL scenarios. 
related works: 
 In this section, we summarize the most related works as two-fold: offline RL and multitask RL.Offline RL. In contrast to traditional RL methods [18, 19] , offline RL focuses on training models and performing trial-and-error using offline data without environmental interaction to arrive at appropriate strategies. These methods primarily address the issue of out-of-distribution (OOD) through strategies such as constraining the learning policies [20] or bounding the overestimated policy values [21] . The integration of transformer architecture in sequence modeling has emerged as a prominent approach for addressing offline RL tasks [22, 23] , further demonstrating the advantages of data-driven policy learning. Decision Transformer (DT) [5] involves encapsulating rewards, states, and actions into triples and training them using autoregressive supervision on offline data. Owing to the transformer's proficiency in capturing and fitting long time series features, it has achieved remarkable results in various offline RL tasks. HDT [17] generalizes new tasks by designing an adaptation module initialized by a hypernetwork. To alleviate the tuning burden while preserving performance, prompt tuning [24, 25] in NLP focuses on optimizing only the input parameters while keeping the majority of the PLMs parameters frozen. However, integrating prompt tuning into RL field poses a challenge, as RL prompts lack semantic information and are challenging to optimize. While Prompt-DT [6] selects pre-defined expert trajectories combined with inputs to guide model training, it primarily relies on the quality of these trajectories for improvement, rather than enhancing prompts directly. On the other hand, Prompt-tuning DT [8] stands out for introducing prompt-tuning techniques using a gradient-free approach, with the goal of preserving environment-specific details and accommodating specific preferences.Multitask RL. Multi-Task Reinforcement Learning (MTRL) [26, 27, 28] aims to address multiple similar reinforcement learning tasks using a unified model. A straightforward approach involves developing a task-conditional multi-task model, akin to those utilized in goal-conditional RL [29] and visual-language grounding [30] . While this method has demonstrated success in certain scenarios, it often encounters challenges stemming from negative interference among tasks. PaCo [31] delves into a compositional structure within the parameter space, distinguishing between task-agnostic and task-specific components. This approach significantly enhances the efficiency and robustness of the MTRL process, leading to more effective training outcomes. Building upon the MTRL paradigm, multi-task prompt [32] aims to acquire transferable, cross-task prompts from multiple tasks, guiding outputs for unseen downstream tasks. Several studies have approached multi-task prompt design from the perspective of prompt decomposition, yielding notable results across various tasks [33, 34] .The most relevant work to ours is Prompt-DT [6] , which utilizes carefully selected prompts for Meta-RL tasks training. Our approach differs in (1) employing trainable prompt decomposition to avoid gradient conflicts among multiple tasks, (2) further optimizing general prompts with TTA without using any test data labels, and (3) extending the model architecture by incorporating PLMs for initialization. To the best of our knowledge, we are the first to implement multi-task prompt tuning based on PLMs parameters in the reinforcement learning domain. 
preliminary: 
 In this section, we provide several concepts and terminologies that will be used in this work.Prompt-DT [6] examines the beneficial effects of incorporating trajectory prompts on the DT [5] in few-shot scenarios. Specifically, the form of trajectory prompts is the same as that of training trajectories, consisting of triplets made up of state s * , action a * , and return-to-go r * . However, these trajectory prompts are significantly shorter than the training trajectories. During training, for each task i, the trajectory prompt τ * i,K * is concatenated with the corresponding training trajectory τ i,K to form τ input i = τ * i,K * , τ i,K , which is then inputted into the model f for training. The concrete form of τ * i,K * and τ i,K are as follows:EQUATIONwhere K * represents the number of environment steps stored in the prompt, and K is the nearest steps of the training trajectory. The prediction head associated with the state token s is designed to predict the corresponding action a. It is noteworthy that during training, the model does not predict the actions of the trajectory prompts. The loss function is formulated as follows: a i,m denotes the actual action at the m-th timestep of the i-th task, while τ i,m-1 encompasses all data up to and including the (m -1) th timestep in the training trajectory of the i th task.EQUATION(2)Test time adaptation (TTA) [16] aims to minimize the gap between training data and testing data distribution during the testing phase. Test-time prompt tuning (TPT) [35] leverages the extensive knowledge in transformer architecture to enhance its generalization capabilities in zero-shot scenarios. During the inference phase, multiple randomly augmented views are created from the provided testing sample X test . Predictions with entropy below a specified threshold are retained, while other views are filtered out using a confidence selection criterion. The average entropy of the filtered predictions is then used to unsupervisedly update the prompts p. Some methods [36, 37, 38] , considering the data instability caused by augmentation techniques, use alignment of feature values in attention layers or feedforward layers to achieve test-time adaptation. Our method falls into this category. 
decomposed prompt decision transformer: 
 In this section, we provide a comprehensive description of the proposed DPDT. Task prompts and training data are combined to calculate the loss through action prediction, which is then used to optimize the cross-task and task-specific prompts in a backward pass. Once optimized, the cross-task prompts serve as a good initialization for use in unseen tasks during test-time adaptation. The objective of decomposed prompt tuning is to learn cross-task prompts and task-specific prompts through prompt decomposition. Test-time adaptation provides an alignment approach for the cross-task prompts. Below, we will describe each module of DPDT in detail.Initialization. Given that Transformers are data-intensive and require pre-training on substantial datasets to achieve satisfactory performance, integrating PLMs from the same architectural family into offline RL is a natural progression. Some existing work has already explored this avenue [13, 39, 40] .Taking inspiration from this, we employ GPT2-SMALL [41] to initialize our DPDT and maintain these parameters frozen throughout training. It is worth noting that, incorporating PLMs into RL is not predicated on the direct applicability of language data to RL tasks. Instead, the advantage lies in leveraging the deep, nuanced representations acquired by PLMs from a variety of datasets. These representations encode a broad spectrum of patterns, relationships, and contexts that can transcend purely linguistic tasks. This taps into the reasoning and few-shot capabilities of language models, addressing challenging scenarios like data scarcity and sparse rewards.Prompt Decomposition. Given a set of training tasks S = {S 1 , S 2 , . . . , S n }, our objective is to learn a general prompt P c that encapsulates common knowledge shared across all tasks in S and can efficiently adapt to unseen tasks. Extracting general task information from tasks with different distributions is often challenging, as gradient conflicts between tasks can lead to suboptimal convergence of information. We adopt prompt decomposition approach to address this issue. As shown in Figure 1 , let P c ∈ R l×s and P k ∈ R l×s . The task prompt P * k for the k-th task is obtained by taking the element-wise product of P c and P k . The goal of prompt decomposition is to enable efficient knowledge sharing across S, while still allowing each task to maintain its own parameters to encode task-specific knowledge. The P c aims to acquire general knowledge from S, while the task-specific prompt P k allows task k to retain its unique knowledge. The task prompts parameterization of the k-th training task is expressed as: In the specific implementation process, inspired by LORA [42] , we further decompose each taskspecific prompt into two lowrank vectors v k ∈ R l×r and u k ∈ R l×s using a low-rank method. We obtain P k through vector multiplication. Here l represents the prompt length, r represents the hidden layer dimension, and s represents the prompt dimension. The hyperparameter r is a manually specified lowrank parameter. Its introduction is crucial for designing prompts for all tasks in the dataset, significantly maintaining model superiority while reducing computational load. We use standard normal distribution to initialize P c , u k and v k . Given the DPDT M, the mean squared loss between the model's predicted actions and the true actions is calculated as:EQUATIONEQUATIONPrompt distillation. Due to the lack of explicit constraints, directly implementing prompt decomposition on the multitask dataset S may lead to an overlap in the information learned by P c and P k , potentially undermining their ability to capture distinct intended details. We employed knowledge distillation techniques to compel the cross-task and task-specific prompts to learn their respective information. We obtained teacher task prompts p teacher k for each task k by using traditional prompttuning methods individually. During training, the mean squared error is calculated directly between p teacher k and p ⋆ k :EQUATIONThe total loss function for training task prompts for obtaining a cross-task prompt to be transferred to the target side is then:EQUATION)where λ is a weight to balance the impact of distillation loss terms. In our experiments, we set λ to 0.5. The overall summary of the multitask training algorithm is presented in Algorithm 1.During the test time adaptation (TTA) phase, we address distribution bias by aligning the distribution of unlabeled test samples with the training samples. For each test task t in the test task set T , we randomly select a subset X of unlabeled test samples, combine them with the cross-task prompts P c and input them into the model.Here, we introduce the data collection method for X. The model's testing phase usually occurs in a simulated environment where we predefine our expected reward values r. The environment provides the initial state s of the environment, consistent with the settings during inference in prompt DT methods. However, unlike in training tasks where ground-truth labels exist, for action a 1 , we assign a value sampled randomly from the action space (which is typically consistent between training and testing tasks). We feed this sequence of Markov chains into the environment, obtaining rewards and the next environment states iteratively, assigning a randomly sampled value to action a 2 in subsequent iterations. This process is repeated |X| times, resulting in data of the form (r0, s 0 , a 0 , r1,s 1 , a 1 , . . . , r|N| , s |N | , a |N | ).In each layer of the model, we calculate the alignment loss based on the means and variances of the training and test samples. Our goal is to update the P c for the given test task through this alignment loss. For each test task t, we denote the distribution of the test samples as T and the distribution of the training samples as D. Specifically, we calculate the aligned token mean and variance via:EQUATIONAlgorithm 2 Test Time Adaptation Input: Test samples set X, Cross-task prompts Pc, µ l (D), σ 2 l (D), The number of layers L.1: for l = 1 to L do 2:for i in X do 3:Calculate H l,i obtained by inputting the concatenation of Pc and i into DPDT.end for 5: end for 6: for l = 1 to L do 7:Compute µ l (T ) and σ 2 l (T ) by Equation 7. 8: end for 9: Compute token distribution alignment loss by Equation 8. 10: Optimize Lalign to update Pc.Here, H l,i represents the state of the i th sample at the l th hidden layer, while µ l (T ) and σ 2 l (T ) denote the mean and variance of all test samples at the l th hidden layer, respectively. Similarly, for each hidden layer of the model, we pre-calculate the statistical measures of mean µ l (D) and variance σ 2 l (D) of the training samples, which are uniformly sampled across all tasks in the training set, in an offline setting to reduce parallel computing costs, since both training samples and labels are accessible. The formula for calculating the alignment loss function is as follows:EQUATIONThe test time adaptation phase process is illustrated in Algorithm 2. 
experiment: 
 In this section, we present an extensive evaluation of our proposed DPDT using widely recognized benchmarks. Additionally, we conduct empirical ablation studies to dissect and understand the individual contributions of the core components of our methodology.Environments. To ensure a fair comparison with existing multi-task offline reinforcement learning algorithms, we conducted verification of DPDT using the MuJoCo [43] and MetaWorld [30] benchmarks, which serve as standard tasks in the domain of sequence offline RL, offering sufficient diversity and representing common challenges in classical RL, such as sparse rewards, complex state spaces, and precise control of robotic systems. Our experiments on the Cheetah-dir, Cheetah-vel, and Ant-dir environments in the MuJoCo benchmark meticulously adhere to the datasets and methodologies outlined in Prompt-DT. These tasks penalize agents for using excessive control signals. In the MetaWorld benchmark, we used the ML10, ML45, MT10 and MT50 environments for Meta-RL. A detailed description of the datasets and the division of training and test tasks is provided in Appendix A.Baselines. We compared DPDT to the following offline RL baselines: (1) Multi-task Behaviour Cloning (MT-BC) [44] : MT-BC optimizes multi-task learning by exclusively simulating trajectories from the original dataset, dispensing with the need for prompts and reward-to-go tokens. This approach emphasizes the utilization of intrinsic task-specific information, adopting a behavior cloning Table 1 : Results for Meta-RL control tasks (zero-shot scenarios). The best mean accumulated returns are highlighted in bold. For each prompt-needing environment, prompts of length K=30 are utilized. Each experiment was run three times to ensure stability and reproducibility of the results. We report the average returns and standard deviations for these three runs (the higher, the better).MT-BC [44] MT-ORL [5] Soft-Prompt [45] HDT [17] Prompt-DT [6] strategy to streamline the learning process. (2) Multi-Task Decision Transformer (MT-ORL) [5] :We train a decision transformer to learn multiple tasks from the training set. To construct the MT-DT, we exclude prompt augmentation present in DPDT, while retaining the rest of the training process identical to that of DPDT. (3) Soft-Prompt [45] : Soft-prompt is trained using a universal prompt across all tasks. (4) Hyper-decision transformer (HDT) [17] : HDT efficiently adapts DT to new tasks by augmenting them with an adaptation module, whose parameters are initialized by a hyper-network, enabling quick and efficient adaptation with minimal data. ( 5) Prompt-DT [6] : Prompt-DT builds on DT, leveraging trajectory prompts and reward-to-go for multi-task learning and generalization to unseen tasks.All experiments were carried out on a server with 8 NVIDIA 3090 GPUs, each with 24GB of memory, using PyTorch [46] and Hugging Face Transformers libraries [47] . The experimental hyperparameter configurations are shown in Appendix B. The computer resources utilized by all methods are shown in Table 12 .Zero-shot Generalization. In Table 1 , we compare the zero-shot generalization ability of DPDT and the baselines to investigate the overall performance of DPDT. For evaluation, we use the average episode cumulative returns in the test task set as the evaluation metric. Additionally, we introduce a variant of DPDT that does not use GPT-SMALL parameters for initialization, referred to as DPDT-WP (DPDT-Without Pretrained). Soft prompts adapt to tasks in a parameter-efficient way. However, training a universal prompt across multiple tasks suffers from significant gradient interference, as demonstrated by the experimental results. The prompt-DT performs well in few-shot scenarios due to its utilization of test data for fine-tuning. However, in downstream tasks where data is scarce, the prompt fails to provide sufficient task-specific guidance to the model, resulting in suboptimal performance. Importantly, our proposed DPDT exhibits significant performance improvements over fine-tuning and prompt-based methods. This vividly demonstrates the distinct advantages offered by our innovative multitask training techniques. It is worth noting that without initialization with PLM, the performance of DPDT-WP is inferior to most methods. We believe this is mainly due to the model lacking sufficient prior knowledge for effective multi-task prompt tuning, resulting in suboptimal performance of DPDT. In Figure 2 , we illustrate the accumulated returns curves of DPDT and other baselines across the Cheetah-vel, MW ML45, and MW MT50 environments. Additional curves for other environments can be found in Figure 4 . We also conducted experiments on Soft-Prompt and Prompt-DT combined with TTA (shown in Figure 11 ). Soft-Prompt-TTA showed performance improvements across all tasks, whereas Prompt-DT-TTA experienced performance declines in some tasks. The main reason for this is that Prompt-DT relies on high-quality trajectory data for prompts during testing, and applying TTA on unlabeled data may have adversely affected prompt optimization.Few-shot Generalization. We explored the performance of DPDT in few-shot scenarios and further investigated whether the prompt decomposition mechanism successfully isolated general knowledge. In this scenario, DPDT does not use TTA to align cross-task prompts P c . Instead, P c is fine-tuned Table 2 : Results for Meta-RL control tasks (few-shot scenarios). The best mean accumulated returns are highlighted in bold. For each prompt-needing environment, prompts of length K=30 are utilized. Each experiment was run three times to ensure stability and reproducibility of the results. We report the average returns and standard deviations for these three runs (the higher, the better).Soft-Prompt [45] HDT [17] Prompt-DT [ directly on a small number of labeled test samples through a self-supervised paradigm. Specifically, we randomly selected only one trajectory from the test dataset for fine-tuning. Other methods adhered to the same amount of fine-tuning data. Furthermore, in few-shot scenarios, we fully fine-tuned DPDT on the complete training and testing data, denoted as DPDT-F. The performance of the DPDT-F method represents the upper bound of all model performances in the current environment. Table 2 shows that the DPDT method, even after fine-tuning, still significantly outperforms or matches the baseline algorithms, demonstrating the effectiveness of prompt decomposition in fewshot environments. Moreover, in some datasets, DPDT approaches the performance of fully fine-tuned models on the test set using only 1.14% of the parameters, as observed in the cheetah-vel environment.It is worth noting that in the Ant-dir environment, the performance of DPDT is slightly inferior to that of Prompt-DT in both zero-shot and few-shot settings. We believe the primary reason for this is the significant domain difference between the language model and environment.In the ablation studies, we conducted research on the components of DPDT as well as the impact of prompt length and model parameters on convergence speed and performance.Impact of model components. As shown in Table 3 , the impact of prompt decomposition was evaluated. Compared to the soft-prompt method in the unseen task settings (first row), substituting it with decomposed prompts P k and P c without distillation (third row) resulted in performance improvements across all three tasks. This ablation highlights the significance of the prompt decomposition strategy in DPDT, demonstrating that the shared component adeptly captures the diverse cross-task knowledge essential for enhancing target downstream tasks.To evaluate the impact of prompt distillation, we trained a standard prompt shared across all training tasks using the same training loss as DPDT. The teacher prompts for each task remained consistent in DPDT. Compared to the basic baseline (first row), incorporating prompt distillation (second row) resulted in a modest improvement in average performance across the three tasks. This emphasizes that prompt distillation from separately trained source prompts is an effective strategy for acquiring high-quality decomposable prompts.To compare the impact of using TTA on model performance, we examined the results in the fourth and fifth rows. We found that the use of TTA affects the model's final performance. The reason is intuitive: cross-task prompts provide a good initialization environment for TTA, but relying solely on the general information from cross-task prompts is insufficient for the model to perform well on unseen tasks. Incorporating TTA allows the model to adapt to the specific nuances of each task during testing, resulting in substantial performance gains.Impact of prompt length. We examined the influence of prompt length on the performance of DPDT by investigating five distinct prompt lengths (3, 6, 30, 60, 90) . Specifically, we explored the effect of prompt length variation on the convergence behavior and generalization capability of the model. It is widely recognized that prompt lengths that are excessively short may impede model convergence, while prompt lengths that are overly long can result in slow convergence rates and potential overfitting. Ablation experiments revealed that a prompt length of 30 is optimal. Further increasing the prompt length to 60 or 90, however, results in minor performance fluctuations but increases the convergence time. Therefore, we used a prompt length of 30 for all our experiments. Impact of model size. We explored the performance of DPDT under three model size configurations. The (3,1,128) configuration uses the pretrained model provided in the original Prompt-DT [6] to initialize DPDT, while the (24,16,768) configuration employs GPT-MIDDLE. Table 4 shows that the size of the pretrained model parameters is correlated with the performance improvement of DPDT. When the model size is expanded to a certain extent (12, 12, 768) , efficient parameter fine-tuning can extract adequate prior knowledge for downstream tasks. However, as the model complexity increases, such as when reaching the size of GPT-MIDDLE, the model exhibits performance improvement on some tasks (Ant-dir) but a decrease in performance on others. This phenomenon can be attributed to the significant gap between the size of the dataset and the complexity of the model. Fine-tuning the model in such scenarios may encounter challenges in appropriately converging for reinforcement learning tasks, potentially leading to overfitting.Impact of data quality. The quality of data used for fine-tuning cross prompts does indeed affect the final model performance. We conducted experiments focusing on data quality. In the Cheetah-vel and ML45 environments, we differentiated the quality of datasets into expert, medium, random, and mixed datasets. Each dataset consists of 200 time steps, which aligns with the setup for few-shot scenarios relative to the size of the training set. As shown in Table 5 , we found that models fine-tuned using expert datasets perform the best, which aligns with our intuition. Additionally, the performance of models fine-tuned on mixed datasets is close to that on expert datasets, suggesting implicitly that the DPDT method can extract information from suboptimal datasets to ensure model performance. Impact of adaptation method. In addition to only utilizing cross-task prompts P c for TTA in zero-shot scenarios, we also investigated (1) combining crosstask prompts P c with the average of all task-specific prompts P k from the training set for TTA, (2) freezing the cross-task prompts P c , we initialized a new task-specific prompt combined with the cross-task prompts for TTA and (3) randomly selecting one P k from a training task and combining it with P c for TTA. However, we found that all of these initialization methods resulted in suboptimal outcomes, shown in Table 10 . Impact of learning rate in prompt decomposition. As shown in Table 6 , we present experimental results on the ML45 dataset where different learning rates were applied to P c and P k in prompt decomposition. We observed optimal performance when both had the same learning rate. We speculate that this occurs because, over training iterations, both prompts converge to their optimal values, and differing learning rates disrupt their joint convergence, leading to poorer performance under similar runtime conditions. Impact of low-rank parameter r. Table 7 presents the results of our ablation experiments focusing on the low-rank parameter r of prompt decomposition for Cheetah-vel and MW ML45. DPDT is relatively insensitive to selecting hyperparameters, a potential advantage of our work. As observed, the model's performance varies with different values of r. These findings suggest that the performance of DPDT remains stable across different values of r. This characteristic can be advantageous, as it allows users to implement the model without extensive tuning, streamlining the deployment process while still achieving competitive results across diverse tasks. 
conclusion, limitation and broader impact: 
 We have introduced a novel approach, the Decomposed Prompt Decision Transformer (DPDT), aimed at efficient generalization to unseen tasks. Through the utilization of PLMs for parameter initialization and the implementation of parameter-efficient multi-task prompt tuning techniques, we have successfully extracted cross-task general knowledge and further fine-tuned it on previously unseen tasks. Our experiments across various Meta-RL environments demonstrated the effectiveness of our components, achieving superior performance with significantly fewer task-specific parameters compared to fully fine-tuned methods. This approach offers a robust framework for future research to further explore and optimize multi-task learning and generalization capabilities.Limitation. Currently, our work primarily involves using large language models to initialize DPDT. While we've utilized parameter-efficient techniques to fine-tune the model and mitigate inter-domain variances, focusing on optimizing these differences could potentially enhance model performance.Broader Impact. Overall, the application of PEFT methods based on PLMs can help users obtain high-quality reinforcement learning decision models at minimal cost. However, this approach may lead to the misuse of language models when users are unaware of the inter-domain differences, potentially resulting in unforeseen negative outcomes in the RL decision-making process.  
broader impacts: 
 Question: If appropriate for the scope and focus of your paper, did you discuss potential negative societal impacts of your work?Answer: [Yes] Justification: We discuss the broader impact in Section 6Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Do you have safeguards in place for responsible release of models with a high risk for misuse (e.g., pretrained language models)?Answer: [NA]Justification: The paper uses standard benchmark datasets and does not involve any high-risk data or models. The pre-trained models and datasets are publicly available.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 
licenses: 
 Question: If you are using existing assets (e.g., code, data, models), did you cite the creators and respect the license and terms of use?Answer: [Yes] Justification: We use standard benchmark datasets, which are properly credited and have open licenses.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We show the hyperparameter of DPDT and other baslines in Table 8 and Table 9 . Justification: We provide the information in Table 12 .Guidelines:• The answer NA means that the paper does not include experiments. Justification: This paper does not involve human subjects, and we have not used any personal data.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper originally received mixed reviews, and concerns regarding:

1. concerns with regard to the comparison to Prompt DT - the main baseline for the paper
2. reliance on long prompts and LLM for initialization
3. clarity
4. fairness of use of test-time adaptation with regards to baselines that do not use it
5. novelty
6. baselines reproduction and choice of HP
7. statistical significance of the gains

the authors seem to have been able to address the concerns of 2/3 reviewers who raised their score to Accept and Weak Accept. However, the reviewer urHM remained unconvinced and provided additional reasons to reject the paper, albeit after the reviewer-authors discussion period concluded, not allowing the authors to respond to these concerns. In light of the above, and observing some overlap between addressed concerns (as confirmed by other reviewers) and new concerns by urHM, the AC is inclined to accept the paper. Yet, the AC urges the authors to seriously consider all concerns raised by the reviewers, and incorporate all the feedback and responses into the revised version of their paper. The AC strongly suggests the authors pay special attention to additional concerns raised by urHM after the discussion period, also addressing them and incorporating them into the final revision.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel Multi-Task Prompt Decision Transformer (MPDT), which leverages pre-trained language models as the initialization and adopts test-time adaptation. This approach achieves efficient generalization to unseen tasks through the prior knowledge from the pre-trained language model and by decomposing the task prompt. In the multi-task training stage, the prompt is decomposed into a cross-task prompt and a task-specific prompt, which can reduce gradient conflicts and computational load. Besides, the task-specific prompt is further decomposed into two low-rank vectors. Prompt distillation is also used to improve the quality of the prompt decomposition. In test-time adaptation, the method further optimizes the cross-task prompts on unseen tasks via alignment loss. An empirical study on Meta-RL benchmarks demonstrates the superior performance of MPDT compared to existing methods.
strength_weakness: strength_weakness: - This paper proposed utilizing the pre-trained language model as initialization but lacks an explanation of why the language pre-trained model could contribute to the RL tasks. 
- This method needs a long prompt to perform well compared with the Prompt DT. 
- This paper lacks detailed explanation on why using the cross-task prompt, task-specific prompt, and the prompt distillation.
questions: questions: - Can you explain why you use the pre-trained embedding layer for word tokens to encode the RL tokens?
- Can you explain more about why you use prompt decomposition and prompt distillation?
- Can you explain why you need to use a long prompt to achieve better performance?
limitations: limitations: See weaknesses and questions.
paper_summary: paper_summary: This paper proposes a new method called Multi-Task Prompt Decision Transformer (MPDT) for efficient generalization to unseen tasks in offline reinforcement learning. MPDT involves two stages. First, the multitask training phase: MPDT is initialized with parameters from a pertained LM, which is GPT2. It decomposes the task prompt into a cross-task prompt shared across tasks and task-specific prompts. Second, the test time adaptation: The cross-task prompt is further optimized on unlabeled test tasks using test time adaptation by aligning the distributions of the test samples and training samples. The paper evaluates MPDT on seven meta RL environments from both MuJoCo and MetaWorld, showing its superior performance over baselines in generalizing to unseen tasks.
strength_weakness: strength_weakness: 1. The paper appears to be hastily written and the presentation is hard to follow. (see question section)

2. The authors claim that a major improvement compared with other PDTs is the decomposition of prompt into the common part and task-specific part. I was expecting to see the common part will be trained across tasks and the task-specific part is within a specific task. However, in line 186, the authors say ‘We use standard normal distribution to initialize Pc, uk and vk.’ Besides, in Eq(4), they are equally optimized for each task. I didn’t see why the common part is the slow weights and the task-specific part is the fast weight, which is claimed by authors in line 167. Possible to explain here?

3. In line 192, the sentence ‘we obtain teacher task prompt $p_k^{teacher}$ for each task by using traditional prompt-tuning method individually. ‘ What method is used? How to learn it?  Is there a separate learning phase for $p_k^{teacher}$? If so, the method requires three training phases instead of two.  Also its quality matters. Can authors explain the details here?

4. In line 201, for test time adaptation, ‘ we randomly select a subset X of unlabeled test samples,….’ Can authors clarify what is this subset? A sequence in the DT is organized  as (r_t, s_t, a_t, r_{t+1}, s_{t+1}, a_{t+1}…). I recommend being specific what you select?
questions: questions: 1. How do tasks for each meta-environment differ? Do they have different reward functions $R(s_t, a_t)$ or transition function $T(s_t, a_t) \to s_{t+1}$. It question is important for the proposed method MPDT. Because I wonder what kind of distribution shift that test-test can handle and what it cannot.

2. How does the quality of the few shot prompt effect the model’s final performance?  Specifically, using expert chunk, random chunks, medium policy chunks, and mixed quality chunks. It would be better to have some analysis here.

Presentation Questions:

3. In algorithm 1, L_{dis} is calculated without being used?

4. In Eq.(4) What is $M((P_k^*, \tau))$? It is used without any explanation? Also, how many input does M take? Why there is a double layer bracket? 

5. In Line 176, the authors state that  $P_k\in \mathbf{R}^{l\times s}$ is the vector multiplication of two low rank vectors $v_k\in R^{l \times r}$ and $u_k \in R^{l\times s}$. First low-rank is a matrix property, there is no low-rank vector. Second, putting the wording issue aside, what is the vector multiplication here? If it is the matrix multiplication, which is used in the low-rank decomposition methods, then the shape of the output should be $r\times s$. Better to clarify here.
limitations: limitations: See my above comments for weakness and questions
paper_summary: paper_summary: Multi-task learning is a critical pursuit in decision-making, and Decision Transformer (DT) is a popular framework in solving various decision-making problems. The author observe the suboptimal performance of prior work utilizing DT for multi-task learning, and propose a new method, Multi-Task Prompt Decision Transformer (MPDT), to alleviate this issue. MPDT is mainly composed of these components:
- GPT2 pre-trained weights for initialization.
- Prompt decomposition: cross-task prompt + task-specific prompt to prevent gradient conflicts.
- Test time adaptation (TTA): dynamically optimizes the cross-task prompts during testing time.

Experiments are conducted on standard Meta-RL tasks, demonstrating prominent improvements compared to prior baselines. Ablation experiments are also extensively conducted.
strength_weakness: strength_weakness: *Significant*
- It is unclear to the reviewer that whether it is fair to compare MPDT with baselines like MT-BC, Soft-Prompt, and Prompt DT. since MPDT adopts much test-time information. How can the authors guarantee that they use the same amount of information in the test set for all baselines? If it is truly unfair, it would make the experiments less convincing. Please provide explanations for this.
- Did the authors reproduce the results of baselines by themselves? If so, how did they pick the hyper-parameters? And for few-shot generalization, which appears to be a new benchmark designed by the authors, did the authors extensively tune the hyper-parameters of baselines?

*Major*
- It is always hard to claim the so-called "SOTA", which demands very rigorous statistical analysis. And it is rarely true in nowadays AI community, see https://rl-conference.cc/review_guidelines.html. Running 3 times is acceptable to present the performance, but not enough to support "SOTA". And the variances in Table 1,2 are very large (some [$\mu-\sigma$, $\mu+\sigma$] intervals are overlapped), thus making it hard to establish statistical improvements. The reviewer recommends removing this claim.
- The framework is a bit complicated and engineering. Specifically, the 3 components, namely pre-training+prompt decomposition+TTA, are orthogonal and not well-connected. And thus the technical novelty of the method is a bit lacking. The novelty concentrates on prompt decomposition+distillation, while solely applying them doesn't achieve good performance (Table 3).


*Minor*
- The wording "Unseen Tasks Generalization" and "Few-shot Generalization" in Section 5.2 are misleading, as few-shot generatlization is also for unseen tasks.
- The name of the proposed method, *Multi-Task Prompt DT*, might not be suitable, since it cannot reflect the differences from Prompt DT, which is already a well-developed algorithm in muti-task RL.
questions: questions: - It looks unnatural to the reviewer that only the weights of prompts are trainable while the embedding layer and GPT blocks are all frozen, as shown in Figure 1. The ablation experiments of [1][2] show that parameter-efficient tuning like LoRA achieves better performance than freezing the whole block. Can the author provide some ablation experiments or reasonable explanations for this?
- Notably, the ablation experiments in Table 3 and performance of MPDT-WP show that, only putting all these 3 components together can achieve best performance. If these components are really orthogonal and helpful, then why simply using one or two of them won't achieve overall improvements? For example, on MW ML45, only using all 3 components can beat Prompt-DT. Could the author share some insights on this?
- How much efforts did the authors spend in tuning the hyper-parameters of MPDT?

[1] Can Wikipedia Help Offline Reinforcement Learning? arXiv preprint arXiv:2201.12122.

[2] Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning. arXiv preprint arXiv:2310.20587.
limitations: limitations: The limitations are discussed in the paper.
paper_summary: paper_summary: This paper proposes the Multi-Task Prompt Decision Transformer (MPDT) algorithm for zero-shot multi-task offline reinforcement learning (RL). Leveraging a pre-trained language model (PLM) with prompt tuning, the MPDT innovatively decomposes multi-task prompts into task-specific and cross-task components. It also achieves zero-shot generalization through test time prompt alignment. Evaluations across various benchmarks demonstrate that MPDT outperforms prior multi-task (meta) offline RL methods.
strength_weakness: strength_weakness: - My main concerns come from the novelty of the paper. The major contribution lies in the use of cross-task prompts and task-specific prompts, specifically the prompt decomposition and prompt distillation in Section 4.1.
- Regarding prompt decomposition, although the authors aim for cross-task prompts to contain common knowledge across tasks, the model and loss design do not ensure this. Specifically, since the authors use element-wise multiplication on $P_c$ and $P_k$, $P_c$ functions more as a common scaling factor for different task-specific prompts. After training, $P_c$ could become a constant scalar, and $P_k$ could simply scale as $1/N$ of $P_k^{teacher}$. The authors could provide the distribution of $P_c$ values to verify if it truly encapsulates common knowledge. And it is better to have a loss function to guide the common knowledge extraction during training.
- About the test time adaptation, if the authors believe that $P_c$ contains the common knowledge, I suppose we should train a randomly initialized $P_k$ (or select one from a training task according to task similarity) for the test task using the alignment loss. This way, the model will still take $P_k \cdot P_c$ as prompt input.
questions: questions: Please refer to the weakness part.
limitations: limitations: The limitations and social impact are provided in the conclusion.

Comments: title: Respectful Request for Reviewer‘s Valuable Feedback on Our Rebuttal
Response to the Reviewer urHM
Response to the Reviewer 6jy3
Response to the Reviewer 6jy3（2）
Response to the Reviewer 6jy3（1）
Response to the Reviewer d5sD
Response to the author
comment: Dear Reviewer zhrS,

We sincerely appreciate the time and effort you have already dedicated to reviewing our submission. We have carefully considered and addressed your initial concerns regarding our paper. Given that the discussion phase is nearing its end, we would greatly appreciate it if you could share any further feedback so we can respond promptly. Additionally, we welcome any new suggestions or comments you may have.

Thank you once again for your valuable insights and continued support.

Best regards,  
The authors of Submission 9319
We sincerely appreciate the reviewers' thorough examination and hold the reviewers' opinions in the highest regard. We strive to do our utmost to address reviewers' concerns, not only about the acceptance of the paper but also in providing meaningful insights for the RL community.

The reviewer's concern can be understood as the possibility that not retraining certain layers of the language model may result in an inadequate understanding of RL data by the model. We believe preserving the language model's knowledge is crucial, and prompts can bridge the gap between the two tasks without retraining the embedding layer or GPT blocks. In AR3, we conducted an ablation study on whether to freeze the embedding layer. The experimental results are intuitive, as training only the embedding layer while freezing the GPT blocks still leads to a suboptimal understanding of input information. Ablation study conclusions from [1][2] also demonstrate that the embedding layer and GPT blocks are highly correlated ([1] performs full fine-tuning of a language model on RL tasks. In contrast, [2] trains a language model on RL tasks using LoRA). Comparing [1][2] with MPDT can further address the reviewer's concerns. The reviewer's concerns can be extended to whether to (1) freeze both the embedding layer and GPT blocks simultaneously and add external components (e.g., prompt) or (2) train both the embedding layer and GPT blocks together (either full fine-tuning or using LoRA). Only these two approaches can achieve better performance. The choice of approach involves a trade-off between model performance, computational cost, and scenario requirements. Our decision to adopt approach (1) is based on the following reasons. 

From the perspective of model performance, the primary purpose of using prompts is to preserve the model's inherent prior knowledge as much as possible. Both [2] and we believe that the full model fine-tuning used in [1] may lead to model overfitting, further disrupting the internal knowledge of the model. We considered adapting the method from [2] to our task (training the embedding layer and LoRA), but the results obtained based on the code provided by [2] differed significantly from those in the original paper. Moreover, when we applied [2] to our datasets, ML45 and MT50 (as shown in the table below), the performance on the training set was far inferior to MPDT, and even careful hyperparameter tuning could not alleviate the significant fluctuations in the reward curve. We speculate that language models using the LoRA structure may still face convergence difficulties on RL data.

||ML45 (Training Performance)|MT50 (Training Performance)
|-|-|-
|LAMO[2]| $367.563\pm129.37$ |$ 930.84\pm 437.62$
|MPDT| $ 604.21\pm11.05$ |  $1687.35\pm4.09$

Furthermore, [2] introduced a language prediction auxiliary loss to ensure the model retains its language task memory during training. However, if the introduction of LoRA is entirely suitable and does not disrupt the internal knowledge of the model, there should be no need for an auxiliary loss. In other words, we must fully understand whether retraining the model internally (including the embedding layer or GPT blocks) fundamentally introduces new knowledge by disrupting existing knowledge (replacement) or guides the model to use existing knowledge to achieve RL tasks (integration). The distinction between these two approaches results in differences in the model's performance ceiling and floor. The extent of adjustments within the model may need to be carefully controlled. Most importantly, freezing the embedding layer and GPT blocks preserves the model's complete knowledge, which we believe is a better choice, and the experimental results also support this.

From the perspective of computational cost, the advantages of MPDT are clear (1.42M vs. 125.5M fine-tune) or (1.42M vs. 3.5M LoRA). Fully fine-tuning a model is undoubtedly costly, and LoRA's 3.5M parameters are based on the adjustable parameter count in a single-task scenario, as reported in [2]. When extending to a multi-task scenario, the parameter count will increase proportionally to the number of tasks.

From the perspective of scenario requirements, prompts have sufficient capacity to guide pre-trained models to adapt to reinforcement learning tasks with smaller data sizes. Additionally, achieving zero-shot generalization with prompts results in significantly less disruption to the model's knowledge and a lower error tolerance rate than fully fine-tuning the model.

We demonstrated from both experimental results and theoretical principles whether it is necessary to freeze the embedding layer and GPT blocks. We sincerely welcome further discussion with the reviewer and will try to address any concerns.

[1] Can Wikipedia Help Offline Reinforcement Learning? arXiv preprint arXiv:2201.12122.  
[2] Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning. arXiv preprint arXiv:2310.20587.
Thank the authors for their detailed responses and additional experiments. The Rebuttal AR3 experiment demonstrates the performance of sticking the word embedding. However, the explanation can not convince me.  I will keep the score.
We agree with the reviewer's point and sincerely thank the reviewer for raising the score.
Thank you for your insights! From the results, the reviewer guesses that LoRA tuning trick might only be superior when learning single task with limited data. Now all my concerns resolved. 

I will raise my score to 7.
From the perspective of computational cost, the advantages of MPDT are clear (1.42M vs. 125.5M fine-tune) or (1.42M vs. 3.5M LoRA). Fully fine-tuning a model is undoubtedly costly, and LoRA's 3.5M parameters are based on the adjustable parameter count in a single-task scenario, as reported in [2]. When extending to a multi-task scenario, the parameter count will increase proportionally to the number of tasks.

From the perspective of scenario requirements, using prompts offers a natural advantage for knowledge transfer. Prompts have sufficient capacity to guide pre-trained models to adapt to reinforcement learning tasks with smaller data sizes. Additionally, achieving zero-shot generalization with prompts results in significantly less disruption to the model's knowledge and a lower error tolerance rate than fully fine-tuning the model.

[1] Can Wikipedia Help Offline Reinforcement Learning  
[2] Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning

**Significant concern: The other hyperparameters of Prompt-DT must be tuned due to a change in the model size.**

We did indeed use the 12-layer, 12-head version of Prompt-DT. To ensure a fair comparison, we reran the experiments and carefully adjusted some hyperparameters (setting the learning rate to 2e-5 and the learning rate decay weight to 1e-5). Other hyperparameters are independent of the model size. The model performance did not change significantly under the optimal hyperparameter configuration. The table below shows the performance of Prompt-DT and Prompt-DT-TTA across all tasks after adjusting the hyperparameters. We will replace the corresponding results in the main text with these updated results.

Table 1: Zero-shot generalization
||Prompt-DT|Prompt-DT-TTA
|-|-|-|
|Cheetah-dir|$-7.92\pm2.97$|$9.03\pm2.11$
|Cheetah-vel|$-192.38\pm11.80$|$-203.07\pm4.01$
|Ant-dir|$123.46\pm10.70$|$121.64\pm3.83$
|MW ML10|$317.31\pm14.98$|$314.08\pm12.93$
|MW ML45|$294.55\pm8.71$|$294.87\pm10.06$
|MW MT 10|$1087.54\pm17.09$|$1030.85\pm14.77$
|MW MT 50|$994.63\pm5.99$|$1137.69\pm13.58$

Table 2: Few-shot generalization
||Prompt-DT
|-|-|
|Cheetah-dir|$934.78\pm5.33$
|Cheetah-vel|$-37.80\pm2.09$
|Ant-dir|$411.96\pm9.28$
|MW ML10|$315.07\pm6.17$
|MW ML45|$473.34\pm4.12$
We sincerely appreciate the reviewers' thorough examination. Below, we address the concerns one by one.

**W1: Prompt DT cannot utilize the same amount of test data, and the TTA is unsuitable.**  
The Prompt-DT+TTA method selects the prompt $p$ that allows the model to perform optimally on the test task from the high-quality trajectories of the training tasks and then fine-tunes $P$ on the same amount of unlabeled test data as MPDT for TTA. This approach ensures zero-shot generalization (without using high-quality trajectories from the test task as prompts) and effectively combines Prompt-DT with TTA. We believe this is the optimal way to adapt Prompt-DT to zero-shot scenarios. The performance fluctuations of Prompt-DT+TTA across different tasks are mainly because Prompt-DT is not inherently a prompt-tuning method (despite using frozen prompts), as it requires high-quality labeled trajectories as prompts during both testing and training, which is a very strong prior condition. It would be more accurate to say that Prompt-DT might not be suitable for zero-shot generalization scenarios rather than stating it is unsuitable for TTA. In few-shot scenarios, MPDT still outperforms Prompt-DT, demonstrating MPDT's superior performance in existing scenarios and its ability to extend to new scenarios that existing algorithms struggle to handle effectively. From a rigor perspective, Prompt-DT could be removed from Table 1, which also indirectly highlights the limited exploration of prompt-based offline RL methods in zero-shot generalization scenarios.

**Q1: Why the embedding layer and GPT blocks are all frozen.**   
In Author Rebuttal AR3, we conducted an ablation study on whether to freeze the embedding layer. Also, based on the ablation study conclusions from [1][2], we speculate that the embedding layer and GPT blocks are highly correlated. To achieve sufficient performance, one should either (1) freeze both the embedding layer and GPT blocks simultaneously and add external components (e.g., prompt) or (2) train both the embedding layer and GPT blocks together (either full fine-tuning or using LoRA). The choice of approach involves a trade-off between model performance, computational cost, and scenario requirements.

From the perspective of model performance, the primary purpose of using prompts is to preserve the model's inherent prior knowledge as much as possible. Both [2] and we believe that the full model fine-tuning used in [1] may lead to model overfitting, further disrupting the internal knowledge of the model. We considered adapting the method from [2] to our task (training the embedding layer and LoRA), but the results obtained based on the code provided by [2] differed significantly from those in the original paper. Moreover, when we applied [2] to our datasets, ML45 and MT50 (as shown in the table below), the performance on the training set was far inferior to MPDT, and even careful hyperparameter tuning could not alleviate the significant fluctuations in the reward curve. We speculate that language models using the LoRA structure may still face convergence difficulties on RL data. Furthermore, the experiments in [2] primarily focus on single-task scenarios, avoiding the challenges of multi-task scenarios where task gradients may mix with the internal knowledge of the model.

||ML45 (Training Performance)|MT50 (Training Performance)
|-|-|-
|LAMO[2]| $367.563\pm129.37$ |$ 930.84\pm 437.62$
|MPDT| $ 604.21\pm11.05$ |  $1687.35\pm4.09$

Furthermore, [2] introduced a language prediction auxiliary loss to ensure the model retains its language task memory during training. However, if the introduction of LoRA is entirely suitable and does not disrupt the internal knowledge of the model, there should be no need for an auxiliary loss. In other words, we must fully understand whether full fine-tuning or LoRA fundamentally introduces new knowledge by disrupting existing knowledge (replacement) or guides the model to use existing knowledge to achieve RL tasks (integration). The distinction between these two approaches results in differences in the model's performance ceiling and floor. The extent of adjustments within the model may need to be carefully controlled. Additionally, current work needs to explore convergence and generalization adequately. MPDT combines transferable prompts with the pre-trained model in a more harmonious and adaptable way. Most importantly, freezing the embedding layer and GPT blocks preserves the model's complete knowledge, which we believe is a better choice.
We sincerely thank the reviewer for kindly raising the score.
I thank the authors for the detailed responses to my questions. On reading the responses, most of my concerns have resolved. 

Two concerns remain:
- *(minor)* W1. Prompt DT cannot utilize the same amount of test data due to its design. To alleviate this(?), the authors conduct experiments on Prompt-DT+TTA for fair comparison, but TTA isn't suitable for Prompt-DT, making it not fair enough. (Anyway, this is not a big problem.)
- *(major)* Q1. The question that "why the GPT blocks are all frozen" is not answered yet.

Another concern:
- *(significant)* The initial hyperparameter of Prompt-DT is #layer=3, #heads=1. The rebuttal states that "To ensure a fair comparison with MPDT, we aligned some general hyperparameters with those used by MPDT (See Author Rebuttal AR4)", and it hence seems that the experiments of Prompt-DT in this paper use #layer=12, #heads=12. If so, that would be unfair, since the other hyperparameters of Prompt-DT must be tuned extensively due to a significant change in the model size.
Thanks for the response. I've raised my rating to 6.