Contextual Transformer for Offline Reinforcement Learning
Authors: Runji Lin
Ye Li
Xidong Feng
Zhaowei Zhang
Xian Hong Wu Fung
Haifeng Zhang
Jun Wang
Yali Du
Yaodong Yang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Offline Meta Reinforcement Learning
Prompt Tuning
Transformer

Abstract: Recently, the pretrain-tuning paradigm in large-scale sequence models has made significant progress in Natural Language Processing and Computer Vision. However, such a paradigm is still hindered by intractable challenges in Reinforcement Learning (RL), including the lack of self-supervised large-scale pretraining methods based on offline data and efficient fine-tuning/prompt-tuning over unseen downstream tasks. In this work, we explore how prompts can help sequence-modeling-based offline Reinforcement Learning (offline-RL) algorithms. Firstly, we propose prompt tuning for offline RL, where a context vector sequence is concatenated with the input to guide the conditional generation. As such, we can pretrain a model on the offline dataset with supervised loss and learn a prompt to guide the policy to play the desired actions. Secondly, we extend the framework to the Meta-RL setting and propose Contextual Meta Transformer (CMT), which leverages the context among different tasks as the prompt to improve the performance on unseen tasks. We conduct extensive experiments across three different offline-RL settings: offline single-agent RL on the D4RL dataset, offline Meta-RL on the MuJoCo benchmark, and offline MARL on the SMAC benchmark. The results validate the strong performance, and generality of our methods.

Under review as a conference paper at ICLR 2023
CONTEXTUAL TRANSFORMER FOR OFFLINE REIN- FORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, the pretrain-tuning paradigm in large-scale sequence models has made significant progress in Natural Language Processing and Computer Vision. How- ever, such paradigm is still hindered by intractable challenges in Reinforce- ment Learning (RL), including the lack of self-supervised large-scale pretraining methods based on offline data and efficient mine-tunin//roompte downstream tasks. In this work, we explore how prompts can help sequence- modeling based offline Reinforcement Learning (offline-RL) algorithms. Firstly, we propose prompt tuning for offline RL, where context vector sequence is con- catenated with the input to guide the conditional generation. As such, we can pretrain model the offline dataset with supervised loss and learn prompt guide the policy to play the desired actions. Secondly, we extend the framework the Meta-RL setting and propose Contextual Meta Transformer (CMT), which leverages the context among different tasks as the prompt to improve the perfor- mance on unseen tasks. We conduct extensive experiments across three different offline-RL RL settings: offline single-agent RL on the D4RL dataset, offline Meta- RL. on the MuloCo benchmark, and offline MARL on the SMAC benchmark. The results validate the strong performance, and generality of our methods.
over unseen
INTRODUCTION
Reinforcement learning algorithms based on sequence modeling (Chen et al., 2021; Janner et al., 2021: Reed et al.. 2022) shine in sequential drcision-making tasks and form new promising paradigm. Compared with classic RL methods, such as policy-based methods and value-based methods (Sutton & Barto, 2018), optimization of the policies from the sequence prospective has advantages in long-term credit assignment, partial observation, etc. Meanwhile, significant gener- alization of large pretrained sequence model in natural language processing (Kenton & Toutanova, 2019; Brown et al., 2020) and computer vision (Liu et al., 2021b; Zhai et al., 2021) not only con- serves vast computation in downstream tasks but also alleviates the large data quantity requirements. Inspired by them, we want to ask the question: whether pretrain technique has similar power in RL? Since limited and expensive interactions impede the deployment of RL in various valuable ap- plications (Levine et al., 2020), pretraining large model to improve the robustness of real-worle wolld gap by a zero-shot generalization and improve data efficiency by few-shot learning offers great sig- nificance. (Team et al., 2021; Meng et al., 2021) pretrains a single model with diverse and abundant training tasks in the maninng domain to generalize in downstream tasks, which proves the feasibility that pretraining enables RL agents harness knowledge for generalization.
Earlier works on sequence modeling RL provide a new perspective on offline RL. However, ex- tending these methods the pretrain domain still confronted with several challenges. One major challenge for generalization (Li et al., 2020b) is how to encode task-relevant information, thereby en- hancing transferring knowledge among tasks. Since discovering the relationship among diverse tasks from data and making decisions conditioned on distinct tasks plays a significant role in generaliza- tion, it is not a trivial modification of existing methods. Another problem is efficient self-supervised learning in offline RL. Specifically, the decision transformer (Chen et al., 202 1) leverages the data to learn return conditioned policy, which ignores the knowledge about world dynamics. In addition, trajectory transformer (Janner et al., 2021) conducts planning based computational intensity and decision latency might be a bottleneck for a large-scale model and hard to fine-tune to other tasks. Therefore, introducing key components transfer the knowledge in
world model, but the high

---

Under review as a conference paper at ICLR 2023
pretrained model and incorporating the advantages in conditioned policy and world model is necessary.
In this work, we propose novel offline RL algorithm, named Contextual Meta Transformer (CMT), aiming to conquer multiple tasks and generalization at one shot in an offline setting from the perspec- tive of sequence modeling. CMT provides a pretrain and prompt-tuning paradigm to solve offline RL problems in the offline setting. Firstly, a model is pretrained on the offline dataset through a self- supervised learning method, which converts the offline trajectories into some policy prompts and utilizes these policy prompts to reconstruct the offline trajectories in the autoregressive style. Then a better policy prompt is learned based on planning in the learned world model attain the ad- vanced policy to generate trajectories with high rewards. In contrast to previous work, CMT learns a prompt to construct policy to guide desired actions, rather than being designed by humans or explic- itly planned by the world model. In the offline meta-learning setting, CMT extends the framework by simply concatenating a task prompt with the input sequence. With a simple modification, CMT is capable of executing proper policy for specific task and sharing knowledge among tasks.
Our contributions are fhree-fold: First, we propose a novel offline RL algorithm based on prompt tuning, in which the offline trajectory is encoded as a prompt, and the appropriate prompt is found to lead a policy for execution to achieve high reward in the online environment. Second, CMT is the first algorithm to solve offline meta-RL RL from sequence modeling perspective. The context trajectory, which represents the structure of the task, is used by CMT as a prompt to guide the policy in specific unknown task. Furthermore, empirical results on D4RL datasets and meta Mujuco tasks show that CMT has outstanding performance and strong generalization.
2 RELATED WORK
Offine Reinforcement Learning. Offline RL is gaining popularity as a data-driven RL method that can effectively utilize large offline datasets. However, the data distribution shift and hyper- parameter tuning in offline settings seriously affect the performance of the agent (Levine, 2021). So far, several schemes have been proposed to address them. Through action-space constraint, BCQ (Fujimoto al., 2019). AWR (Peng et al., 2019), BRAC (Wu et al., 2019), and ICQ (Yang et al., 2021) reduce extrapolation error caused by policy iteration. Noticing the problem of overestimation of values, CQL (Kumar al.. 2020) keeps reasonable estimates by looking for pessimistic expec- tations. UWAC (Wu et al., 2021) handles out-of-distribus during training by estimating the uncertainty of (s, a). MOPO (Yu et al., 2020) and MOReL (Ki- dambi et al., 2020) solve the offline RL problem from the model- perspective while ensuring rational control by adding penalty items to uncertain areas. Decision Transformer (DT) (Chen al., 2021) and Trajectory Transformer (TT) (Janner et al., 2021) reconstruct the RL problem into a quential decision problem, extending the LangeeLLnguug-MMode-liin RL area, which inspires many follow-up works them. However, the relevant works offline RL are still insufficient due to the lack of self-supervised large-scale pretraining methods and ef- ficient prompt-tuning over unseen tasks, and CMT proposes a pretrain-and-tune paradigm to deal with them.
(OOD) data by weighting the Q value
(LLM-like) structure to the
Pretrain and Sequence Modeling. Recently, much attention has been attracted to pretraining big models on large-scale unsupervised datasets and applying them to downstream tasks through fine- tuning. In language process tasks, transformer-based models such as BERT (Kenton & Toutanova, 2019), GPT-3 (Brown et al., 2020) overcome the limitation that RNN cannot be trained in parallel and improve the ability to use long sequence information, achieving SOTA results on NLP tasks such as translation, question answering systems. Even the CV field has been inspired to reconstruct their issues as sequence modeling problems, and nigh-performance models like the swin transformer (Liu et al., 2021b) and scaling ViT (Zhai et al., 2021) have been proposed. Since the trajectories in of- fline RL datasets have Markov properties, they can be modeled through transformer-Iie like structures. Decision transformer (Chen et al., 2021) and trajectory transformer (Janner et al., 2021) propose condition policy on return to go (RTG) and behavior cloning policy improved by beam search to solve RL problems in offline settings respectively. Inspired by these works, we bring prompt tuning from NLP into the RL domain, then propose a potential path to pretrain a large-scale RL model and efficiently transfer the knowledge to downstream tasks.
2

---

Under review as a conference paper at ICLR 2023
Offline meta-RL and Task Generalization. Offline meta-RL shines recently since it allows al- gorithms to adapt to new tasks quickly without interacting with the environment. Targeting it, an optimization-bet based method with advantage weighting loss called MACAW (Mitchell et al., 2021) is proposed, which learns the initialization of both the value function and the policy. FOCAL (Li et al., 2020b) combines the deterministic context encoder with behavior regularization and achieves in- spiring results based on an off- Meta-RL method called PEARL (Rakelly et al., 2019). Then it improved by combining the intra-task attention mechanism and the inter-task contrastive learning objective, which is named FOCAL+ (Li et al.. 2021), to deal with sparse reward and distribution shift. BOReL (Dorfman et al., 2020) aims learn Bayesian optimal policies from discrete data for the mentioned problems, whereas SMAC (Pong et al., 2021) learns meta-policies from reward- labeled data and then fine-tunes on new tasks. From the model-based perspective, MerPO (Lin al., 2022) proposes a meta-model for efficient task structure inference and a meta-policy for safe explo- ration of OOD data. It is worth mentioning that recent work on general model construction, such as SayCan (Ahn et al., 2022), and Gato (Reed et al., 2022), has achieved exciting results, demonstrat- ing the huge potential of LLM-lliee architectures. Just like them, CMT is also a general LLM-like model that can solve offline me problems effectively.
3 PRELIMINARY
Meta Reinforcement Learning. The major purpose of meta-RL is to leverage multi-task experi- ence to enable fast adaptation to new unseen tasks. A task T; is defined a Markov Decsion Process (MDP) Ti = A, R, P), where S is the state space, A is the action space, R is reward function, and P transition function. In deep RL, the policy Ty( ( which specifies the probability that the agent takes action a, in state st at time t, is described by neural network with parameters B. The goal in a MDP to learn a optimal policy x¹ = arg max, Ess0. $1.............................. (Ei X Xt((s,,0) which can maximize the expected discounted return, where À is discounted factor. In meta-RI,, tasks are drawn from a task distribution T ~ p()), the state space S and the action space A are common across tasks, and reward function R, and transition function P, are task specific. During meta- training, the meta policies are trained based on some training tasks sampled from task distribution to achieve fast adaptation to new unseen tasks in meta tests.
Offline Reinforcement learning. In offline RL setting, the trajectory dataset D is collected from unknown behavior policy μu, which might be expert policy, sub-optimal policy, random policy, or a mixture policy (e.g. corresponding the replay buffer of an RL agent). A offline trajectory r consists of states, actions, and scalar rewards: т = {st, a re) LT-1 A trajectory fragment Tex denotes transitions from time-step to time-step tz. This paper aims to learn an optimal policy n from the fixed dataset D without interaction with the environment.
Conditional generation tasks are common in NLP, where the input is a context x and the output y is a sequence of tokens. Autoregressive model LM (Brown et al., 2020) is powerful tool to solve this kind of tasks, which concatenates the context and the output as a whole sequence u = [z. yl and models the probability for the next token u, based on the previous tokens uez.
Prompt and Prompt-Tuning
by = AMMuss)ee))
p(u:uue)) softmax(Wh.),
where u, denotes i-th token in the sequence u, h; E IRd denotes the activation in transformer at time step i, and W the learning parameter matrix. To leverage the knowledge in the pretrained large-scale model, prompts are designed to improve the few-shot performance in the downstream task. A prefix-style prompt z, also a sequence of tokens, are concatenated with input u = [z, x, y] to guide the model to generate the desired output. Besides hand-designed prompts z, prompt-tuning(Li & Liang, 2021) is proposed to learn a continuous prompt that can be learnt from data.
METHOD
In this section, we introduce CMT, an RL framework for offline RL and offline meta-RL. We de- scribe CMT with policy prompts for offline RL in Section 4.1, and CMT extended with task prompts and policy prompts for meta-RI. in Section 4.2.
3

---

Under review as a conference paper at ICLR 2023
linear decoder
Contextual Meta Transformer
Automgressive Transformer Ag
Adaptor Layer
Bidireaction Transfomer A,
[+]
Future
position embrelding
History
Figure I: The framework for CMT in the offline learning Setting. (a) In the representation stage, CMT pretrains an auto-encoder model in the offline dataset, which predicts the future action, reward, and state with the policy prompt from the history trajectory. The adaptor layer is identity function during this stage, which mean = z'. (b) In the improvement stage, we freeze the pretrianed model, and tune the prompt reach a better performance.
4.1 OFFLINE SEQUENCE LEARNING
The main assumption of our method is that offline trajectories can be viewed as samples from un- known policies, and the optimal policy can be represented mixture of these basic policies. Our method contains two stages of training, the representation stage and the improvement stage. CMT learns model convert an offline trajectory into a policy prompt with some characteristics to rep- resent these deterministic policies. In the second stage, policy prompt is learnt to mix up basic policies by planning the world model to attain an advanced policy.
Representation Stage. Fig.1 shows the whole architecture, which constitutes auto-encoder A in trajectory-level. CMT consists of two components: trajectory encoder A, with parameter and an autoregressive generator A, with parameter c. Trajectory encoder A, is bi-direction transformer (Kenton & Toutanova, 2019), which gets a history trajectory and gives the policy prompt シ, for the trajectory 2, Ar; 0). Autoregressive generator Ag is GPT--tyle (Brown al., 2020) conditional generator, which predicts the policy prompt 2, and the next token in the future based on (<<))
the previous history trajectory: T
In this stage, CMT introduces two loss terms to update and d. The major loss Li supervised loss, which is used reconstruct the whole trajectory, and auxiliary loss La help improve policy in the next stage. The loss is linear weighted as L Li + 7L2, in which is the contrastive loss coefficient. For the supervised loss L. since A, predicts the future action, reward and state one by one, it employs as an union of a policy TTass) = Ag(Er, st, Tet), dynamic model P(s'ls. |s. a) Ay(2,, Tet) and reward function R(s,a) Ag(2,,0, St, Tet). The prediction and the ground truth form a supervised loss in (2):
T-1 Σ (D(s,, P(reis o 0)) + D(a,, (81, Teas 0)) + Drr,, R(ar, St, Tet 0, 0))), (2)
L (T 0)
in which distance matrices adopts MSE loss for deterministic output and cross-entropy loss for stochastic prediction. Since the entire architecture is differentiable, the supervised loss be used to update A, and Ay-
An auxiliary loss L2 constrains the distance between prompts coming from similar trajectories by supervsper learning. Inspired by (Liu et al., 2021a), an effective and stable policy improvement based imitation learning often satisfies properties: (a) Keeping new behavior close to pre-
4

---

Under review as a conference paper at ICLR 2023
vious ones. (b) Getting higher rewards than the previous ones. As we desire to improve the policy by prompt tuning, it is natural to facilitate the similarity of prompts from similar trajectories. For this purpose,
introduce an InfoNCE contrastive loss (Van den Oord et al., 2018) to constrain the prompt in a self-supervised method to meet the aforementioned requirements. The auxiliary contrastive loss is given as E3.(3):
exp(An(TT; 0) Ar(T4;0)/a) E exp((e (T;:0))aa)
exp(zg 2+/0)
C2(Tg, fr log
log E ixxeeppz zi/ 21/a)
in which a is temperature coefficient. For the anchor policy prompt by encoded from trajectory Tq. batch of K policy prompts )=,),-] encoded from set of trajectories {T: sampled from the offline dataset. {zi} consists of K - negative samples z_ and one positive sample 24. The definition of the positive and negative samples influence the property of the policy prompt. To ensure similar behavior trajectories be encoded into close prompts, the auxiliary loss defines the pair of policy prompts samples from the same trajectory and different trajectories as the positive and negative sample pair.
Improvement Stage. Since the behaviour policy can be sub-optimal in the offline dataset, we consider prompt tuning to boost the agent performance, with the purpose to transfer the knowledge in the pretrained model. As shown in Fig. the key idea is simple: we can freeze the pretrained model, and learn prompts that can guide the pretrained model to generate trajectory with high reward. Specifically, improvement stage consists of relabeling the offline dataset and prompt tunning by adaptor layer.
Relabeling the offline dataset to replace the raw ordinary action with better action to provide new supervised target for prompt tumning. Concretely, sample mini-batch of data, and then adopts the beam search method proposed by trajectory transformer Janner et al. (2021) as a planning algorithm to find the better action, in which the autoregressive generator Ag works a world model.
To improve the performance by prompt tunning, we should tune the policy prompt 2n for a better policy prompt 2, to guide generator Ag generate trajectory with a higher reward. For this purpose, we freeze the pretrained model parameters, denoted 0 and ø and only tune the parameter & for the adaptor layer L on the relabeled dataset. The adaptor layer L is trained by the following (4)
T-1
Ca(T: = Σ D(ars n ( 0,), + B(= L(=E)))²
(4)
in which âr is the relabeled action, and the second term constrains behavior changes to alleviate distribution shift in the offline setting, like (Fujimoto & Gu, 2021) and a weight coefficient for behavior constraint. This method can be regarded as using prompt tunning to remember planning results in the world model, which significantly reduce the computation cost and decision delay in evaluation. However, it should be noticed that we use planning method as the improvement method, but any other improvement algorithm of which loss function based on the output of generator Ag can be easily plugged in.
4.2 CONTEXTUAL SEQUENCE META LEARNING
Extended from the section 4.1 which introduces policy prompts to solve offline RL problem, we sim- ply incorporates a task prompts in CMT to achieve generalization ability in downstream unseen task in offline meta RL setting. The task encoder F ((t is used to encode transitions into a task hidden variable / and learn contextual policy π (a|s, () in classical context meta-RI. methods. Therefore, CMT is feasible to extend to the meta-learning domain by simply plugging in task prompts. Fig. shows the minor modifications supporting CMT have impressive generality.
Meta Training. To contain the task information, CMT simply concatenates a context trajectory, which is trajectory fragment coming from the same task, with the input. During offline meta- training, the context trajectory is randomly sampled from the offline dataset. To avoid information fusion, we separate the context and history trajectories with special token ([SEP), whose param- eters can be learned. Then we adopt a contrastive learning method similar to E3) to learn a stable and consistent task prompt, similar to (Fu et al., 2021) in online meta-RI.. The major difference
5

---

Under review as a conference paper at ICLR 2023
Bidireaction Transtormer A₂
Task
r-2
t-1
Policy t-2
Figure 2: The framework of CMT in Offline meta reinforcement Learning Setting. Based the ba- sic framework in Figure.1, CMT introduces a context trajectory as task prompt in trajectory encoder Ag guide Ag.
between contrastive loss in task prompts and policy prompts that task prompts form positive and negative sample pairs in task-level, while policy prompts form positive and negative sample pairs at trajectory-level.
Meta Test. After training on diverse tasks, meta test stage requires agent rapidly adapting in the unseen task. context meta RL, agent permitted collect few context trajectories understand the task. The context trajectory in the meta test could come from offline dataset in an unknown task or trajectory that has interacted with the online world. The second setting is more challenging (Dorfman et al., 2021) due to the exploration problem. To verify the strong capacity of CMT, we evaluate CMT the second setting. Furthermore, CMT discards recursive component, so is suitable for zero-shot setting, which means CMT collects the context during online evaluation, rather than in advance. To the best our knowledge, there is no existing method to solve this one-shot setting in offline meta RL. As result, construct context buffer to store the history of interactions, and the context trajectories are randomly chosen from the context buffer.
5 EXPERIMENT
In this section, evaluate the performance of CMT in terms of offline RL tasks in D4RL bench- marks (Fu et al., 2020), offline meta-RL tasks in meta Mujoco benchmarks (Todorov et al., 2012). Additional offline multi-agent experiments are conducted on StarCraft Micromanagement (Pong et al., 2021). Simply replacing sequence of transition by a sequence of agent, CMT can be eas- ily extended solve multi-agent offline-RI. tasks and evaluated in popular MARI. benchmark (SMAC). The results on SMAC are reported in Appendix A.3. Apart from the performance in vari- ous settings, we design experiment for ablation study to demonstrate the validity of the components contained in CMT. Our experiments are conducted on server with Nvidia Tesla A100 GPU and AMD EPYC 7742 CPU.
5.1 OFFLINE LEARNING TASKS
We evaluate CMT on the continuous control tasks from D4RL benchmarks. The experiments on four standard Mujoco locomotion environments (HalfCheetah, Hopper, Walker, and Ant) are conducted with three kinds of dataset quality (Medium, Meedium-Replan and Medium-Experti. The differences between them are as follows: Medium contains million timesteps generated by a "medium" policy interacting with the environment, with an intelligence level of around 1/3 that of experts. Medium- Replay contains the replay buffer generated during the medium policy training process, and about 25k-400k timesteps timesteps generated by the medium policy concatenated with another million timesteps generated by the expert policy.
included in the tested environments. Medium-Expert consists of million
Five baselines are considered, including behaviour cloning (BC) (Torabi et al., 2018), behavior regularized ActorCritic (BRAC) (Wu al., 2019), conservative Q-learning (CQL.) (Kumar et al.,

---

Under review as a conference paper at ICLR 2023
2020), implicit Q-leaming (IQL) (Kostrikov et al., 2021), and decision transformer (DT) (Chen al., 2021). BC realizes intelligence by learning from expert datasets, which actually a supervised learning process that learns the states to predict actions. Because of severe extrapolation emoos caused by the policy evaluation, traditional offline RI. algorithms perform poorly. And the methods such BCQ, BRAC, and IQI., avoid extrapolation errors by constraining the behavior space. While CQL solves it by finding a conservative Q function that keeps the policy function's 's expected value less than the value. Starting from another perspective, DT transforms the RL problems into sequence modeling problems and attempts to find the optimal actions. The detail about hyper- parameter lists is Appendix. A.2.
The results for D4RL datasets are shown in Table. 1. CMT performs excellently on the Medium and Medium-expert datasets, but not so well on the Medium-replay dataset, indicating that CMT prefers to learn from data generated by stable policies. Compared with DT, which is also transformer- based structure, CMT outperforms it in most of the tasks. Moreover, although IQL. is the SOTA algorithm currently, the performance of CMT the Medium and Medium-expert datasets meets or exceeds it, demonstrating that our method has huge potential.
Table 1: Results for D4RL datasets. Here report the mean for three seeds, and the reward is normalized so that 100 represents an expert policy and 0 represents a worst policy in D4RL. PT abbreviation stands for prompt tunning. In addition, our method name and the best performances are bold font.
CMT
CMT
Dataset
Environment
DT
BRAC-v
CQL
IQL
BC
with PT
w/o PT
Medium- Medium-Expert
halfcheetah
92.9
59.8
88.0
41.9
91.6
86.7
65.6
Medium-Expenn
hopper
106.5
102.0
103.3
0.8
105.4
91.5
55.4
Medium-Ea Expert
walker
97.6
83.5
108.4
81.6
108.8
109.6
11.2
Medium-Expert
ant
101.3
67.1
89.3
115.8
125.6
71.2
Medium halfcheetah
43.6
40.1
42.1
46.3
44.0
47.4
41.6
Medium hopper
68.9
62.8
62.0
31.1
58.5
66.3
48.6
Medium
walker ant
75.0
69.6
71.6
81.1
72.5
78.3
47.8
Medium
71.8
61.3
64.6
90.5
102.3
63.7
Medium-replay
halfcheetah
38.7
16.5
36.3
47.7
45.5
44.2
2.2
Medium-replay
hopper
84.9
58.4
67.8
0.6
95.0
94.7
30.8
Medium-replay
walker
49.5
37.3
47.8
0.9
26.7
73.9
5.9
Medium-replay ant
40.6
42.1
61.7
93.9
88.8
30.1
5.2 OFFLINE META LEARNING TASKS
We explore four tasks to evaluate CMT on zero-shot generalization: Half-Cheetah- Vel, Ant-Fwd- Back, and Ant-Fwd-Back. The same data collection method is used as described the literature (Li et al., 2020b). The following baselines are taken into account: Batch PEARL (Rakelly et al., 2019); modified version of PEARL which can be used for offline RL tasks. CBCQ (Fujimoto al., 2019): An advanced version of the BCQ that has been adapted to offline RL tasks by incorporating latent variables into state information. MEML (Li et al., 2020a): A multi-task offline RL method with metric learning. FOCAL (Li et al., 2020b): A model-free offline Meta-RI. method with state- of-the-art performance based on the deterministic context encoder. These baselines are trained on set of offline RL tasks and are tested on the set of unseen offline RL tasks.
The results for meta Mujoco environments are shown in Fig. 3. Once again, we should emphasize the results of CMT is zero-shot setting, while all the other baselines requires context from offline dataset or online interactions in advance. As we can see, CMT can outperform most baselines, including CBCQ, batch PEARL, and MBML. Besides, FOCAL is the SOTA algorithm currently, while CMT can outperform in different tasks except Dalker-2D-Parraee showing that our algo- rithm also has great potential in the area of offline meta-RI.

---

Under review as a conference paper at ICLR 2023
Ant-F Font-Fwd-BBr Bwd
Half- Cheetahe Fwd-Bwd
600
1000
800
Averaged Return
200
400
Averaged
200
-200
-200
-600
POCA.
Beich TEMIL 100L
CMT Thters
CMT 77P
400
100A.
-200
Butth TOAL
HB%
OST Fffiaal
noo 7ioaa
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3,0
Gradient Stepsix10m5)
Gradient Steps(x10"5)
Paint- Robitt-Root--- Wind
Walker 20 Paramns
Averaged Return
Averaged Return
-5.0
-5.5
-6.0
-6.5
-7.0
50
-7.5
Pignes)
100ML
-8.0
0.0
0.2
0.4
0.6
0.0
0.2
1.0
Gradient St
Gradient Steps(x100
Figure 3: Results for Meta Mujoco Environment. In all benchmark tasks, CMT obviously learns that a policy can face adaptation into a new task, and provides evidence that sequence modeling method is promising. Noticed that CMT have two training stage, it is difficult to align the x-axis. Therefore, we report the training curve of CMT representation stage and the final evaluation results of CMT after prompt-tunning dotted line with standard deviation.
5.3 ABLATION STUDY
In this section, we formulate experiments to investigate the following research questions: Q1: How important prompt-tunning for performance? Q2: Does contrastive loss benefit the prompt-tuning? Q3: Does the behavioral constraint affect the results? Q4: In offline meta RL setting, does quality of task prompts affect the performance in downstream tasks? Without loss of generality, we completed the following ablation experiments in the Ant-Fwd-Bwwd environment, and the results are shown in Figure 5.
(Q1) Prompt-tuning. Prompt-tuning is utilized in the second stage to enhance the model's perfor- mance based on the pre-trained model. With the help of it, the average return will increase by 65.8%, which shows that prompt-tuning is effective in improving model effects. Furthermore, the results of CMT with prompt-tuning and without prompt tuning in Table and Figure strongly demonstrates the benefit in performance from improvement stage.
(Q2 Contrastive Loss. Contrastive loss plays a key role clustering similar trajectories, which ensures that the model can find the correct prompts guide better trajectories. To investigate its influence, we utilized two extreme coefficients, the minimal value of 10 and the maximal value of 1. As shown in Figure 5, when the coefficient gets to its minimal value, the average return drops by 50.9%. A possible explanation is that the model lacks the clustering process, preventing it from generating effective prompts to distinguish different types of trajectories. Finally, the model will be heavily affected by data distribution shifts during the tuning process. When the coefficient gets maximal, the average return falls slightly by 11.1%. 1,1% showing that an overly strict constraint will also affect the model. Moreover, we conduct visualization analysis in Figure 4 to demonstrate the significant effect on the distribution of prompts.
(Q3) Behavioral Constraint. Behavioral constraint is utilized in the second stage to enhance the model, which has significant impact on the final effect. Similarly to Q2, we use the minimum coefficient of 0 and the maximum coefficient of 50 to show its impact. When the coefficient is 0. the average return will dramatically drop by 128%. Despite the fact that the loss decreases on offline datasets during the tuning process, the test results are still poor. This is the typical overfitting situation, indicating that the model suffers from severe data distribution shifts. When the coefficient is 50, the average return will also be reduced by 36.4%. It shows that extremely severe behavioral constraint will also lead to inefficient policy boosting, resulting in slightly better performance than that of the model without prnipt- Therefore, it very important to find a suitable coefficient.
8

---

Under review as a conference paper at ICLR 2023
With Constrasiive Loss
Without Constrasivee Loss
L-SNE dimrension
1-SNE
-130
-28
22
L-SNE dimansion
E-SNE dimansion
Figure 4: Visualization for Policy Prompts in halfcheetah task. We visualize prompts from three pairs of trajectories with contrastive loss and without contrastive loss. Each pair have similar behav- ior and reward sampled from offline reply dataset, and use similar colors.
(Q4) Quality of task prompts. Task content is constructed to accurately identify Meta-Ra RLL tasks. To demonstrate its effect, the experiments are divided into no context, medium context, and expert context groups based content quality. In the first set of experiments, the CMT full model collects the context during online evaluation to support zero-shot adaption. The absence of task prompts significantly deteriorates the performance by 18% due the inability accurately identify tasks. The performance is improved when task contents from the offline datasets are employed. The results utilizing the medium and expert datasets increase by 4% and 7%, respectively. Both of them are better than the results in the first set of experiments. In fact, if offline task contents are used, the tasks will become few-shot tasks since the offline datasets co-distributed with the tuning datasets. It is simpler and better than the zero-shot tasks using the online task contents.
600
500
Average Retur
200
100
CMI-F
PIN
CL CL-10-0 10
CL-1
BC--9
B-C+1
Note C- Medium Expert
CMT Full Model (CMT-F)
Contrastive Loss (C-L)
Context (C)
CMT wio PT (PT-N)
Behavioral Constraint (B-C)
Figure 5: Results for ablation. We explore the impact of prompt-tuning, contrastive loss, behavioral constraint, and context on the model by whether use or set extreme values. And each group has different color. The first group is the full CMT model, which can be used the benchmark. CONCLUSIONS
In this paper, we present CMT, an offline RI. algorithm based on prompt tuning, with the goal of training large-scale model that be utilized various downstream tasks from the sequence modeling perspective. The prompt tuming is designed for offline RL to pre-train the model and guide the autoregressive model to generate trajectories with high rewards. Besides, variety of experiments are conducted in three different RL settings, offline single-agent RL (D4RL), offline Meta-RL (MuJoCo), and offline MARL (SMAC), and the model's performance is evaluated with different baselines. The results show that CMT has strong performance, and generality. To our best knowledge, CMT also first General decision models like CMT enhance the efficiency of model training and lower the threshold for the applications RL algorithms.
al algorithm for offline meta-RL. RL problems.

---

Under review as a conference paper at ICLR 2023
Reproducibility Statement To reproduce the results in this paper, effort are listed as following:
Environment. We give the detail information about our hardware platform in section 5. And the software version and dependencies are included in requirement file.
Method. In section4. we give all the loss function formula. The network architecture are shown in Figure For the better understanding of the relationship of data and loss function, we illustrate the detail dataflow in appendix A.1. We also list all the hyper-parameler for the experiment in Appendix A.2.
Data, Since the performance in offline RL highly depends on the dataset, we clearly report the dataset resource. In the offline RL setting, adopt the popular benchmark (D4RL https: : / / thub. com / /aa//aaraaaa Foundatiinn/DRR)) In the offline meta setting, we follow the FOCAL (https: httttp///ggttub..cm/m//aiiioggep to generate data.
FFOCAL-
Randomness. To alleviate the randomness and noise, we run three seeds to reduce the bias. And we randomly sample the seed from a uniform distribution from (0, 65536) without any selection.
Code. Since we has not been completed code clean yet, we are glad to open source our code in more readability format in rebuttal.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01699 2022.
Tom Brown et al. Language models are few-shot learners. Advances in Neural Information Pro- cessing Systems, 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in Neural Information Processing Systems, 34, 2021.
Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta learning of exploration. arXiv preprint arXiv:2008.02599, 2020.
Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta reinforcement learning identifiability challenges and effective data collection strategies. Advances in Neural Information Processing Systems, 34, 2021.
Haotian Fu, Hongyao Tang, Jianye Hao, Chen Chen, Xidong Feng, Dong Li, and Wulong Liu. Towards effective context for meta-reinforcement learning: an approach based on contrastive learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7457- 7465, 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2000 2062. PMLR, 2019.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in Neural Information Processing Systems, 34, 2021.
Jacob Devlin Ming Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HI.T, 2019.
10

---

Under review as a conference paper at ICLR 2023
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 21810-21823, 2020.
Iya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:21 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179 13:1179-1111 1191, 2020.
Sergey Levine. Understanding the world through action. CoRR,
2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review. and Perspectives on Open Problems, 2020.
Jiachen Li, Quan Vuong, Shuang Liu, Minghua Liu, Kamil Ciosek, Henrik Christensen, and Hao Su. Multi-task batch reinforcement learning with metric learning. Advances in Neural Information Processing Systems, 33:6197-6210 2020a.
Lanqing Li, Rui Yang, and Dijun Luo. Focal: Efficient fully-offline meta-reinffrrcement learning via distance metric learning and behavior regularization. arXiv preprint arXiv:2010.01112 2020b.
Lanqing Li, Yuanhao Huang, Mingzhe Chen, Siteng Luo, Dijun Luo, and Junzhou Huang. Provably improved context-based offline meta-rl with attention and contrastive learning, arXiv e-prints, pp. arXiv-2102, 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582-4597, 2021.
Sen Lin, Jialin Wan, Tengyu Xu, Yinghin Liang, and Junshan Zhang. Model-based offline meta- reinforcement learning with regularization. arXiv preprint arXiv:2202.02929, 2022.
Minghuan Liu, Hanye Zhao, Zhengyu Yang. Jian Shen, Weinan Zhang. Li Zhao, and Tie- Yan Liu. Curriculum offline imitating learning. Advances in Neural Information Processing Systems, 34, 2021a.
Ze Liu et al. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the JEEE/CVF ICCV, 2021b.
Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang. and Bo Xu. Offline pre-trained multi-agent decision transformer: One big sequence model conquers all starcraftii tasks. arXiv preprint arXiv:2112.02844 2021.
Eric Mitchell, Rafael Rafailov, Xue Bin Peng. Sergey Levine, and Chelsea Finn. Offline meta- reinforcement learning with advantage weighting. In International Conference on Machine Learning, pp. 7780-7791. PMLR, 2021.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint 190000 2019.
Vitchyr H Pong, Ashvin Nair, Laura Smith, Catherine Huang, and Sergey Levine. Offline meta- reinforcement learning with online self-supervision. arXiv preprint 0rXiv:2107.03974, 2021.
Kate Rakelly, Aurick Zhou, Chelsea Finn. Sergey Levine, and Deirdre Quillen. Efficient off off-policy meta-reinforcement learning via probabilistic context variables. In International Conference on Machine Learning, pp. 5331-5340. PMLR, 2019.
Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards. Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent, 2022.

---

Under review as a conference paper at ICLR 2023
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al. Open-ended learning leads generally capable agents. arXiv preprint arXiv:2107. 1177.128000 12808, 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint 2018.
Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv e-prints, pp. arXiv-1807, 2018.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: /911. 1361, 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08144 2021.
Yiqin Yang, Xiaoteng Ma. Li Chenghao, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offiine multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances Neural Information Processing Systems, 33:14129-14144 2020.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. CORR, abs/2106.00566, 2021.
12


Reviews:
[{"review_id": "r4JCxGc30L", "reviewer": "Reviewer_7JkM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates how to improve decision transformers with prompt and meta-learning. Evaluation on D4RL and MuJoCo benchmark tasks shows that the proposed method outperforms other baselines in some cases. ", "summary_of_the_review": "This paper investigates how to improve decision transformers with prompt and meta-learning. But the proposed method does not show significant and consistent improvements over other baselines. More importantly, prompting decision transformers has been introduced in an ICML paper before. ", "strength_and_weaknesses": "# Strength\n\n- Improving decision transformers with prompting is natural.\n- The proposed prompt mechanism is well-designed.\n\n# Weakness\n\n- This paper failed to provide a complete literature survey. Especially, \"Prompting decision transformer for few-shot policy generalization, ICML 2022” leverages the sequential modeling ability of the Transformer architecture and the prompt framework to achieve few-shot adaptation in offline RL, which is very similar to this paper’s approach. The authors of this submission should clearly explain what are the differences and properly cite this ICML paper.\n- The performance of CMT doesn't seem to be impressive and robust. It would be better if the authors could conduct experiments to probe how performance varies according to different task prompts.\n- Figure 5 is not clear enough for me. Fig. 5 should be segmented into 3 or 4 different tables or figures to separately investigate different hyper-parameters' influence. Only providing C-L=10^{-6}, C-L=1, B-C = 0, and B-C = 1 settings cannot completely show how these hyper-parameters influence the model.\n- There is a performance gap between the proposed model and IQL. The performance should be further improved. Or the author should explain why the performance gap is acceptable at least.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "pty3Rg1MTH", "reviewer": "Reviewer_SQ2X", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method based on utilizing the \"offline trajectories\" as a latent variable (prompt), and then conditioning the behavior on the online agent on the latent variable. The paper also explores using a model based planner to achieve higher reward by adapting the latent variable which the paper mentions as \"prompt tuning\". The paper evaluates the proposed method across different settings: offline single-agent RL on the D4RL dataset, offline MetaRL on the MuJoCo benchmark. ", "summary_of_the_review": "The paper explores an interesting problem i.e., how to improve the policy given some offline data. The way the paper instantiates the proposed idea is interesting. It would be useful to evaluate the method on more complex tasks to further study the behavior of the proposed model. ", "strength_and_weaknesses": "Strengths\n\n- The paper is very well written, and motivated. \n- The underlying idea is not \"new\" per se, but the way the idea is instantiated is interesting and very relevant. The paper also explores the idea in various different settings such as offline RL and offline MetaRL.\n- The reviewer appreciate the ablations provided in the paper to dissect where the improvement is coming from.\n- The reviewer appreciates the comparison with various different baselines for offline meta-RL tasks such as CBCQ, FOCAL, Batch-Pearl.\n\nWeaknesses\n\n- One way to improve the paper, is perhaps building on the offline MetaRL results. It's interesting to see that the performance of the proposed method is zero-shot as compared to other baselines which requires some context. Further evaluating the method on more complex tasks which requires adaptation as well as exploring how the adaptation varies by varying the nature of the data in the offline dataset would be very interesting. \n- There are aspects of the work, which are already explored in the literature like learning a policy conditioned on the \"past\" experiences as done in Retrieval Augmented RL [1, 2]. These methods learn an encoding function using self-supervised learning, and then based on the context of the agent, figures out what information to retrieve to condition the current policy. The interesting difference is A_{g} in the current work is parameterized as  auto-regressive transformer which is trained via state, action and reward prediction, whereas in [1] it's parameterized as a neural network (non-LSTM) and trained with RL/behavior cloning (A_{e} is still parameterized as a bidirectional transformer. \n\nMinor point:\n\n- The reviewer is not sure if it's really important to mention the point \"\"CMT is the first algorithm to solve offline meta-RL from a sequence modeling perspective\". Not sure what \"solving\" means here especially after only evaluating on Ant, Half-Cheetah, Walker. \n\n\n[1] Retrieval-Augmented Reinforcement Learning, https://arxiv.org/abs/2202.08417 (ICML'22) ,\\\n[2] Large-Scale Retrieval for Reinforcement Learning, https://arxiv.org/abs/2206.05314 (NeurIPS'22) \n  \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kc2GiiYCcZ", "reviewer": "Reviewer_9AEp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents Contextual Meta Transformer (CMT) which is an offline RL algorithm based on prompt tuning. It targets on training a large-scale model that can be utilized on various downstream tasks from the sequence modeling perspective. The training of CMT consists of two stage, i.e. representation stage and improvement stage. The prompt tuning is designed for offline RL and guide the autoregressive model to generate trajectories with high rewards. Experiments are conducted in three different RL settings, offline single-agent RL (D4RL), offline Meta-RL(MuJoCo). and offilien MARL (SMAC).", "summary_of_the_review": "This work is well motivated which is exploring an interesting problem: how prompts can help sequence modeling for offline RL algorithms. To achieve this, a Transformer encoder-decoder architecture is presented, technically it is a combination of a bidirectional transformer and a causal transformer (DT). To pretrain such a model, two pretraining objectives are leveraged, i.e. autoregressive prediction and a contrastive loss as auxiliary loss during training. To evaluate the proposed approach, three different settings are been conducted, i.e. meta RL, offline RL and prompt tuning. \n\nHowever, there are some concerns:\n1.the writing is hard to follow. Some statements are conflicted overall, e.g. in abstract, it is claim that this approach is training with supervised loss ('as such, we can pretrain a model ..... with supervised loss and .....'). While in Introduction, it claims that CMT is trained in a self-supervised manner (... Firstly, a model is pretrained on the .... through a self-supervised learning method). Based on my understanding, the training of CMT leverages two losses, where the autoregressive prediction is a supervised loss while the contrastive loss is a self-supervised loss. I would recommend the authors to keep the descriptions consistent in the whole paper. Otherwise, it is very confusing to the readers.\n2. lack of implementation details and experiment settings. For example, as a pretraining work, I did not find the pretraining and finetuning datasets details, especially the coverage of tasks. Are you evaluating and/or finetuning on the same tasks or unseen tasks? Without such information, it is hard the validate the generalization ability of CMT.\n3. The technical novelty is limited. From model architecture aspect, CMT is a combination of existing approaches, DT and bidirectional transformer. From algorithm aspect, it is an autoregressive loss (DT style) and contrastive loss. I can understand that this work is trying to propose a new pretraining regime, however, the usage is also not natural to me. Since the prompt  strategy can be directly utilized with only a causal transformer, in this case the DT. Then why an extra bidirectional transformer is needed? I did not find a clear answer in this paper. The rational under the contrastive loss is also unclear to me. For a specific task in a certain environment, the policy should be the same. However, based on the definition of positive and negative samples, it assumes that only the segments from the same trajectory share a policy while the other trajectories have different policies. I am confused about this setting, unless this contrastive loss are capturing other information or factors lie in trajectories, e.g. spatio-temporal information.\n4. the gains of CMT are marginal when comparing with other approaches. ", "strength_and_weaknesses": "Pros:\nThis work is well motivated which is exploring an interesting problem: how prompts can help sequence modeling for offline RL algorithms. To achieve this, a Transformer encoder-decoder architecture is presented, technically it is a combination of a bidirectional transformer and a causal transformer (DT). To pretrain such a model, two pretraining objectives are leveraged, i.e. autoregressive prediction and a contrastive loss as auxiliary loss during training. To evaluate the proposed approach, three different settings are been conducted, i.e. meta RL, offline RL and prompt tuning. \n\nCons:\n1.the writing is hard to follow. Some statements are conflicted overall, e.g. in abstract, it is claim that this approach is training with supervised loss ('as such, we can pretrain a model ..... with supervised loss and .....'). While in Introduction, it claims that CMT is trained in a self-supervised manner (... Firstly, a model is pretrained on the .... through a self-supervised learning method). Based on my understanding, the training of CMT leverages two losses, where the autoregressive prediction is a supervised loss while the contrastive loss is a self-supervised loss. I would recommend the authors to keep the descriptions consistent in the whole paper. Otherwise, it is very confusing to the readers.\n2. lack of implementation details and experiment settings. For example, as a pretraining work, I did not find the pretraining and finetuning datasets details, especially the coverage of tasks. Are you evaluating and/or finetuning on the same tasks or unseen tasks? Without such information, it is hard the validate the generalization ability of CMT.\n3. The technical novelty is limited. From model architecture aspect, CMT is a combination of existing approaches, DT and bidirectional transformer. From algorithm aspect, it is an autoregressive loss (DT style) and contrastive loss. I can understand that this work is trying to propose a new pretraining regime, however, the usage is also not natural to me. Since the prompt  strategy can be directly utilized with only a causal transformer, in this case the DT. Then why an extra bidirectional transformer is needed? I did not find a clear answer in this paper. The rational under the contrastive loss is also unclear to me. For a specific task in a certain environment, the policy should be the same. However, based on the definition of positive and negative samples, it assumes that only the segments from the same trajectory share a policy while the other trajectories have different policies. I am confused about this setting, unless this contrastive loss are capturing other information or factors lie in trajectories, e.g. spatio-temporal information.\n4. the gains of CMT are marginal when comparing with other approaches. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DSVRc_nFiON", "reviewer": "Reviewer_yV6V", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a transformer based method to solve offline meta reinforcement learning and offline reinforcement learning tasks. An auto-regressive transformer is learned on the existing trajectory, and prompt tuning is used to guide the prediction towards policy improvement. There is an additional contrastive loss term used to ensure better clustering properties. ", "summary_of_the_review": "Paper is good overall. It would be good to address the weaknesses listed in a revised version of the paper. ", "strength_and_weaknesses": "Strength:\n- The paper is well-written and easy to follow.\n- Experiments are thorough, for offline, meta-offline, and ablation studies. \n- The proposed method aligns well with existing literature\n\nWeaknesses:\n- Hyper-parameters have been tuned to improve performance, but the process has not been explained. It is unclear if there is over-fitting to test data. \n- In LLMs, the size of the transformer model makes a large difference in performance. This has not been discussed or investigated. \n- Prompt-tuning relies on beam search. This will only work well if the replay buffer has sufficient \"good\" data points. This is evident with poor performance on Medium-replay buffers. This should be analyzed and discussed. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject