Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervision
Authors: Shilin Zhang
Zican Hu
Wenhao Wu
Xinyi Xie
Jianxiang Tang
Chunlin Chen
Daoyi Dong
Yu Cheng
Zhenhong Sun
Zhi Wang
Venue: NeurIPS 2025 poster ()
Keywords: Offline meta-reinforcement learning
reinforcement learning
world model
Primary area: reinforcement_learning
TL;DR: We learn offline meta-policies from natural language supervision with contrastive language-decision pre-training, aligning text embeddings to comprehend environment dynamics.

Abstract: Offline meta-RL usually tackles generalization by inferring task beliefs from high-quality samples or warmup explorations. The restricted form limits their generality and usability since these supervision signals are expensive and even infeasible to acquire in advance for unseen tasks. Learning directly from the raw text about decision tasks is a promising alternative to leverage a much broader source of supervision. In the paper, we propose **T**ext-to-**D**ecision **A**gent (**T2DA**), a simple and scalable framework that supervises offline meta-RL with natural language. We first introduce a generalized world model to encode multi-task decision data into a dynamics-aware embedding space. Then, inspired by CLIP, we predict which textual description goes with which decision embedding, effectively bridging their semantic gap via contrastive language-decision pre-training and aligning the text embeddings to comprehend the environment dynamics. After training the text-conditioned generalist policy, the agent can directly realize zero-shot text-to-decision generation in response to language instructions. Comprehensive experiments on MuJoCo and Meta-World benchmarks show that T2DA facilitates high-capacity zero-shot generalization and outperforms various types of baselines. Our code is available at [https://github.com/NJU-RL/T2DA](https://github.com/NJU-RL/T2DA).

Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervision
Shilin Zhang"
Zican Hu
Wenhao Wuln Xinyi Xie Jianxiang Tang'
Chunlin Chen'
Daoyi Dong²
Yu Cheng²
Zhenhong Sun 45s Zhi Wang"
Nanjing University University of Technology Sydney 3 The Chinese University of Hong Kong Australian National University University of New South Wales
(shilinzhang, zicanhu, wenhaowu, xinyixie, j jianxiangtang @smaml.mjjmlnna edu en {clchen, zhiwang)@nju. edu. cn daoyi dong@uts edu. au chengyuêcse. cuhk. edu. zhenhong. suncanu. aataauuedd.aaa edu.au
Abstract
Offline meta-RL usually tackles generalization by inferring task beliefs from high- quality samples or warmup explorations. The restricted form limits their generality and usability since these supervision signals are expensive and even infeasible acquire in advance for unseen tasks. Learning directly from the raw text about decision tasks is promising alternative to leverage much broader source of supervision. In the paper, we propose -ee--to-ee Agent (T2DA), a simple and scalable framework that supervises offline meta-RL with natural language. We first introduce a generalized world model to encode multi-task decision data into dynamics-aware embedding space. Then, inspired by CLIP, we predict which textual description goes with which decision embedding, effectively bridging their semantic gap via contrastive language-decision pre-training and aligning the text embeddings to comprehend the environment dynamics. After training the text-conditioned generalist policy, the agent can directly realize zero-shot text-to-decision generation in response to language instructions. Comprehensive experiments on MuloCo and Meta- World benchmarks show that T2DA facilitates high-capacity zero-shot generalization and outperforms various types of baselines. Our code is available at https://github.co/// www.///Oo///////DD
Introduction
Reinforcement learning (RL) has emerged as effective mechanism for training autonomous agents to perform complex tasks interactive environments [[] unleashing its potential across frontier problems including preference optimization [3] diffusion model training [4], and reasoning (5
such as in OpenAl ol [7] and DeepSeek-RI g Despite its achievements, one of the grand challenges
is generalization: building general-purpose agent capable of handling multiple tasks in response to diverse user commands [9] Due to distribution shift and the lack self-supervised pre-training techniques, RL agents typically struggle with poor generalization to unseen tasks [10]. Offline meta- RL tackles generalization via training on a distribution of offline tasks (1I. However, they usually rely high-quality samples
12 or warmup explorations [K] to infer task beliefs for generalization at test time, while these supervision signals are expensive and even infeasible to acquire in advance for unseen tasks [14]. This inspires us to explore much broader source of supervision.
Recently, large language models (LLMs), pre-trained on extensive text corpora, can encode wealth of semantic knowledge with remarkable representation power and transferability [5] The typical "text-
"Equal contribations. "Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

to-text" interface enables task-agnostic architectures to achieve broad generalization with minimal or no domain- sppecissspecic data [[6]. The availability of large-seale text collections for aggregate supervision during model pre-training has revolutionized the language and multi-modal communities in recent years [7]. Naturally, the above advancements highlight a promising question: Could scalable pre-training methods that learn perception from the supervision embedded in natural language lead to a similar leap forward in developing generalist agents for offline meta-RL. RLL?
RL typically learns from active in- teractions with the outer world tied to specific environment dynamics, with unique decision-leve' represen- tation distinct from the unbounded perception-level representation of nat- ural language [8]] Recent stud-
3n/2
2n
ies facilitate language grounding to
decision domains from various as-
pects, such as LLMs as policies [19
Raw Decision Embeddings
Dynamics-Aware Dedsion Embeddings
20 21. LLMs as rewards [22] 23
24 and
policy
learning [25 26 27]. Despite these efforts. how to harness unbounded representations of natural language knowledge for building generalist de- cision agents remains the following challenges: i) LLMs, trained on text corpora, typically lack grounding in the physical world and fail to capture any environment dynamics; ii) Lever- aging LLMs for decision- maaiing is prone knowledge misalignment due to the semantic gap between the text and decision modalities; iii) Scalable implementations are necessary to fully harness language knowledge to train generalist decision models.
Raw Text Embeddings
Aligned Text Embeddings
Figure 1: t-SNE visualization of Ant-Dir where tasks with target directions in [0, 2 are mapped to rainbow-colored points. Top: we encode multi-task data into dynamics-aware decision embeddings to capture task-specific environment dynamics. Bottom: we bridge the semantic gap between text and decision via contrastive pre-training The aligned text embeddings follow cyclic spectrum that exactly matches the periodicity of angular directions in physical sense. This interesting finding shows that we effectively align text em- beddings to comprehend environment dynamics and facilitate convincing language grounding in decision domains.
To tackle these challenges, we pro- pose Ceat-to-Decision Agent (T2DA),
a simple and scalable pre-training framework for offline meta-RL via aligning language knowledge with environment dynamics of decision tasks. First, we pre-train a generalized world model to encode multi-task data into dynamics-awarr decision embeddings, effectively capturing task-specific envi- ronment dynamics. Second, inspired by CLIP [16]. we predict which textual description goes with which decision embedding, efficiently bridging their semantic gap via contrastive language-deciste pre-training. It distills the world model structure to the text modality and aligns text embeddings to comprehend the environment dynamics. Finally, we deploy T2DA on two mainstream conditional generation architectures, training generalist policies conditioned on aligned text embeddings. During evaluation, natural language is used to reference learned decision perceptions or describe new ones, and the agent can directly realize zero-shot text-to-decision generation according to textual instruc- tions at hand. Extensive experiments show that T2DA transfers nontrivially to downstream tasks, achieves high-capacity zero-shot generalization, and outperforms various types of baselines.
In summary, our main contributions are follows:
We present a generalized world model designed to capture the environment dynamics, facilitating effective grounding of linguistic knowledge in decision domains.
We introduce contrastive language-decision pre- pre-training bridge their semantic gap, aligning the text embeddings to comprehend the environment dynamics.
We propose simple and scalable framework that supervises offline meta-RL. with natural language, and develop scalable implementations with the potential to train decision models at scale: Text-to- Decision Diffuser and
Transformer.
2

---

2 Related Work
Offline Meta-RL learns adaptable policies from pre-collected datasets without requiring online environment interaction [1 [8]_ .. It addresses two fundamental challenges: the distributional shift between behavior and learned policies in offline RL.
[29 30), and the quick adaptation to unseen tasks with minimal data. Existing approaches can be broadly categorized into the memory-based
(c.g., RL [31] and LLIRL 132 33 the optimization-base
(MAML (34] and MACAW (IT),, and the context-basse based methods (c.g., PEARL (35), VariBAD [36], etc. [28] 13 37)) Recent studies of in-context RL attempt to leverage the in-context learning capability of transformers to improve RL's generalization via casting RL as an across-episodic sequential prediction problem with few-shot prompting test time [38 39 40 41) such as AD [42] and DPT 45.. In general, the above approaches rely on high-quality samples or domain knowledge to infer task beliefs during evaluation, limiting their generality with this restricted form of supervision. This inspires us to explore a much broader source of supervision from natural language.
LLMs for RL. Existing studies on leveraging LLMs for RI. domains be categorized into three main threads: LLMs as rewards, LLMs as policies, and
conditioned policy learning. The first kind uses LLMs or VLMs Wisiio-language models) to automate the generation and shaping of reward functions 44 22
24 or state representations 45). providing denser information to facilitate RL training. The second kind utilizes LLMs or VLMs as the policy backbone [8| 46 47 48
23
Vision- laanoo-lnnuage-acttinnn (V (VLA) models provide direct instantiation of using pre- trainaindd VLMs for robotics, fine-tuning iisually-condition
VLMs to generate robot control actions, such RT-2 ( OpenVLA [50]. and mo.5 [51]. In contrast, we leverage LLMs to obtain task representations for offline Meta-RL. and train conventional RL policies based on decision transformer or diffuser architectures.
Landuage-condittonee policy. We focus on the third kind that learns a language-conditionee policy, extracting the world knowledge encoded in LLMs to help train offline meta-RL agents. Previous studies acquire language embeddings using pre-trained LLMs like BERT and GPT [52]. while these embeddings are learned independently from decision tasks and may fail to capture domain-related information. Early methods develop rule-basee [33 or task-specific (54) intermediate representations to capture taskerelated semantics, which can require laborious design and lack scalability. Under an imitation learning paradigm, BC-Z [25] trains a vision-based robotic manipulation policy conditioned on pre-trained language embeddings, and BAKU 27 produces simple transformer architecture
that fuses multi-modal vasion-language information and temporal context for multi-task decision- making. To improve language grounding, (55) translates natural language to task language with a referential game, and [26] uses supervised fine-tuning to enable bidirectional translation between lunguage-descrbee skills and rollout data. Overall, the challenges include the inability to capture environment dynamics, knowledge misalignment due to semantic gaps, and limited model scalability. In the paper, we tackle these challenges to fully harness LLMs for building generalist decision agents.
3 Method
In this section, we present -extito-Decision Agent (T2DA), simple and scalable framework train generalist policies. Figure the subsequent subsections. Corresponding algorithm pseudocodes are given in Appendix A
Sequre@]llustrran illustrates the overall pipeline, followed by detailed implementations in
3.1 Problem Statement
We consider a language-conditionee offline meta-RL setting. Tasks follow a distribution Ms- = (S, A. Tx, R4: ((M), sharing the same state-action spaces while varying in the reward and state transition functions, i.c., environment dynamics. For each training task k. our system receives user-provided natural language supervision ly that describes the task (c.g., open the door), paired with an offline dataset Dj = Es at rk. s collected by arbitrary behavior policies. The agent can only access the offline datasets E (le, D&) to train generalist policy π (a]ss l) to follow language instructions. At test time, natural language used to reference learned decision perceptions or describe new ones. The agent can perform text-to-decision generation in test environments in zero-shot manner, given any language instruction lnew at hand. The primary objective is to obtain a highly generalizable policy that achieves strong zero-shot performance on unseen tasks.
3

---

Decision embedding
Environment dynamics
R
R
13/
Text-to-Dacision Dilliser
maniing
Trajectory Encoder
Decoder World Model
21 Test-to -eeeisinee Transformer
LoRA
Trajectory Encoder
Text Encoder
Text Encoder
8-
Multi-Task Dataset D2
Open
Open Gemerallist policy training
Language Instructions
(a) Dynamic-waare decision embedlingg
Contrrstive langaggedettison pre-training
Figure 2: The overall pipeline of T2DA. (a) We encode the multi-task trajectories into dynamics- aware decision embeddings and decode the generalized world model conditioned on that embedding, effectively capturing the environment dynamics. (b) We bridge the semantic gap between decision and text by fine-tuning the text encoder (initialized from popular language models such as CLIP or T5) to align the produced text embeddings with dynamics-aware decision embeddings using contrastive loss. It distills the world model structure from decision embeddings to the text modality, aligning text embeddings to comprehend the environment dynamics. (c) We condition the generalist policy on the aligned text embeddings, and develop scalable implementations with the potential to train decision models at scale: Text-to-Dee Deees---D Diffuser and Text-to-Decision Transformer. During evaluation, the agent directly realize zero-shot eex-ito-eeiiioon generation according to textual instructions hand, enabling high-capacity zero-shot generalization to downstream tasks.
3.2 Dynamics-Aware Decision Embedding
Natural language is complex and unbounded representation of human instruction, and training policies to harness language supervision is nontrivial. A standard approach is to convert language instructions to simpler latent embeddings using popular pre-traing rre-trained language models that can draw on a wealth of knowledge learned from copious amounts of text. However, directly applying these models to decision domains could fail to solve user-specific problems, they are typically learned independently of decision-making tasks. A major challenge that LLMs trained on text corpora typically lack grounding in the interacting environment and fail to capture its dynamics.
For effective grounding, the agent should comprehend environment dynamics of underlying decision tasks. RL typically learns from active interactions with the outer environment. The world model, i.e., the reward and state transition functions p(s', r|s, a), fully characterizes the environment and presents promising alternative capturing environment dynamics. Hence, we introduce generalized world model to encode the multi-tase task decision data into a aanamics-aware embedding space. We separate the reasoning about the world model into two parts: i) encoding the dynamics-spectie information into a latent embedding, and ii) decoding environment dynamics conditioned on that embedding.
First, we use a trajectory encoder & to abstract multi-task decision trajectories into compact embedding z that captures task sask-specific dynamics. Specifically, we "tokenize" different elements (i.e., state, action, and reward) in the raw sequence (50, do, To. GL, Dr. by lifting them to common representation space using element-specific tokenizers f² f.. f
(e) eū. ej, ... EZ, et, eT), where = =f fr())) Vt [0, LJ. (I)
We use a bi-directional transformer Egg extract the dynamics-aware embedding from input tokens as 2か = Eg The bi-directional structure allows for capturing the forward and inverse dynamics.
Second, we introduce decoder &o containing reward model R. and state transition model Ty. The latent embedding is augmented into the input to predict the instant reward next state $1+1 (st, ax; 2k). The pecoder-decoder pipeline is jointly trained by minimizing the reward and state transition prediction error conditioned on the decision embedding as
(st, a, az2z)) and
L( up) = EyaDi [BsNO(T)
[Eg (re (o, az =x)) + (8141 - To (8), a, xx) =4))
Vk. (2)

---

In practice, we randomly sample a trajectory from the dataset as ~~ DR and use its decision embedding to decode the dynamics of other trajectories in the same dataset as r" E D& The reason is that, since the embedding has access to the entire trajectory's information, using it to decode the same trajectory can lead to deceptive traps during training. After proper training, we freeze the world model for the subsequent learning phases.
3.3 Contrastive Language-Deciin Decision Pre-training
To successfully instruct generalist training with human language, must bridge the semantic gap between text and the "decision modality". Inspired by CLIP [16), we predict which textual description goes with which training task, bridging their semantic gap via contrastive learning. The core idea is to connect the language supervision with corresponding decision embeddings, distilling the world model structure to the text modality and aligning text embeddings comprehend environment dynamies.
Formally, given a batch of (trajectory т, textual task description () pairs, we predict which of the N N possible (trajectory, text) pairings across batch actually occurred. We use the pre- trained trajectory encoder in Sec. [3.2] to extract the dynamics- aaaaicc--wware decision embedding from the trajectory as é(T). The text encoder is transformer- based model & that could be initialized from of wide variety of popular language models, such as CLIP [66] or T5 [56]. Then, the text embedding 27 easily derived by tokenizing the textual instruction / and feeding it to the encoder as 2T (l)..
We model the pre-training task as a text-decision multi-modal learning problem by fine-tuning the text encoder while keeping the trajectory encoder fixed. The objective is to maximize the cosine similarity of the decision and text embeddings of the N real pairs in the batch while minimizing the cosine similarity of the embeddings of the N - N incorrect pairings. The two directional similarities between the modalities, which are transposes of each other, are defined as
WDD (U),, [ (1)Wl) sim(l, r) - exp(aa)
w(),, ((T)
sim(r, (, exx()a)))
(3)
(1) [Wll"
where a is a learnable temperature controlling the range of similarities, and WD and WT are learnable projections that map feature representations of each modality to a joint multi modal embedding space. The decision-to-text p(r) and text-to-decision p(l) similarity scores in the batch are calculated as
sim(le:FR)
p(Tk) = esim(redi))
p(1e) = Σn esim(le.7s)
Let q(T) and ql) denote the ground-truth similarity scores, where negative pairs have a probability of O and positive pairs have 1. We optimize symmetric cross-entropy loss over similarity scores as
(( 0.5 E,,, [CE (p(T). +
CE
(5)
where CE is the cross-entropy between two distributions. To effectively adapt the text encoder to the semantics of decision modality at a lightweight computing cost, we employ low-rank adaptation (LoRA) [57] fine-tune the text encoder with the above contrastive loss. Experiments in Appendix E present the analysis of its efficiency and superiority compared to full- farl-parameter fine-tuning.
This alignment process distills the environment dynamics from the decision embedding to the text modality. The aligned text embeddings not only serve as mere linguistic representations but also establish stronger comprehension of underlying decision tasks, effectively bridging the semantic gap between language supervision and decision dynamics.
3.4 Generalist Policy Learning
A widely adopted manner to tackle generalization in RL domains is to condition the policy on some representation that can capture task-relevant information, such as context-based meta-learning [35] 36) and prompt- based methods [2 [4]. Under this unified framework, we formulate the generalist agent as task- cond-onnttioeedd policy model. There exists a true variable that represents the task identity, and we use latent representation h to approximate that variable. Then, a base policy TT-, h) can be shared across tasks by conditioning on the task representation as TH (a (s) = (a) (s; l1)). Wk. Previous studies usually approximate the task representation by acquiring expert decision data [12] or exploring the
unseen task first [33. This restricted form of supervision limits their universality and practicability since expensive high-quality samples explorations are needed to specify any new task.
5

---

Motivated by the recent success of language models, we study promising alternative that leverages a much broader source of supervision. Natural language is flexible representation for transferring variety of human ideas and intentions. Learning from its supervision can not only enhance the representation power but also enable efficient knowledge transfer. We map the textual task description 7 to latent embedding (l) via the fine-tuned text encoder in S.c.3.3 and use it to approximate the task representation as h (1). Then, the task-agnostic generalist policy is approximated by
T(a
| 8; 14) = π (a s; (b+)), vk.
(6)
At test time, the agent can directly perform text-to-decision policy generation according to any textual instruction lew hand, enabling high-capacity zero-shot generalization downstream tasks without the need for expensive decision demonstrations or warmup explorations.
3.5 Scalable Implementations
We develop scalable implementations of T2DA using two mainstream conditional generation archi- tectures that hold the promise to train RI. models at scale: the decision diffuser [58] and the decision transformer (59), yielding the following T2DA variants.
-ee-iiDoe Diffuser (T2DA-D). Inspired by the recent success of text-to-image diffusion models [60] 61]. we implement the text-to-decision counterpart by modeling the policy return- conditioned diffusion model with action planning. For model simplicity and scalability, we choose to diffuse over the //-step state-action trajectory as
(T)
[st
Sc4-1
814 Ge+ 11
Ot
0++11
where denotes the timestep in the forward diffusion process, and c= 0 corresponds to the unperturbed data. To enable generalist policy with natural language supervision, we include the fine-tuned text embedding (1) an additional condition to the diffusion model. To this end, the generalist policy training is formulated as the standard problem of conditional generative modeling as
max EYWD [lo pe (mo(T) | R(()ss v0)]
(8)
where A denotes diffusion parameters and R is the return-to-go. During evaluation, we can cast planning in RL as sampling from the diffuser. At each timestep. we observe a state 8, in the environment, sample an H-step trajectory mo(T) with the diffusion process conditioned
target return-to-go and task prompt (I), execute the first action de in (o()), and transition to the next state
8.411 More details can be referred as in Algorithms and 5 in Appendix A
Text-to-Decision Transformer (T2DA-T). We also implement T2DA with the causal transformer architecture, drawing upon the simplicity, scalability, and associated ad in language modeling such as BERT and GPT. We prepend a task prompt to the input of the causal transformer to realize generalization across tasks, akin to Prompt-DT [12]. For each task, we take the prompt-augmented trajectory 7+ as the input, which contains both the fine-tuned text embeddings (i( and the most recent H step history sampled from offline datasets as
+ = (vb(T); R4-H+1:8 8t- 01 H+1, Re, 81,0)) at).
(9)
where the return-to-go R the cumulative rewards from the current time step till the end of the episode. Then, the model predicts actions autoregressively using causal self-attention mask. This architecture introduces only one additional token compared to the standard decision transformer, allowing minimal architecture change and lightweight cost for generalist training. More details can be referred to as in Algorithms 4 and in Appendix A
Decision diffuser diffuses over only states and trains another inverse dynamics model to generate actions. In contrast, we only keep a unitied diffusion model to i) ensure simplicity with a single end-to-ent model in a multi task setting, and ii) unlock the potential of diffiusion models at scale, rather than depending on additional MLPs generate part of the information, e.g., actions.

---

T2DA-T (Ours)
T20N-D (Ours)
BC-2
C5RO
UNICORN
AD
BAKU
Chestah-ve
-40
.402
2403
1203
2.03
Retums
Danurns
-350
Danurnn
603
-17
-20
330
200
Timesops
Timesteps
Figure 3: Zero-shot test return curves of T2DA against baselines using Mixed datasets.
Table 1: Zero-shot test returns of T2DA against baselines using Mixed datasets, i.e., numerical results
of converged performance from Figure
Best results bold and second best underlined.
Environment
T2DA-T
T2DA-D
BC-Z
CSRO
UNICORN
AD
BAKU
Point-Robot
-7.2=0.1
-84402
-15.5002
-13.710.
-11.4+13
-15.8402
-66.4+0.1
Cheetah-Vel
- -00.3388
-00.4118
-107.4181
-92.6459
-94.14200
-107.6223
-104.5448
Ant-Dir
970.3287
570.6213.1
700.04433
317.1
407.3=216
268.9114
786.9417.
1
1376.4:396
1053.8#33.8
1005.2
754.6=627
921.0+66.
1187.6+18.2
4
Experiments
We comprehensively evaluate and analyze our method on popular benchmarking domains across datasets of varying qualities, aiming to answer the following research questions:
Can T2DA achieve consistent performance gain zero-shot generalization capacity to unseen tasks? We compare it to various types strong baselines, including offline meta-RL, in-context RL, and language-conditioned policy learning approaches. (Sec.
What is the contribution of each component to T2DA's performance? We ablate both the T2DA-D and T2DA-T architectures to analyze the respective impact of world model pre-training, contrastive language-decision pre-training, and language supervision. (Sec. 4.2)
How robust is T2DA across diverse settings? We evaluate T2DA against baselines using offline datasets of varying qualities, and assess T2DA's performance when initializing the text encoder from different LLMs. (Sec. 4.3
Environments. We evaluate T2DA three benchmarks that are widely adopted assess general- ization capacities of RL algorithms: i) the 2D navigation Roint-Robot; ii) the multi-task MuJoCo locomotion control, containing Cheetah-Ves -eel and Ant-Dir; and iii) the Meta-Worll platform for robotic manipulation, where a robotic arm is designed to perform a wide range of manipulation tasks, such close faucet, lock door, open door, and press button. For each domain, distribution of tasks captioned with text descriptions, and split them into training and test sets. We employ SAC (62) to independently train single-task policy on each training task for offline dataset collection. We develop three types of offline datasets: Mixed, Medium, and Expert, following the common practice in offline RL [63] Appendix presents more details of environments and datasets.
randomly sample
Baselines. We compare T2DA to five competitive baselines that cover three representative paradigms in tackling RL generalization: the context-based offline meta-RL approaches, including 1) CSRO [13] and 2) UNICORN [37]: the in-context RL method of 3) AD [42]: : and the aunuuag-conditione policy
learning methods including 4) BC-Z 25 and 5) BAKU 27
For evaluation, experiments are conducted using five different random seeds. The mean of received return is plotted as the bold line, with 95% bootstrapped confidence intervals of the mean indicated by the shaded region. Appendix gives detailed implementations of T2DA. Appendix gives analysis on fine-tuning of the text encoder.
7

---

T2DA
we world
wio align
w/o text
Fains
1600
-10
Rezurms
600
Returss
400
103
38
-25
Timestags (1w4))
Timesops
(1 Ablation of T20A-T
(i]] Ablation of T2D44-
Figure 4: Ablation results using Mixed datasets. w/o world omits pre-training the trajectory encoder, w/o align omits contrastive pre-training, and w/o text omits the language supervision.
Table 2: Numerical results of ablation study on both T2DA-D and T2DA-T using Mixed datasets, i.c., the final test returns after learning convergence from Figure 4 Best results in bold.
Environment
T2DA-T
w/o world w/o align
w/o text
T2DA-D
w/o world w/o align
w/o text
Point-Robot
-7.2+0.1
-7.7+03
-9.2+002 -6..9+0.1
-8.400.
-9.5+04 -10.1+02 -17.2=04
Ant-Dir
963.4+153
891.0+544 792.3=48.3 242.6+137
100
111.1 513.8+11.0 105
4.1
Main Results
We compare our method against various baselines under an aligned zero-shot setting. Figure and Table ppelsenttt test return curves and numerical results of converged performance on various benchmark environments using Mixed datasets. A noteworthy point is that language-conding conditione baselines such as BC-Z and BAKU generally achieve better performance than offline meta-RL RL and in-context RL baselines, especially in harder environments like Ant-Dir and Meta- Deta-World. This again validates our motivation of exploring a much broader source of supervision from natural language, harnessing the representation power and knowledge transferability embodied in pre-trained LLMss
In these diversified environments, both T2DA-D and T2DA-T consistently achieve significantly superior performance regarding learning speed and final asymptotic results compared to the three types of baselines. most cases, T2DA-D and T2DA-T take the top two rankings for the best and second- best performance, and showcase comparably strong zero-shot generalization capacities average across all evaluated environments. It highlights the effectiveness of both T2DA implementations using the two mainstream conditional generation architcctures. Furthermore, method typically demonstrates lower variance during generalist policy learning, signifying not only enhanced learning efficiency but also improved training stability.
T2DA-
UNICORN
Potnt Robot
World
Point-Robe
Meto World
1420
21033
10:0
-12-
650
Timesteps (1e4)
Timesteps (1e6)
Timestops (Zed)
Timesteps (1e4)
Expert Datasets
Medium
Figure 5: Results of robustness to data quality, where T2DA is compared to baselines using Expert and Medium datasets. T2DA-D and T2DA-T achieve consistent superiority across various datasets.
4.2 Ablation Study
We compare T2DA three ablations: 1) w/o world, it eliminates pre-training the trajectory encoder and instead updates it jointly with the text encoder during contrastive pre-training; 2) w/o align,

---

Table 3: Numerical results of robustness data quality where T2DA is compared against baselines using Expert and Medium datasets, i.e,, the final test returns after learning convergence from Figure 5 Both T2DA-D and T2DA-T achieve consistent superiority across datasets of varying quality. Best results in bold and second best underlined.
Expert
T2DA-T
T2DA-D
BC-Z
CSRO
UNICORN
AD
BAKU
Point-Robot
1.24
-0.
14.4104
-2..8.0.2
-11.1177
-17.1102 -15.4_0.11 0.1
Meta-World
1142.8 i 106.6
1387.7 1.166
935.512667
1087.0178.
787.1 991.0137.7
833.618.3
Medium
T2DA-T
T2DA-D
BC-Z
CSRO
UNICORN
AD
BAKU
Point-Robot
10.7104
-1..0...
16.6102
16.1
14.9105
16.9100
16.1 0.1
Meta-Words World
1033.311306
1025.3
156.911024
854.111799
715.8193.
585.71126.1
669.5.1114
omits contrastive language-decision pre-training and fixes the text encoder initialized from pre-trained language models; and 3) w/o text, it completely removes the text encoder.
Figure 4 presents ablation results of both architectures. First, the erasure of pre-training the trajectory encoder (T2DA vs. w/o world) results in decreased test returns with greater variances. It indicates that capturing environment dynamics via the world model can enable more precise and stable knowledge aligmment between text and decision. Second, omitting the language-decision pre- pre-training (w/o world vs. w/o align) further decreases the performance. confirms the existence of a semantic gap between text and decision, whereas the gap can be effectively bridged by contrastive knowledge alignment. Finally, removing the language supervision (w/o align vs. w/o text) can produce catastrophic performance with poor zero-shot generalization, highlighting the necessity of leveraging a wealth of knowledge from natural language. In summary, the performance T2DA declines when any component is omitted, verifying that all components are essential to its final capabilities.
4.3 Robustness Study
A reliable framework with scalability should be robust across diverse settings. We assess T2DA's robustness in two critical aspects: data quality and choice of text encoders.
Robustness to Data Quality. We compare T2DA against baselines using Expert and Medium offline datasets. As shown in Figure and Table 3 both T2DA-D and T2DA-T consistently outperform baselines on these datasets. Notably, while most baselines suffer significant performance degradation when trained on Medium datasets, T2DA maintains high-capacity generalization despite the lower data quality. It showcases the promising applicability of T2DA in real-world scenarios where the agent often needs to learn from sub-optimal data, In summary, T2DA achieves consistent superiority across datasets of varying quality, yielding excellent robustness.
Robustness to Text Encoders. We also investigate T2DA's sensitivity to different representations of language supervision by initializing the text en- coder from three distinct language models: CLIP [16), BERT [64]. and T5 (56). As shown in Figure 6 the near-identical performance across all language models highlights that T2DA's effectiveness is independent of the specific text encoder architec- ture. It suggests that T2DA can ef- fectively leverage the semantic under- standing capabilities inherent to differ- ent language models while maintain- ing consistent performance.
CLPP
BERT
T5
Point-Robot
Ant-Dir
Returns
Returns
-14
-16
Timestops (1o4)
Timestaps (1c4)
Figure 6: Results robustness to text encoders where T2DA initializes the text encoder from CLIP, BERT, and T5, show- ing that T2DA can efficiently harness language knowledge embedded in different language models.

---

4.4
Visualization Insights
We gain deep insights into the knowledge alignment between text and decision through t-SNE visualization on the example Cheetah- Vel task as shown in Figure 7
Decision Embeddings. Raw deci- sion embeddings are derived from randomly initialized trajectory coder. In the dynamics-aware embed- ding space, sample points from dif- ferent tasks are more clearly distin- guished, and similar tasks grouped more closely. The initially entangled trajectories are transformed into well- separated clusters, highlighting the successful capture of task task-specific en-
Raw Decision Embeddings
Dynamics-Aware Dedsion Embeddings
vironment dynamics via the general- ized world model.
40
Text Embeddings. Raw text embed- dings are derived from pre-trained lan- guage models, and are scattered with vague distribution structure. In the aligned space, text embeddings fol- low more distinguished distribution that matches the metric of task sim- ilarity, highlighting successful bridg- ing of the semantic gap. Notably, the aligned embeddings form clear rec- tilinear distribution from red (low ve- locity) to blue (high velocity), exactly matching the rectilinear spectrum of target velocities in a physical sense. verifies efficient comprehension of the environment dynamics.
Embedddings
Aligned Embeddings
Figure 7: t-SNE visualization of Cheetah- Vel where tasks with target velocities in [0.075, 3.0] mapped into rainbow- colored points. Top: the evolution from Raw Decision Embeddings to Dynamics-Aware Decision Embeddings, where the initially entangled trajectories are transformed into well-separated clusters that highlight the successful cap- ture of task-specific environment dynamics. Bottom: using dynamics- -ware decision embeddings to align text embed- dings with contrastive loss. This inspiring finding verifies the effective alignment of text embeddings comprehend envi- ronment dynamics and interpretable language grounding.
Conclusions, Limitations, and Future Work
In the paper, tackle offline meta-RI. challenge via leveraging much broader source of supervision from natural language. Improvements in zero-shot generalization capacities highlight the potential impact of our scalable implementations: text-to-decision diffuser and text-to-decision transformer. These findings open up new avenues for building more flexible and accessible generalist agents that can understand and upon natural language instructions without extensive Nomain-speciin training.
Though, our agent is trained on lightweight datasets compared to popular large models. An essential step is to implement vast datasets with diversified domains, unleashing the scaling law with the diffusion or transformer architectures. Another step is to deploy the idea of knowledge alignment in VL. models, or deploy the text-to-decision paradigm real robots toward embodied intelligence.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China (Nos. 62376122 and 72394363), and in part by the AI & AI for Science Project of Nanjing University.
10

---

References
[1]
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei Rusu, Joel Veness, et al. Human- level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.
[2] Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Comera-Paredess al. Discovering faster matrix multiplication algorithms with reinforcement learning, Nature, 610(7930):47-53 2022.
[3] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, 2023.
[4] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In Proceedings of International Conference on Learning Representations, 2024.
[5] Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505. 226117, 2025.
[6] Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang. Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang Learning to reason under off-policy guidance. In Advances in Neural Information Processing Systems, 2025.
[7] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, et al. OpenAl ol system card. arXiv preprint arXiv:24/2.16720 2024.
[8] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, et al. DeepSeek- RI: Incentivizing reasoning capability Ilms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
[9] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. No: A model for general robot control. arXiv preprint arXiv:2410. 24 1614, 2024.
flow
[10] Thomas Schmied, Markus Hofmarcher, Fabian Paischer, Razvan Pascanu, and Sepp Hochreiter. Learning to modulate pre-trained models in RL. Advances in Neural Information Processing Systems, 2023.
[11] Eric Mitchell, Rafael Rafailov, Xue Bin Peng, Sergey Levine, and Chelsea Finn. Offline meta- reinforcement learning with advantage weighting. In Proceedings of International Conference on Machine Learning, pages 7780-7791, 2021.
[12] Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In Proceedings of International Conference on Machine Learning, volume 162, pages 24631-24645, 2022.
[13] Yunkai Gao, Rui Zhang, Jiaming Guo, Fan Wu, Qi Yi, Shaohui Peng. Siming Lan, Ruizhi Chen, Zidong Du, Xing Hu, et al. Context shift reduction for offline meta-reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, 2023.
[14] Zhi Wang, Li Zhang, Wenhao Wu, Yuanheng Zhu, Dongbin Zhao, and Chunlin Chen. Meta- DT: Offline meta-rl as conditional sequence modeling with world model disentanglement, In Advances in Neural Information Processing Systems, 2024.
[15] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, et al. GPT-4 technical report. arXiv preprint arXiv:2303.0 087744 2023.
[16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, et al. Learning transferable visual models from natural language supervision. In Proceedings of International Conference Machine Learning, pages 8748-8763 2021.

---

[17] Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust CLIP: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision- -aiiu-la models. In Proceedings of International Conference on Machine Learning, 2024.
[18] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as I can, not as say: Grounding language in robotic affordances. In Proceedings of Conference on Robot Learning, 2022.
[19] Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Rin Metcalf, Walter Talbott, Natalie Mackraz, R Devon Hjelm, and Alexander T Toshev. Large language models as general- izable policies for embodied tasks. Proceedings of International Conference on Learning Representations, 2024.
[20] Zican Hu, Wei Liu, Xiaoye Qu, Xiangyu Yue, Chunlin Chen, Zhi Wang, and Yu Cheng. Divide and conquer: Grounding LLMs as efficient decision-making agents via offline hierarchical reinforcement learning. In Proceedings of International Conference on Machine Learning, 2025.
[21] Runzhe Zhan, Yafu Li, Zhi Wang, Xiaoye Qu, Dongrui Liu, Jing Shao, Derek F Wong, and Yu Cheng. ExGRPO: Learning to reason from experience. arXiv preprint arXiv:2510.02245, 2025.
[22] Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision- language models are zero-shot reward models for reinforcement learning. In Proceedings of International Conference on Learning Representations, 2024.
[23] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Texr2Reward: Reward shaping with language models for reinforcement learning. In Proceedings of International Conference on Learning Representations, 2024.
[24] Yuwei Fu, Haichao Zhang, Di Wu, Wei Xu, and Benoit Boulet. FuRL: Visual-language models fuzzy rewards for reinforcement learning. In Proceedings of International Conference on Machine Learning, 2024.
[25] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler. Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. BC-Z: Zero-shot task generalization with robotic imitation learning. In Proceedings of Conference on Robot Learning, pages 991-1002, 2022.
[26] Jing-Cheng Pang, Si-Hang Yang, Kaiyuan Li, Jiaji Zhang, Xiong-Hui Chen, Nan Tang, and Yang Yu. Knowledgeable agents by offline reinforcement learning from large language model rollouts. In Advances in Neural Information Processing Systems, 2024.
[27] Siddhant Haldar, Zhuoran Peng, and Lerrel Pinto. BAKU: An efficient transformer for multi-task policy learning. In Advances Neural Information Processing Systems, 2024.
[28] Haoqi Yuan and Zongqing Lu. Robust task representations for offline reinfrrennrree learning via contrastive learning. In Proceedings of International Conference on Machine Learning, pages 25747-25759, 2022.
[29] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[30] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pages 1179-1191, 2020.
[31] Yan Duan, John Schulman, Xi Chen, Peter I. Bartlett, Ilya Sutskever, and Pieter Abbeel. RI.2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 161 1.02779, 2016.
[32] Zhi Wang, Chunlin Chen, and Daoyi Dong. Lifelong increeental reinforcement learning with online Bayesian inference. IEEE Transactions on Neural Networks and Learning Systems, 33(8):4003 2022.
12

---

[33] Zhi Wang, Chunlin Chen, and Daoyi Dong. A Dirichlet process mixture of robust task models for scalable lifelong reinforcement learning. IEEE Transactions on Cybernetics, 53(12),7599-7520 2023.
[34] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learing for fast adapta- tion of deep networks. In Proceedings of International conference on machine learning, pages 1 1126-1 11 135, 2017.
[35] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In Proceedings of International Conference on Machine Learning, pages 5331-5340, 2019.
[36] Luisa Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl. Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. VariBAD: Variational Bayes-adaptive deep RL via meta-learning. The Journal of Machine Learning Research, 22(1):1118--32233 2021.
[37] Lanqing Li, Hai Zhang, Xinyu Zhang, Shatong Zhu, Junqiao Zhao, and Pheng- Heng. Towards an information theoretic framework of context-based offline meta-reinforcement learning. In Advances in Neural Information Processing Systems, 2024.
[38] Sili Huang, Jifeng Hu, Hechang Chen, Lichao Sun, and Bo Yang. In-context decision trans- former: Reinforcement learning via hierarchical chain-of-thought. In Proceedings of Interna- tional Conference on Machine Learning, 2024.
[39] Jake Grigsby, Linxi Fan, and Yuke Zhu. AMAGO: Scalable in-context reinforcement learning for adaptive agents, In Proceedings of International Conference on Learning Representations, 2024.
[40] Wenhao Wu, Fuhong Liu, Haoru Li, Zican Hu, Daoyi Dong, Chunlin Chen, and Zhi Wang. Mixture-of-experts meets in-context reinforcement learning. In Advances in Neural Information Processing Systems, 2025.
[41] Jinmei Liu, Fuhong Liu, Jianye Hao, Bo Wang, Huaxiong Li, Chunlin Chen, and Zhi Wang. Scalable in-context Q-learning. arXiv preprint #rXiv:2506.01299 2025.
[42] Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, et al. In-context reinforcement learning with algorithm distillation. In Proceedings of International Conference on Learning Representations, 2023.
[43] Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. Advances in Neural Information Processing Systems, 36, 2023.
[44] Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In Proceedings of International Conference on Learning Representations, 2023.
[45] Boyuan Wang, Yun Qu, Yuhang Jiang, Jianzhun Shao, Chang Liu, Wenming Yang, and Xi- angyang Ji. L.M--eppwereed state representation for reinforcement learning. In Proceedings of International Conference on Machine Learning, 2024.
[46] Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision-language models as decision- making agents via reinforcement learning. In Advances in Neural Information Processing Systems, 2024.
[47] Ruizhe Shi, Yuyao Liu, Yanjie Ze, Simon Shaolei Du, and Huazhe Xu. Unleashing the power of pre-trained language models for offline reinforcement learning. In Proceedings of International Conference on Learning Representations, 2024.
[48] Zican Hu, Shilin Zhang, Yafu Li, Jianhao Yan, Xuyang Hu, Leyang Cui, Xiaoye Qu, Chunlin Chen, Yu Cheng, and Zhi Wang. Diversity-incentivized exploration for versatile reasoning. arXiv preprint 2xXi:25009.62299 2025.
13

---

[49] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. RT-2: Vision-language- action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 2023.
[50] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, al. VLA: An open-source vision-language-action model. In Proceedings of Conference on Robot Learning. 2024.
[51] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, et al. no.s: vision-language-actiinn model with open-world generalization. arXiv preprint arXiv:2504, 16054, 2025.
[52] Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human deep reinforcement learning via transfer-lerring from text. arXiv preprint XXXi:2005500382 2020.
with
[53] Muhannad Alomari, Paul Duckworth, David Hogg, and Anthony Cohn. Natural language acquisition and grounding for embodied robotic systems. In Proceedings of AAAI Conference on Artificial Intelligence, volume 31, 2017.
[54] Ahmed Akakzia, Cédric Colas, Pierre-Y Yves Oudeyer, Mohamed Chetouani, and Olivier Sigaud. Grounding language to autonomously-acquiee skills via goal generation. In Proceedings of International Conference on Learning Representations, 2021.
[55] Jing-Cheng Pang, Xin-Yu Yang, Si-Hang Yang, Xiong-Hui Chen, and Yang Yu. Natural language cnstruction-followiit with task-related language development and translation. In Advances in Neural Information Processing Systems, volume 36, 2023.
[56] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-66 67, 2020.
[57] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In Proceedings of Interna- tional Conference on Learning Representations, 2022.
[58] Anurag Ajay. Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for makknn?? In Proceedings of International Conference on Learning Representations, 2023.
[59] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems, volume 34, pages 15084-15097, 2021.
[60] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, volume 35, pages 36479-36494, 2022.
[61] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to -ext-to-image diffusion models. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[62] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of International Conference on Machine Learning. pages 1861-1870, 2018.
[63] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.0022/1 2020.
[64] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. BERT: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the North American Chapter of the Association for Computational Linguistics Human Language Technologies, volume 1, 2019.
14

---

[65] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-Worldd benchmark and evaluation for multi-task and meta reinforcement learning. In Proceedings of Conference Robot Learning, pages 1094 1 100, 2020.
[66] Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang. MetaDif- fuser: Diffusion model as conditional planner for offline meta-RL. RL. Proceedings of the 40th International Conference Machine Learning, pages 26087-26105, 2023.
[67] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[68] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUS). arXiv preprint arXiv:1606.08411 2016.
[69] Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Hu- manoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation. arXiv preprint arXiv:2403. 10506, 2024.
15

---

Appendix
Algorithm Pscudocodes
17
B Evaluation Environments and Dataset Construction
17
The Details of Environments
17
B.2 The Details of Datasets
19
Baseline Methods
20
Implementation Details of T2DA
21
D.i Network Architecture and Hyperparameters
21
D.2 Computation
22
Analysis of Parameter-Eftichee Efficient Fine-tuning
23
Analysis of Training Pipline
23
Meta-RI, Baselines with Goal Information
24
H Performance Unstructured Language Instructions
24
Experimental Results on Humanoid
24
Comparison More OMRL Baselines
25
Limitations
25
16

---

A Algorithm Pseudocodes
Based the implementations in Sec. B] this appendix gives the brief procedures of each component in T2DA. First, Algorithm presents the pre-training of the generalized world model. Based on the dynamics-aware decision embedding encoded by Algorithm [ Algorithm 2 shows the contrastive language-decision pre-training that bridges the semantic gap between text and decision, and aligns text embeddings comprehend the environment dynamies of decision tasks.
Then, Algorithm 3 and A present the model training processes of Text-to-Decision Diffuser and Text-to- Decision Transformer, respectively, where the generalist policy is conditioned on the text embeddings fine-tuned in Algorithm Finally, Algorithm and Algorithm 6 show zero-shot evaluations test tasks, where the agent directly realize text-to-decision generation according textual instructions at hand.
Algorithm 1: Pre-training the generalized world model
Input:
Offline datasets D Trajectory encoder Φb for each iteration do
Batch size m
Reward decoder R2 and Transition decoder T
for i 1, m do
Sample task M& and obtain the corresponding dataset Dr from Drrain Sample two trajectories from DD as T = (80, (o, ro, 81, (11, ri,... and
T" (sg, ag. ro si aj, .... Obtain the decision embedding by feeding 7 to the trajectory encoder as z = ((T)
for t 0, 1,... do
Compute the predicted reward fi = (sp (t; and next state $ (s 07; 2) trajectory 7"
end end
Update and (p jointly using loss
C(o. ④) m Σ E, (rt Ro(s7, 2))² + (si+1 as;
end
Algorithm 2: Contrastive Language-Decisinn Pre-training
Input: Offline datasets D
Batch size N
Pre-trained trajectory encoder o;
Text encoder eb
Freeze the trajectory encoder pre-trained in Algorithm
2 Initialize text encoder &b from popular pre-trained language models like CLIP or T5
for each iteration do Sample batch of N (trajectory textual task description l) pairs from Dimin
Compute the symmetric similarities between the modalities, sim(r, l) and sim (1, r), as in Eq. (3)
Derive the decision-to-text and text-to-decision similarity scores, p(r) and p(l). in Eq. Update the text encoder zi using LoRA by minimizing the symmetric cross-entropy loss C(w) as Eq.
end
B Evaluation Environments and Dataset Construction
B.1 The Details of Environments
We evaluate T2DA and the baselines on three classical benchmarks that are widely adopted to assess the generalization capacity of RI. algorithms, containing the following four environments:
Point-Robot is a 2D navigation environment where an agent starts from a fixed origin and navigates to target location g. The agent receives its coordinate position the observation
17

---

Algorithm 3: Model Training of Text-to-Decision Diffuser
Input:
Offline datasets Dirinn = (Dk,4))
Pre-trained text encoder A Batch size N
Noise model cØ:
Freeze the text encoder & pre-trained in Algorithm
while not converged do
Sample batch of N (trajectory zo(r"), language supervision (x) pairs from different tasks
in D vain
a Uniform( (1....,C)) C})
1(0,1)
xo(T2))
I/ Constrain the first state of the plan
Take gradient descent step V Ek=1 ||e ca (x=((*),, (((T)) (()) ||
end
Algorithm 4: Model Training of -eeission Transformer
Input:
Offline datasets Dirain
Σx (Pk:
Pre-trained text encoder &
Causal transformer Fo:
Batch size N
Freeze the text encoder và pre-trained in Algorithm
2 while not converged do
Sample batch of N (trajectory Tk, language supervision 1x) pairs from different tasks in Drrain
Get batch of inputs B (T ME= where
+
TH (1x), RK H+1 St S H--HH1 Hi-H+1:".
R st at
ardd
(T ) VT E B
Minimize loss (()) N/2 Enter B (a apnod)²
end
Algorithm 5: Zero-Shot Evaluation of
Diffuser
Input:
Language instruction lawwi
Pre-trained text encoder &b Diffusion steps C;
Trained noise model Eg;
Guidance scale
Obtain the text embedding from language instruction as (i(weww)
while not done do
Observe state 8;; Initialize (T(TT) ~ (T,
for c C...,, do
(T,,)) + St.
Constrain the first state of the plan
& EB (II(T)) + (T)(T), R(T), (bloow)) .)) co (oe(T),c) Classifier-free guidance
11
(Ma-1, (T- ~ N
E.-1) + (enoise(x,,)) x. (T)
end
10
Execute first action of plan as + xo(To.)
++t+1
end
and outputs actions in the range of [-0.1, 0.13, representing displacement in the X- and Y-axis directions. The reward is the negative Euclidean distance to the goal: r, = - - g/|2, where 8t denotes the current position. The maximal episode step is location that is randomly sampled within a unit square of [-0.5, 0.5] Each task is captioned by text description as "Please navigate the goal position of g".
20. Tasks differ in the target
Cheetah- Vel multi-task MuJoCo environment where planar cheetah robot aims to run target velocity v along the X X-axis. The reward function combines quadratic control cost and
18

---

Algorithm 6: Zero-Shot Evaluation of Sext-to-Decision Transformer
Input:
Language instruction newi
Pre-trained text encoder vo
Trained casual transformer Fo:
Target return G
Obtain the text embedding from language instruction as ()(www)
2 Initialize desired return R G
for each timestep do Observe state 8t
Obtain the H Htsppp history trajectory as T (R- St-Ht,, a1 H+ Re
Augment the trajectory with the text embedding as r = (so()oww), Get action 0r Fg (r+)
7
Step env. and get feedback a, R --r
Append (R, s, a) to recent history 7
10 end
velocity matching term as Tr = -0.05| 0 - 10y- where a and iz represent the action and current velocity, respectively. The maximal episode step is set 200. Tasks differ in the target velocity Ug that is randomly sampled from a uniform distribution as vg ~ Uniform(0.. (0. 3.0]. Each task captioned by a text description as "Please run at the target velocity of vig".
Ant-Dir is also multi-task MuloCo environment where 3D ant robot navigates toward target direction 0. The primary reward component measures directional alignmentas Tforward Us cos 0 + Vy sin 0, where vy) represents the horizontal and vertical velocities. The final reward incorporates additional terms for the control cost, contact penalty, and survival bonus. The maximal episode step is set 200. Tasks differ in the goal direction 0 that is uniformly sampled from the full space as A ~ Uniform (0, 2T... Each task is captioned by text description as "Please walk toward the target direction 6"..
Meta- Meta-World comprises 50 robotic manipulation tasks where robotic arm interacts with various objects on a tabletop. Each task's reward function (ranging from 0 to 10) combines multiple com- ponents for fundamental behaviors (such reaching, grasping, and placing), with 10 indicating successful completion. The maximal episode step is set to 200. Table 4 illustrates the detailed description of the training and test tasks. More details can be found in 65
For each domain of Point-Robot, Cheetah- Vel, and Ant-Dir, we sample 50 tasks in total and split we use 18 training tasks and 4 test tasks
them into 45 training tasks and 5 test tasks. For as detailed as shown in Table
B.2 The Details of Datasets
We employ the soft actor-critic (SAC) algorithm 62 to independently train a policy for each task and save policy checkpoints different training steps. The hyperparameters used for SAC training across all environments are listed in Table We consider three types of datasets as
Medium: We load the checkpoint of medium policy that achieves approximately half the performance of expert policies, and use the medium policy to generate number of trajectories to construct the Medium dataset.
Expert: We load the checkpoint of expert policy that achieves the highest return after the convergence of training, and use the expert policy to collect several trajectories to construct the Expert dataset.
Mixed:
We load all saved checkpoints of policies at different training steps, and use these policies to generate a diverse set trajectories to construct the Mixed dataset.
Specifically, collect 200 trajectories for each kind of dataset in all evaluated environments.
19

---

Table 4: Details of the training and test tasks of Reta-World.
Training Task
Description
faucet-close
rotate faucet clockwise
door-lock
lock door by rotating clockwise
door-unlock
unlock door by rotating counter-clockwise
window-close
push and close window
window-open
push and open window push button on coffee machine open drawer open door with revolving joint press button press button from top bypass wall and press button from top bypass wall and press button press handle down pull handle up slide plate back
coffee-button
drawer-open
door-open
button-press
button-press-topdown
button-ppess-topdlonn-wlll
button-press-wall
handle-press
handle-pull
plate-slide-back
plate-slide-side
slide plate side
plate-slide
slide plate
slide plate back side
Test Task
Description
faucet-open
rotate faucet counter-clockwise push and close drawer bypass wall and reach goal press handle down sideways Pull handle up sideways Insert the gripper into a hole
drawer-close
reach-war wall
handle- handle-press-sidd sidss-side
handle-pullsside
hand-insert
Table 5: Hyperparameters of SAC used to collect multi-task offline datasets.
Environments
Training steps
Warmup steps
Save frequency
Learning rate
Soft update
Discount factor
Entropy ratio
Point-Robot
2000
100
40
3e-4
0.005
0.99
0.2
Cheetah- Vel
500000
2000
10000
3e-4
0.005
0.99
0.2
Ant-Dir
500000
2000
10000
3e-4
0.005
0.99
0.2
Meta-World
1000000
5000
100
le-3
0.005
0.99
0.2
C Baseline Methods
This section gives details the competitive baselines, including context-based offline meta-RL (OMRL), in-context RL, and iinguag--conitioonl policy learning approaches. These baselines are thoughtfully selected to cover the main studies that tackle the generalization problem of RL algorithms. The baselines are introduced as follows:
CSRO Context Shift Reduction for OMRL. addresses the context shift problem in offline meta-RL through two key components: 1) During meta-training, it employs a max-min mutual information representation learning mechanism to minimize mutual information between the task representation and the behavior policy while maximizing mutual information between the task representation and the task information; 2) During meta-test, introduces a non-priov context collection strategy to first randomly explore the environment and then gradually update the task representation.
UNICORN 37 Unified Information Theoretic Framework of Context-Bassee Offline Meta- Reinforcement Learning. provides general framework to unify several context-hased offline meta-RL algorithms by proving that they optimize different bounds of mutual information between
20

---

the task variable M and latent representation Z. Based on this theoretical insight. it proposes supervised and sulf-supervised implementation of the mutual information derivation (Z; M).
AD 42
Algorithm Distillation. It distills RL algorithms into neural networks by modeling their training histories with a causal sequence model. Using a dataset of learning histories generated by a source RL algorithm, AD trains a causal transformer autoregressively predict actions given their preceding learning histories as the context, enabling in-context policy improvement without updating network parameters.
BC-Z [25], Behavior Cloning Z. It an imitation learning system designed for zero-shot generalization to novel vision-based manipulation tasks. It flexibly conditions the policy on pre-trained embeddings of natural language or video of perroonnn tasks. To align BC- Z to the benchmarks investigated in this paper, we replace the video data with RL datasets D = (s,. ag raa, s/) and modify its video encoder to trajectory encoder 6 identical to that used in T2DA. This adjustment for a fair comparison results in the optimization objec- tive: min Einkk [(s.a)>D -- log π (als, (bxx)) Deos( (v))), ((Tx))), where Dens denotes the cosine distance.
BAKU (27). an efficient transformer architecture for multi-task policy learning that improves upon prior work in offline imitation learning. It carefully integrates multiple key components: observation trunks for multi- multi-modal fusion, action chunking for smoother control, multi- sensory observation processing, and modular action heads for flexible prediction. The model is optimized through behavior cloning with the objective: min Ensk & (o.,)>> la ralo, I where represents multi-modal observations and Ti denotes the task instruction. For a fair comparison, we adjust BAKU to the
zero-shot setting.
Prompt-DT 12 Prompt-based Decision Transformer. It is DT-based offline meta-RL method that leverages the transformer architecture and prompting mechanisms to enable few-shot adapta- tion. It introduces trajectory prompts, which consist of short demonstration segments from the target task, to encode task specific context and guide policy generation. Without requiring any fine-tuning unseen tasks, Prompt-DT achieves strong few-shot performance through expert demonstrations in contructed prompts.
MetaDiffuser [66], a diffusion- based offline eta-RL method that considers the generalization problem as conditional trajectory generation task with contextual representation. It trains a context conditioned diffusion model to generate task-oriented trajectories for planning. To further enhance the dynamics consistency of the generated trajectories while encouraging trajectories to achieve high returns, introduces a dual-guided module the sampling process of the diffusion model.
Meta-DT [4, Meta Decision Transformer. It harnesses the sequential modeling capability of the transformer architecture and enables robust task representation learning through a disentangled world model, thereby achieving efficient generalization in offline meta-RI.. learns a compact task representation via context-aware world model, used as contextual condition to guide task- oriented sequence generation. Complementary to the task representation, selects self-guided trajectory segment from online interactions to further exploit the architectural inductive bias. Thereby, Meta-DT exhibits strong few- and zero-shot performance without expert demonstrations.
For a fair comparison, all baselines are adjusted to an aligned zero-shot setting, where no warm-start data is available before policy evaluation. All these baselines can only use samples generated by the trained meta-policy during evaluation to infer task representations.
D Implementation Details of T2DA
D.1 Network Architecture and Hy Hyperparameters
World Model. We implement the generalized world model using simple architecture: a trajectory encoder and two decoders for the reward and state transition prediction. Specifically, the trajectory encoder consists of a transformer encoder with 6 layers, where each layer contains a multi-head self-attention module (8 heads) and a feed-forward network with ReLU activation. States, actions, and rewards the raw trajectory are first tokenized into 256-dimensional vectors through separate linear layers. These embeddings are concatenated and processed by the transformer encoder to produce 256-dimensional decision embedding z via mean pooling and linear projection. Two MLPs serve as decoders: reward decoder that predicts the instant reward given the tuple
21

---

(s, s'; 2),, and a state transition decoder that predicts the next state s' given the tuple (s, 7; 2) Both MLPs utilize two hidden layers of 256 dimensions each.
Text Encoder. We initialize text encoders from pre-trained CLIP [6). BERT 644
and T5 [56] in this paper. Specifically, we use openai//lip--vt-aaaiee pattt--bas--ppchh... borgle-bert/bert-beee -nceeeducce, and google-t5/t5-small, respectivlly
Diffusion Architecture of T2DA. We implement the diffusion architecture of T2DA-D based on the decision diffuser [58] framework Following DiT [67]. we replace U-Net backbone with transformer represent the noise model eø. Each block employs multi-head self-attention module followed by a feed-forward network with GELU activation
[68], using adaptive layer normalization (adaLN) for conditioning. The model processes state-action trajectories through linear embeddings, which are combined with positional embeddings derived from RL timesteps. The diffusion time embeddings are encoded via sinusoidal embeddings processed through an MLP. For conditioning, text embeddings and returns-t--g are concatenated with diffusion time embeddings. The detailed configurations and hyperparameters of T2DA-D are presented in T
Transformer Architecture of T2DA. We implement the transformer architecture of T2DA-T based on the decision transformer [59] framework Specifically, we employ modality-specifff embeddings for states, actions, returns-to to-go, and timesteps. The timestep embeddings serve as positional encodings and are added to each token embedding. These tokens are concatenated with text embedding of task description and fed into GPT architecture which predicts actions autore- gressively using causal self-attention mask. The detailed configurations and hyperparameters of T2DA-T are presented Table
Table 6: Configurations and hyperparameters in the training process of T2DA-T.
Configuration
Point-Robes
Cheetah- Vel
Ant-Dir
Meta- World
layers num
3
3
3
attention head num
1
1
1
embedding dim
128
128
128
128
horizon
20
200
40
50
training steps
100000
70000
100000
100000
learning rate
lc-4
2e-4
Ic-4
lc-4
Table 7: Configurations and hyperparameters the training process of T2DA-D.
Configuration
Point-Robot
Cheetah- Vel
Ant-Dir
Meta- World
DiT layers num
8
4
DiT attention head num
8
8
8
embedding dim
128
128
256
128
horizon
10
50
40
50
diffusion steps
20
20
20
20
training steps
100000
70000
100000
100000
learning rate
5e-5
le-4
le-4
le-4
D.2 Computation
We train our models on one Nvidia RTX4080 GPU with the Intel Core i9- 10900X CPU and 256G RAM. Dynamic- decision embedding and constrastive language-deciste pre-traine pre-training cost about 0.5 hours. The generalist policy training cost about 0.5-3 hours depending on the complexity of the environment.
22

---

E
Analysis of Parameter-Efficient Fine-tuning
we employ LoRA, a classical parameter-efficient fine-tuning method, to fine- tune the text encoder at lightweight cost. As shown in Table the number of trainable parameters substantially smaller than full-parameter fine-tuning (less than 1%), enabling much more efficient computation and memory usage. To investigate whether limited number of trainable parameters constrains the capacity for knowledge alignment, we evaluate T2DA's performance by comparing LoRA fine-tuning with full-parameter fine-tuning during contrastive anguage-decision pre-trat mre-training. Figure "ecureBBllusiee Billustrates the results on representative environments with the text encoder initialized from CLIP. Obviously, the two fine-tuning approaches yield nearly identical performance, highlighting T2DA's superior parameter efficiency of T2DA without sacrificing effectiveness.
As stated Sec. 3.3
Table 8: The number and memory usage of parameters for LoRA fine-tuning and full-parameter fine-tuning.
Method
Trainable parameters (Memory)
Total parameters (Memory)
Percentage
LoRA Tuning Full Tuning
$89,824 (2.25MB)
63,755,776 (243.21MB)
0.93%
63,165,952 (240.96MB)
63,165,952 (240.96MB)
100%
LoRA
Full Tuning
Cheetah-Vel
Ant-Dir
1000
-100
800
1150
Returns
600
220
400
-20
-300
Timesteps (1e4)
(1e4)
Figure 8: Test return curves of T2DA using LoRA-tuned and fully-tuned CLIP text encoders. With less than 1% parameter count, T2DA using LoRA fine-tuning yields nearly identical performance compared to full-perameter fine-tuning, highlighting T2DA's superior parameter efficiency without sacrificing effectiveness.
F Analysis of Training Pipline
T2DA separately trains three key components: (a) decision embedding, (b) knowledge ali alignment, and (c) generalist policy. To demonstrate the effectiveness of this modular ap- proach, conducted ablation studies comparing method against alternatives: jointly training (a)+(b), and jointly training all three components (a)+(b)+(C) simultaneously. As shown in Table our separate training pipeline achieves substantial performance gains of 20%-56% across different en- vironments. This modular architecture not only provides a more stable training process but also offers superior scalability, as individual components can be easily substituted or upgraded independently without affecting the entire system.
Table 9: Ablation study results on modular architecture. Best results in bold.
Environment
T2DA
(a)+(b
(a)+(b)(CC)
Point-Robot
-7.2
-8.9 1008
-16.3
Woold
1274.5
817.3
23

---

G Meta-RL Baselines with Goal Information
For comprehensive evaluation, we enhanced meta-RL baselines by incorporating goal information into their observation space. This approach provides standardized framework where all compared methods have access to equivalent task information.
The results in I T2DA's performance advantages over the goal-enhanced baselines, further validating the effectiveness of leveraging natural language supervision. This -all-interrated setting was maintained consistently for all meta-RI. baseline comparisons throughout the paper.
Table 10: Performance comparison with goal-enhanced meta-RL baselines. Best results in bold.
Environment
T2DA
CSRO (w/ goal)
UNICORN (w/ goal)
Cheetah- Vel Ant-Dir
-60.4 4 + 1.8 970.3 + 8.7
-84.0 + 4.3
-82.7 9.2
384.4 30.1
427.1 25.3
H Performance on Unstructured Language Instructions
This section presents an evaluation of the method's generalization capabilities when confronted with less structured, ambiguous, or noisy natural language instructions. Experiments were conducted where tasks were captioned using two additional types of textual descriptions.
Task descriptions reconstructed in two distinet linguistic styles:
Noisy style: "Um, there's this location I think... somewhere around (roundruumber=task(() ndigits=2)J-ish in the x direction? And maybe (rondrssubbe=sask// ndigits=2)) so up?"
Conversational style: "Could you move over to the spot that's roughly (round(numbe==ass// ndigits=2)] units the right and (round(
ndigits=2)) units up? Thanks!"
Results demonstrate T2DA's consistent performance across these different textual description formats:
Table 11: Performance comparison across different instruction styles.
Environment
T2DA
T2DA (Noisy)
T2DA (conversational)
Point-Robot
-7.2 + 0.1
77.6 0.1
8.0 + 0.1
These findings highlight the robustness of leveraging natural language supervision, as the language representations effectively capture rask-relevant information and successfully extrapolate meta-level knowledge across tasks presented with varying textual descriptions.
I Experimental Results on Humanoid
This section presents experimental results on stimulated humanoid robot benchmark 69 Table 12 shows the converged performance of T2DA and baselines using Mixed datasets under aligned zero-shot setting. The results demonstrate that T2DA consistently outperforms baselines this more challenging environment.
Table 12: Performance comparison of different methods on Humanoid. Best results in bold and second best underlined.
Method
T2DA-T
T2DA-D
BC-Z
DD
CSRO
UNICORN
AD
Returns
610.4
575.4
531.7
520.3
381.3
520.3
505.6
427.4

---

Comparison to More OMRL Baselines
In this section, we compare T2DA to three additional OMRL baselines: Meta-DT 14). MetaDif- fuser [66], and Prompt-DT [2. More details about these baselines can be found in Appendix C The results in Table demonstrate T2DA's consistent superiority over these OMRL baselines.
Table 13: Performance comparison with OMRL baselines. Best results in bold.
Environment
T2DA-T
T2DA-D
Meta-DT
MetaDiffuser
Prompt-DT
Point-Robot Cheetah- Vel
-7.2 1 0.1
0.2
-8.4 0.4
-11.7 1 0.6
144.3 + 1.7
-70.3 3.8
1.8
-78.2
-86.5 = 4.8
194.688.1
Ant-Dir
970.3
570.6 13.1
922.1 = 11.9
858.5 + 42.5
895.2 17.3
K
Limitations
We tackle the offline meta-RL. challenge via leveraging the supervision from natural language. Our scalable implementations demonstrate that the generalist RL agent can understand and act upon natural language instructions zero-shot manner. While our approach uses relatively lightweight datasets compared to large models, it demonstrates promising results. An essential step of future work implement on vast datasets with diversified domains and deploy the text-to-decision paradigm on real robots toward embodied intelligence.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer the Abstract and Section |
Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Section 5 Conclusions, Limitations, and Future Work, and Appendix K
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
Our work does not include theory assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please refer to Appendix A for algorithm pseudocodes. Appendix B for the details of environments and dataset construction, and Appendix for the implementation details and hyperparameters of our method. Our source is included in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
27

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Please refer to Appendix A Appendix B and Appendix for details of our method and dataset construction. We also provide our source code in the supplementary materials
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
sooolioiccii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Section Experiments, Appendix B Appendix D
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please refer Section 4
Experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Appendix D and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aaaaaaaa.cppaaaaaa/aattta
Answer: [Yes]
Justification: Please refer to Appendix A Appendix B and Appendix for details of
implementation and dataset construction. We also provide our source code in the supplementary materials.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Section Conclusions, Limitations, and Future Work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: We construct the datasets used in this work ourselves, and not use existing assets, please refer to Appendix B for details.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does involve crowdsoureing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in our work does not involve L.MSs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
httpp:///oorrps.cc////fffrrrrrr/
32


Reviews:
[{"review_id": "98FxKxb3fb", "reviewer": "Reviewer_qsSY", "summary": "This paper introduces Text-to-Decision Agent (T2DA), a novel framework for offline meta-reinforcement learning (meta-RL) that leverages natural language supervision to achieve zero-shot generalization across decision-making tasks. Traditional offline meta-RL approaches often rely on expensive task-specific signals (e.g., warmup explorations or expert data), whereas T2DA proposes to replace such supervision with textual task descriptions. Experiments on MuJoCo, Meta-World, and Point-Robot tasks demonstrate that T2DA significantly outperforms baselines from offline meta-RL, in-context RL, and language-conditioned imitation learning, particularly in challenging zero-shot settings.", "questions": "1. The test tasks are variants within the same environment families (e.g., new angles, velocities, or target locations). Have you explored cross-environment generalization, such as transferring from Point-Robot to Ant-Dir or from MuJoCo to Meta-World?\n\n2. While you mention LoRA as a lightweight fine-tuning method for the text encoder, what is the performance or efficiency tradeoff compared to full fine-tuning?\n\n3. In your experiments, the task descriptions are relatively structured and templated. How well does the method generalize to less structured, ambiguous, or noisy natural language instructions? Can you comment on robustness to varied linguistic phrasing or human-written descriptions?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper presents a well-engineered system that integrates offline meta-RL, world model pretraining, contrastive representation learning, and language-conditioned policy generation in a coherent and technically sound way.\n\n2. The paper is clearly written, with excellent visual aids (e.g., t-SNE plots, training diagrams) that intuitively convey the ideas behind decision-text alignment and dynamics-aware embedding.\n\n3. The proposed T2DA framework addresses a long-standing challenge in RL: enabling agents to generalize across tasks without task-specific adaptation or exploration.\n\n4. The use of a dynamics-aware embedding space as an intermediate for text grounding is a unique conceptual advance.\n\nWeakness: \n\n1. While the framework is modular, it is unclear how robust it is to noisy or ambiguous natural language instructions, or how it performs in real-world environments beyond the benchmarks.\n\n2. Some notations (e.g., $x_0(\\tau)$ in diffusion model training) are dense and may require more intuitive explanation for non-specialists in generative models.\n\n3. Some components build upon prior work (e.g., Prompt-DT, Decision Diffuser), so the originality mainly lies in their integration and adaptation rather than invention from scratch.", "was_revised": true, "final_justification": "I appreciate the authors' thoughtful responses and the revisions made in the manuscript. Overall, the revisions have significantly improved the clarity and strength of the paper. I maintain my positive assessment and recommend acceptance.", "rebuttal": "***\n\n**`Q1. How well does the method generalize to less structured, ambiguous, or noisy natural language instructions?`**\n\nA6. Thank you for your insighful question. Following your suggestion, we have conducted new experiments where tasks are captioned by two additional kinds of textual descriptions. We reconstructed task descriptions in two linguistic styles. **The first is a noisy style**: *\"Um, there's this location I think... somewhere around {round(number=task[0], ndigits=2)}-ish in the x direction? And maybe {round(number=task[1], ndigits=2)} or so up?\"*. **The second is a conversational style**: *\"Could you move over to the spot that's roughly {round(number=task[0], ndigits=2)} units to the right and {round(number=task[1], ndigits=2)} units up? Thanks!”.*\n\nThe following result shows **T2DA’s consistent performance across different textual descriptions**. It also demonstrates the robustness of leveraging natural language supervision, as the powerful language representations can still capture task-relevant information and extrapolate the meta-level knowledge across tasks from different textual descriptions.\n\n| **Method** | **T2DA**  | **T2DA (Noisy)** | **T2DA (conversational)** |\n| --- | --- | --- | --- |\n| Point-Robot | -7.2 $\\pm$ 0.1 | -7.6 $\\pm$ 0.1 | -8.0 $\\pm$ 0.1 |\n\n***\n\n**`Q2. Some components build upon prior work (e.g., prompt-DT and decision diffuser), rather than invention from scratch.`**  \n\nA3. Foremost, **we address the key challenge in offline meta-RL** (OMRL) that usually requires either high-quality samples or warmup explorations to infer task beliefs for generalization. Instead of relying on these expensive supervision signals, we propose a novel solution using natural language supervision. Then, **we indeed address three fundamental challenges in learning generalist decision agents from natural language supervision**: \n\n- LLMs failing to capture any environment dynamics (we learn dynamics-aware decision embeddings);\n- Knowledge misalignment due to the semantic gap between text and decision (we align text embeddings to comprehend environment dynamics);\n- The necessity of scalable implementations (we develop scalable implementations of text-decision diffuser and text-to-decision transformer).\n\n**Our method is the first to align decision and language embeddings through world model dynamics**, which enables natural language understanding for decision task representation. Our primary contribution is a language-conditioned scalable framework for offline meta-RL. **The downstream policies in our framework are modular and can be implemented using various methods, even simple MLPs**. To promote scalability, we deploy T2DA on two mainstream generation architectures that hold the promise to train RL models at scale: autoregressive transformers and non-autoregressive diffusion models.\n\n***\n\n**`Q3. What is the performance or efficiency tradeoff of LoRA fine-tuning compared to full fine-tuning?`** \n\nA4. Due to space constraints in the main paper, we've included a detailed analysis of the performance and efficiency tradeoffs between LoRA fine-tuning and full-parameter fine-tuning in Appendix F: \"Analysis of Parameter-Efficient Fine-tuning.” The number of trainable parameters of LoRA is substantially smaller than full-parameter fine-tuning (less than 1%), enabling much more efficient computation and memory usage. Results show that with less than 1% parameter count, **T2DA using LoRA fine-tuning yields nearly identical performance compared to full-parameter fine-tuning, highlighting T2DA’s superior parameter efficiency without sacrificing effectiveness**.\n\n***\n\n**`Q4. How does the framework perform in real-world environments?`** \n\nA4. While our current work focuses on tackling zero-shot generalization in offline meta-RL through text embedding alignment with environment dynamics, we have not yet conducted real-world robotic experiments. **The offline meta-RL community [1-3] primarily evaluates methods on simulation benchmarks such as MuJoCo and Meta-World** rather than physical robots. However, our framework shows promising potential for real-world applications - **the text understanding capabilities could help robots interpret natural language commands and generalize to new tasks**. We are willing to extend our method to real robotic systems in future work.\n\n[1] Offline Meta-RL with Advantage Weighting, ICML 2021.\n\n[2] Context Shift Reduction for Offline Meta-Reinforcement Learning, NeurIPS 2023.\n\n[3] Towards an information-theoretic framework of context-based offline meta-reinforcement learning, NeurIPS 2024.\n\n***\n\n**`Q5. Have you explored cross-environment generalization, such as transferring from Point-Robot to Ant-Dir or from MuJoCo to Meta-World?`**\n\nA5. **Cross-environment generalization remains challenging in RL**, unlike natural language processing where semantically similar concepts share similar vector representations. The fundamental challenge lies in the state-action spaces: each environment has its unique structure and meaning, making direct transfer between different environments (like Point-Robot to Ant-Dir) impractical. While general RL aims to address cross-environment transfer [4], **our work focuses specifically on in-domain generalization, which is the standard of the offline meta-RL community** [1-3]. \n\nThe cross-environment generalization is a challenging yet significant problem to be investigated in RL domains, especially in the era of large foundation models. As mentioned in our limitations, an essential step is to implement on vast datasets with diversified domains, unleashing the scaling law with the diffusion or transformer architectures. We leave it as future work.\n\n[4] Towards General-Purpose Model-Free Reinforcement Learning, ICLR 2025.\n\n***\n\n**`Q6. More intuitive explanation on diffusion model training for non-specialists in generative models.`** \n\nA6. We kept our explanation of the diffusion model concise due to paper space constraints. The diffusion model operates in two phases: In the forward process, noise is progressively added to a robot trajectory (states and actions) until it becomes random, analogous to blurring an image. In the reverse process, the model learns to reconstruct meaningful trajectories from noise by conditioning on task descriptions $\\psi(l)$ and target returns $\\hat{R}(\\tau)$, similar to image denoising. During inference, the model generates appropriate action sequences by iteratively denoising random trajectories guided by natural language instructions. This enables zero-shot task execution through language conditioning without requiring additional training. **We will include more intuitive explanations in the main paper and more detailed descriptions in the appendix.**\n\n***\n\n### **`Summary Response`**\n\nThank you for your valuable review comments, which have provided critical insights that helped strengthen our work. We are honored to have **your positive comments on our motivation and contribution** (”addresses a long-standing challenge…a unique conceptual advance…presents a well-engineered system”), **empirical evaluation** (”excellent visual aids, e.g., t-SNE plots, training diagrams”), **and the writing** (”the paper is clearly written”). Also, we are grateful that **Reviewer sYVw make positive comments on our novelty** (”providing a novel framework”) **and the empirical evaluation** (”comprehensive experimental validation”). We hope that our contributions can be adequately justified. \n\nWe summarize that your main concerns may include: 1) robustness to ambiguous natural language instructions, 2) the choice of downstream components, and 3) discussions on real-world applications and cross-environment generalization capabilities. **We have extended a number of justifications and experimental analyses to address your concerns**. Please let us know if we have addressed your concerns. We are more than delighted to have further discussions and improve our manuscript. **If our response has addressed your concerns, we would be grateful if you could re-evaluate our work**."}, {"review_id": "S54DfkOBVB", "reviewer": "Reviewer_sYVw", "summary": "This paper proposes the Text-to-Decision Agent (T2DA) framework, which enhances the generalization capability of offline meta-reinforcement learning through natural language supervision and addresses the issue of traditional methods relying on high-quality samples. Its core approach involves encoding decision data into dynamics embeddings using a world model, aligning text with decision modalities via contrastive learning, and finally achieving text-to-decision generation through diffusion models and Transformers. Experiments demonstrate that T2DA outperforms baselines significantly on MuJoCo and Meta-World benchmarks, and exhibits robustness to data quality and text encoder selection.", "questions": "1. Eq. 3 calculates the cosine distance, the cosine distance is independent of the order of multiplication. Aren't these two similarities the same? \n2. Eq. 4 uses contrastive learning, and $e^{sim(\\tau_{i},l_{i})}$ in the denominator should be $e^{sim(\\tau_{k},l_{i})}$? Otherwise, the non-diagonal elements of the similarity matrix in Figure 2b aren't used. Additionally, there is a minor error: the index of the second element on the diagonal of Figure 2b is incorrectly labeled.\n3. The input of T2DA’s policy requires a text description, but under this setting, it resembles goal-based reinforcement learning or multi-task reinforcement learning rather than meta-reinforcement learning. For example, the text description of Point Robot directly includes the goal position, whereas in meta learning, the goal position cannot be known directly. Therefore, is it fair to compare T2DA with meta-learning algorithms like CSRO and UNICORN?\n4. What is the impact on the results if the task is described in another kind of sentence?\n5. Since a world model is used, T2DA should also be applicable to tasks with different dynamic functions. Could the authors supplement this part of the experiment?\\\nIf the author answers my question, I am willing to increase my score.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\\\nThe authors introduce natural language into offline meta-reinforcement learning, providing a novel framework by aligning text embeddings with environmental dynamics embeddings, thus breaking through the traditional methods' reliance on high-quality samples. Comprehensive experimental validation has demonstrated the necessity of each component of the method and the robustness of the algorithm.\\\nWeaknesses:\\\nThe evaluation time of the method is relatively long. The description of the embeddings alignment module is unclear. The comparison with baseline algorithms is unfair. There is no description of the impact of textual changes.", "was_revised": true, "final_justification": "The author's answer solved my problem", "rebuttal": "***\n\n**`Q1. Difference from goal-based RL or multi-task RL.`**\n\nA1. Goal-based RL augments the observation with an additional goal to tackle multiple goals simultaneously, and the reward function is usually defined as a binary bonus of reaching the goal. In contrast, **T2DA belongs to a language-conditioned policy learning paradigm, where a meta-policy learns to follow language instructions**. In simple environments like Point-Robot, the language instruction can contain the goal, so it looks a bit like goal-based RL. However, in general scenarios like robotic manipulation, language instructions are usually highly abstract descriptions like “*pull a handle up sideways*” or “*bypass wall and press button from top*”. Apparently, it is hard to solve with goal-based RL. \n\nT2DA falls into the meta-RL category as we care about the zero-shot generalization to unseen test tasks. In contrast, multi-task learning focuses on maximizing performance on a set of training tasks.\n\n***\n\n**`Q2. Fairness of comparing to meta-RL baselines.`**\n\nA2. As T2DA learns a language-conditioned meta-policy, we cannot ablate the language instructions. The key motivation of leveraging natural language supervision is to realize zero-shot generalization without inferring task beliefs from high-quality samples or warmup explorations (as meta-RL baselines do). **From both perspectives, T2DA uses language instructions to eliminate the need for few-shot samples, while meta-RL baselines need few-shot samples for task inference without language instructions**. Experiments in our manuscript have verified the significant superiority of T2DA’s paradigm over that of meta-RL baselines, which comes from harnessing the representation power and knowledge transferability embodied in pre-trained language models.\n\nFurther, we modify the meta-RL baselines by adding the goal information to the observation, resembling a goal-conditioned setting. This ensures that the baselines can also access the goal information for a fairer comparison. **The result in the following table demonstrates T2DA’s consistent superiority over the goal-augmented baselines**. It further validates our motivation for leveraging natural language supervision. We will use this goal-augmented setting for meta-RL baselines in the revised paper.\n\n| **Method** | **CSRO (w/ goal)** | **UNICORN (w/ goal)** | **T2DA** |\n| --- | --- | --- | --- |\n| Cheetah-Vel | -84 $\\pm$ 4.3 | -82.7 $\\pm$ 9.2 | **-60.4 $\\pm$ 1.8** |\n| Ant-Dir | 384.4 $\\pm$ 30.1 | 427.1 $\\pm$ 25.3 | **970.3 $\\pm$ 8.7** |\n\n***\n\n**`Q3. The impact on the results when the task is described by different textual descriptions?`**\n\nA3. Following your suggestion, we have conducted new experiments where tasks are captioned by two additional kinds of textual descriptions. We reconstructed task descriptions in two linguistic styles. **The first is a noisy style**: *\"Um, there's this location I think... somewhere around {round(number=task[0], ndigits=2)}-ish in the x direction? And maybe {round(number=task[1], ndigits=2)} or so up?\"*. **The second is a conversational style**: *\"Could you move over to the spot that's roughly {round(number=task[0], ndigits=2)} units to the right and {round(number=task[1], ndigits=2)} units up? Thanks!”.*\n\nThe result in the following table shows **T2DA’s consistent performance across different textual descriptions**. It also demonstrates the robustness of leveraging natural language supervision, as the powerful language representations can still capture task-relevant information and extrapolate the meta-level knowledge across tasks from different textual descriptions.\n\n| **Method** | **T2DA**  | **T2DA (Noisy)** | **T2DA (conversational)** |\n| --- | --- | --- | --- |\n| Point-Robot | $-7.2 \\pm 0.1$ | $-7.6 \\pm 0.1$ | $-8.0\\pm 0.1$ |\n\n***\n\n**`Q4. Experiments on tasks with different dynamic functions.`**\n\nA4. We agree with you that it is necessary to evaluate T2DA on both variations of the reward function and dynamics function, since these two functions constitute the whole world model. **Our experiments on Meta-World have already included tasks on different dynamic functions** $p(s’|s,a)$. While utilizing the same robotic arm, the interactions with different objects vary across tasks. These objects have different shapes, joints, and connectivity. For instance, the door has a revolute joint, while the drawer has a sliding joint, resulting in distinct dynamic functions.\n\n***\n\n**`Q5. The evaluation time of the method is relatively long.`**\n\nA5. We train T2DA and baselines on one Nvidia RTX4080 GPU with the Intel Core i9-10900X CPU and 256G RAM. The following table shows the running time of T2DA and three types of baselines on the most complex environment Meta-World. The result demonstrates that **the computation time cost of T2DA is comparable to various baselines**.\n\n| **Method** | **Meta-RL baselines** | **AD baseline** | **Language-conditioned baselines** | **T2DA** |\n| --- | --- | --- | --- | --- |\n| Meta-World | 3h (CSRO) - 3.5h (UNICORN) | 22h | 1.5h (BAKU) - 3.5h (BC-Z) | 1.5h |\n\n***\n\n**`Q6. The description of the embeddings alignment module is unclear. Two similarities in Eq. (3), and the use of non-diagonal elements in the similarity matrix of Eq. (4).`**  \n\nA6. **The two similarities are the transpose of each other**, i.e., $sim(\\tau,l)=sim(l,\\tau)^T$. The reason we use two similarities is to highlight the difference between the text-to-decision score along the decision axis and the decision-to-text score along the text axis. Following CLIP, we use **a symmetric loss function** that contains cross-entropy losses along both the text and the decision axes for **contrastive alignment in both modalities**. \n\n**Non-diagonal elements.** Thank you for pointing out this typo. The correct decision-to-text score is calculated as $p(\\tau_k)=\\frac{e^{sim(\\tau_k,l_k)}}{\\sum_{i=1}^Ne^{sim(\\tau_k,l_i)}}$ and the text-to-decision score is $p(l_k)=\\frac{e^{sim(l_k,\\tau_k)}}{\\sum_{i=1}^Ne^{sim(l_k,\\tau_i)}}$. \n\nThank you for your professional comments on the contrastive alignment module.  We will improve the notations in this section to avoid potential misunderstandings, e.g., using a unified one similarity matrix. \n\n***\n\n### **`Summary Response`**\n\nThank you for your valuable review comments, which help us gain more critical insights and further enhance our empirical evaluation. **We are honored to have your positive comments on our novelty** (”providing a novel framework”) **and the empirical evaluation** (”comprehensive experimental validation”). Also, we are grateful that **Reviewer qsSY is affirmative to our technical challenge** (”addresses a long-standing challenge in RL”), **the novelty** (”a well-engineered system…a unique conceptual advance”), **and the empirical evaluation** (”with excellent visual aids, e.g., t-SNE plots, training diagrams”). We hope that our contributions have been adequately justified. \n\nWe summarize that your main concerns may include: 1) the difference from goal-based RL and fairness of comparing to meta-RL baselines, 2) T2DA’s performance with different textual task descriptions, and 3) experiments on tasks with different dynamics functions. **We have extended a number of justifications and experimental analyses to address your concerns**. Please let us know if we have addressed your concerns. We are more than delighted to have further discussions and improve our manuscript. **If our response has addressed your concerns, we would be grateful if you could re-evaluate our work**."}, {"review_id": "pxzMAtGeiF", "reviewer": "Reviewer_EUfm", "summary": "The paper proposes the Text-to-Deicision Agent (T2DA) framework, designed to tackle the offline mtea-RL problem by supervising the policy using natural language. While prior works typically require expert trajectories or environment interactions to infer the task embeddings, the proposed method instead leverages language inputs to directly guide policy inference.", "questions": "1. Can authors report the “success rate” for Meta-World environment?\n\n2. Why T2DA outperforms other context-based offline meta-RL approaches (CSRO, UNICORN)? Given that T2DA simply aligns language embeddings with decision embeddings, it is not evident what fundamentally enables its superior performance.\n\n3. Similarly, it is unclear why T2DA maintains superior performance despite the use of lower-quality data, as shown in Figure 5.", "limitations": "Yes. Limitations are discussed in Section 5.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "1. **Lack of technical challenge**: The main weakness of the paper lies in its problem formulation, which lacks technical depth or significant challenge. The core task is reduced to aligning text embeddings with decision embeddings, which, in its current form, appears to be a relatively straightforward application of existing techniques without addressing more fundamental or difficult aspects of the problem domain.\n\n2. **Lack of novelty**: The individual components of T2DA lack originality. (a) “dynamics-aware decision embedding” resembles standard techniques for task embedding widely adopted in meta-RL literature [1,2,3]. (b) “contrastive language-decision pre-training” simply follows CLIP method without significant methodological innovation. \n\n    [1] VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning (ICLR20)\n\n    [2] Meta-Reinfrocement Learning Based on Self-Supervised Task Representation Learning (AAAI23)\n\n    [3] MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL (ICML23)\n\n3. **Design choice for dynamics-aware decision embedding**: Did the authors consider alternative methods for learning task embedings, such as mutual information optimization [4,5] or contrastive representation learning [6] ? It is unclear whether the proposed dynamics-aware approach is the most effective, especially given that recent studies often adopt information-theoretic or contrastive objectives.\n\n    [4] Context Shift Reduction for Offline Meta-Reinfrocement Learning (NeurIPS23)\n\n    [5] Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitation (AAAI24)\n\n    [6] Robust Task Representation for Offline Meta-Reinforcement Learning via Constrastive Learning (ICML22)\n\n4. **Baseline choices**: Some essential language-based policy learning baselines are missing [7,8]. Including such works would provide a more comprehensive evaluation, especially given that original meta-RL baselines used in the paper (CSRO, UNICorn, and AD) do not incorporate language inputs.\n\n    [7] LISA: Learning Interpretable Skill Abstractions from Language (NeurIPS22)\n\n    [8] Language Control Diffusion: Efficiency Scaling Through Space, Time and Tasks (ICLR24)", "was_revised": true, "final_justification": "Most of my concerns have been addressed through discussion. Accordingly, I have increased my score", "rebuttal": "***\n\n**`Q1. Lack of technical challenge.`**\n\nA1. We disagree with you. Foremost, **we address the key challenge in OMRL** that usually requires either high-quality samples or warmup explorations to infer task beliefs for generalization. To omit these expensive supervision signals, we propose a novel solution using natural language supervision. Then, **we address three fundamental challenges in learning generalist decision agents from natural language supervision**: \n\n- LLMs failing to capture any environment dynamics (we learn dynamics-aware decision embeddings);\n- Knowledge misalignment due to the semantic gap between text and decision (we align text embeddings to comprehend environment dynamics);\n- The necessity of scalable implementations (we develop scalable implementations of text-decision diffuser and text-to-decision transformer).\n\nThrough this language-supervised OMRL framework, **T2DA gains a 16% to 200% improvement** over various types of baseline methods. The ablation study also verifies the effectiveness of leveraging language supervision, capturing environment dynamics, and aligning the text-decision semantic gap. In summary, **the significant results validate the effectiveness of tackling the above-mentioned challenges**.\n\n***\n\n**`Q2. Lack of novelty.`**\n\nA2. We disagree with you. **T2DA is the first to align decision and language embeddings through world model dynamics**. Also, we develop **two new implementations** for realizing language-conditioned generation: text-to-decision diffuser and text-to-decision transformer. Next, we continue to defend our novelty from various aspects as \n\n- **Our dynamics-aware decision embedding extends beyond standard techniques for task embedding**. To our knowledge, the way of using an encoder-decoder world model to learn task embedding was first proposed by VariBAD [1] in meta-RL literature. Existing works [1-3] usually compute a local task embedding $z_t$ per timestep, and use $z_t$ to decode the same trajectory. In contrast, we learn a single-step task embedding $z$ for an entire trajectory, and use $z$ to decode the dynamics of other trajectories. **Our task embedding is more computationally efficient, more generalizable, and more applicable for text-decision alignment**.\n- In literature, it is very common for follow-up studies to adopt standard techniques, while we can still justify the novelty through significant modifications or tackling significant challenges for new problems. For example, MoSS [2] adopts the VariBAD style and improves its task embedding by contrastive learning. MetaDiffuser [3] also adopts the VariBAD style and tackles the diffusion-based OMRL problem. **While our method adopts the basic principle of VariBAD to learn task embedding, we make an innovative improvement upon it and use it to tackle the new challenge for text-to-decision alignment.**\n- **Our novelty of contrastive language-decision pre-training is to extend the CLIP principle to RL domains**, aligning text embeddings to comprehend environment dynamics (**rather than proposing a new CLIP method**). CLIP is a very popular and efficient method for contrastive alignment between different modalities, and has been successfully extended to various domains like VideoCLIP [4], PointCLIP [5], and DiffusionCLIP [6].\n\nIn summary, our method is not a straightforward application of existing techniques. **We focus on addressing key challenges in OMRL from language supervision, with significant extensions of existing techniques like VariBAD and CLIP.**\n\n[1] VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning, ICLR 2020.\n\n[2] Meta-RL Based on Self-Supervised Task Representation Learning, AAAI 2023.\n\n[3] MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL, ICML 2023.\n\n[4] VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding, EMNLP 2021.\n\n[5] PointCLIP: Point Cloud Understanding by CLIP, CVPR 2022.\n\n[6] DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation, CVPR 2022.\n\n***\n\n**`Q3. Design choice for dynamics-aware decision embedding.`**\n\nA3. Since we aim to align text embeddings to comprehend the environment dynamics, we use the world model to learn task embeddings rather than information-theoretic or contrastive objectives like in CSRO or CORRO. **Through the encoder-decoder world model structure, we can explicitly distill the environment dynamics into the obtained task embedding**. This is the key property that alternative methods do not possess.\n\nIn experiments, we have included CSRO and UNICORN [7] as baseines. UNICORN can be considered an improvement over CORRO. Section 4.1 Main Results show that **T2DA gains a 36% to 200% improvement over CSRO and UNICORN**. Further, we have conducted a new ablation experiment where we use the CORRO’s contrastive approach to learn task embeddings for T2DA. The result shows **the superiority of utilizing world modeling to learn dynamics-aware task embeddings for text-decision alignment**. \n\n|Abaltion|**T2DA (CORRO)**|**T2DA**|\n|---|---|---|\n|Cheetah-Vel|-125.3±9.7|**-60.4±1.8**|\n\n[7] Towards an Information Theoretic Framework of Context-Based OMRL, NeurIPS 2024.\n\n***\n\n**`Q4. Baseline choices.`**\n\nA4. **We include five baselines that cover three representative paradigms in tackling RL generalization**: CSRO, UNICORN, AD, BC-Z, and BAKU. We select CSRO and UNICORN to validate our motivation for leveraging language supervision over traditional OMRL methods. \n\nBoth [8] and [9] learn hierarchical policies and fall into the paradigm of hierarchical RL. We did not compare T2DA to them due to not involving such a hierarchy. Instead, **we carefully select the most relevant and most representative baselines**: the classical BC-Z (citations: 649) [10] and the newer BAKU [11], to demonstrate our superiority in harnessing language knowledge.\n\nIn summary, we justify that **we have included adequate and competitive baselines to verify the claim and contribution of our method**.\n\n[8] LISA: Learning Interpretable Skill Abstractions from Language, NeurIPS22.\n\n[9] Language Control Diffusion: Efficiency Scaling Through Space, Time and Tasks, ICLR24.\n\n[10] BC-Z: Zero-shot task generalization with robotic imitation learning, CoRL 2022.\n\n[11] BAKU: An efficient transformer for multi-task policy learning, NeurIPS 2024.\n\n***\n\n**`Q5.  Report the “success rate” for Meta-World environment.`** \n\nA5. We selected returns rather than success rate as our evaluation metric for the Meta-World environment because the tasks vary significantly in difficulty. Some tasks readily achieve near 100% success rates, while others rarely succeed at all. **Returns provide a more nuanced and appropriate measurement for generalization problems across this spectrum of task difficulties**. Following your advice, we have reported the success rate of T2DA on Meta-World test tasks.\n\n|**Method**|faucet-open|drawer-close|reach-wall|handle-press-side|handle-pull-side|hand-insert|\n|---|---|---|---|---|---|---|\n|**T2DA**|0.11|0.82|0.13|0.38|0.21|0.34|\n\n***\n\n**`Q6. The reason why T2DA outperforms other context-based OMRL approaches, and what fundamentally enables T2DA’s superior performance.`**\n\nA6. The superiority over traditional OMRL methods verifies our motivation of harnessing the representation power and knowledge transferability embodied in pre-trained language models. The Section 4.2 Ablation Study has thoroughly analyzed the respective contributions to T2DA. We compare T2DA to three ablations: w/o world, w/o align, and w/o text. In a progressive manner, we demonstrate that **T2DA’s performance comes from three aspects: 1) leveraging a wealth of knowledge from natural language, 2) contrastive knowledge alignment, and 3) capturing environment dynamics via the world model.** Also, the visualization insights in Figure 1 and Figure 3 again verify T2DA’s efficient comprehension of environment dynamics.\n\n***\n\n**`Q7. Why T2DA maintains superior performance despite the use of lower-quality data, in Figure 5.`**\n\nA7. The absolute performance of any offline RL methods will be bottlenecked by the dataset quality. Both T2DA and baselines suffer a performance degradation when switching from Expert to Medium datasets. **T2DA’s superiority mainly comes from aligning text embeddings to comprehend environment dynamics**. The world model $p(s’,r|s,a)$ fully characterizes the environment, and remains invariant to behavior policies or collected datasets. Hence, **we can still distill the environment dynamics from lower-quality datasets using world modeling**. We evaluate T2DA on three types of datasets (Mixed, Expert, and Medium) to demonstrate our robustness to dataset quality.\n\n***\n\n### **`Summary Response`**\n\nThank you for your valuable review comments, which help us gain more critical insights and further enhance our empirical evaluation. Also, we are grateful that **other reviewers** (we refer to qsSY as R1 and sYVw as R2) **make positive comments to our contributions, including the technical challenge** (’addresses a long-standing challenge in RL’ by R1 and ‘breaking through the traditional methods reliance on high-quality samples’ by R2), **the novelty** (’a well-engineered system…a unique conceptual advance’ by R1 and ‘a novel framework by…’ by R2), **and empirical evaluation** (’with excellent visual aids, e.g., t-SNE plots, training diagrams‘ by R1 and ‘comprehensive experimental validation’ by R2). We hope that our overall contributions have been adequately justified. \n\nWe summarize that your main concerns may include the technical challenge, the novelty, and the baseline choice of our work. **We have extended a number of justifications and experimental analyses to address your concerns**. Please let us know if we have addressed your concerns. We are more than delighted to have further discussions and improve our manuscript. **If our response has addressed your concerns, we would be grateful if you could re-evaluate our work**."}]

Author rebuttal:

Meta-review: The reviews (4,4,4) for this paper have been collected and discussed. There is a general consensus among the reviewers that the paper, in its current form, is suitable for publication. After careful consideration of the feedback and the paper itself, the recommendation is to accept this submission.
Decision: Accept (poster)