DAIL: Beyond Task Ambiguity for Language-Conditioned Reinforcement Learning
Authors: Runpeng Xie
Quanwei Wang
Hao Hu
Zherui Zhou
Ni Mu
Xiyun Li
Yiqin Yang
Shuang Xu
Qianchuan Zhao
Bo XU
Venue: NeurIPS 2025 poster ()
Keywords: Offline Reinforcement Learning
Language-conditioned Reinforcement Learning
Representation Learning
Primary area: reinforcement_learning
TL;DR: This work presents DAIL, a semantic aligned reinforcement learning method that improves task discrimination and generalization in language-conditioned problems.

Abstract: Comprehending natural language and following human instructions are critical capabilities for intelligent agents. 
However, the flexibility of linguistic instructions induces substantial ambiguity across language-conditioned tasks, severely degrading algorithmic performance.
To address these limitations, we present a novel method named DAIL (Distributional Aligned Learning), featuring two key components: distributional policy and semantic alignment.
Specifically, we provide theoretical results that the value distribution estimation mechanism enhances task differentiability.
Meanwhile, the semantic alignment module captures the correspondence between trajectories and linguistic instructions.
Extensive experimental results on both structured and visual observation benchmarks demonstrate that DAIL effectively resolves instruction ambiguities, achieving superior performance to baseline methods. Our implementation is available at https://github.com/RunpengXie/Distributional-Aligned-Learning.

DAIL: Beyond Task Ambiguity for Language-Conditiones Reinforcement Learning
Runpeng Xie* Quanwei Wang" Hao Hu Zherui Zhou', Ni Mu Xiyun Li Yiqin Yang"!
Shuang Xu Qianchuan Zhao?, Bo Xu
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing. China *Department Automation, Tsinghua University "Moonshot
Department of Computer Science and Engineering, Washington University TTecent Al xierunpeng2021 1iii. aia..ac.com en, wqp210mails tsinghua. edu.cuu
Abstract
Comprehending natural language and following human instructions are critical capabilities for intelligent agents. However, the flexibility of linguistic instruc- tions induces substantial ambiguity across language-conditioned tasks, severely degrading algorithmic performance. To address these limitations, we present novel method named DAIL (Distributional Aligned Learning), featuring two key components: distributional policy and semantic alignment. Specifically, we provide theoretical results that the value distribution estimation mechanism en- hances task differentiability. Meanwhile, the semantic alignment module cap- tures the correspondence between trajectories and linguistic instructions. Exten- sive experimental results on both structured and visual observation benchmarks demonstrate that DAIL effectively resolves instruction ambiguities, achieving superior performance to baseline methods. Our implementation is available at http::/gggtttlooookkuuueeeeeeeto
Introduction
Artificial agents are anticipated master diverse skills while effectively interpreting human in- structions and generalizing across various tasks. Therefore, comprehension and following of nat- ural language emerges as critical capabilities for agents in this context. For example, language- conditioned agents have achieved remarkable success in robotic manipulation [36 6), text-based
environments [35] 7], visual navigation [62 (22, and autonomous driving [14 51]. The fundamental
requirement has propelled anguage-conditiones reinforcement learning (RL) to the forefront of
research, which focuses on enabling agents to interpret and execute natural language instructions through RL frameworks.
Recent advancements in the language-conditione conditioned RL domain have focused on bridging the gap between linguistic understanding and decision-makin processes, aiming to create agents capable of executing complex instructions with human-like adaptability. For example, some works [12] integrate aegguage-conditione policy with trial-and-error learning, significantly improving the performance and sample efficiency robot task acquisition. Meanwhile, some studies [T8 19 leverage expert demonstrations to map language instructions reward signals directly to address the issue of sparse rewards in language-conditione flexibility, which induces exponential growth in task space. In this case, identical tasks may have
RL. However, linguistic instructions exhibit high
"Equal contribution. Corresponneecee to Yiqin Yang and Bo Xu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Entedkingg
Trajectory Sequence Encoder
function
Q (6, E(Z)
Offire Trajectories 1
Policy
Totn left odvance next to the slove und not past to right....
Language Encoder
Embedding
x argma (6..)
Distributional Palicy
Language Task Instruction I
Semantic Alignment via ①
Ordinary suffers task ambiguity
Contrastive Learning
DAIL clearly idemiifies the task through ① ?
Figure 1: The framework of our method. The key ideas are: (1) use the distributional language-gun guided policy to aid task discrimination, (2) use the trajectory -wise semantic alignment module to extract task representation.
divergent expressions, while distinct tasks share overlapping language instructions. This variability makes language-conditione RI. methods face the significant challenge of task ambiguity, which hinders the agent from discerning the connection between rewards and task objectives, thereby significantly impairing learning efficiency.
To solve this issue, we propose novel method, DAIL, which consists of two main components: distributional language-guided policy and a trajectory-wise semantic alignment module. Specifically, the distributional policy module [3] estimates the value distribution, preserving more information to aid task discrimination. Theoretically, we analyze the sample complexity required to guarantee task disambiguation and establish that distributional estimation methods are sample-efficien in offline settings. On the other hand, the semantic alignment module constrains the language instruction representations by maximizing the mutual information between trajectories and the instructions, thereby achieving better differentiation across instructions. With theoretical guarantees, the two modules enable precise disambiguation and execution of linguistic instructions, thereby improving learning efficiency.
We conduct extensive experiments on both structured observation [0 and visual observation [54] benchmarks. This design is validate the external validity of the DAIL agent, progressing from less complex structured inputs to more complex and expressive visual observations. The experimental results show that DAIL outperforms the state-of-the-arr benchmarks. Further, the visualization analysis demonstrates that DAIL can learn a non-ambin non-ambiguous task representation compared with baselines. Our main contributions are summarized follows:
segguage--onditionnee RL methods both
First, we highlight the critical issue of task ambiguity and empirically analyze the limitations of current mainstream methods. We define the task distinction in our setting and analyze the sample complexity to avoid task ambiguity theoretically.
Second, we propose DAIL, a simple yet efficient language-condition learning framework, which addresses the task ambiguity issue based distributional policy and semantic alignment.
Lastly, we conduct extensive experiments to show that DAIL significantly outperforms conventional language-conditioned methods. The results indicate that by improving task discrimination, we can effectively mitigate the task ambiguity issue, thereby broadening the application of language-condition RL.
2 Preliminaries
Language-conditionee RL Based on contextual markov decision process (CMDP) 20]
consider Language-conditioned Markov Decision Process (LCMDP) as model consisting of tuple M (S, A, P,r.y Pu,Pt))) Pt), where S denotes the state space, A represents the action space, L is the language instruction space, P(s'ls, a. represents the probabilistic transition model, r: x A
+ the reward function conditioned on language instructions and y the discount factor. Pn represents the probability distribution of the initial state, and pt denotes the probability distribution of language instruction. We establish language instructions as task descriptors, and each instruction uniquely specifies task.

---

conditioned RL aims to obtain a policy π T-1(s,) that maximizes the cumulative discounted returns under a specific distribution of language instructions:
arg max, of EE
Σ (r(
(1)
where so po(:), pı(+), at (-|| and 8t41 P(- ac, (c.1). The temporal difference loss in lunguage-conditioned RL is adapted as follows:
(I) (s, (1, (r(s, a) ymax, Q5(5', a', (o.s.,
(2)
where Qa(s, a, 1) is the parameterized Q-function conditioned language instruction I, and Q3(s, a,
the target Q-network.
Offline RL Due to the high cost of real-time interaction with the environment, we consider the offline learning setting, in which we learns a policy æ without interacting with an environment. Rather, the learning is based on a dataset 'D generated by a behavior policy T.. One of the major challenges in offline RL is the issue of distributional shift [17), where the learned policy is different from the behavioral policy. Existing offline RL methods apply various forms of regularization to limit the deviation of the current learned policy:
π arg max, (JD(T) - (IT,T, m+))
where (Tz is the cumulative discounted return of policy π on the empirical MDP induced by the dataset D, and D(T, #g) is a divergence measure between and gg. As for the language-conditioned task, we will provide the language instruction in the evaluation. To make our writing more concise, let be a full trajectory, and Te be the trajectory ending time-step Let pD(T, represents the joint distribution of trajectory-instructon pairs in the dataset D.
3 Ambiguity on Language-Cnddtionee Tasks
In practical tasks, language instructions have high flexibility. For example, similar tasks may employ divergent expressions, while distinct tasks share overlapping language instructions. The variability induces exponential growth of the task space. Therefore, when the number of language instructions increases, the agent is required to accurately identify the tasks; otherwise, it will significantly affect the agent's 's performance. We name this issue the ambiguity in kanguage-conditione tasks. We give formal definition of semantics distinction from instructions as follows.
Definition (Semantics Instructions Distinction). In a multi-task RL setting with known task instruc- tion space L and unknown semantics space G. for task distinction threshold 8 and sub-optimality gap e, two task instructions le E are considered with different underlying semantics, Is 4 9g. the expected Q-values under any shared e-optimal policy πT satisfy:
Ex as as. 8.
Conversely, if
Bn [1Q (s, a, 4) (x a, 4 s 8/2,
then 4, and I, are considered with the same underlying semantics, gi + 9j. where so ~ po(:), T((ss), s a()ss,,)). (s(s) 2 (s) Vs S, V* is optimal value function.
To make this point more straightforward, we introduce a toy experiment based on the Minigrid
environment [1]] The setup consists of 10 accessible goal positions G {go, 91, ..., ......9), ). where G represents the set of all possible goal positions. The agent (red riangle-shaped, must follow a given instruction / E L to navigate using numerical task IDs, making C N. We employ a random mapping F L - G to assign each / to goal position g. which simulates semanties of instructions and is hidden from the agent (Middle of Figure 2). With these settings, we can control the number instructions (C] by simply adjusting the set of valid task IDs and the mappings. We conduct 10 experiments, varying the number of instructions from 1 to 20 For each experiment, we generate a new mapping F and collect 1024 random trajectories the offline dataset.
specific goal position (Left of Figure 2). We simulate instructions
We evaluate the standard offline RI. algorithm, CQL. 30,, on the above settings. In addition, to test how model size affects the task ambiguity, create enhanced version called CQL-double by
3

---

Tasks
Instructions
100%
to (1,1)
Success Rate.
90%
Go to (1.2)
Task "1"
80%
Go to (1.1)
Task "2"
DAIL
CQL
Go to (1,2)
Task "512"
60%
CQL double
2 2
2 2
20 tasks
512 instructions
Number of Instrunctions
Figure 2: Left: The green flags in the map denote the accessible goals. Middle: An illustration of the mapping between goal positions and instructions. Right: Average success rates over 100 evaluations for each number of instructions and 3 seeds.
Grainary RL methoos
Q fanction (5.0.)) Policy
Oedinary mothodi obtan similar PQ value
Agent suthers task ambiguity
Difirrnt Tork
4
Distrbutionni method
BO 1
D Distibutional Ro retain difference between dutributions
Ditributional Pollcy
The dizr bution @65.a.
Identily
Figure 3: An illustrative case where Q(s, a) functions in two tasks share the same expectations but have different distributions. In this case, traditional RL cannot discriminate between the tasks while distributional RL can. Please refer to Appendix D for more details.
doubling its parameters. The experimental results in Figure 2 show that if the number of instructions small (e.g., under 16), agents can understand instructions and reach the goal position with high success. However, instructions multiply, the number of demonstrations for each instruction declines, which increases task ambiguity and consequently makes CQL and CQL-double performance drop significantly. These observations reveal fundamental limitations in language-conditionee RL. regarding task ambiguity, highlighting that model scaling alone remains insufficient to resolve this challenge. For detailed information the toy experiment, please refer Appendix E
Method
To address the issue of task ambiguity, hope improve the algorithm's ability to distinguish tasks. In this section, we propose the Distributional Aligned Learning algorithm (DAIL), which enhances the task differentiability from policy and representation. Specifically, DAIL adopts the distributional language-guided policy to estimate the value distribution, preserving more information aid task discrimination. On the other hand, DAIL the trajectory-wise semantic alignment module to extract task representations and help discriminate different instructions by maximizing the mutual information between trajectories and instructions. We show the overall framework of DAIL in Figure I and Algorithm Ii in Appendix
4.1 Distributional Language-Guided Policy
Current RL methods aim to estimate the expectation of the cumulative discounted reward. However, as shown in Section 3 when the number of instructions increases while keeping the number of actually-distinct tasks constant, the estimated expectation for different task instructions becomes similar. a result, difficult for the agent to complete the task instructions accurately. Differently, the distributional technique addresses this issue by calculating the distribution of the cumulative discounted reward, which is more distinguishable than expectation, as illustrated through an extreme straightforward example in Figure For fixed policy let the random variable ZT represent

---

the cumulative discounted reward obtained along the policy and it has the following relationships:
= E
frr( 01, 1)|80 8,
E
Σ 7'r(s,, 0c 1)|50 8,
Let 2 denote the space of value distributions. In the following, for simplicity of notation, we denote 7" 2 as Z. Instead of estimating the expectation, we calculate the probability distribution of the random variable Z:
T 7 a, 22 R(s, a, + 7
where T the distributional Bellman operator, and A _ denotes that A equals B by probability laws. R E Z is function depicting the reward distributions. Since modeling continuous distributions challenging, we can discretize the value function distribution and train it by minimizing the cross-entropy loss:
Dist (0) = (r Σ DxL. (DT a. a.))
(6)
where T' is the trajectory length, & the sample Bellman update, which projects the value function distribution onto the parametric discrete distribution. Zo and Za are estimated value distributions parameterized by ! and Ó, respectively. By optimizing Equation 6 obtain the distribution Ze that exhibits strong discriminative power across different instructions L. Please refer to Appendix C for the details.
In addition, we conduct the following theoretical analysis. Let Tlaaue Tdis denote the number of samples needed to avoid task ambiguity for malue-based and distributional settings, respectively. As shown in Theorem and Corollary 2 distributional RI. achieves better sample efficiency compared with estimating the expectation when the number of tasks is sufficiently large, Thvalue and further details can be found in Appendix [
Tidist- Proofs
Theorem 1 (Sample Complexity for Task Instruction Disambiguation). Consider an offline multi-task RL setting with M distinct tasks (with different semantics). In direct Q-value estimate setting, suppose Q(s, a, 1) E [0, Quax)) ((8 a,1) 'D with finite Quaxo and the task distinction threshold is > O. When the number of training samples Twvauue satisfies:
Ovalue log(3M2 /1/
Tevalue
The mean value estimate algorithm achieves task-level semantic disambiguation with confidence at least 1- TJ. In the distributional RL setting, let Z(s, a,1) denote the learned return distribution, and suppose the task distinction threshold is given by a I-Wasserstein distance O. Then, to ensure semantic disambiguation of task instructions with confidence at least 1 it suffices that:
Caist log(oo2
Tidist
d2
where Cvalues Caist are universal constants depending on certain attributes of Q-value distribution.
4.2 Trajectory-WW Wise Semantic Aligmment
In this subsection, we focus on learning trajectory embedding that enhances the correspondence between instructions and trajectories, thereby reducing task ambiguity the representational level. To achieve this, we attempt to maximize the mutual information between the language instructions and trajectory:
max T (w): (w)).
where w is the parameter the representation module, X,, X₁ are the random variables of trajectory embedding and language instruction, respectively. For the trajectory embedding Ir, adopt the
5

---

sequence model. Specifically, we first use (:, to encode the state-action pairs, and then pass the sequence of embeddings through sequence model (-):
hww (51, a1), the (82, (2). .... the (ST, aT)).
(8)
As for the language instruction representation, we employ a language encoder to tokenize and encode the instructions into language embeddings z1. Please refer to Appendix F for the detailed model architecture. Let (T, 1) T || || measure the similarity between the trajectory embedding and language instruction representation. Since minimizing InfoNCE is equivalent maximizing the lower bound of the mutual information [45], we can maximize the mutual information in Equation 7
x_
by minimizing the following NCE loss:
(w) (B. (otr:==))pppt:lllooo(((( (t)) + log(1 o(f((T, (ー~
where 1+ denotes the positive samples, which are sampled from the distribution of trajectory- instruction pair pp(::-. | denotes the negative samples, generated by uniform sampling over the language instructions from the offline datasets.
4.3 Practical Implementation
To address the partial observability issue in practical implementation, the trajectory encoding 2 defined in Equation g is incorporated as an additional input (8t, at It-1, The Equation 5
is
transformed into correspondingly:
7st, at, 11. _ R(se, ae, 1) + 7 (St+1, 01,1,
(10)
In practice, estimate the value distribution Zo with discrete distribution, using set of atoms
{& VMIN + i M-1 A V VMIN, VMAX R the lower and upper bounds of the distributions with support, respectively, and M N is the number of atoms. Then the discrete value distribution is modeled as:
fi=1
e (i 1.f)
Zo Zr 24, with probability Pr(Se, ae, Z4 21-1, E, (c
(11)
where 0; : S x A x X' x C - a parametric model employed to approximate Zo and updated
by Equation Based on the analysis in Section 4.1 compute Q-function with distributional mechanism by Qa(st, Tt.1). E[Ze(S (t ae 21) Ei= M-1 pi(st, a, Tz-1, Please refer to Appendix for the details.
In addition, we consider the offline learning setting in this work, which learns policy without interacting with the environment. For this reason, we adopt the standard offline learning term, CQL(H() [30], to address the distribution shift issue in offline RL learning [77:
CCoL(() -( Σ log Σ exp(QQo(s, a, a,11, Ea-r Qolse 1t-,, [)),
(12)
Combining all the above loss functions, the total loss function is:
L Liitt + XCe + aLcoL
(13)
where X, are weights of the trajectory- semantic alignment module and the offline learning term, respectively. Finally, we select the action with the highest Q-value:
argmaxx2 (St, a, 21-1.()
(14)
The complete process of our method is shown in Algorithm T in Appendix Appendix F
B and Figure 14 in
5 Experiments
We designed our experiments to answer the following questions: Q1: How does DAIL compare to other state-of-the-arr methods on offline language-conditioeed tasks? Q2: How does DAIL perform as the number of tasks explodes? Q3: Can DAIL. learn a meaningful alignment between trajectories and instructions? Q4: What is the contribution of each of the proposed techniques in DAIL?

---

Go blue door, then pick up the green bull, then
with then ....
Figure 4: Left: An example of the SynthLoc task in the BabyAl Al environment: "put the yellow key next to green ball".. Right: Two example scenes with instructions from ALFRED, Agents are asked to finish specific tasks in the 3D household environment according to received instructions.
Table 1: Success rate of out-of-distriiutiin Baby AI tasks. Each score is evaluated over 3 seeds.
Tasks
GCBC
BC-Z
GRIF
IQL
CQL
DAIL (ours)
Open
94.4+2.5
93.7+1.0
95.9+1.7
98.0=0.4
98.8+0.5
99.0=0.2
Goto
90.3#1.6
76.943.0
88.8+2.6
86.1+1.2
88.942.1
91.3+1.0
PickUp
78.4#2.1
15,4+1.5
75.6+3.9
70.443.6
71.9+2.2
87.6+2.0
PutNext
27.4=1.6
11.243.3
22.5+2.7
21.4+3.1
27.6+0.8
49.1
All
74.1+0.7
57.9+1.8
71.242.6
69.7+2.3
72.6+0.4
81.7+1.3
5.1 Experimental Setting
We evaluate various methods in language-conditionee tasks, with detailed introductions as follows:
BabyAI 10 is a language learning research platform with different levels of tasks, shown on the We choose level SynthLoc for evaluation, which contains four major groups of tasks Open, Goto, PickUp, and PutNext. The agent is asked operate with assigned object, like "open a red door" or "put the gray box in front of you next to the blue key". Tasks vary as the colors, types, or locations of objectives change, making around 6000 different tasks total. To evaluate the algorithms' generalizations, we divide the task space into in-distribution tasks and out-of-distribun tasks. In-distribution tasks account for approximately 60% of the total number of tasks, with a total of 3325. For the offline learning, we construct an offline dataset with 50k expert trajectories, 50k imitation learning agent trajectories, and 25k random trajectories.
Left of Figure 4
ALFRED 54 benchmarks sequential decision decision-making making tasks involving household activities (e.g, cleaning, heating food) through language instructions and first-person vision, shown on the Right of Figure 4 The dataset provides 8055 expert demonstrations with 25k numan-annotater language instructions detailing both high-level goals and sub-goal atep-by-step guidance. As our work primarily focuses low-level policy learning rather than high-level planning, on the GOTO sub-goal setting for our evaluation. In this task set, the agent must go to specific locations according instructions like "Move other side of couch the right side of the table before the door". To simulate the presence of noisy data in real-world applications, we augment the training set with 30k random-agent trajectories, resulting in 97896 total trajectories with 53442 unique instructions across 108 household scenes.
specifically concentrate
Baselines We choose three state-of-the-art offline RL algorithms, CQL 30 IQL 28 and adapt them into a language-conditioned manner as our RI baselines. In addition, for imitation baselines, we include GCBC [55] BC-Z 25 and GRIF 42, which are also adapted into a language-condition
manner by the benchmarks [10 54]. Further details about baselines are shown in Appendix

---

GCBC
BC2
GRIF
IQL
CQL
DAIL
GoTo
Open
Pickup
PutNext
1.
14
1.0
Sucoess Rate
Success Rate
Sucoess Rate
Success Rate
2.8
0.4
26
0.2
02
0.2
82
08
22
Training Steps (x109))
Training Steps (x1105)
Training Steps (x110)
Training Steps (x(10F)
Figure 5: Training curves on in-distribution Baby Al tasks. Success rates are evaluated over 3 seeds.
right and move to face the tub.
the countor left the fridge.
Take the heated mug coffee machine, right of sink.
Figure 6: Example trajectories of DAIL in ALFRED validation set instructions with varied instructions and scenes.
5.2 Main Results
BabyAl experimental results. We provide training curves of in-distribution tasks in Figure and out-of-distribttion experimental results in Table The complete results are shown in Table 5 in Appendix H It shows that our method achieves superior performance compared with other baselines, especially in the PutNext task category. Vanilla offline RL algorithms like CQL and IQL underperform compared to imitation learning methods like GCBC, which attribute the adverse impact of task ambiguity on RL-based approaches as discussed in Theorem [ On the other hand, modified algorithms designed
(BC-Z and GRIF) perform poorly under our setting. This is primarily because their contrastive learning objectives are not robust in the presence of noisy or suboptimal data. In contrast, our alignment- approach, built on an offline RI. framework, maintains strong performance. We further evaluate various algorithms on a more challenging dataset with fewer expert trajectories (Table Appendix H. Our method still achieves the optimal results. Further details about the experiment are shown in Appendix B
annuuag--conditooe
ALFRED experimental results. The complexity and variety of instructions in ALFRED challenge the agent's ability to discern instructions. The experimental results Table show that our approach demonstrates the highest success rate (SR), validating its positive impact on instruction recognition capability compared to other baselines. GCBC exhibits poorer resistance to suboptimal data, resulting in performance comparable success rates (PLW SR), which considers the length of expert demonstration and demonstrates the effectiveness of the behavior policy following [54] We further illustrate the observation trajectories generated by DAIL under the validation set instructions of ALFRED with varied instructions and scenes in Figure 6 This visualization demonstrates DAIL's robust task execution in complex scenes under diverse instructions, with additional trajectory demonstrations provided Appendix[[]
other RL baseline models. We also report path-length weighted
5.3 Visualization
To better understand how our proposed method enhances the performance, we show the t-SNE (57 results of the language instructions internal embedding on BabyAl SynthLoc. Each point repre-

---

Table 2: Success rate (SR) and path-length weighted success scores (PLW SR) in the ALFRED tasks. The results are shown on the training set and validation set respectively. Each score is evaluated over 3 seeds.
Tasks
GCBC
BC-Z
GRIF
IQL
CQL
DAIL (ours)
SR (Training)
87.9+2.4
86.5+0.8
87.8+1.6
88.4+0.6
87.2+1.2
92.3+2.2
PLW SR (Training)
84.3+2.6
82.3+2.7
82.342.6
84.4+2.7
83.0+2.9
90.2+3.1
SR (Validation)
47.1+2.4
43.0#2.5
48.6+1.0
52.0=3.0
50.4
56.842.1
PLW SR (Validation)
40.5+3.1
39,5+2.2
43.242.5
47.0+3.2
44.4+1.
50.3+2.4
Distributional
Alignment
Goto, door
Goto, key
Goto, box
Goto, Open, door
PickUp, key
PickUp, box
PickUp,
Figure 7: T-SNE visualization of instructions BabyAl tasks. The figure distinguishes between different task categories (e.g.. Open) and target object types (e.g., box), using marker colors and shapes to represent each separately. For example, "pick up a red box" corresponds to m, "go to green box" corresponds to
sents the internal representation of unique language instruction within the policy network. The experimental results in Figure 7 show that vanilla RL can only marginally separate some broad task categories, but fails distinguishing PickUp and Goto. Moreover, it is completely confused between Open (door) and Goto (door). Alignment and distributional methods help discriminate tasks between and within categories. Our method substantially enhances task representation by clearly differentiating between task categories and target object types (for example, separating and and *).
For more precise visualization, we use the same method to visualize the task representations of algorithms the same task category, illustrated in Figure Appendix [ Our method effectively distinguishes all tasks in these two categories without overlapping confusion and group similar tasks into smaller clusters training seeds, all of which yield consistent experimental conclusions. Moreover, we quantitatively measure the clustering quality of our proposed components with the Silhouette score [52) using the
+, and so on). We conduct the visualization experiments under
learned language embeddings. The experimental results in Table g in Appendix H demonstrate that our method effectively enhances clustering performance, which aligns with the experimental findings from the visualization. Details of the quantitative evaluation and more visualizations of task representations can be referred in Appendix and respectively.
5.4 Ablation Studies
Ablation of components. To study the contribution of each component in learning framework. we conduct the following ablation study. We compare the performance of algorithms that only apply trajectory-wiss alignment or distributional language-guided policy alone with our method on SynthLoc. The experimental results in Table 3 show that both modules significantly improve the performance vanilla CQL on in-distribution and out-of-distribution tasks. Further, combining both components can achieve the best performance compared to other approaches.
Ablation of alignment weight X. In Equation 13 X is the weight of the alignment loss. For this reason, we evaluate the choice of BabyAl tasks with various The experimental results in Figure g show that there is noticeable performance difference between À = 0.2 and X 1. The

---

Table 3: Ablation results for components of our method. Each score is evaluated over seeds.
In Distribution
Out of Distribution
Algorithm
PutNext
All
PutNext
All
CQL
25,6+2.5
78.111.
27.6+0.8
72.6+0.4
DAIL w/o Distributional
39.6#0.6
83.3+0.2
39.3+1.7
77.340.8
DAIL w/o Alignment
39.1#1.0
82.2+1.3
32.0=0.9
75.1
DAIL
57.9=0.9
89.2+0.5
49.1+1.8
81.7#1..
performance begins to degrade when the influence of the loss either too small (A = 0.01) or too large (A 2). Therefore, recommend choosing value between 0.2 and 1.
In Distribution
Success Rate
85
0
0.01
0.1
0.2
0.5
1
Out of Distribution
Success Rate
85-
80
75-
70
0.1
Figure 8: Percent difference of the performance of an ablation over A, compared to the average results of all As evaluated.
6 Conclusion and Future Work
In this work, aim to address the task discrimination and comprehension challenges in language- conditioned RL. To achieve this, we propose a novel method called DAIL, which incorporates distributional trrguage-guider policy and trajectory- semantic alignment module. We first theoretically demonstrate that distributional RL methods are more sample-efficient than traditional RL methods for learning in language-comee conditioned tasks. We then perform extensive experiments on both structured and visual observation benchmarks. The experimental results and visualization analysis show that method learns language instruction representations with clearer semantics. The simplicity and robustness of DAIL make it easily adaptable as a plug-in for other methods tackling iagguage-conditoone problems. While method is theoretically and empirically validated, demonstrating its significant advantages in seaguage-conditione First, due experimental constraints, are unable validate the method's s effectiveness and robustness, which we will consider in future work. Second, our theoretical analysis distributional RL's advantages relies on the assumptions of offline RL. Although we believe that this approach remains effective in online settings, we defer the theoretical analysis to future work.
tasks, several limitations remain. method in real-world scenes further
Acknowledgments and Disclosure of Funding
This work is supported by Strategic Priority Research Program of the Chinese Academy of Sciences (NAA722200))000
10

---

References
[1] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International conference machine learning, pages 166-175. PMLR, 2017.
[2] Marcin Andrychowiez, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong. Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[3] Mare G Bellemare, Will Dahney, and Rémi Munos. A distributional perspective reinforce- ment learning. In International conference on machine learning, pages 449-458, PMLR, 2017.
[4] Zhenshan Bing, Alexander Koch, Xiangtong Yao, Kai Huang, and Alois Knoll. Meta- reinforcement learning via language instructions. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5985-5991. IEEE, 2023.
[5] Stéphane Boucheron, Gábor Lugosi, and Olivier Bousquet. Concentration inequalities. In Summer school on machine learning, pages 208-2082000 240. Springer, 2003.
[6] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, al. Rt-2: Vision-language- action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 15818, 2023.
[7] Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In International Conference on Machine Learning, pages 3676-3713. PMLR, 2023.
[8] Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International conference on machine learning, pages 1430-1440. PMLR, 2021.
[9] Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[10] Maxime Chevalier-Boisver Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint arXiv: 1810.0822, 2018.
[11] Maxime Chevalier- Bolun Dai, Mark Towers, Rodrigo Perez-Vicente Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Systems 36, New Orleans, LA, USA, December 2023.
[12] John D Co-Reyes, Abhishek Gupta, Suvansh Sanjeev, Nick Altieri, Jacob Andreas, John DeNero, Picter Abbeel, and Sergey Levine. Guiding policies with language via meta-learning, arXiv preprint arXiv: 18/1.7888 2018.
[13] Cédric Colas, Ahmed Akakzia, Pierre-Y Yves Oudeyer, Mohamed Chetouani, and Olivier Sigaud. Language-conditiones goal generation: a new approach to language grounding for rl. arXiv preprint arXiv:2006.07043 2020.
[14] Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, and Ziran Wang. Drive as you speak: Enabling human-like interaction with large language models in autonomous vehicles. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 902-909, 2024.
[15] Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning, Advances in neural information processing systems, 32, 2019.

---

[16] Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision-based instruction following. arXiv preprint arXiv: 1922.0774 2019.
[17] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[18] Prasoon Goyal, Scott Niekum, and Raymond Mooney, PixI2r: Guiding reinforcement learning using natural language by mapping pixels to rewards. In Conference on Robot Learning, pages 485-497. PMLR, 2021.
[19] Prasoon Goyal, Scott Niekum, and Raymond Mooney. Using natural language for reward shaping reinforcement learning. arXiv preprint arXiv:1903.02024 2019.
[20] Assaf Hallak, Dotan Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint (502.0:550002255555 2015.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[22] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded language learning in simulated 3d world. arXiv preprint 1rXiv:1706.000 2017.
[23] David Yu-Tung Hui, Maxime Bhevalier-Boisvert, Dzmitry Bahdanau, and Yoshua Bengio. Babyai 1.1. arXiv preprint arXiv:2007 12770, 2020.
[24] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
[25] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991-1002. PMLR, 2022.
[26] Russell Kaplan, Christopher Sauer, and Alexander Sosa. Beating atari with natural language guided reinforcement learning. arXiv preprint arXiv:1704.05539 2017.
[27] Diederik Kingma. Adam: method for stochastic optimization. arXiv preprint
arXiv: 14/2.6980, 2014.
[28] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning arXiv preprint arXiv:2110.061666 2021.
[29] Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine, Offline q- learning on diverse multi-task data both scales and generalizes. arXiv preprint arXiv:2211.15144 2022.
[30] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179 1191, 2020.
[31] Jing Lei. Convergence and concentration of empirical measures under wasserstein distance in unbounded functional spaces. 2020.
[32] Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierar- chies with hindsight. arXiv preprint arXiv:1712.00990 2017.
[33] Alexander Li, Lerrel Pinto, and Pieter Abbeel. Generalized hindsight for reinforcement learning. Advances in neural information processing systems, 33:7754-7767, 2020.
[34] Jinning Li, Chen Tang, Masayoshi Tomizuka, and Wei Zhan. Hierarchical planning through goal- conditioned offline reinforcement learning. IEEE Robotics and Automation Letters, 7(4): 102)1 10223, 2022.
12

---

[35] Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems, 35:31199-31212, 2022.
[36] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. arXiv preprint arXiv: 190.00:990000 2019.
[37] Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint (rXi:20005007645 2020.
[38] Jason Yecheng Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. Offline goal-conditioned reinforcement learning via f-advantage regression. Advances in neural information processing systems, 00:310-3223, 2022.
[39] Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 7(4): (14):112053) 1121- 2022.
[40] Tomas Mikolov. Efficient estimation of word representations in vector space. arXiv preprint 1301 3781, 2013.
[41] Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. arXiv preprint arXiv:1704.08795 2017.
[42] Vivek Myers, Andre Wang He, Kuan Fang, Homer Rich Walke, Philippe Hansen-Estruch, Ching-An Cheng, Mihai Jalobeanu, Andrey Kolobov, Anca Dragan, and Sergey Levine. Goal representations for instruction following: A semi-supervised language interface to control. In Conference Robot Learning, pages 3894-3908. PMLR, 2023.
[43] Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation. Conference on Robot Learning, pages 1303-1316 PMLR, 2022.
[44] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12600) 2022.
[45] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03744 2018
[46] Jing-Cheng Pang, Xin-Ye Yu Yang, Si-Hang Yang, and Yang Yu. Natural language-conditione reinforcement learning with inside-out task language development and translation. arXiv preprint vrXiv:2322.09366 2023.
[47] Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal- conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 36, 2024.
[48] Shaohui Peng, Xing Hu, Rui Zhang, Jiaming Guo, Qi Yi, Ruizhi Chen, Zidong Du, Ling Li, Qi Guo, and Yunji Chen. Conceptual reinforcement learning for language-condttioned tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9426 99444, 2023.
[49] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
[50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
[51] Junha Roh, Chris Paxton, Andrzej Pronobis, Ali Farhadi, and Dieter Fox. Conditional driving from natural language instructions. In Conference on Robot Learning, pages 540-551. PMLR, 2020.
13

---

[52] Peter Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65, 1987.
[53] Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg. Concept2robot: Learning manipulation concepts from instructions and human demonstrations. The International 2021.
Journal of Robotics Research, 10(12-14)
[54] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10740-10749, 2020.
[55] Shawn Squire, Stefanie Tellex, Dilip Arumugam, and Lei Yang. Grounding english commands to reward functions. In Robotics: Science and Systems, 2015.
[56) Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Stefan Lee, Chitta Baral, and Heni Ben Amor. Language-conditonn conditioned imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems, 33: 53:13139-13150 2020.
[57] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[58] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems,
2017.
[59] Rui Yang. Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking condiconnd supervised learning and its connection to offline rl. arXiv preprint MXXiv:2202.04777 2022.
[60] Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems, 34:1 1 1501-11516. 2021.
[61] Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Generating adjacency- constrained subgoals in hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 77-991590, 2020.
[62] Hongkuan Zhou, Xiangtong Yao, Yuan Meng. Siming Sun, Zhenshan Blng, Kai Huang, and Alois Knoll. condtage-oonditioned learning for robotic manipulation: A survey, arXiv preprint 1xXi:2332.110077 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims in the abstract and introduction accurately summarize the paper's contributions, particularly in highlighting the proposed method's effectiveness in reducing task ambiguity and its robustness under noisy, offline settings.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our method Section including experimental
settings and theoretical assumptions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: In the main text, we mainly conduct theoretical analysis in Section 3 and 4.1 The paper provides the full set of assumptions and complete proof in Appendix Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We discuss the implementation of our method in Section 4.3 and details of experiments in Section and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide data and code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iisscc//pil wwtpcccpppppppppeeeeeeeeeeeeeeee httpe//ggideee/eeeeebbissssssss /Cc////oddiiisssseeeeeee sswiissiiciiiyyyycc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We discuss the experimental details of experiments in Section 5 and Appendix E We provide the hyperparameters and network architectures in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We provide the standard deviation in all the experimental results in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information or computer resources are provided in Appendix F
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification:
We confirm that our research conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work focuses on addressing task ambiguity in offline reinforcement learning: this work does not present any foreseeable societal consequences.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly eredit the existing assets and respect the terms of use in research.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The model and code we propose are well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

A Related Work
Language-conditioned Agents. Two primary approaches for enabling an agent follow hu- man instructions are reinforcement learning (RL) and imitation learning (IL). Some approaches in language-conditoon RL focus on aligning language with policies or performing feature extraction to integrate language information into the learning process [6] 41 53 4. while others prioritize
reward shaping [55] [6. 19 18 53 to formulate language-conditionee reward functions. However, most methods rely on simulators and are limited in scalability when applied in the offline setting. On the other hand, II designed to learn from large datasets but is heavily dependent on the quality of the data [24]. To mitigate this dependency and enhance data efficiency. recent works have leveraged crowd-sourced annotations [43], multimodal alignment [25] 42 44], or carefully designed model architectures [39, [5).. These methods enhance IL by supplying richer supervisory signals or building more robust model structures. Despite the success IL methods in solving complex tasks, the scale and quality of the dataset remain significant constraints, partieularly without external annotations auxiliary datasets.
Offline Goal-condee conditiodd RL. Learning task- policy from demonstrations with different
goals presents a significant challenge offline goal-conditioned reinforcement learning (GCRL). By relabeling trajectories and treating intermediate states additional goal states [2
32
33
59
60
offline GCRL has achieved notable improvements in sample efficiency. However, this approach is challenging to replicate in Iinguage-condition settings. First, replicating specific goal state is particularly challenging in environments characterized by randomness or partial observability. Second, directly using goal states that lack semantic context labels proves ineffective for learning. While some works use language instructions to guide planning [33 48 46), the ambiguity of language can complicate this process. Several studies have applied hierarchical RL to guide policy through subgoals [32, 61 8 (7.34)) and demonstrate strong performance. However, they rely heavily on high-level policy that accurately decomposes language instructions into subgoals.
22

---

B
Algorithm
Algorithm 1 Distributional Aligned Learning
Require: Offlin dataset D {(T, 1) target network update frequency K Kupdate and support atoms
7atomss
Initialize policy parameters and target policy parameters θ.
for each gradient step do
Sample batch B {(T, 1) ~D D
Encode instructions: {21]/=1
Compute the history information {.o, 01, } using (8)
// Distributional Language-Guide Policy
for each transition (st, az Tt,, Sr+-1 () batch B do
Compute estimated value distribution Zo Oe, 2x -r- 1.0 () using 15
Compute projected update 6720 (s, 0, x, 1) using 18 (0) DKL end for
ZZ ( 02, 2t-1, [)] Zo Oe, 11-1.
// Trajectory- Wise Semantic Alignment for each trajectory-instructin pair (r, do
for each pair (7', other than (r.1) do
View as negative instruction 1-, and compute contrastive loss le by using end for end for
Add up the losses above to get Liat + + Dist. and Le + Evro) le
Compute conservative Q-learning loss LeQu using 12
Liot + Dist + ALe + aLcQL
Update 0 t V (0)
if step % Kupdate=0 then
Update target network: <- 0
end if end for
Extract policy: (ss x. + arg max, EM a, return ((,, I,
23

---

C Details on Distributional Language-Guided Policy
In this section, we provide more detailed introduction to the specific computational workflow of our proposed Distributional Language-Guui Guided Policy. We follow the approach in [3] employing discrete atoms to model and approximate the original value distribution. To facilitate understanding, we have included the calculation methodology for this section of the work here.
Specifically, we model the discrete value distribution with discrete units called atoms (z: VIIIN " which uniformly spaced support points that discretize the range of possible returns into [VMIN, VMAx]. M E N is the number of atoms. VMAX,, VMIN, M are pre-defined parameters, which together decide the step between the atoms of the categorical value distribution: A == M
To represent the discrete value distribution, we need to estimate the probability pi, which means the probability for the discrete value equaling In practical implementation, we approximate the probabilities P. by parametric model 0 S x A x L TR We use (.(:)))')) to denote the i-th dimension of this model's output. Therefore, the discrete value distribution can be written as:
ella,,111)
Zo(s,a,z,1)- = w.p. 2, E, (l((aazz.))
(15)
M-1
((2o))))((((s,,,,,,,) Σ p,,,,,,01122 2=0
where Σ M-1 P: 1.
We now explain how to learn 0 through RL. Given a transition (s,a, 8 the Bellman update for each atom 2/ is computed as:
T2, = 724
(16)
The Bellman update T2, maps the original support points to locations that do not align with predefined discrete atoms & M-M making impossible to represent valid discrete distribution over the fixed atoms. Therefore, we need to project these values back onto the fixed support {2o..... ..., 2M-1). To do so, distribute probability mass pi(s', a',x', x,,'1) to the nearest two atoms in [VMIN, [MAA], where a' ( is the output of the greedy policy T(' "))) x' is computed
as x' (2, (s, a)). Ultimately, we can compute the projected probability for Tz, via a local interpolation mechanism:
B:1--a pi As
assigned to 2ÿ
Projected probability
(17)
pr(s', a' x/,1).
assigned to Zj+1
A=
Sum all the projected probabilities from all i 2zz, we can compute the projected update probabilities by:
M-1
[
VMLx VNIN
Zo(, = Σ [1
V
1p, d','',
(18)
A
1=0
where 1.1& bounds the argument in the range (a, bj. Given the project update o and the current estimates of the discrete value distributions Zo. we can update 0 by minimizing the KI. divergence:
DKL DKL(T (TT a, x, 1) Zo a,,,,
(19)
where i the target network.
24

---

D Theoretical Analysis
D.1 Insights Using Distributional RL
The key insight of applying distributional RI. to language-conditionee tasks lies in its capacity to capture value distributions, which provides fine-grained task differentiation across various tasks. Traditional RL methods, however, rely on learning scalar value expectations, discarding critical distributional information, making it require more samples discriminate between tasks properly. We first demonstrate this key insight through an extreme yet illustrative example as illustrated in Figure
Orainary RL methods
Q function (5.o,,)
Dedivery methodi oboan similar EQ value
Policy
Agent sullisulles task ambiguity
Dnferent Tosk
disribution
Distrbutionai method
Dxtributional retainn difference betwoe disributions
Dstributional Pollcy
The dier bution Q (t)
identify
Figure 9: An illustrative case where Q(s, a) functions in two tasks share the same expectations but have different distributions. In this case, traditional RL cannot discriminate between the tasks, while distributional RI. can.
Consider two distinct tasks and la that share identical expected returns Qs, a, (1) Q= a, (2)
for a specific state-action pair (s, a). but exhibit fundamentally divergent Q value distributions (ss, 41 A (s, a, 12). In this case, traditional RL methods relying on value estimation would inevitably conflate tasks 11, le at (s, a). regardless of sample size, while distributional RL resolves this ambiguity through approximating the full distributions values.
To formally validate this insight, we present the following theorem, accompanied by rigorous proof.
D.2 Theoretical Proof
Definition 1 (Restatement of Definition D.. In multi-task RL RLsting with known task instruction space C and unknown semanties space 9. for a task distinction threshold 8 and sub-optimality gap c, two task instructions li, E L are considered with different underlying semantics, 9i 47 9j if the expected Q-values under any shared e- e-optimal policy n satisfy:
Ex 11Q (s, 0,4,
8.
Conversely, if
E, (sQ 4) - Q=(s, a. 11 S 8/2,
then I and Ij are considered with the same underlying semantics, 9r 9j. where 80 po(.), s' ~ ~p((-ss, a). (s(s) 2 (s) es E S, V* is optimal value function.
In the case of distributional reinforcement learning, the Wasserstein distance Wi between the return distributions Z(s,0 a) is used as the criterion. Specifically, l and l, are considered to represent different semantics if
En W1 (Z (s, a, 1t), (s, a, 1,)1 2 d,
and the same semantics this expectation is less than equal d/2.
Theorem 1 (Sample Complexity for Task Instruction Disambiguation). Consider an offline multi-task RL setting with M distinct tasks (with different semantics). In direct Q-value estimate setting, suppose Qss, a. 1) E [0, Quax]. V(s, a, 1) ~ 'D with finite Quaxs and the task distinction threshold is > O. When the number of training samples Tevalue satisfies:
Oralus log(3M2 (3322/n)
Tevalue
82
The mean value estimate algorithm achieves task-level semantic disambiguation with confidence at least 1 - TJ. In the distributional RL setting, let Z(s, a, 1) denote the learned return distribution, and
25

---

suppose the task distinction threshold is given by a 1- I-Wasserstein distance d > O. Then, to ensure semantic disambignation of task instructions with confidence at least 77, suffices that:
Caist log(3o3 d²
Tidist
where Cvalue: Cdist > are universal constants depending on certain attributes of Q-value distribution.
Proof. We denote Qi a) as shorthand for (s, a. 44) and (s a) as a shorthand for 2 (s. 4) in the following discussion. For direct offline Q-learning, when the task distinction threshold is > we define the semantic ambiguity event S₂ as:
E (R,a)D [[ -(((s,a)-Q((s,a))) ||] - ~/4
8
(20)
2'
Ex [1Q((s.)) Q,(s, a)ll 2 8. 4 9ソ.
(21)
where D is the offline dataset, Q the optimal Q function, π the optimal policy along with Q, and Q is the learned Q function.
Following the triangle inequality, we have:
((,))= 2111 E, -
(22)
B ( i,1l -ED - Uill [ وار 11-1-- 2i1l
(23)
In general, we assume that the offline RL dataset is collected by e-optimal policy, i.e., there exists c-optimal policy π that 'D satisfies
ED -211 = 1Q.. - (,1l. Qx =Q
(24)
Besides, we have:
Em 11 - -111--1111100
11: - 2, = Er (Q0--0)1-11-- 2,1)
(25)
< E -0- -QQ Q31
(26)
< ( ()) -Q)) - Q7)
(27)
E +
+
2
(28)
Therefore,
E [[ssa)DDlQi - - - 2/1l 2 ;
Pr
(29)
E [( - Q,1l E. - 2,1 E,1Q - Q.1 + - 2,1 2 8)
;)
< Pr
(30)
2
E [[s,a))D11 - Q E, 1 - 2,1 2 8
$ Pr
2
(31)
8
+ Pr E Q,1 2 5)
+ Pr
E - Qi1 -0:@--Qil20 Qil2
(32)
2
2
Since Q,Q E [0, max]] 1-: -0,-2,1110 Q Q.1, 14, - -2² E [0, Qmax], together with Equation
24
we can apply Theorem (Hoeffding's inequality) in [5]. and obtain that for Vl,, L C,
Pr (( 2,1 E. 11
2 2 6/2
< 2 expp - /Cvalue)
(33)
Pr [ Q 2 5/2
< 2 exp( -Teatued /Cratue)
(34)
Pr E Q2 2 5/2)
< 2 expl -Trvatudd /Cratue)
(35)
26

---

where Revatos [D] is the size of dataset, Cvalue E (0, ( is constant value.
Combining with Equation
32 have
Pr (B(e.))DllQ: 2 - 11, - -5/2) - -
/Caale))
(36)
Consider the extreme case in task semantics distinction, when 315. L, 4 95. By 1Q: that can be exactly partitioned under the threshold. We have
Pr (E(,,a)DD Q31l - 8/2) s sexp((nhvauue" /C
(37)
Consider all task pairs with different real semantics
M
M²
2
Pr Ji, 4 9js B( - 2,1 - 6/2)
s 3M² exp( = / /Cvalue)
(38)
To ensure confidence level of least 1) for any task ambiguity, require that the probability of the event S, satisfies:
Pr(Sa) S ny
Thus, we need to let:
3M2 exp(-nrvate 8 /Cvatue) S
Then,
Cvalue 8²
3M2
Tvaauee
log
(39)
For the distributional RL. setting, we have the estimated return distributions 2, 2, and real distribu- tions Zi, Zj. with the threshold d O. Similar Equation 21 we have the semantic ambiguity event
S, as:
E (s,ay>D (i((((s,)), d
(40)
2
E, Wi (Zi (s) d, 9a 4 9j.
(41)
where the optimal distribution Q-value, π is the optimal policy along with Z, and 2 learned distribution.
Similarly, we have:
E(DD)D Wi (2, 23) -Ex [Wr (Zi, Zi)|
(42)
s E(sDyDD [Wi 23)] EDDE [Wi(2,, 23)
(43)
+ EDDEMaaND (2, - -Ex [ (Z;, 2,)1
(44)
E W, (2,, |,)] Fy [Wi 23)| + F.r [Wi -
Wi, 23)
(45)
Following the triangle inequality of Wasserstein distance, we have:
W2,2, 2,) wi w1, 2,) + W2(Z,,2,)
(46)
27

---

Then.
[BIsa)D W2(2,, 2)] Ex [ (2,, 2y>1 2うē
Pr
(47)
[(sa)\D [ 2,1 - Exx [Wi(2,, + | 2 w2(,,
$ Pr
2
(48)
E ((sa))D (1) -Ex Ww (2,, 2,,) || + Ex [w,, 2,) + Wi 23) 12, < Pr
2)
(49)
< Pr BISS)ND Wi, [3)] - Exx [Wi (()))22)) / 5
(50)
+ Pr (B Wi [W](2 (2 Z+)
2 5) + Pr (E Wi [W](2 2))
d' 7
(51)
Since 2 is the empirical measure of Z and the estimated return is 1-dimensional, Q E [0, maxx] (Z,,,, Zz) [0, max]]. Following Theorem 1 (Hoeffding's inequality) in (5) Corollary 5.2 and Remark 5 in [1], we have the following equation where Caisty Cdista (0, ((Q max)) are constant values.
Pr (Binaay [wi(2,,
W1(2, 2
2 d/2)
< 22 exxp( dd / /Ciiss )
(52)
Pr
Ex
[W, 2,)
2 d/2
< 2 cxp( - /Cansa)
(53)
Pr Ex [Wi (2j, 27)
2 d/2) < 2 - /Caisz)
(54)
Combining with Equation
51 we can obtain the following equation where Cdist
max Cuisty Caissa)
Pr (B(La) (2., E. (Z,, - d/2) 66epl( -asap //Ciis)
(55)
In the same way, consider the extreme when Eki,4j E L. have
gs + gj. E, (W1 (W, Z,)) d. We
Pr (B(e.a)D (2, S <d/2
S 6exp( - ef/ /Caist)
(56)
Consider all task pairs with different real semantics
M
M² 2
2
Pr j. 4 Gj. (e.))=D (2,, 2,)) < d/2) < 3M² exp(-naiad /Casa)
(57)
Similarly, to ensure a confidence level of at least 17 for avoiding task ambiguity, we need
2 exp(- /Caist) S 17.
Then,
Cdist 3M2 d2
Rddist 2
log
(58)
1)
Corollary mullii-askk RL setting, to avoid task ambiguity with confidence level 1 1- TJ, learning the distribution over Q-values fewer samples than learning point estimates of Q-values when the number of tasks M is sufficiently large. Formally, Twvauue 2 Ridist where Theauee: Rddist denote the samples needed to avoid task ambiguity for value-based and distributional settings.
Proof. From Theorem 1 have
Cvatus
3M
Dyyalue
log
(59)
82
17
Cdist d
3M/
Midist
log
(60)
17
28

---

First, Q E [0, m,ax] Wi 2,) E [0, Qmazl, following Theorem (Hoeffding's inequality) in
5
Corollary 5.2 and Remark [T we obtain that the constant C satisfies:
Cvalue O(Q Cdist I <O(O
(61)
Then, we can prove that for any Q-value distribution Zi, Zz.
Wi 2,) 2 IE (Q) - E2z (Q)1
(62)
The definition of Wasserstein- I distance is
(2, 2,) = inf Exx Yl xeI((Z.,,)
Here, (T, 2z) denotes the set all joint distributions with marginals Z;, Zz.
For (Z,, 2n) т,, we have
E||X YI [E(X - Y) [x) Ez, (X)
[x, |Bx,
Ez, (X)
Thus, for any joint x.
E E XX Y Ezz (X) Ežz (X)
For the optimal joint n.
Wi (2 2,) 2 Ez, (Q) - Ez, (Q))
(63)
Thus, we must choose much smaller threshold å for point estimates than that for distribution learning, 6 S d. In some scenarios, when the mean difference of the Q function is small but the distribution difference is large, << < d.
Combining Equation 61
63 we can obtain that
Cvaluc
3M
Cdist 3M² log
log
2
(64)
82
"
d²
"
we prove that Tyyauu 2 Ridists even 7lyalue
>>
Yldss in some scenarios.
29

---

E Experimental Details
E.1 Toy Experiment
In this toy experiment based on the Minigrid environment [T we demonstrate that vanilla offline RL fails to establish the relationship between the tasks and their underlying reward functions as the number of tasks explodes.
The setup consists of 10 accessible goal positions G {gu, 91, ... 9o where g represents the set of all possible goal positions. The agent (red triangle-shapets must follow a given instruction E C to navigate to specific goal position. We simulate instructions using numerical task IDs, making L C N. We employ a random mapping F + 9 to assign each to goal position g, which simulates semantics of instructions and is hidden from the agent. With these settings, the number instructions L] by simply adjusting the set of valid task IDs and the mappings. We conduct 10 experiments, varying the number of instructions from 2 For each experiment, generate a new mapping F' and collect 1024 random trajectories as the offline dataset.
control
The agent always starts from the center of the map. For each step, the agent receives the task ID and current state as input, and the agent can choose to move forward, turn left, or turn right. A reward of 1 0.9 x (STEP_ CCUET_ /MAX_STT (_STEP) is given for success and 0 for failure. MAX S_EESPTPP/ is fixed to 12 in our toy experiment. For offline datasets, a random policy with MAX_STEP = 12 is used to generate
Tasks
Instructions
100%
Go to (1,1)
Task "0"
Success Rate
90%
to (1,2)
Task "1"
80%
to (1.1) Task "2"
DAIL
70%
COL
(1.2)
"512"
60%
CQL-double
2 2 2 2
2 2 2
10 tasks
512 instructions
Number of Instrunctions
The green flags in the map denote the accessible goals. Middle: illustration of the mapping between goal positions and instructions. Right: Average success rates over 100 evaluations for each number of instructions and 3 seeds.
Figure 10: Left:
64 trajectories for 8 fewer tasks, where is the number of tasks, and 1024 trajectories for 16 or more tasks, respectively. For all these datasets, the success rate is fixed to be 0.5.
For the observation signals in this toy experiment, Bag-of- Wordd encoding [40] and two-layer CNN. We use 1 fully connected layer to embed task IDs into task representations and two-layer MLP as the output network. We use 64 the feature size for CQL and our method, 128 for CQL-double separately. For each number of tasks, each agent trained with these 3 seeds and record the best performance among them as the performance for this number of instructions, illustrated in the Right of Figure 10
simple architecture, which combines
{0, 0.01, 0.1, 0.5, 1, over 3 seeds. We calculate the average of the results from
E.2
Offline Datasets
E.2.1
BabyAI
BabyAl [10 is a language-conditioned research platform built on MiniGrid [Л], which provides different levels of tasks equipped with varied language instructions. We choose level SynthLoc as the benchmark, which is the union of all instructions from PutNext, Open, Goto, and PickUp. The agent needs to deal with synthetic Baby Language and interact with the specified objects at the goal position. Some examples of language instructions are "put the green key behind you next to a box". "go to the red ball behind you", and "pick up green box".
Baby AI, where each room has a size of x 7, arranged in a 3 x 3 grid with total of 9 rooms. Each room may be connected to others via door, illustrated in Figure TT. The agent has field view of x front of it, with the observation size being (7 x x 3). Each grid cell in the observation contains the values (OBJECT IDX, COLOR_ IDX, STATE),
We follow the default map configuration
30

---

Figure 11: Level SynthLoc in BabyAI
all represented in a structured format. A reward of '1 0.9 (step_count/ / max_steps)" is only given for success, and in all other cases. Agents are permitted to take up to 300 steps before truncation in our setting. In SynthLoc, the total number of tasks is large, and their distribution is sparse. We
303
201
2003
2000
2000
4000
lask occumeneesss
The histogram shows the frequency distribution of tasks in 10 samples. The x-axis represents the total frequency of each task, while the y-axis indicates the number of tasks that fall within each frequency interval.
Figure 12:
reset the environment 106 times and obtain over 5500 unique instructions. From the histogram of task count (Figure 12 can be observed that most tasks only appear less than 500 times in 10 while some tasks appear over 4000 times. The distribution of tasks highlights the highly uneven distribution of tasks. We divide the task set into two subsets, designating approximately 60% of the tasks as in-distribution tasks. All trajectories in the offline dataset are collected under in-distribution instructions, while tasks encountered during testing outside this set are considered out-of oui-of-diitribution tasks.
To construct the offline dataset, collect three types of data: expert data, gathered by pre-designed bot within the environment; medium data, collected by a well-trained agent; and random data. The built-in bot has access global information accomplish every possible task with near-optimal solution. We train an IL agent following BabyAI 1.1 233, the state-of-the-ar model proposed by the original environmental authors. Trained on a dataset of 100k expert trajectories, it achieved approximately 87.9% success rate across all tasks. Random agent achieves 10.5% success rate during data collection. We conduct a high-quality dataset with 50k expert trajectories. 50k IL agent trajectories, and 25k random trajectories; a ceddium-qualig dataset with 12.5k expert trajectories, 25k agent trajectories, and 40k random trajectories. All the trajectories the dataset are generated under in-distribution instructions.
31

---

E.2.2 ALFRED
benchmarks sequential decision makinn tasks involving household activities (e.g.. cleaning, heating food) through language instructions and first-person vision, shown Figure [3] and T The dataset provides 8055 expert demonstrations with 25k human-annotated language
ALFRED 54
instructions detailing both high-level goals and sub-goal step-by-step guidance. As our work primarily focuses on low-level policy learning rather than high-level planning, we specifically concentrate on the GOTO sub-goal setting for our evaluation. locations according to instructions like "Move other side of couch on the right side of the table before the door". To simulate the presence of noisy data in real-world applications. we augment the training set with 30k random-agent trajectories, resulting in 97896 total trajectories with 53442 unique instructions across 108 household scenes.
this task set, the agent must go to specific
As for the experiment, we use the Modeling Quickstart dataset, which is recommended [54] including trajectory JSONs and pre-g rre-generated ResNet features. The ResNet features are obtained using a pre- trained ResNet-18 [2]] to extract 512 7 x 7 features from the conv5 layer, which are used as observation input during training and evaluation.
Figure 13:
Two example scenes from ALFRED.
Table 4: Instruction examples in ALFRED COTO task set.
#
Instructions
Go left and turn to the right to face the couch.
2
Turn around and make a left immediately after the toilet turn a quick left to face the side of the toilet.
3
Move other side of couch the right side of the table before the door.
4
back to your right to the fridge and open the door
5
Turn around and walk to the white stove on the right.
6
Turn to the right and go to the sink across from you.
7
Turn around and walk towards the toilet, then turn right and walk towards the door, turn left to face the counter.
8
Walk forward, then hang right and walk across the room, turn left and walk up to the chair.
For sub-goal evaluation, we follow [54] use the expert trajectory to move the agent until the start state of the tested sub-goal, and the agent takes over to operate based on the instructions and observations. Episodes with more failed actions or exceeding the MAX_STES as failures immediately. A reward of is given for success and for failure. Failed actions refer to actions that cannot be successfully executed in the current state (for example, moving against the walls or other obstacles in the room).
32 are counted
32

---

Architecture, Training, and Evaluation Details
F.1 Details of Encoders
This section describes the network architecture of we used for language and observation encoding in BabyAl and ALFRED environments. In all the experiments, all models share the same encoders, detailed as follows, unless otherwise specified.
Language Encoder For the language signals, we use the Transformer model 58 from the pre- trained CLIP network (50) for language encoding in BabyAl and ALFRED experiments, freezing the entire model during training and adding an additional fully connected layer at the end for fine-tuning.
Observation Encoder Given the differing input structures of BabyAl and ALFRED, we employ separate observation encoders.
For BabyAI, we adopt the original visual encoding framework from BabyAI 23 which integrates a Bag-of- Words embedding layer [40]. convolution backbone, and a linear layer. The BOW module first turns the structural inputs with size X 7 into 7 256 embeddings. The subsequent convolutional backbone processes these features through two sequential blocks and max- max-pooling layer: each block contains a 3 x 3 convolutional layer, followed by batch normalization and ReLU activation. The output is then processed with a 7 7 max-pooling layer. The features are then flattened and projected to 256 dimensions through a linear layer, producing a 256-dimensional vector as the observation encoder's final output.
For ALFRED, we use the original encoding framework in ALFRED 54 for all implemented methods, which contains two sequential blocks: each block contains a 1 X 1 convolutional layer, followed by batch normalization and ReLU activation. The features are then flattened and projected to 512 dimensions through linear layer, producing a 512-dimensional vector as the observation encoder's 's final output.
FiLM and Sequence Encoder We follow 23 to use FiLM 49 to fuse language and observation encodings through feature-wise affine transformations. For history encoding, all baseline methods employ a two-layer unidirectional LSTM to model temporal dependencies.
F.2
Architecture Details of DAIL
The overview architecture of DAIL is shown in Figure 14. We adopt the language and observation encoders described above to encode instructions and observations separately. To use the same sequence encoder for both trajectory-wiss semantic alignment and history encoding, modify the sequence encoder to process observation-acen action trajectories jointly and output history information zt-
The computation process of state-action value g(st, a, 11,1, is shown on the Left of Figure 14 given instruction I,, observation 8t. The outputs of the FiLM network are concatenated with the outputs of the sequence encoder, then processed through Multi-Layer Perceptron (MLP) and Softmax layer to generate the final value distribution with dimension M. For Trajectory-Wins Semantic Alignment as shown the Right of Figure 14, derive embeddings from instruction and trajectory т. The trajectory embedding is represented by the sequence model's final output IT
E3 Training Details
All models are implemented with PyTorch, and trained with batch size of 64, using the Adam optimizer 277 at a learning rate of 3c 4. All layers in the networks utilize Py PyTorch's 's default weight initialization, and the network outputs fixed-dimessie
embeddings suitable for downstream tasks.
In BabyAl experiments, all methods were trained for 50 epochs over 3 seeds. And in the ALFRED experiments, all methods
trained for 20 epochs over 3 seeds following (54).
As for DAIL, we fix a and 0.2 except for the toy experiment and ablation experiment of X.
We use VAXX - VMIN 20, M 51 in all our experiments following (29].
33

---

Instuctioo
CLIP
Transformer
Layers
CLIP
Linear
Trandormer
Sequence Encoder
LSTM//Trrrforme
Linear
FILM
Linear
History Info
Sequence
Encoder
Concat
Intrrccion embedding
Contrastive Learning
sppctory-mbbedddy
MLP
Softmex
Discrete
Fatomr
Figure 14: Overview of our algorithm. Left: Computation process of state-action value. Right:
trajectory-wii wise semantic alignment.
F.4 Baseline Details
In this work, all the baselines share the same language encoder and similar observation encoder, and only differ the decision module. In the following part, we introduce the decision modules of several baselines used in our paper:
GCBC: language-conditioned behavior cloning with all data to maximize
(BC( (........ [log me (ax (.r, [)]
(65)
BC-Z:
learning two task encodings from language and trajectory (video), and aligning them through p"q to measure the similarity. We denote (r,1) as
similarity. We use cosine distance Deas (p. g) |n]a pair of trajectory and instruction.
JBC-Z(T1 (T Es
..)....))D......... al Dog (a, Fr, fo()) (Doo))DD(Ds((g()))))))
(66)
where (.).) is the video encoder proposed by BC-Z to encode trajectory information, and fx(+) is the instruction encoder.
explicitly aligning the representations of language-conditioned tasks through contrastive learning with similarity measure (.). = Deos (f), hy (Su, (&u,g)), where y,, t are learnable parameters of the language encoder and goal encoder respectively and g the last state of trajectory, Positive data (r+, (+) PD(-.) are uniformly sampled from the dataset, with st, g being the start and end state of ,+ respectively. Negative examples 8 . are the start state and end state of a randomly sampled trajectory from the dataset. Negative instruction 1- Pr(-) is the instruction of another random trajectory. For each positive example, i negative examples are sampled noted as {s, 94 ===1 and {1, }
GRIF:
exp(C(st /+,/+//T)
Llang-sgoal(,, v/) = log
Σ exp(C(ct,, 1+),))/+)++))))) - exp(C(e, .9. (1)/T/
(67)
exp(C(st Exp((sst,tt.t*1) g+,1+)/7)
Lgoal (4,, v)
log
exp(C(ct,gt,
exp(C((t .t (111)
where the temperature parameter. We employ the last state sT the trajectory as the goal state: (y g) = has (Sp, ST). Then the policy network is trained with behavior cloning by maximizing likelihood actions:
Joo) = D)- [log (a (())
(68)
We the GRIF(Joint) setting to train the model 42
34

---

CQL: we implement CQL (w/o distributional) based on DDPG,
JeQL = E (s D. [Q To( ().0)
(69)
where Q function learned by minimizing:
(CQL(H) (a /a [(Qolzt ar - 15 Qo(ar. ar, 1))
(70)
( [lo
exp(Qo(21, a,l)) (an (Q0 (2 1)]]
where 2 is the behavior policy, 8 is the Bellman operator, and the balance ratio in experiment.
IQL: training additional value network and extracting policy through advantage weighted regression.
(v(b)) D. (Q ar 1) Vp(
(Ø( = (ee [Ir: + + V (x - Qo az, 1) (71)
[(T)) 11- (etp:///eeeeppo0((ee a, 0r,
- (x I)) log Ta (a/xr; 0)]
The expectile 0.7, 3 5. We follow the authors' suggestions and subtract from the reward it equals 0.
To follow the original IL setting and simplicity, merely use observation without action information in the history state encoding in BC and IQL. Our primary experiments show that it has little impact on the results. We also investigate recent approaches of Iinguag--condtionee R3M [44], but they perform poorly in prior experiments, so only BC-Z is chosen as the representative in the final baselines.
such as LLfP (77 and
35

---

G Visualization Supplementary Results
We apply t-SNE visualization in different algorithms and task types to show that our method substan- tially improves task representation on offline language-conditioned RL. Beyond the main text, here are some supplementary visualization results.
CQL
DAIL wio Distributional
DAIL Alignment
DAIL
PickUp, box Pickup, ball Goto, bell Figure 15: The t-SNE visualization of instructions from various tasks in BabyAl Al for different algorithms. The figure distinguishes between different task categories (e.g., PutNext) and target object types (e.g., box), using marker colors and shapes to represent each separately.
Putteext door
PutNext, box
Open, door door
Pickup, key
PutNest PutNext,
Goto, box
GCBC w/o Alignment
GCBC with Alignment
IQL w/o Alignment
IQL with Alignment
Open, Picktip, PickUp, Puthicst, PutNiext Goto, door
Picklpp key
Goto, ball Goto, key Figure 16: The t-SNE visualization of instructions from various tasks in Baby Al for different algorithms. The figure distinguishes between different task categories (e.g.. PutNext) and target object types (e.g.. box), using marker colors and shapes to represent each separately.
Puttiext ball
Goto, box
Overall task representation As introduced Section 5 in BabyAI SynthLoc tasks, there are four main categories of tasks: Goto, PickUp, PutNext, and Open. We sample tasks from all four categories to visualize in Figure [5) Our method demonstrates superior task embedding capabilities compared other approaches in the Goto, PickUp, and Ofen-effectively reducing confusion as highlighted by the red circles. Some degree of confusion remains inevitable in the PutNext tasks, due to their complexity and variability indicated by the green circles).
We also visualize representations from GCBC and IQL (with and w/o alignment) in Figure 16 GCBC shows reliable task representation, but confuses tasks from Goto and PickUp, "Open, door" eand "Goto, door" where their embeddings are tightly gathered into small clusters. Our further results in Figure 18 show that each cluster represents a specific color. Similarly, alignment significantly eases this issue by separating each instruction. IQL shows similar results to CQL. while alignment has more significant influence CQL. This result explains why simple GCBC shows comparable performance in our settings while vanilla offline RL fails due to confusion in task encoding.
Task PickUp and Goto We take closer look at tasks in GoTo and PickUp that only have one target object. The task representations of DAIL and ablation algorithms are shown in Figure 17 and GCBC and IQL (with and w/o alignment) are shown in Figure [8] The color of the markers indicates that of the target objects other than black markers. As illustrated in Figure / our method further subdivides tasks (for example, sand into smaller clusters. Upon closer observation, these smaller clusters correspond to different target object colors. Similarly, CQL performs poorly in object color recognition, while distributional representation and semantic alignment substantially help task discrimination. As illustrated by the red circles, when the target object type is key, only our method
36

---

CQL
DAIL wio Distributional
DAIL w/o Alignment
DAIL
blue, Figure 17: The 1-SNE visualization of instructions from the same task categories with more detailed distinction. The figure distinguishes between different target object types (c.g., door) and target object colors (e.g., blue), using marker colors and shapes to represent each separately, For example, "go to the red door" corresponds to e, "go to red ball behind you" corresponds to "pick up the ball in front of you" corresponds *.
red, blue, key
red, key
GCBC Alignment
GCBC with Alignment
IQL Alignment
IQL with Alignment
blue, door red, door blue, key Figure 18: The t-SNE visualization of instructions from the same task categories (PickUj and Goto) with more detailed distinction. The figure distinguishes between different target object types (e.g., door) and target object colors (e.g., blue), using marker colors and shapes to represent each separately. The legend has the same meaning Figure I7
red, key blue, box red, box blue, ball
red, ball
succeeds in separating embeddings of different target colors while forming clusters for targets of the same color. In contrast, other methods tend to produce entangled representations, leading to less distinguishable task embeddings.
GCBC yields strong results this scenario, whereas IQL relatively performs poorly. However, excessive overlap in GCBC suggests a confusion between the PickUp and Goto tasks. IQL can distinguish between different types of targets (different shapes of markers) but fails in differentiating target colors (for example, ...). After our alignment method was applied, its performance significantly improved.
Representation with instruction texts We add some instruction texts to the representation map to better demonstrate the language instructions' representation results. We draw around 300 instructions from BabyAI AI SynthLoc level to visualize and sample around 20 tasks and include their original text in the figure, displayed to the right the corresponding marker. Figure 19 shows the detailed representation result of vanilla CQL, and Figure shows that of method DAIL. The red dashed circles highlight the "Goto, door and "Open, door +" tasks. Our method DAIL. successfully separates the two task categories and further organizes them into clusters (based colors, Figure 17p while CQL fails. Similarly, the green dashed circles denote the "PickUp, key e" and "Goto, key tasks. While CQL fails disentangle the Pi PickUp and Goto task types, our method achieves clear separation between them.
37

---

Goto, ball
key
tig the Lefani
Goto, door
Goto, box
rnd Key hort you
PickUp,
PickUp, key
ap betaxd you
PickUp, box Open,
beidd
kyy of
pick OPEE box
if tebend
bue olyr
impo purpletiox
Figure 19: The t-SNE visualization of instructions with text annotations from Open, Goto, PickUp in BabyAl for CQL. The figure distinguishes between different task categories (e.g., PickUp) and target object types (e.g., box), using marker colors and shapes to represent each separately.
Goto, ball
Goto, key
Goto, door
Goto, box
PickUp,
#Pcck up ball behird
PickUp, key
PickUp, box
gggy key nore you
Open,
So to tngas your right
your right
the ball
Figure 20: The t-SNE visualization instructions with text annotations from Open, Goto, PickUp in BabyAl for our method. The figure distinguishes between different task categories (e.g., PickUp) and target object types (e.g., box), using marker colors and shapes to represent each separately.
38

---

H
Additional Results
Results on the hight-quality dataset BabyAI The success rates of in-distribution and out- of-distribution tasks on the high-quality dataset are shown in Table 5 in both in-distribution and -uttr--distribution tasks. Our method demonstrates significant advantages over other approaches on out-of-distribution tasks, particularly achieving substantial performance improvements on complex tasks such as PutNext compared to the baseline RL method CQL (49.1% 1% vs. 27.6%). Vanilla offline RL algorithms like CQL and IQL underperform compared to imitation learning methods like GCBC, which we attribute to the adverse impact of task ambiguity on RL-based approaches as discussed in Theorem [ On the other hand, modified algorithms designed for language-conditionel conditioned IL (BC-Z and GRIF) perform poorly under our setting. This is primarily because their contrastive learning objectives
robust in the presence of noisy suboptimal data. In contrast, alignment-bane -aase approach, built on an offline RL framework, maintains strong performance.
Table 5: Success rate of in-distribution tasks and out-of-distribution Baby Al tasks, Each score is evaluated over seeds.
Algorithm
Open
Goto
PickUp
PutNext
All
In Distribution
GCBC
96.9=0.8
91.8+1.1
85.6+0.0
27.6#3.3
79.1+1.3
BC-Z
96.3+0.7
77.5#1.0
49.9*4.4
14.2+0.7
6..0+1.8
GRIF
96.6+0.8
89.4+2.5
87.6+0.1
27.7+3.7
78.642.5
IQL
98.2+0.4
87.9+1.4
73.7#1.1
26.243.5
75.2+0.7
CQL
98.7+0.3
92.240.9
83.8+1.7
25.6+2.5
78.1
DAIL (ours)
97.2+0.2
96.5+11.4
94.9+1.4
57.940.9
89.2+0.5
Out of Distribution
GCBC
94.4+2.5
90.3+1.6
78.4+2.1
17.4
74.1+0.7
BC-Z
93.7=1.0
76.9+3.0
45.4+1.5
11.243.3
57.9+1.8
GRIF
95.9*1.7
88.842.6
75.6+3.9
22.5+2.7
71.242.6
IQL
98.0=0.4
86.1+1.2
70.4+3.6
21.4+3.1
69.742.3
CQL
98.8+0.5
88.942.1
71.9+2.2
27.6+0.8
72.640.4
DAIL (ours)
99.0+0.2
91.3+11.0
87.6+2.0
49.1
81.7+1..
Results on the medium-quality dataset of BabyAl The success rates of in-distribution and out-of- distribution tasks the medium-quality dataset are shown in Table Due to the lower proportion of successful trajectories, learning in this dataset is more challenging. As a result, all methods show significant decline in performance compared results on the high-quality dataset (Table I. However, our method still achieves optimal results, especially in the PutNext task category.
Table 6: Success rate of on the medium-quality dataset. Each score is evaluated over 3 seeds.
In Distribution
Out of Distribution
Algorithm
PutNext
All
PutNext
All
GCBC
15.6+1.5
58.0+2.2
10.
52.6+2.3
BC-Z
4,4+1.0
54.2+0.2
5,0+1.6
49.7+1.4
GRIF
7,0+1.5
61.4+0.2
4.9+2.5
54.740.6
IQL
17.742.8
67.3+0.5
12.3+0.9
61.440.0
CQL
13.5+1.8
69.2+0.7
14.5+1.9
63.4+11.
Ours
32.812.7
81.3+0.7
26.420.6
73.740.6
Ablation components
To study the contribution each component our learning framework, conduct the following ablation study. We compare the performance of algorithms that only
39

---

Table 7: Ablation results of AUC on the high-quality dataset. Each score is evaluated over seeds.
Algorithm
10
20
CQL
38,385.34853.5
81,876.048488
DAIL w/o Alignment DAIL w/o Distributional DAIL
40,374.74601.3
15,247.1+540.2 1+540.2
40,752.94248.7
85,416.54543.6
42,370..:+404.88
88,450.5+291.6
apply trajectory-wii sajectory-wiie alignment distributional language-guides policy alone with method on SynthLoc. The experimental results in Figure[2] show that both modules significantly improve the performance over vanilla CQL on in-distribution and both components can achieve the best performance compared to other approaches.
tasks. Further, combining
We further evaluate the sample efficiency of each method by calculating the Area Under the Curve (AUC) for the success rates of their learning curves, and present the AUC comparisons of in- distribution learning curves at the key milestones of 10 and 20 epochs in Table Statistical analysis confirms that at 20 epochs, DAIL holds a significant lead over the other three ablation models, as demonstrated by Kolmogorov-Smirov tests on the AUC results (p 0.004).
COL
DAIL w/o Distributional
DAIL we Alignment
DAIL
GoTo
Open
Pickup
PutNext
Successs Rate
0.8
Successs Rate
Successs Rate
Success Rate
0.8
0.4-
0.1
0.4
0.4
0.0
08
80
00
06
Training Steps (x10*)
Training Stops (x10²)
Training Steps (x110*)
Training Steps (x110))
Figure 21: Ablation experiments on Baby tasks. The success rates are evaluated over 3 seeds.
Quantitative measure of clustering. To quantitatively demonstrate the impact of different compo- nents in DAIL on representation clustering, we measure the clustering quality with the Silhouette score[52], and present the result in Table The labels required to calculate the score are defined as follows: two instructions share the same label if they require the agent to perform the same action on the same kind of object with the same color as the final target object. The distance between language embeddings is measured using cosine distance.
The results demonstrate that in the All-task setting, the clustering metric of CQL even exhibits negative values, indicating pronounced task confusion. In contrast, both proposed methods evidently improve clustering performance.
Table 8: Silhouette score of in-distribution BabyAl AI tasks.
Algorithm
All
PutNext
CQL
-0.030+0.006 0.024+0.016
0.004+0.007
DAIL w/o Alignment
0.048+0.008
DAIL w/o Distributional
0.110#0.019
0.088+0.020
DAIL
0.127=0.013
0.10740.012
Ablation of a In Equation 13 a is the weight of the CQL loss. The ablation done on the high-quality dataset in Al tasks with various The experimental results in that a wide range of a values (from 0.5 to 2) yield comparable performance, which drops off the 5). We therefore recommend setting between 0.5 and 2.
extremes (a = 0.2 and
40

---

Table 9: Ablation experimental results
Each score is evaluated over 3 seeds.
In Distribution
Out of Distribution
PutNext
All
PutNext
All
0.2
44.0+3.6
83.7=0.7
40.8+5.1
78.4+1.7
0.5
57.4+2.2
88.1+0.3
50.7+2.7
83.1+0.8
56.2+3.6
87.7#1.3
50.8+3.7
84.1+0.7
57.9+0.9
89.2+0.5
49.1+1.8
81.7+1.3
5
44.8+3.2
85.2+0.2
37.1+5.1
77.4+1.7
35.7+2.3 10
82.8+1.1
39.144.4
77.2+0.7
41

---

I Demonstration Trajectories in ALFRED
We present extended trajectory visualizations of DAIL's task execution in the ALFRED benchmark, illustrating its semantic comprehension and generalization capabilities.
Turn left, take a right and walk to the black tridge on the left
Taka leddle right,
right turn right go the step
Turn around and werk alread, then
roam.
Figure 22: Extended trajectories of DAIL in ALFRED validation tasks.
42


Reviews:
[{"review_id": "waR7dZ655u", "reviewer": "Reviewer_gNoU", "summary": "The paper addresses the task ambiguity problem in language-conditioned RL tasks: proximity in the language instruction space does not necessarily imply task similarity. It introduces DAIL, a method that combines distributional RL with trajectory-based contrastive learning of task representations. They show the efficacy of their method through the BabyAI and ALFRED benchmark tasks.", "questions": "1. How does this work fit into the prior work in RL that studies learning task representations?\n2. Could the authors provide a justification for Definition 1?\n3. How tight are the sample complexity upper bounds? Are gains only in constants inside the log sufficient? Could one use a different concentration inequality to arrive at better constants?", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- Task ambiguity is a problem very relevant for the RL community.\n\n**Weaknesses**\n- Language-conditioned RL is an instantiation of the contextual MDP setting [1], and prior work has studied learning useful task representations in this setting already [2, 3, 4, 5]. Surprisingly, this paper conveniently ignores these works and does not reference them at all. \n- The trajectory-based contrastive learning of task representations is already well studied in the community [2, 3, 4, 5].\n- Definition 1 is stated without sufficient justification. Why should similar tasks have similar Q-values?\n- It is unclear how the authors draw the conclusion that distributional RL achieves better sample complexity for task disambiguation than value-based RL. Since the gains are not in the exponents but only in the constants, how do the authors arrive at the conclusion? The relation between C_value and C_dist should dictate the final conclusion.\n- Overall, the method appears to be incremental and as such does not meet the standards for NeurIPS.\n\n[1] Hallak et al. Contextual Markov Decision Processes, arXiv, 2015.\n\n[2] Fu et al. Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning, AAAI 2020.\n\n[3] Agarwal et al. Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning, ICLR 2021.\n\n[4] Li et al. FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization, ICLR 2021.\n\n[5] Mahajan et al. Learning Embeddings for Sequential Tasks Using Population of Agents, IJCAI 2024.", "was_revised": true, "final_justification": "The authors clarified the positioning of their work during the rebuttal.", "rebuttal": "Dear Reviewer,\n\nThank you for your valuable comments. We address the points you raised in the following, and we hope our responses can resolve your concerns.\n\n---\n\n**W1, W2, and Q1: Lack of references to and comparisons with existing research on task representations in RL.**\n\n**A for W1, W2, and Q1:**\n\nThanks for pointing out the relevant papers—these references are indeed valuable. We have studied all and reproduced several of the papers as the reviewer suggested, and our analysis and experimental results are summarized below:\n\n- **LCMDP**: While language-conditioned MDP can be viewed as a special case of contextual MDP, the key challenge differs a lot. LCMDP is mainly obstructed by the complexity and diversity of language instructions, which makes it difficult for the agent to ground them to the intended task goals correctly.\n\n- **Related works:** In our original paper, the Related Work section (Appendix A) focused on task representation of goal-conditioned reinforcement learning.\nWe incorporate an extra discussion of the papers and related research that the reviewer mentioned in the revised version.\n\n- **Supplementary experiments：** Some papers the reviewer mentioned have a different research focus from our paper. They contribute to learning implicit task information from trajectories, while task information is explicitly provided through instructions in language-conditioned RL. As a result, the improvements of these algorithms are minor, and some cannot even be reproduced due to the experimental setting. We provide the extra experimental results and analysis as follows:\n\n**Table 1**: Performance of in-distribution BabyAI tasks.\n\n|   Algorithm   | Open | Goto | PickUp | PutNext | All |\n|:-------------:|:----:|:----:|:------:|:-------:|:---:|\n|      CCM      | 94.9±4.5 | 88.3±2.6 | 83.0±2.4 | 29.9±1.1 | 78.4±1.8 |\n|     FOCAL     | 98.6±0.3 | 90.5±2.6| 87.5±1.6  |  37.7±0.9 |81.8±0.8  |\n|      CSRO     | 98.0±0.5|93.1±1.6 | 87.6±1.5 | 36.7±1.7 | 82.2±1.5 |\n|    UNICORN    | 97.3±0.2|90.4±0.4|84.4±1.8|35.3±0.7|80.3±0.7|\n\n**Table 2**: Performance of out-of-distribution BabyAI tasks.\n\n|   Algorithm   | Open | Goto | PickUp | PutNext | All |\n|:-------------:|:----:|:----:|:------:|:-------:|:---:|\n|      CCM      | 98.6±1.2 | 87.5±1.3 | 68.0±4.2 | 26.1±4.5 | 69.5±0.5 |\n|     FOCAL     | 98.2±0.4 | 86.9±2.2 | 74.3±5.7| 27.7±0.7|72.5±2.8|\n|      CSRO     | 98.8±0.2 | 89.8±1.6 | 74.5±2.1 | 26.1±1.8 | 72.5±0.8 |\n|    UNICORN    | 99.0±0.5 | 88.0±3.6 | 73.3±4.6 | 29.9±1.2 | 73.5±2.4 |\n\n- CCM[1]: It introduces contrastive learning methods to enhance transition encoding, and we adapt it to language instruction encoding. The results indicate that this method provides only marginal improvement over CQL, primarily because it overlooks the relationship between language instructions and trajectories.\n\n- PSM[2]: This paper focuses on optimal policy transfer, which relies on learning state representations in similar scenarios and is limited to a single or a small number of tasks. In contrast, our work considers a partially observable environment that varies across runs, with diverse and abundant tasks. As a result, it is not applicable to our setting.\n\n- FOCAL[3]: It introduces distance metrics to separate different trajectory embeddings. We adapt it to language embedding, which shows some improvements on CQL. However, it still fails to account for the relationship between language instructions and trajectories, and thus does not fundamentally resolve the problem of task ambiguity.\n\n- [4]: This paper leverages the idea that two tasks are similar if observing an agent’s performance on one task reduces the uncertainty about its performance on the other. We did not implement it because the task similarity estimate needs rollouts from the environment, which is not available in offline settings.\n\n- CSRO[5] and UNICORN[6]: We adapt the task representation methods from two state-of-the-art meta-RL approaches to language-conditioned setting and conduct reproduction experiments. Their performance was comparable to that of FOCAL.\n\n---\n\n**W3 and Q2: Definition 1 is stated without sufficient justification.**\n\n**A for W3 and Q2:**\n\nFor language-conditioned policy $\\pi(a|s,l)$, when task representations are similar, the corresponding policies should also exhibit similar behaviors. This not only helps prevent abrupt behavioral shifts but is also an important condition for designing robust control systems[8]. Conversely, if two different tasks are mapped to nearby positions in the task representation space, the agent may fail to distinguish between them, potentially resulting in incorrect behaviors. \n\nMoreover, given the complexity and vast number of language instructions, estimation errors in the Q-function can easily propagate backward and cause confusion in the task embeddings. To quantify this phenomenon, we introduce Definition 1.\n\n\nConsequently, in Definition 1, the \"similarity\" between tasks specifically refers to the similarity of their true Q-values. \n\n---\n\n**W4: Theoretical analysis of sample complexity upper bounds.**\n\n**A for W4:**\n\nAlthough the sample efficiency gains are not in the exponents, our main focus is on the difference between the predefined task discrimination threshold of value-based ($\\delta$) and distributional-based RL ($d$). Following the proof in Appendix D from our paper, we obtain that $\\delta \\leq d$, when the mean difference of the Q function is small but the distribution difference is large, $\\delta<<d$.\n\nWe also prove that both $C_{\\text{value}}, C_{\\text{dist}}\\in (0, \\mathcal{O}(Q_{\\max}^2)]$ are nearly identical in terms of magnitude. In particular, since $Q_{\\max}=1$ in most language-conditioned scenarios, their impact on the overall sample efficiency becomes even less.\n\n**Q3: Theoretical problems**\n\n**A for Q3:**\n\n- **Theoretical analysis:** The sample complexity upper bounds are quite tight. Our gains are not in constants inside the log sufficient but in the denominator ($\\delta$ and $d$).\nWe believe that even others use different concentration inequalities to achieve better bounds, they can only achieve smaller constant coefficients [9], which have little effect while the index of other variables remains the same.\n\n- **Experiments:** Moreover, we have computed the AUC(Area-Under-the-Curve) for the learning curves and statistical significance tests on these AUC results at 20 epochs. We present the AUC comparisons of ID learning curves at the key milestones of 10 and 20 epochs below. \n\n    **Table 3**: AUC of in-distribution BabyAI tasks. Each score is evaluated over 3 seeds.\n    \n    |Algorithm|10(↑)|20(↑)|\n    |--|--|--|\n    |CQL|38,385.3±853.5|81,876.0±848.0|\n    |DAIL w/o Alignment|40,374.7±601.3|85,247.1±540.2|\n    |DAIL w/o Distributional|40,752.9±248.7|85,416.5±543.6|\n    |DAIL|**42,370.6±404.8**|**88,450.5±291.6**|\n    \n    The statistical significance tests show that, at the $\\alpha=0.05$ significance level, paired t-tests demonstrate statistically significant performance improvements for the distributional component (p=0.0342). Furthermore, DAIL yields highly significant enhancements (p=0.002). The above results indicate that the distributional component indeed contributes to achieving higher sample efficiency.\n\n---\n\n**W5: The method appears to be incremental.**\n\n**A for W5:**\n\nWe respectfully disagree that our method appears to be incremental for the following reasons:\n\n- **Theory**. To the best of our knowledge, this work is the first to focus on the task ambiguity problem inherent in language instruction learning. We clearly define this problem within the language-conditioned RL framework and empirically demonstrate its prevalence in existing RL methods through experimental analysis (Figure 2 and the results for IQL and CQL in Table 1).\n\n- **Method**. To address this problem, we attempt to tackle it from two perspectives: policy learning and representation learning.\n  * For policy learning, we prove, for the first time, that distributional reinforcement learning methods can effectively enhance the agent's ability to distinguish tasks, and apply this approach to language-conditioned tasks.\n  * For representation learning, we propose Trajectory-Wise Alignment, which leverages the correspondence between language instructions and trajectories unique to language-conditioned tasks. By introducing self-supervised constraints, this approach directly enhances the distinguishability of different tasks in the language embedding space.\n\n- **Experiment results**. Comprehensive experiments validate the substantial performance gains achieved by DAIL (Table 6 and Figure 20). These significant gains also underscore the critical importance of addressing the task ambiguity problem in LCMDP, as we have claimed.\n\nThanks again for your timely and valuable comments. We sincerely hope that our response has cleared the concern you had regarding our problem settings and novelty, and we are looking forward to more discussions.\n\n---\n\n**Reference:**\n\n[1] Fu et al. Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning, AAAI 2020.\n\n[2] Agarwal et al. Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning, ICLR 2021.\n\n[3] Li et al. FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization, ICLR 2021.\n\n[4] Mahajan et al. Learning Embeddings for Sequential Tasks Using Population of Agents, IJCAI 2024.\n\n[5] Gao, Yunkai, et al. Context shift reduction for offline meta-reinforcement learning, NeurIPS 2023.\n\n[6] Li, Lanqing, et al. Towards an information theoretic framework of context-based offline meta-reinforcement learning, NeurIPS 2024.\n\n[7] Weng T W, Zhang H, Chen P Y, et al. Evaluating the robustness of neural networks: An extreme value theory approach. ICLR 2018.\n\n[8] Boucheron, et al. Concentration inequalities. 2003. 208-240."}, {"review_id": "E4RcSQvB4u", "reviewer": "Reviewer_ZEoJ", "summary": "The paper proposes a better algorithm for language-conditioned offline RL tasks. It identifies one key challenge in such kind of RL tasks, that is to handle the vast space of language instructions and potential ambiguity in its relation to the final goal. To address this issue, the proposed method DIAL build upon CQL with two additional change\n1. distributed value function to handle the ambiguity more directly.\n2. cross similarity to enhance the representation.\nThe experiment shows better performance than baselines with the two addons.", "questions": "1. The difference of GoTo and Open can be differs if one use some exisitng pre-trained models like (RT-1). It is unclear if we use pre-trained components that it will solve the ambiguity problem directly.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The proposed new loss are simple and easy to understand.\n2. The experiments is conducted on a large set of benchmarks set which is comprehensive.\n\nWeakness.\n1. The proposed losses can be found origin from existing literature (distributed RL, and contrastive learning for similarity metrics), which reduces the novelty.\n2. I don't understand the confusion of the language embedding relates to the final outcome of the policy. The t-sne is a lower dimension space, the points can still be different in high dimension. Moreover, go to door and open door shares a lot in common in going to the door first, which should have similar embedding. To my knowledge, as long as the Q is correct, it should learn each task individually. Does the benefit comes from distributed value itself (like in single-task RL policy literacture) rather than the latent embedding?", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThanks for finding our paper easy to understand and empirically comprehensive. The points you raised are explained in the following.\n\n---\n\n**W1: Proposed losses can be found origin from the existing literature**\n\n**A for W1**:\n\nWe appreciate your perspective that proposed losses can be found originating from existing literature. While we agree with this perspective, we wish to emphasize that our core contribution lies not in refining specific losses but rather in identifying critical challenges inherent to language instruction tasks and proposing targeted solutions to address them. \n\nTo the best of our knowledge, this work is the first to focus on the task ambiguity problem inherent in language instruction learning. We clearly define this problem within the language-conditioned RL framework and empirically demonstrate its prevalence in existing RL methods through experimental analysis (Figure 2 and the results for IQL and CQL in Table 1).\n\nTo address this problem, we attempt to tackle it from two perspectives: policy learning and representation learning.\n- For policy learning, we prove, for the first time, that distributional reinforcement learning methods can effectively enhance the agent's ability to distinguish tasks, and apply this approach to language-conditioned tasks.\n- For representation learning, we propose Trajectory-Wise Alignment, which leverages the correspondence between language instructions and trajectories unique to language-conditioned tasks. By introducing self-supervised constraints, this approach directly enhances the distinguishability of different tasks in the language embedding.\n\nComprehensive experiments validate the substantial performance gains achieved by DAIL (Table 6 and Figure 20). These significant gains also underscore the critical importance of addressing the task ambiguity problem as we have claimed.\n\n---\n\n**W2.1: Confusion of the language embedding relates to policy**\n\n**A for W2.1**: \n\nWe claim that the confusion of language embeddings is closely related to the policy for the following reasons.\n\nLanguage embeddings provide the only information to distinguish between different tasks in Language-conditioned MDP settings. Therefore, when two distinct instructions are mapped to similar language embeddings, the agent struggles to differentiate between them, potentially leading to erroneous behaviors deviating from the original instructions.\n\nOn the other hand, when language embeddings are similar, policy behaviors should also exhibit similarity in robust control systems (so-called Lipschitz continuity[1]). As a result, when embeddings exhibit ambiguity, this confusion propagates into the agent's behavioral patterns.\n\nBesides, our ablation studies (Table 6 and Figure 20) also support our claim that policies with more discernible language embeddings have better performances.\n\n**W2.2: The points can still be different in high dimensions**\n\n**A for W2.2**:\n\nTo address your concern, we quantitatively measure the clustering quality with the Silhouette score[2] using language embeddings before dimensionality reduction. We present the scores for all tasks and PutNext sub-tasks below.\n\n**Table 1**: Silhouette score of in-distribution BabyAI tasks.\n\n|Algorithm|All(↑)|PutNext(↑)|\n|--|--|--|\n|CQL|-0.030±0.005|0.004±0.007|\n|DAIL w/o Alignment|0.024±0.016|0.048±0.008|\n|DAIL w/o Distributional|0.110±0.019|0.088±0.020|\n|DAIL|**0.127±0.013**|**0.107±0.012**|\n\nThe results demonstrate that both proposed improvements contribute to enhancing the performance of language embeddings in high dimensions.\n\n**W2.3: Similar behaviors should have similar embeddings**\n\n**A for W2.3**:\n\nYes, we agree. The results of our visualization also match your intuition. For example, in the third figure of Figure 7, 'Goto the door' is significantly closer to \"Open the door\" rather than to 'Goto the ball' since the former two share a lot more in behaviors.\n\n**W2.4: Does the benefit come from the distributed value itself rather than the latent embedding?**\n\n**A for W2.4**:\n\nOur partial experimental results support that the advantages of distributional RL are not solely due to the distributional value itself: \n- In the toy experiment, when the number of tasks is low, the performance gap between the distributional method and CQL is not substantial. However, as the number of tasks increases, its performance advantage becomes pronounced. This indicates that its benefits extend beyond the distributional value itself.\n- Comparing the results of CQL and DAIL w/o Alignment shown in Figure 7 (where the latter figure represents results applying the distributional method on top of CQL), the distributional method clearly enhances the discriminative power of the language embeddings. This improvement is particularly evident in tasks within the \"Pickup/Goto the box\" and \"Pickup/Goto the key\" groups, where tasks involving key versus box are distinctly separable.\"\n\n---\n\n**Q1: Would pre-trained components solve the problem directly?**\n\n**A for Q1**: \n\nWhile pre-trained components can alleviate this issue, they cannot directly solve it. To illustrate this claim, we employ a more recent language encoder [3] (CLIP+LLM2CLIP version), which leverages large language models to enhance the training of linguistic encoders, thereby achieving superior capabilities in language understanding and discrimination. Using this encoder, we retrain our method along with CQL on the BabyAI high-quality dataset. Notably, we introduce an additional baseline, CQL-LLM2CLIP, which directly feeds the LLM2CLIP-embeddings as language embeddings into CQL's policy network without fine-tuning the language encoder. The experimental results are as follows:\n\n**Table 2**: Success rate of BabyAI tasks with LLM2CLIP.\n\n|Algorithm|ID PutNext|ID All|OOD PutNext|OOD ALL|\n|--|--|--|--|--|\n|CQL|25.9±2.1|78.8±0.9|22.8±2.3|70.2±0.5|\n|CQL-LLM2CLIP|36.2±1.2|81.5±0.5|28.7±2.7|70.3±1.4|\n|DAIL(ours)|**58.7±1.5**|**89.1±0.6**|**55.7±1.5**|**84.7±0.6**|\n\nThe experimental results demonstrate that:\n\n- Adopting a stronger pre-trained component shows measurable gains for both CQL-LLM2CLIP and DAIL, confirming that a better pre-trained component can help alleviate the ambiguity problem. \n\n- The CQL vs. CQL-LLM2CLIP comparison demonstrates that despite possessing a powerful pre-trained component, CQL fails to effectively leverage these pre-trained embeddings, with its performance slightly underperforming even the non-finetuned version.\n\nAll the above experimental results substantiate our claim that pre-trained components cannot fully resolve the ambiguity problem, emphasizing the critical importance of effectively learning frameworks, precisely the focus of our work.\n\nThanks again for the valuable comments. We hope our response has cleared your concern. We are looking forward to more discussions.\n\n---\n\n**References:**\n\n[1] Evaluating the robustness of neural networks: An extreme value theory approach[J]. ICLR, 2018.\n\n[2] Silhouettes: A Graphical Aid to the Interpretation and Validation of Cluster Analysis, Journal of computational and applied mathematics, 1987.\n\n[3] https://github.com/microsoft/LLM2CLIP"}, {"review_id": "FWoHrNelr7", "reviewer": "Reviewer_SRtx", "summary": "This paper proposes a novel method called DAIL to address the task ambiguity problem in language-conditioned reinforcement learning. DAIL integrates two key components: a distributional policy that enhances task discrimination by estimating full return distributions, and a semantic alignment module that optimizes task representations by maximizing mutual information between trajectories and language instructions. Extensive experiments conducted on both structured observation (BabyAI) and visual navigation (ALFRED) benchmarks validate DAIL's effectiveness, with visualization analyses revealing its ability to learn task representations and exhibit generalization to out-of-distribution tasks.", "questions": "1. The current experimental setup appears to inversely model the task ambiguity problem - while true task ambiguity involves multiple instructions mapping to a single task, the experiments conversely examine multiple tasks sharing identical instructions. This fundamental mismatch raises questions about whether the evaluation properly addresses the core challenge the paper aims to solve.   \n2. The similar clustering patterns observed between DAIL and its ablated variant (w/o Distributional) in Figure 7 may potentially undermine the claimed contribution of the distributional component.  \n3. The paper may lack ablation studies on the α coefficient in the CQL loss term?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths：\n1. The paper is well-written, with a logical flow and accessible language that enhances readability.\n2. It introduces and formally defines the task ambiguity problem in language-conditioned reinforcement learning, providing a fresh perspective on this challenge.\n3. Empirical results support the claims—DAIL outperforms established baselines across all reported tasks.\n\nWeaknesses：\n1. The paper inadequately justifies the practical significance and broader implications of addressing the task ambiguity problem, failing to establish its real-world relevance or potential impact on the field.\n2. The selected baseline methods are insufficient for rigorous comparison, as they: (a) fail to include state-of-the-art offline RL approaches, and (b) omit relevant multi-task/meta-RL methods (e.g., CSRO[1] and UNICORN[2]) that could better demonstrate the proposed method's advantages in handling task ambiguity.\n3. A fundamental disconnect exists between the proposed technical solution and the core challenges stemming from task ambiguity. Moreover, the design of Trajectory-Wise Semantic Alignment lacks sufficient explanation and empirical validation. \n\n[1] Gao, Yunkai, et al. \"Context shift reduction for offline meta-reinforcement learning.\"   \n[2] Li, Lanqing, et al. \"Towards an information theoretic framework of context-based offline meta-reinforcement learning.\"", "was_revised": true, "final_justification": "Thank you for the author's reply. At present, I think the my main concern has been resolved, so I have raised the score.", "rebuttal": "Dear Reviewer,\n\nThanks for finding our work fresh and accessible. We hope the following statement clears your concern.\n\n---\n\n**W1: Inadequately justifying the practical significance**\n\n**A for W1:**\nLanguage-conditioned RL holds significant practical importance. There is a growing expectation that a single agent should be able to accomplish various tasks specified by human instructions[1].\n\nWe present that task ambiguity is a realistic problem in language-conditioned RL. It is mainly obstructed by the complexity and diversity of language instructions. \n\nTo make our methods practical in the real world, we choose ALFRED as one of benchmarks, which includes sequential decision-making tasks involving household activities through language instructions and first-person vision. ALFRED is widely used for testing visual navigation and embodied agents[2], our experiments on ALFRED show high real-world relevance.\n\n---\n\n**W2: Offline RL baselines and multi-task/meta-RL methods**\n\n**A for W2:**\n\nAs suggested, we conduct extra experiments on state-of-the-art offline RL (MCQ[3], TD3+BC[4]) and meta RL methods (CCM[5], FOCAL[6], CSRO[7], UNICORN[8]). The experimental results of training from the high-quality dataset of BabyAI are as follows:\n\n**Table 1**: Performance of out-of-distribution BabyAI tasks.\n\n|   Algorithm   | Open | Goto | PickUp | PutNext | All |\n|:-------------:|:----:|:----:|:------:|:-------:|:---:|\n|   MCQ  | 91.9 ± 6.2 | 80.1±2.1 | 59.9±2.1 | 18.9±1.6 | 63.8±1.7 |\n|     TD3+BC    | 96.5±0.5 | 90.4±2.0 | 85.4±2.0 | 35.8±2.6 | 77.5±1.3 |\n|      CCM      | 98.6±1.2 | 87.5±1.3 | 68.0±4.2 | 26.1±4.5 | 69.5±0.5 |\n|     FOCAL     | 98.2±0.4 | 86.9±2.2 | 74.3±5.7 | 27.7±0.7 | 72.5±2.8 |\n|      CSRO     | 98.8±0.2 | 89.8±1.6 | 74.5±2.1 | 26.1±1.8 | 72.5±0.8 |\n|    UNICORN    | 99.0±0.5 | 88.0±3.6 | 73.3±4.6 | 29.9±1.2 | 73.5±2.4 |\n\n- Offline RL: The focus of offline RL methods is to avoid the phenomenon where policies perform poorly in unknown state-action spaces, which differs significantly from the task ambiguity problem addressed in this paper. Consequently, the performance gains prove ineffective or even detrimental.\n\n- Meta-RL: They focus on encoding trajectories to infer implicit task information. In our setting, however, task information is explicitly provided through instructions, and the real challenge lies in the vast number and diversity of instructions, as well as their complex relationships with tasks. Since the emphasis of meta-RL research does not address these challenges, its improvements are limited, and it is ineffective in resolving task ambiguity.\n\n---\n\n**W3.1: Disconnect between the proposed technical solution and core challenges**\n\n**A for W3.1:**\n\nWe respectfully disagree with your view that our method fails to address the core issue of task ambiguity, for reasons below:\n\n- **Definition**: In our work, task ambiguity refers to the phenomenon in language-conditioned tasks where agents confuse similar tasks due to the vast scale and fuzzy boundaries of instructions, leading to performance degradation. To address this, we tackle task ambiguity from both policy learning and representation learning perspectives.\n- **Policy Learning**: We are the first to theoretically demonstrate that distributional RL methods can effectively enhance an agent's ability to distinguish between tasks, and we apply this to language-conditioned problems.\n- **Representation Learning**: We propose a Trajectory-Wise Semantic Alignment, which leverages the correspondence between instructions and trajectories unique in language-conditioned tasks to apply contrastive learning, which directly enhances task discrimination in the language embedding space.\n- **Experimental Validation**: Our comprehensive experimental results demonstrate the effectiveness of the aforementioned methods. Furthermore, visualization experiments (Figure 7) qualitatively validate that our proposed method effectively solves the task ambiguity. Further, we quantitatively measure the improvement in embedding quality using the Silhouette score[9]. The results demonstrate that both proposed methods contribute to better language embeddings.\n\n**Table 2**: Silhouette score of task embedding on BabyAI.\n\n|Algorithm|All|PutNext|\n|--|--|--|\n|CQL|-0.030±0.005|0.004±0.007|\n|DAIL w/o Alignment|0.024±0.016|0.048±0.008|\n|DAIL w/o Distributional|0.110±0.019|0.088±0.020|\n|DAIL|**0.127±0.013**|**0.107±0.012**|\n\n**W3.2: Trajectory-Wise Semantic Alignment lacks explanation and empirical validation**\n\n**A for W3.2:**\n\nAs discussed in **A for W3.1**, we design Trajectory-Wise Semantic Alignment to enhance the distinguishability of different tasks. To address your concern about the design of this component, we elaborate detailed implementations and experimental validations as follows:\n\n- Implementation: We utilize the correspondence between instructions and trajectories to implement contrastive learning. In the dataset, each trajectory is annotated with an instruction describing its behavior. For a given trajectory, we treat its annotated instruction as a positive sample and select any other instruction annotation from the dataset as a negative sample. The trajectory encoding is derived from the output of a sequence model encoding history information (Equation 8), which is then used alongside language embeddings for contrastive learning training. The implementation details are provided in Figure 13 in Appendix F.\n\n- Empirical validation: We further validate the component's efficacy through ablation studies in Sections 5.3 and 5.4: 1) Figure 7 visually demonstrates how alignment sharpens task representations. 2) Table 6 quantitatively shows significant performance gains, particularly on PutNext tasks: CQL (25.6±2.5) $\\to$ DAIL w/o Distributional (39.6±0.6), and DAIL w/o Alignment (39.1±1.0) $\\to$ DAIL (57.9±0.9). These results confirm the contribution of the proposed improvement. Additional data and analysis are available in Appendix H. \n\n---\n\n**Q1: The fundamental mismatch in experimental setup**\n\n**A for Q1:**\n\nThe reviewer’s understanding of task ambiguity aligns with ours, and our experiments have been specifically designed to model this issue. We elaborate on this from two perspectives: (i)relationships between tasks and instructions, and (ii)errors caused by task ambiguity.\n\n- The relationship between instructions and tasks is complex. In practice, multiple instructions may correspond to the same task (e.g., in ALFRED, a single trajectory can be described by three different instructions). This makes task ambiguity highly likely to occur, where different instructions are incorrectly treated as corresponding to the same task.\n\n- Our experiments are designed to address task ambiguity. We observe that baseline methods often fail to learn task embeddings correctly, thereby confusing distinct task objectives(e.g., encoding \"pick up a red/blue ball\" into similar embeddings). Our goal is to separate the representations of inherently different policies.\n\nAs a supplement, in our formulation, the definition of a \"task\" is solely determined by the reward distribution $p(r|s, a, l)$, and is independent of environment configurations such as map layouts or different initial states.\n\n---\n\n**Q2: Clustering patterns between DAIL and DAIL w/o Distributional**\n\n**A for Q2:**\n\n- To address this issue, we quantitatively measure the clustering quality with the Silhouette score[9]. We present the scores in *Answer for W3.1* (Table 2). Results indicate that distributional indeed improves task embedding. \n\n- Appendix H presents ablation study results on the distributional component, as shown in Table 6. DAIL significantly outperforms DAIL w/o Distributional, demonstrating the effectiveness of distributional component. \n\n- Moreover, in Figure 7 and the ablation experiment, the difference between CQL and DAIL w/o Alignment also stems from the distributional component. We can observe a substantial improvement in both task embeddings and experimental performance. \n\nFrom these comparisons, we conclude that the distributional design makes a noteworthy contribution to the task embedding.\n\n---\n\n**Q3: Ablation studies on $\\alpha$**\n\n**A for Q3:**\n\nAs suggested, we do the ablation studies on $\\alpha$ coefficient on BabyAI. We present the results for PutNext and All tasks here with complete results added to the revised manuscript.\n\n**Table 4**: Ablation results on $\\alpha$.\n\n|$\\alpha$|ID PutNext|ID All|OOD PutNext|OOD All|\n|--|--|--|--|--|\n|0.2|44.0±3.6|83.7±0.7|40.8±5.1|78.4±1.7|\n|0.5|57.4±2.2|88.1±0.3|50.7±2.7|83.1±0.8|\n|1|56.2±3.6|87.7±1.3|50.8±3.7|84.1±0.7|\n|2|57.9±0.9|89.2±0.5|49.1±1.8|81.7±1.3|\n|5|44.8±3.2|85.2±0.2|37.1±5.1|77.4±1.7|\n|10|35.7±2.3|82.8±1.1|39.1±4.4|77.2±0.7|\n\nThe ablation results above show that DAIL is not very sensitive to the choice of $\\alpha$ between 0.5 and 2. The performance begins to degrade when the influence of the loss is either too small($\\alpha=0.2$) or too large($\\alpha=5$). \n\n---\n\n**References:**\n\n[1] Rt-2: Vision-language-action models transfer web knowledge to robotic control. CoRL. PMLR, 2023.\n\n[2] Context-aware planning and environment-aware memory for instruction following embodied agents. ICCV, 2023.\n\n[3] Mildly conservative q-learning for offline reinforcement learning, NeurIPS 2022.\n\n[4] A minimalist approach to offline reinforcement learning, NeurIPS 2021.\n\n[5] Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning, AAAI 2020.\n\n[6] FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization, ICLR 2021.\n\n[7] Context shift reduction for offline meta-reinforcement learning, NeurIPS 2023.\n\n[8] Towards an information theoretic framework of context-based offline meta-reinforcement learning, NeurIPS 2024.\n\n[9] Silhouettes: A Graphical Aid to the Interpretation and Validation of Cluster Analysis. Journal of computational and applied mathematics, 1987."}, {"review_id": "tc7cYCEPMq", "reviewer": "Reviewer_eo9K", "summary": "# Summary:\n\n## Problem:\n\nNatural Languages (NLs) is highly expressive but can sometimes be ambiguous. Current AIs struggle to maintain high performance when NL instructions for a given task are ambiguous. How to address this limitation?\n\n## Contributions:\n\n(i) Empirical analysis of the limitations of current mainstream methods in the face of NL-induced task ambiguity ;\n\n(ii) Theoretical analysis of the relationship between task ambiguity and sample complexity ;\n\n(ii) Present Distributional Aligned Learning (DAIL) agent, which leverages distributional RL to better map the relationships between complex observations, possibly-ambiguous language instructions, and reward signals. It also employs semantic alignment between the representations of trajectories and language instructions in order improve task discrimination by reducing the impact of ambiguous(/similar) NL instructions onto the policy learning process;\n\nEvaluations on ALFRED and BabyAI benchmarks show that DAIL achieves high performance compared to baselines, even in visually complex settings (ALFRED).", "questions": "Please see Strengths and Weaknesses points for all question.\nHowever, the two main actionable points that I would like to highlights are:\n\n1. WQ3\n2. WQ4\n3. WC13\n\n# POST-REBUTTAL :\n\nFollowing the proposed changes and clarification from the authors' rebuttals, I am raising my scores:\n- Quality: 1 -> 3 \n- Clarity: 1 -> 2\n- Significance: 2 -> 3\n- Originality: maintained at 3\n\nI am also raising my overall score from 3 to 5.\nI am waiting for subsequent authors-reviewers discussion before completely filling in my final justification.", "limitations": "While Section 6 discusses 2 limitations pertaining to external validity of DAIL, and albeit vaguely, the paper does not discuss limitations of the experimental protocols and techniques it employs, principally in terms of internal validity (e.g. 'is the experimental protocol providing strong or weak evidence of the claims?' or 'is t-SNE the most appropriate approach to evidence the claim that DAIL's alignment improves task discrimination?' or 'why using t-SNE over other methods?').\n\nThe paper did not address potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths & Weaknesses:\n\n## Quality:\n\nSQ0: I appreciate section 3’s formal definition and discussion and experiment, as it provides a solid problem framing.\n\nWQ1: Regarding definition 1, I think it is important to further highlight to the reader that this definition is dependent on (1) a specific distrinction threshold $\\delta$, and (2) a specific policy $\\pi$ ; the latter feels like a possible limitation of the work and problem framing, which ought to be flagged so that it can later be improved upon.\n\nWQ2: The experiment in Section 3 contains a few confounders that are not accounted for:\n\n1. How does DAIL’s size compare to CQL’s ?\n    \n2. How would a CQL-half (or even -quarter) compare? I mean that model scaling might not require increase to resolve the challenge, but maybe rather decrease.\n    \n\nWQ3: As highlighted in WC7-11, the paper’s theoretical results are very difficult to comprehend due to lack of proper definition and notations. As the theoretical analysis is a very important contribution of the paper, I would advise the author to make the theorems and their proofs in the paper as detailed as possible, as there needs to be no doubt about their veracity for the paper to be possibly accepted in the conference. As it stands, I cannot recommend anything but rejection.\n\nWQ4: Experimental protocol fails to emphasise measures of sample-efficiency. Indeed, only performance is being reported on in the form of success rates under a fixed budget of observations. While showing the learning curves provides some qualitative understanding of the sample efficiency of each method, it fails to provide a quantitative measure or, better yet, statistically-significantly-comparable measures about sample-efficiency. Thus, I would advise the authors to report the Area-Under-the-Curve (AuC) of the learning curves, for instance, as a quantitative measure of sample-efficiency, for starters. Then, in order to be as thorough as possible, I would advise the authors to perform statistical-significance tests on the results to provide stronger evidence. I believe this is a critical aspect of the paper’s quality, especially given that the theoretical improvement that the paper claims is along the line of sample-efficiency.\n\nSQ5:Section 5.3 presents a very insightful experiment. However, I would advise the authors to consider adding a quantitative measure of clustering, if possible, in order to increase the strength of their evidence towards showing that DAIL’s Alignment is impacting the language instructions’ representations.\n\n## Clarity:\n\nWC1: ln24-26: the paper claims that MILLION[14] improves robot task acquisition in terms of (sample-)efficiency. However, MILLION[14] does not claim improvement in sample-efficiency but only in terms of success rate (performance). Thus, I think the second part of this sentence needs refactoring.\n\nWC2: ln100 should probably point to Figure 2 (left). Then ln101 to middle.\n\nWC3: ln116-122: this paragraph is repeating some sentences from ln35-42, it might be valuable to remove the repeated sentences to make for some space to address other issues. Moreover, this paragraph seems to be meant to work as an introduction for Section 4 but it fails to summarise the content of the subsections of Section 4.\n\nWC5: ln126: ‘when the number of instructions increases’ is slightly misleading on its own. I would advise the author to also add: ‘… while keeping the number of actually-distinct tasks constant’ . By this, I mean to convey the nuance between learning difficulties due to the increase in size of the goal (and task) space from the learning difficulties due to the increased ambiguity between tasks, as the paper is focused on the latter only. Indeed, an increase in the number of instructions could be due to an increase in the number of actually-distinct tasks, which is not the context of the paper.\n\nWC6: Section  4.1 employs many Bellman-related operators which have not been formally defined in the preliminaries (Section 2). Clarity and readability of the paper would be enhanced by providing their definitions and some motivating examples, possibly.\n\nIf not in the preliminaries section, at least in Appendix.\n\nWC7: ln138: ‘we can discretize the value function distribution Z and train it by…’ : My understanding is that Z is the shorthand for $Z^\\pi\\in\\mathbb{Z}$, which is the random variable that represents ‘the cumulative discounted reward obtained along the policy $\\pi$’. Thus, I do not understand what the sentence mean when it says it, the random variable, can be trained.\n\nWC8: In a similarly puzzling fashion, $\\hat \\theta$ is introduced in ln141 and equation 6 without definition. It seems to be used as the target of the optimisation of equation 6, but it is not explained how to obtain it in the first place.\n\nWC9: ln552 describe an assumptions without justification and therefore is not rigorous enough for a proof. I advise the author to possibly introduce a formal lemma to address this assumption, after the vocabulary has been made more formal. Indeed, ‘can be regarded as’ is an informal expression whose formal meaning has not been detailed.\n\nWC10: ln554 : ‘Following [5,29]’ is a very ambiguous reference. This is rather detrimental to the clarity and understandability of the proof, and therefore should probably be addressed with a lemma leading to equation 22 or equivalent. As it stands, I cannot evaluate the veracity of said proof due to WC9 and WC10.\n\nWC11: ln553 : In a minor fashion, the terms ‘sub-gaussian’ and ‘1-Lipschitz’ have been used without definition. This hurts the clarity and readability of the proof and overall claim.\n\nWC12: Figure 5’s right graphs (Pickup and PutNext) are using different y-axis scales compared to the left graphs, which makes it possibly misleading to a distracted reader. I would advise the author to set a fixed y-axis scale between 0 and 1.\n\nWC13: ln206-209 highlights division of the task space between in-distribution and out-of-distribution, but it is not clear how the division is performed. Is it investigating combinatorial/systematic generalisation, i.e. is the division performed over instructions’ action-object pairs, e.g. ‘open red/blue door’ in the training set but ‘open yellow door’ is only found in the test set, and similarly with goto and pickup-based tasks, except with different colour being held out for the test set ?\n\nWC14: What about the test-train split in the context of the ALFRED benchmark?\n\nWC15: In equation 11, what is the meaning of ‘w.p.’ ? ‘z_i’ and ‘p_i’ are also left undefined. Please clarify.\n\n## Significance:\n\nSS1: The interplay between language instructions and behaviours is a critical topic, and I think that this paper makes both valuable theoretical and practical contributions to address this novel perspective on task ambiguity.\n\nSS2: I appreciate the BabyAI test-train split being highlighted towards evaluating generalisation capabilities, as I think it is a valuable evaluation to make. However, the current amount of details about the test-train split does not allow me to evaluate what kind of generalisation is actually evaluated, as mentioned in WC13.\n\n## Originality:\n\nSO1: To my knowledge, this is the first paper to highlight the problem of task-ambiguity in the context of language-conditioned RL, and its formalisation of the problem is appreciated.\n\nWO1: the related work paragraphs in the introduction could be improved to be more thorough, starting with, e.g. [Misra2017] for its original model that first attempted to ‘jointly reason about linguistic and visual input’, moreover within an reward-shaped RL context ; [Fried2018] for further characterisation of the difficulties caused by the flexibility of NLs, there in terms of NL instructions ‘identify[ing] only a few high-level decisions and landmarks rather than complete low-level motor behaviours’ ; [Co-Reyes2018] for proposing a trail-and-error-and-correction learning paradigm (going a step further than MILLION[14]) to deal with the ambiguities of NL instructions in language-conditioned RL, and actually claiming improved sample-efficiency.\n\nWO2: I am now realising about Appendix A- Related Works. I would advise the author to reference the Appendix A from the introduction. Moreover, ln466 is an ambiguous statement without clear reference, I would advise the authors to rephrase and add a proper citation.\n\nWO3: Semantic alignment of languages  with a contrastive learning approach is also investigated in a similar sparse-reward and language-conditioned RL context in [Denamganai2023]. The related works discussion (ln24-33) should probably include it and highlight how differently useful semantic alignment can be in RL contexts.\n\n## Misc:\n\nln36 and ln119: ‘retains’ is an ambiguous term to be used here and I don’t think it unequivocally conveys the meaning that is intended. The whole sentence seems to want to emphasise how the distributional aspect of the RL module (policy and value function) allows for a more expressive representation, compared to non-distributional counterpart, and therefore should be more equipped to dealing with the ambiguities in the NL instructions. If I understand correctly, then I would advise a reformulation of this important sentence that introduces the specificity of DAIL. (ln37-39 is doing a far greater job driving the point home, though).\n\nln39: ‘constrains the language instructions’ --> I do not think I understand, as I do not think an input to the model, here the language instruction, can be constrained by the model. I assume that it is the ‘language instruction’ representation\\* that is constrained?\n\nln36-42: The paragraph is constructed in a bottom-up approach, by going from the fine details about the policy and semantic alignment to the coarse claim of DAIL ‘improving learning efficiency’. As a neurodivergent individual, I appreciate this construction, but neurotypical readers might find it difficult to read. They might find more clarity through a top-down approach, starting with the coarse claim and then describing what the finer details that enable it.\n\nln43: It might be valuable to add a sentence that explicitly states why using structure vs visual observations (e.g. to check external validity of the DAIL agent, from structured to visual, and therefore from less to more complex and more expressive, observations).\n\nln95: ‘sematics’ -> ‘seman\\*tics’\n\nln144: ‘denote the number\\* of\\* samples’ maybe ? Also, there might be a need to add a ‘respectively’ at the end of the sentence.\n\n# References:\n\n[Misra2017]: [Mapping Instructions and Visual Observations to Actions with Reinforcement Learning](https://aclanthology.org/D17-1106/) (Misra et al., EMNLP 2017)\n\n[Fried2018]: Fried, Daniel, et al. \"Speaker-follower models for vision-and-language navigation.\" Advances in neural information processing systems 31 (2018).\n\n[Co-Reyes2018]: Co-Reyes, John D., et al. \"Guiding Policies with Language via Meta-Learning.\" International Conference on Learning Representations. 2018.\n\n[Denamganai2023]: Denamganaï, Kevin, et al. \"ETHER: Aligning Emergent Communication for Hindsight Experience Replay.\" arXiv preprint arXiv:2307.15494 (2023).", "was_revised": true, "final_justification": "Main issues resolved:\n\n- WQ3 : lack of clarity was preventing me from properly evaluating the theoretical contributions. As far as I now understand, thanks to the authors' rebuttal clarifications, the quality of the theoretical contributions is high.\n\n- WQ4: The paper was previously not including any quantitative measure of sample-efficiency whereas the paper was claiming improvements on that end. The authors' rebuttal provided statistically-significant results on that end. Thus, as far as I understand those results, the quality and significance of the paper has been raised by these additions.\n\n- WC13: It was previously not clear what kind of generalisation improvement was being claimed. The authors' rebuttal have clarified that it is combinatorial generalisation (/systematicity). As I believe this kind of generalisation to be very important towards AGI, I think this clarification not only raises clarity but also the significance and quality of the paper, as the results present are subsequent (and likely statistically-significant).\n\nI was assigning equal weights between those three main issues.\n\nThe rebuttal and discussions further highlighted the strenghts of the paper and addressed the major weaknesses, therefore I am recommending acceptance.", "rebuttal": "Dear Reviewer,\n\nThanks for finding our paper novel, empirical rigor and practical relevance. We hope the following statement clears your concern.\n\n**WQ1: Emphasize the key points of Definition 1.**\n\n**A for WQ1:**\n- In the revised version, we have updated Definition 1 to emphasize predefined task distinction thresholds $\\delta$ and $d$. \n\n- We have refined the requirement on the policy from \"a specific policy $\\pi$\" to \"any $\\epsilon$-optimal policy $\\pi$ that $V_{\\pi}(s)\\geq V^*(s)-\\epsilon, \\forall s \\in \\mathcal{S}$\". \n\n---\n\n**WQ2: A few confounders that are not accounted for.**\n\n**A for WQ2:**\n\n-  The sizes of these models in the toy experiment are as follows: CQL-double 56,075, CQL 23,947, DAIL 33,697. DAIL's size is between CQL and CQL-double.  When in BabyAI, the numbers of parameters are: CQL 1,481,217, DAIL 1,488,179, where the impact of this parameter scale becomes even less significant. \n\n-  As suggested, we run a CQL-half and a CQL-quarter on the toy experiment, and here are the results. Due to space constraints, we have presented only the results for 32, 128, and 512 tasks for reference (in percentage). \n\n**Table 1**: Average success rates over 100 evaluations for each number of instructions.\n\n|Model|32|128|512|\n|--|--|--|--|\n|CQL-quarter|91.2±5.8|86.0±2.1|70.9±0.9|\n|CQL-half|80.5±7.4|74.9±0.1|74.4±3.6|\n|CQL|89.9±3.2|77.4±1.1|70.6±2.3|\n|DAIL|**96.6±2.1**|**96.1±2.0**|**95.3±1.2**|\n\nResults above indicate that even when the model's parameter scale is halved or reduced to one-quarter, CQL's performance remains nearly identical, which demonstrates that the model size is not the key confounder of the performance differences between CQL and DAIL.\n\n---\n\n**WQ3 (WC7-11)**\n\n**A for WQ3:**\n\nWe have reconstructed part of the proof and updated the theoretical results accordingly. The new sample efficiency result is as follows: \n\n$$n_{\\text{value}} \\geq \\frac{C_{\\text{value}} \\log(3M^2 / \\eta)}{\\delta^2}, n_{\\text{dist}} \\geq \\frac{C_{\\text{dist}} \\log(3M^2 / \\eta)}{d^2}$$\n\nTo ensure clarity and coherence, we first provide a brief summary of the revised proof, followed by our responses to each of the comments.\n\n**Proofs:(modifying from line 552)**\n\nFollowing the triangle inequality of the absolute value, we have: **(WC9)**\n$$\n\\left|\\mathbb{E}\\_{(s,a)\\sim\\mathcal{D}}[|\\hat{Q}\\_i-\\hat{Q}\\_j|]-\\mathbb{E}\\_{\\pi}[|Q\\_i-Q\\_j|]\\right|\n\\leq \\left|\\mathbb{E}\\_{(s,a)\\sim\\mathcal{D}}[|\\hat{Q}\\_i-\\hat{Q}\\_j|]-\\mathbb{E}\\_\\mathcal{D}\\mathbb{E}\\_{(s,a)\\sim\\mathcal{D}}[|\\hat{Q}\\_i-\\hat{Q}\\_j|]\\right|+ \\left|\\mathbb{E}\\_\\mathcal{D}\\mathbb{E}\\_{(s,a)\\sim\\mathcal{D}}[|\\hat{Q}\\_i-\\hat{Q}\\_j|]-\\mathbb{E}\\_{\\pi}[|Q\\_i-Q\\_j|]\\right|\n$$\n\nIn general, we assume that the offline RL dataset is collected by an $\\epsilon$-optimal policy, i.e., there exists an $\\epsilon$-optimal policy $\\pi$ that $\\mathcal{D}$ satisfies \n$$\n\\mathbb{E}\\_\\mathcal{D}\\,\\mathbb{E}\\_{(s,a)\\sim\\mathcal{D}}[|\\hat{Q}\\_i-\\hat{Q}\\_j|] \n= \\mathbb{E}\\_{\\pi}[|\\hat{Q}\\_i-\\hat{Q}\\_j|], \n\\quad \\mathbb{E}\\_\\mathcal{D}(\\hat{Q}) = Q\\_{\\pi} = Q\n$$\n\nBased on the above derivation steps (with some intermediate steps omitted for word limit), we can upper-bound the original expression by following the three expressions, \n$\\left|\\mathbb{E}\\_{(s,a)\\sim\\mathcal{D}}[|\\hat{Q}\\_i-\\hat{Q}\\_j|]-\\mathbb{E}\\_\\mathcal{D}\\mathbb{E}\\_{(s,a)\\sim\\mathcal{D}}[|\\hat{Q}\\_i-\\hat{Q}\\_j|]\\right|$ (term 1), $\\mathbb{E}\\_{\\pi}|\\hat{Q}\\_i-Q\\_i|$ (term 2), $\\mathbb{E}\\_{\\pi}|\\hat{Q}\\_j-Q\\_j|$ (term 3).\n\nSimilar to the derivation of probabilities in Line 569 of the original text, we can also convert the original probability into the sum of the probabilities that each of the three terms exceeds\n$d/2$. Moreover, each of these three terms takes the form of an empirical estimate approximating the true expectation. Based on the existing conditions, we can apply Theorem 1 (Hoeffding's inequality) in[1], and obtain that for $\\forall l_i,l_j\\in \\mathcal{L}$, **(WC10, 11)**\n$$\n\\mathrm{Pr}\\left(\\text{term 1, 2, 3}\\geq \\delta/2\\right)\\leq 2\\exp\\left(-\\tfrac{n\\delta^2}{C\\_{\\text{value}}}\\right)\n...\n$$\n\nThe subsequent proof follows the same reasoning as in the original manuscript. Besides, we modified the proof of distributional RL accordingly, where we upper-bound the original expression by similar three expressions. \n\n**Answers:**\n\n- *A for WC7:* Z is a shorthand of $Z^\\pi$ and denotes a random variable. However, we often do not have access to the random variable Z but only its empirical version $Z_{\\theta}$, which is a parameterized model to be learned.\n\n- *A for WC8:* Yes, $\\hat{\\theta}$ denotes the parameters of a target network, a common stabilization technique in deep RL. This network is initialized identically to the learning model but updated periodically (e.g., via polyak averaging \n$\\hat{\\theta}\\leftarrow \\tau \\hat{\\theta}+(1-\\tau)\\theta$). Its primary role is to provide stable regression targets for the Bellman equation.\n\n- *A for WC9:* We have removed the vague assumption in Line 552 and given a more rigorous derivation.\n\n- *A for WC10:* In the revised version, we have explicitly indicated the theorem from the referenced work. In addition, we use Corollary 5.2 and Remark 5 in [2] for proofs of distributional RL.\n\n- *A for WC11:* We have modified the proof to eliminate the dependence on arguments related to \"sub-Gaussian\" and \"1-Lipschitz\" properties.\n---\n\n**WQ4: Quantitative measures of sample-efficiency.**\n\n**A for WQ4:** \n\nWe have computed the AUC for the learning curves, and present the AUC comparisons of ID learning curves at the key milestones of 10 and 20 epochs below. The complete results are comprehensively detailed in the revised manuscript.\n\n**Table 2**: AUC of in-distribution BabyAI tasks.\n\n|Algorithm|10(↑)|20(↑)|\n|--|--|--|\n|CQL|38,385.3±853.5|81,876.0±848.0|\n|DAIL w/o Alignment|40,374.7±601.3|85,247.1±540.2|\n|DAIL w/o Distributional|40,752.9±248.7|85,416.5±543.6|\n|DAIL|**42,370.6±404.8**|**88,450.5±291.6**|\n\n**Statistical-significance test**. As suggested, we perform statistical significance tests on these AUC results at 20 epochs. The result shows that, at the $\\alpha=0.05$ significance level, paired t-tests demonstrate statistically significant performance improvements for both the distributional component (p=0.0342) and the semantic alignment module (p=0.0332) individually. Furthermore, DAIL yields highly significant enhancements (p=0.002), confirming the synergistic efficacy of the complete methodology.\n\nThese results of both epochs confirm the significant sample efficiency gains of our method.\n\n---\n\n**SQ5: Quantitative measure of clustering**\n\n**A for SQ5:**\n\nAs suggested, we measure the clustering quality with the Silhouette score[3]. The labels required to calculate the score are defined as follows: two instructions share the same label if they require the agent to perform the same action on the same kind of object with the same color as the final target object. \nThe distance between language embeddings is measured using cosine distance. \n\n**Table 3**: Silhouette score of in-distribution BabyAI tasks.\n\n|Algorithm|All(↑)|PutNext(↑)|\n|--|--|--|\n|CQL|-0.030±0.005|0.004±0.007|\n|DAIL w/o Alignment|0.024±0.016|0.048±0.008|\n|DAIL w/o Distributional|0.110±0.019|0.088±0.020|\n|DAIL|**0.127±0.013**|**0.107±0.012**|\n\nThe results demonstrate that in the All-task scenario, the clustering metric of CQL even exhibits negative values, indicating a severe task confusion phenomenon where it fails to effectively partition tasks, while both proposed methods evidently improve clustering performance.\n\n---\n\n**WC1-WC6 and WC12: Clarity issues.**\n\n**A for WC1-WC6 and WC12:** \n\nWe sincerely thank you for identifying the areas needing clarification or reference in our manuscript. These points have been carefully addressed in the revised version.\n\n**WC13, SS2: Test-train split in BabyAI**\n\n**A for WC13, SS2:**\n\nOur experimental design indeed aims to evaluate compositional generalization. We use only the full given instruction (verbatim text) as the unit for splitting. We sampled approximately 60\\% of the unique instructions, and the training set contains only the trajectories in which the agent attempted to complete these tasks. We only constrain whether an instruction's text is included in the set. In Appendix E.2.1, we describe more details of ID/OOD split. \n\n**WC14: Test-train split in ALFRED**\n\n**A for WC14:**\n\nFor the ALFRED dataset, we strictly adhere to the original work's train-test split between the Training set and Evaluation set (see Appendix E.2.2 for details). While this setup inherently tests generalization capabilities, it differs from BabyAI’s ID/OOD split: the original ALFRED split allows partial instruction overlap between training and evaluation sets (despite environmental variations). \n\n**WC15: Clarify ``w.p.'', $z_i$, and $p_i$ in Equation 11.**\n\n**A for WC15:** \n\nEquation 11 models the discrete value distribution $Z_\\theta$ with two key components:\n- $z_i$ (termed *atoms* in our manuscript), which are support points on the quantized value domain.\n- $p_i$, representing the probability mass that the state-action value matches the support point $z_i$.\n\n\"w.p.\" here stands for \"with probability\". Please refer to Appendix C for further explanations. We have revised this paragraph to make it clearer.\n\n---\n\n**WO1-WO3, Misc:**\n\n**A for WO1-WO3:**\n\nWe sincerely appreciate your detailed suggestions on related works, word choice, paragraph organization, and the logical presentation. They have been comprehensively addressed in the revised manuscript.\n\nThanks again for your detailed and valuable comments. We sincerely hope our response has cleared your concerns, and we are looking forward to more discussions.\n\n---\n\n**References:**\n\n[1] Concentration inequalities. 2003.\n\n[2] Convergence and concentration of empirical measures under Wasserstein distance in unbounded functional spaces. 2020.\n\n[3] Silhouettes: A Graphical Aid to the Interpretation and Validation of Cluster Analysis, 1987."}]

Author rebuttal:

Meta-review: The authors study the problem of language-conditioned (offline) RL tasks, identifying and rigorously examining the challenge of ambiguity of tasks in language instruction learning and their interpretation in relation to the final goal. Methodologically, they build on existing RL algorithms (e.g., conservative Q-learning (CQL)) from two perspectives: 1) from a policy learning perspective, they prove that distributional RL methods can improve the agent's ability to distinguish tasks and 2) from a representation learning perspective, they use 'trajectory-wise alignment' to optimize task representations through their correspondence to the language instructions (i.e., mutual information). The resulting algorithm, Distributional Aligned Learning (DAIL), is empirically evaluated on structured observations (BabyAI) and visual navigation (ALFRED) benchmarks to validate performance improvements over imitation learning (GCBC, BC-Z, GRIF) and offline RL algorithms (IQL, CQL) -- all adapted for language-conditioned settings. In summary, DAIL shows improvements in terms of success rate for BabyAI and path-length weighted success scores for ALFRED. Additionally, visualizations and ablation studies are also performed to better understand the dynamics of DAIL.

Strengths identified by the reviewers include:
- Task ambiguity is an important problem in contextual RL in general and language-conditioned RL specifically given recent increased interest in language representations (e.g., LLMs).
- The authors rigorously characterize the task ambiguity on language-conditioned tasks and theoretically analyze the proposed methods regarding distributional language-guided policy learning and trajectory-wise sentence alignment.
- The theory emits a practical implementation that is empirically evaluated against relevant baseline methods on challenging datasets -- and shown to perform well.
- The paper is well-written, the proposed method is conceptually appealing, and they key points are well-explained.

Weaknesses identified by the reviewers include:
- Reviewer gNoU had concerns regarding sufficient contextualization with respect to existing work, specifically contextual MDP and contrastive task representations. However, this was well-addressed in rebuttal both conceptually (i.e., pointing out distinctions) and empirically. Some of this should make its way into the main body of the paper and some to an appendix.
- Multiple reviewers had concerns regarding the significance of the methodological novelty (e.g., distributed RL, contrastive learning of similarity functions). While there is some merit to these concerns, the problem being solved, motivation, analysis, etc. are different (i.e., it is not rare to apply an existing solution once the problem is well-understood).
- Reviewer SRtx had concerns regarding sufficiency of the experiments in terms of comparing to state-of-the-art models (e.g., MCQ, TD3+BC, CCM, FOCAL, CSRO, UNICORN). These were adequately addressed in rebuttal -- although it would obviously be nice to see this integrated into the main text/appendices to expand the discussion more.
- There were multiple technical details from reviewers (e.g., eo9K, gNoU) that were addressed during rebuttal. Some of this expanded discussion, etc. should be integrated into the paper.

In summary, I believe the reviewer consensus is that the task ambiguity formulation for language-conditioned RL is an interesting perspective that leads to a practical algorithm that is well-analyzed (especially after rebuttal) for a timely and important problem. The experimental results are sufficiently convincing and the additional results provided during rebuttal strengthen the claims and can easily be integrated into the existing manuscript. Of particular note is that the rebuttal addressed all major concerns (in my assessment), even if not always noted by the reviewers. That being said, many of the weaknesses addressed still need to be integrated into the paper before it is ready for publication.
Decision: Accept (poster)