Title: DAIL: Beyond Task Ambiguity for Language-Conditioned Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Offline Reinforcement Learning

Language-conditioned Reinforcement Learning

Representation Learning
Abstract: Comprehending natural language and following human instructions are critical capabilities for intelligent agents. 
However, the flexibility of linguistic instructions induces substantial ambiguity across language-conditioned tasks, severely degrading algorithmic performance.
To address these limitations, we present a novel method named DAIL (Distributional Aligned Learning), featuring two key components: distributional policy and semantic alignment.
Specifically, we provide theoretical results that the value distribution estimation mechanism enhances task differentiability.
Meanwhile, the semantic alignment module captures the correspondence between trajectories and linguistic instructions.
Extensive experimental results on both structured and visual observation benchmarks demonstrate that DAIL effectively resolves instruction ambiguities, achieving superior performance to baseline methods. Our implementation is available at https://github.com/RunpengXie/Distributional-Aligned-Learning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n# Summary:\n\n## Problem:\n\nNatural Languages (NLs) is highly expressive but can sometimes be ambiguous. Current AIs struggle to maintain high performance when NL instructions for a given task are ambiguous. How to address this limitation?\n\n## Contributions:\n\n(i) Empirical analysis of the limitations of current mainstream methods in the face of NL-induced task ambiguity ;\n\n(ii) Theoretical analysis of the relationship between task ambiguity and sample complexity ;\n\n(ii) Present Distributional Aligned Learning (DAIL) agent, which leverages distributional RL to better map the relationships between complex observations, possibly-ambiguous language instructions, and reward signals. It also employs semantic alignment between the representations of trajectories and language instructions in order improve task discrimination by reducing the impact of ambiguous(/similar) NL instructions onto the policy learning process;\n\nEvaluations on ALFRED and BabyAI benchmarks show that DAIL achieves high performance compared to baselines, even in visually complex settings (ALFRED).\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths & Weaknesses:\n\n## Quality:\n\nSQ0: I appreciate section 3’s formal definition and discussion and experiment, as it provides a solid problem framing.\n\nWQ1: Regarding definition 1, I think it is important to further highlight to the reader that this definition is dependent on (1) a specific distrinction threshold $\\delta$, and (2) a specific policy $\\pi$ ; the latter feels like a possible limitation of the work and problem framing, which ought to be flagged so that it can later be improved upon.\n\nWQ2: The experiment in Section 3 contains a few confounders that are not accounted for:\n\n1. How does DAIL’s size compare to CQL’s ?\n    \n2. How would a CQL-half (or even -quarter) compare? I mean that model scaling might not require increase to resolve the challenge, but maybe rather decrease.\n    \n\nWQ3: As highlighted in WC7-11, the paper’s theoretical results are very difficult to comprehend due to lack of proper definition and notations. As the theoretical analysis is a very important contribution of the paper, I would advise the author to make the theorems and their proofs in the paper as detailed as possible, as there needs to be no doubt about their veracity for the paper to be possibly accepted in the conference. As it stands, I cannot recommend anything but rejection.\n\nWQ4: Experimental protocol fails to emphasise measures of sample-efficiency. Indeed, only performance is being reported on in the form of success rates under a fixed budget of observations. While showing the learning curves provides some qualitative understanding of the sample efficiency of each method, it fails to provide a quantitative measure or, better yet, statistically-significantly-comparable measures about sample-efficiency. Thus, I would advise the authors to report the Area-Under-the-Curve (AuC) of the learning curves, for instance, as a quantitative measure of sample-efficiency, for starters. Then, in order to be as thorough as possible, I would advise the authors to perform statistical-significance tests on the results to provide stronger evidence. I believe this is a critical aspect of the paper’s quality, especially given that the theoretical improvement that the paper claims is along the line of sample-efficiency.\n\nSQ5:Section 5.3 presents a very insightful experiment. However, I would advise the authors to consider adding a quantitative measure of clustering, if possible, in order to increase the strength of their evidence towards showing that DAIL’s Alignment is impacting the language instructions’ representations.\n\n## Clarity:\n\nWC1: ln24-26: the paper claims that MILLION[14] improves robot task acquisition in terms of (sample-)efficiency. However, MILLION[14] does not claim improvement in sample-efficiency but only in terms of success rate (performance). Thus, I think the second part of this sentence needs refactoring.\n\nWC2: ln100 should probably point to Figure 2 (left). Then ln101 to middle.\n\nWC3: ln116-122: this paragraph is repeating some sentences from ln35-42, it might be valuable to remove the repeated sentences to make for some space to address other issues. Moreover, this paragraph seems to be meant to work as an introduction for Section 4 but it fails to summarise the content of the subsections of Section 4.\n\nWC5: ln126: ‘when the number of instructions increases’ is slightly misleading on its own. I would advise the author to also add: ‘… while keeping the number of actually-distinct tasks constant’ . By this, I mean to convey the nuance between learning difficulties due to the increase in size of the goal (and task) space from the learning difficulties due to the increased ambiguity between tasks, as the paper is focused on the latter only. Indeed, an increase in the number of instructions could be due to an increase in the number of actually-distinct tasks, which is not the context of the paper.\n\nWC6: Section  4.1 employs many Bellman-related operators which have not been formally defined in the preliminaries (Section 2). Clarity and readability of the paper would be enhanced by providing their definitions and some motivating examples, possibly.\n\nIf not in the preliminaries section, at least in Appendix.\n\nWC7: ln138: ‘we can discretize the value function distribution Z and train it by…’ : My understanding is that Z is the shorthand for $Z^\\pi\\in\\mathbb{Z}$, which is the random variable that represents ‘the cumulative discounted reward obtained along the policy $\\pi$’. Thus, I do not understand what the sentence mean when it says it, the random variable, can be trained.\n\nWC8: In a similarly puzzling fashion, $\\hat \\theta$ is introduced in ln141 and equation 6 without definition. It seems to be used as the target of the optimisation of equation 6, but it is not explained how to obtain it in the first place.\n\nWC9: ln552 describe an assumptions without justification and therefore is not rigorous enough for a proof. I advise the author to possibly introduce a formal lemma to address this assumption, after the vocabulary has been made more formal. Indeed, ‘can be regarded as’ is an informal expression whose formal meaning has not been detailed.\n\nWC10: ln554 : ‘Following [5,29]’ is a very ambiguous reference. This is rather detrimental to the clarity and understandability of the proof, and therefore should probably be addressed with a lemma leading to equation 22 or equivalent. As it stands, I cannot evaluate the veracity of said proof due to WC9 and WC10.\n\nWC11: ln553 : In a minor fashion, the terms ‘sub-gaussian’ and ‘1-Lipschitz’ have been used without definition. This hurts the clarity and readability of the proof and overall claim.\n\nWC12: Figure 5’s right graphs (Pickup and PutNext) are using different y-axis scales compared to the left graphs, which makes it possibly misleading to a distracted reader. I would advise the author to set a fixed y-axis scale between 0 and 1.\n\nWC13: ln206-209 highlights division of the task space between in-distribution and out-of-distribution, but it is not clear how the division is performed. Is it investigating combinatorial/systematic generalisation, i.e. is the division performed over instructions’ action-object pairs, e.g. ‘open red/blue door’ in the training set but ‘open yellow door’ is only found in the test set, and similarly with goto and pickup-based tasks, except with different colour being held out for the test set ?\n\nWC14: What about the test-train split in the context of the ALFRED benchmark?\n\nWC15: In equation 11, what is the meaning of ‘w.p.’ ? ‘z_i’ and ‘p_i’ are also left undefined. Please clarify.\n\n## Significance:\n\nSS1: The interplay between language instructions and behaviours is a critical topic, and I think that this paper makes both valuable theoretical and practical contributions to address this novel perspective on task ambiguity.\n\nSS2: I appreciate the BabyAI test-train split being highlighted towards evaluating generalisation capabilities, as I think it is a valuable evaluation to make. However, the current amount of details about the test-train split does not allow me to evaluate what kind of generalisation is actually evaluated, as mentioned in WC13.\n\n## Originality:\n\nSO1: To my knowledge, this is the first paper to highlight the problem of task-ambiguity in the context of language-conditioned RL, and its formalisation of the problem is appreciated.\n\nWO1: the related work paragraphs in the introduction could be improved to be more thorough, starting with, e.g. [Misra2017] for its original model that first attempted to ‘jointly reason about linguistic and visual input’, moreover within an reward-shaped RL context ; [Fried2018] for further characterisation of the difficulties caused by the flexibility of NLs, there in terms of NL instructions ‘identify[ing] only a few high-level decisions and landmarks rather than complete low-level motor behaviours’ ; [Co-Reyes2018] for proposing a trail-and-error-and-correction learning paradigm (going a step further than MILLION[14]) to deal with the ambiguities of NL instructions in language-conditioned RL, and actually claiming improved sample-efficiency.\n\nWO2: I am now realising about Appendix A- Related Works. I would advise the author to reference the Appendix A from the introduction. Moreover, ln466 is an ambiguous statement without clear reference, I would advise the authors to rephrase and add a proper citation.\n\nWO3: Semantic alignment of languages  with a contrastive learning approach is also investigated in a similar sparse-reward and language-conditioned RL context in [Denamganai2023]. The related works discussion (ln24-33) should probably include it and highlight how differently useful semantic alignment can be in RL contexts.\n\n## Misc:\n\nln36 and ln119: ‘retains’ is an ambiguous term to be used here and I don’t think it unequivocally conveys the meaning that is intended. The whole sentence seems to want to emphasise how the distributional aspect of the RL module (policy and value function) allows for a more expressive representation, compared to non-distributional counterpart, and therefore should be more equipped to dealing with the ambiguities in the NL instructions. If I understand correctly, then I would advise a reformulation of this important sentence that introduces the specificity of DAIL. (ln37-39 is doing a far greater job driving the point home, though).\n\nln39: ‘constrains the language instructions’ --> I do not think I understand, as I do not think an input to the model, here the language instruction, can be constrained by the model. I assume that it is the ‘language instruction’ representation\\* that is constrained?\n\nln36-42: The paragraph is constructed in a bottom-up approach, by going from the fine details about the policy and semantic alignment to the coarse claim of DAIL ‘improving learning efficiency’. As a neurodivergent individual, I appreciate this construction, but neurotypical readers might find it difficult to read. They might find more clarity through a top-down approach, starting with the coarse claim and then describing what the finer details that enable it.\n\nln43: It might be valuable to add a sentence that explicitly states why using structure vs visual observations (e.g. to check external validity of the DAIL agent, from structured to visual, and therefore from less to more complex and more expressive, observations).\n\nln95: ‘sematics’ -> ‘seman\\*tics’\n\nln144: ‘denote the number\\* of\\* samples’ maybe ? Also, there might be a need to add a ‘respectively’ at the end of the sentence.\n\n# References:\n\n[Misra2017]: [Mapping Instructions and Visual Observations to Actions with Reinforcement Learning](https://aclanthology.org/D17-1106/) (Misra et al., EMNLP 2017)\n\n[Fried2018]: Fried, Daniel, et al. \"Speaker-follower models for vision-and-language navigation.\" Advances in neural information processing systems 31 (2018).\n\n[Co-Reyes2018]: Co-Reyes, John D., et al. \"Guiding Policies with Language via Meta-Learning.\" International Conference on Learning Representations. 2018.\n\n[Denamganai2023]: Denamganaï, Kevin, et al. \"ETHER: Aligning Emergent Communication for Hindsight Experience Replay.\" arXiv preprint arXiv:2307.15494 (2023).\n\nQUESTIONS:\nPlease see Strengths and Weaknesses points for all question.\nHowever, the two main actionable points that I would like to highlights are:\n\n1. WQ3\n2. WQ4\n3. WC13\n\n# POST-REBUTTAL :\n\nFollowing the proposed changes and clarification from the authors' rebuttals, I am raising my scores:\n- Quality: 1 -> 3 \n- Clarity: 1 -> 2\n- Significance: 2 -> 3\n- Originality: maintained at 3\n\nI am also raising my overall score from 3 to 5.\nI am waiting for subsequent authors-reviewers discussion before completely filling in my final justification.\n\nLIMITATIONS:\nWhile Section 6 discusses 2 limitations pertaining to external validity of DAIL, and albeit vaguely, the paper does not discuss limitations of the experimental protocols and techniques it employs, principally in terms of internal validity (e.g. 'is the experimental protocol providing strong or weak evidence of the claims?' or 'is t-SNE the most appropriate approach to evidence the claim that DAIL's alignment improves task discrimination?' or 'why using t-SNE over other methods?').\n\nThe paper did not address potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method called DAIL to address the task ambiguity problem in language-conditioned reinforcement learning. DAIL integrates two key components: a distributional policy that enhances task discrimination by estimating full return distributions, and a semantic alignment module that optimizes task representations by maximizing mutual information between trajectories and language instructions. Extensive experiments conducted on both structured observation (BabyAI) and visual navigation (ALFRED) benchmarks validate DAIL's effectiveness, with visualization analyses revealing its ability to learn task representations and exhibit generalization to out-of-distribution tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1. The paper is well-written, with a logical flow and accessible language that enhances readability.\n2. It introduces and formally defines the task ambiguity problem in language-conditioned reinforcement learning, providing a fresh perspective on this challenge.\n3. Empirical results support the claims—DAIL outperforms established baselines across all reported tasks.\n\nWeaknesses：\n1. The paper inadequately justifies the practical significance and broader implications of addressing the task ambiguity problem, failing to establish its real-world relevance or potential impact on the field.\n2. The selected baseline methods are insufficient for rigorous comparison, as they: (a) fail to include state-of-the-art offline RL approaches, and (b) omit relevant multi-task/meta-RL methods (e.g., CSRO[1] and UNICORN[2]) that could better demonstrate the proposed method's advantages in handling task ambiguity.\n3. A fundamental disconnect exists between the proposed technical solution and the core challenges stemming from task ambiguity. Moreover, the design of Trajectory-Wise Semantic Alignment lacks sufficient explanation and empirical validation. \n\n[1] Gao, Yunkai, et al. \"Context shift reduction for offline meta-reinforcement learning.\"   \n[2] Li, Lanqing, et al. \"Towards an information theoretic framework of context-based offline meta-reinforcement learning.\"\n\nQUESTIONS:\n1. The current experimental setup appears to inversely model the task ambiguity problem - while true task ambiguity involves multiple instructions mapping to a single task, the experiments conversely examine multiple tasks sharing identical instructions. This fundamental mismatch raises questions about whether the evaluation properly addresses the core challenge the paper aims to solve.   \n2. The similar clustering patterns observed between DAIL and its ablated variant (w/o Distributional) in Figure 7 may potentially undermine the claimed contribution of the distributional component.  \n3. The paper may lack ablation studies on the α coefficient in the CQL loss term?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper proposes a better algorithm for language-conditioned offline RL tasks. It identifies one key challenge in such kind of RL tasks, that is to handle the vast space of language instructions and potential ambiguity in its relation to the final goal. To address this issue, the proposed method DIAL build upon CQL with two additional change
1. distributed value function to handle the ambiguity more directly.
2. cross similarity to enhance the representation.
The experiment shows better performance than baselines with the two addons.

STRENGTHS AND WEAKNESSES:
Strength:
1. The proposed new loss are simple and easy to understand.
2. The experiments is conducted on a large set of benchmarks set which is comprehensive.

Weakness.
1. The proposed losses can be found origin from existing literature (distributed RL, and contrastive learning for similarity metrics), which reduces the novelty.
2. I don't understand the confusion of the language embedding relates to the final outcome of the policy. The t-sne is a lower dimension space, the points can still be different in high dimension. Moreover, go to door and open door shares a lot in common in going to the door first, which should have similar embedding. To my knowledge, as long as the Q is correct, it should learn each task individually. Does the benefit comes from distributed value itself (like in single-task RL policy literacture) rather than the latent embedding?

QUESTIONS:
1. The difference of GoTo and Open can be differs if one use some exisitng pre-trained models like (RT-1). It is unclear if we use pre-trained components that it will solve the ambiguity problem directly.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the task ambiguity problem in language-conditioned RL tasks: proximity in the language instruction space does not necessarily imply task similarity. It introduces DAIL, a method that combines distributional RL with trajectory-based contrastive learning of task representations. They show the efficacy of their method through the BabyAI and ALFRED benchmark tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- Task ambiguity is a problem very relevant for the RL community.\n\n**Weaknesses**\n- Language-conditioned RL is an instantiation of the contextual MDP setting [1], and prior work has studied learning useful task representations in this setting already [2, 3, 4, 5]. Surprisingly, this paper conveniently ignores these works and does not reference them at all. \n- The trajectory-based contrastive learning of task representations is already well studied in the community [2, 3, 4, 5].\n- Definition 1 is stated without sufficient justification. Why should similar tasks have similar Q-values?\n- It is unclear how the authors draw the conclusion that distributional RL achieves better sample complexity for task disambiguation than value-based RL. Since the gains are not in the exponents but only in the constants, how do the authors arrive at the conclusion? The relation between C_value and C_dist should dictate the final conclusion.\n- Overall, the method appears to be incremental and as such does not meet the standards for NeurIPS.\n\n[1] Hallak et al. Contextual Markov Decision Processes, arXiv, 2015.\n\n[2] Fu et al. Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning, AAAI 2020.\n\n[3] Agarwal et al. Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning, ICLR 2021.\n\n[4] Li et al. FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization, ICLR 2021.\n\n[5] Mahajan et al. Learning Embeddings for Sequential Tasks Using Population of Agents, IJCAI 2024.\n\nQUESTIONS:\n1. How does this work fit into the prior work in RL that studies learning task representations?\n2. Could the authors provide a justification for Definition 1?\n3. How tight are the sample complexity upper bounds? Are gains only in constants inside the log sufficient? Could one use a different concentration inequality to arrive at better constants?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}