REI-Bench: Can Embodied Agents Understand Vague Human Instructions in Task Planning?
Authors: Chenxi Jiang
Chuhao Zhou
Jianfei Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Robot task planning
vagueness
LLMs
Primary area: applications to robotics, autonomy, planning

Abstract: Robot task planning decomposes human instructions into executable action sequences that enable robots to complete a series of complex tasks. Although recent large language model (LLM)-based task planners achieve amazing performance, they assume that human instructions are clear and straightforward. However, real-world users are not experts, and their instructions to robots often contain significant vagueness. Linguists suggest that such vagueness frequently arises from referring expressions (REs), whose meanings depend heavily on dialogue context and environment. This vagueness is even more prevalent among the elderly and children, who are the groups that robots should serve more. This paper studies how such vagueness in REs within human instructions affects LLM-based robot task planning and how to overcome this issue. To this end, we propose the first robot task planning benchmark that systematically models vague REs grounded in pragmatic theory (REI-Bench), where we discover that the vagueness of REs can severely degrade robot planning performance, leading to success rate drops of up to 36.9\%. We also observe that most failure cases stem from missing objects in planners. To mitigate the REs issue, we propose a simple yet effective approach: task-oriented context cognition, which generates clear instructions for robots, achieving state-of-the-art performance compared to aware prompts, chains of thought, and in-context learning. By tackling the overlooked issue of vagueness, this work contributes to the research community by advancing real-world task planning and making robots more accessible to non-expert users, e.g., the elderly and children.

Published as a conference paper at ICLR 2026
REI-BENCH: CAN EMBODIED AGENTS UNDERSTAND VAGUE HUMAN INSTRUCTIONS IN TASK PLANNING?
Chenxi Jiang Chuhao Zhou Jiangfei Yang" MARS Lab, School of Mechanical and Aerospace Engineering Nanyang Technological University Singapore 639798
(chenxi003, chuhao002, jianfei. yang edu. sg
Vague Instructions
Benchmark Framework
Instruction with Explicit REs
Context Memory
Prompt
Previous diologue between
Contest momory instruction a formatted prompt
Move the from the sink kitchen island.
and rabot
REs Instruction implicit RE: hinder robot comprehersion
Anerix:
Instruction with Implicit REs
Contact Viesory
the pot.
Are all the ingredients ready?
Hussnn Instruction
Skill Set
Yes, everything's all right.
heawy
Could you please the is clean?
Simulator
LLM-Based
The robot execuées one step of the planner.
Task Planner
will inspect the pot.
State
One Step Plan
SayCan
Then please move heavy stuff it outbide.
Per-step Outcome
LLM+P
Figure I: Left: Robots using existing LLM-based task planners can understand clear instructions with explicit referring expressions (REs), but they struggle to resolve implicit REs in multi-turn dialogues. Right: We propose the REI-Bench framework that aims to study real-world HRI scenarios where coreferential vagueness exists in human instructions.
ABSTRACT
Robot task planning decomposes human instructions into executable action se- quences that enable robots to complete a series of complex tasks. Although recent large language model (LLM)-based task planners achieve amazing performance, they assume that human instructions are clear and straightforward. However, real- world users are not experts, and their instructions to robots often contain significant vagueness. Linguists suggest that such vagueness frequently arises from referring expressions (REs), whose meanings depend heavily dialogue context and envi- ronment. This vagueness is even more prevalent among the elderly and children, who the groups that robots should serve more. This paper studies how such vagueness in REs within human instructions affects LLLM-based robot task plan- ning and how overcome this issue. To this end, we propose the first robot task planning benchmark that systematically models vague REs grounded in pragmatic theory (RFI-Bench), where we discover that the vagueness of REs can severely degrade robot planning performance, leading to success rate drops of up to 36.9%. We also observe that most failure cases stem from missing objects in planners. To mitigate the REs issue, we propose simple yet effective approach: task-oriented context cognition, which generates clear instructions for robots, achieving -f the-art performance compared learning. By tackling the overlooked issue of vagueness, this work contributes to the research community by advancing real-world task planning and making robots more accessible to non-expert users, e.g., the elderly and children.
prompts, chains of thought, and in-context
"Corresponding author.
Project page and code available http:///c0000.000..ta...o/////hd
10

---

Published as a conference paper at ICLR 2026
INTRODUCTION
In recent years, large language models (LLMs) have shown strong capabilities in tackling open-
world tasks across diverse domains. Their use in robot planning indicates a promising shift: unlike
traditional task planning methods (Hart et al.. 1968 Aeronautiques et al. T998: Chaslot et al. 2008).
which are often constrained by specific environments and narrow task domains, LLMs enable robots
to tackle tasks outside of conventional planning domains (Kim et al. 2024b; Zhou & Yang 2025
Wu
ct al.
2024 An et al. 2024).
Although existing LLM-based task planners have shown remarkable performance (Ahn et al. 2022 Huang et al., 2022a; Wong et al. 2023), they rely on an idcalized assumption: human instructions are always clear, complete, and unambiguous. However, in practice, human language often exhibits vagueness. Figure shows how vague terms like "it" or "heavy stuff" (refer to a pan) can make a kitchen robot grab the wrong item (e.g., a plate frying pan) instead of the pot. This challenge becomes even more pronounced for individuals with impaired memory or limited expressive abili- ties (Robinson & Apperly 2001 So et al. 2010 Hendriks et al., 2014), including young children, older adults, and people with Alzheimer's s disease- groups that rely most on robotic assistance.
I Linguists suggest that various forms of linguistic vagueness can impact human-robot interaction, including syntactic, scopal, and coreferential vagueness (Li et al. 2024b). Among these, coreferential vagueness is particularly common and impactful in robot task planning, as it can affect the planner's understanding of noun phrases in instructions and lead to incorrect identification of task objects. The coreferential vagueness arises because humans employ not only explicit referring expressions (REs) (e.g., "pot"), which directly identify their referents, but also implicit REs (e.g.. "it" or "this heavy thing"), which require contextual and environmental reasoning to resolve their meaning. Unlike robots, humans naturally use and interpret implicit REs in communication (Drave, 2002 Jucker ctt al. 2003 Paris et al. 2021| Peter 2018 Alkhhtnaii 2017 Wasow et al. 2005). Linguists have Tound that about 20% of expressions news content are descriptive (a kind of implicit REs) (Hervaš
& Finlayson 2010), with this percentage being even higher in everyday life. Bridge inference theory (Clark 1975) explains the process by which humans resolve implicit REs. When listener hears an expression like "this heavy stuff," they naturally identify several possible referents based on contextual memory: the "pot," the "ingredients." and the "sink." Among these, the "pot" best matches the description. This explains why the word "refer" is composed of two parts: the prefix re- ("back/again") and the root -fer ("to carry/ to bring"), ", which together embody the core mechanism of linguistic reference -namely, carrying meaning back from previously established contextual information. Inspired by linguistic studies, our research investigates key questions regarding task planners in embodied agents: Do implicit REs in human instructions impact the performance of bLsdd task planners in robots? How does the frequency of implicit REs influence the success rate of these task planners? What are the underlying causes of this impact, and what strategies can be employed to mitigate it?
To evaluate the impact of implicit REs the success rate of planners, we first build the referring expressions instruction (REI) dataset and then propose the first benchmark that systematically models coreferential vagueness grounded in pragmatic theory for robot task planners, namely REI-Bench. In this benchmark, our work systematically models the use of REs in human-robot instructions by defining three levels of referential difficulty, based on the ratio of explicit to implicit REs. As robots need context to understand REs in human instruction, we propose three levels of real-world contexts in multi-turn dialogues with irrelevant or missing information. Combining REs' difficulties with context memory types, the REI dataset includes nine levels of coreferential vagueness.
Then we evaluate task planners based on the REI dataset, including mainstream LLMs and robot planning frameworks. Because most deployed robot systems currently rely on small-scale language models, our analysis focuses on planners operating within this model regime (Kim et al.) 2024b). The results show that existing planners generally perform poorly in the presence of vagueness, with task success rates dropping between 7.4% and 36.9% in the baseline models. We attempt to mitigate the issue using novel basic NLP methods, including aware prompt (AP) (Gao et al. 2024a), suuin-of-thour (CoT) (Wei et al. 2022). and in-context learning (ICL) (Brown et al. 2020). but observe limited gains. Meanwhile, because they devote excessive attention to plan generation while failing to perform their inherent language understanding abilities. This challenges the common assumption that simply embedding
find that these performance declines in LLMs occur primarily
2

---

Published as a conference paper at ICLR 2026
an LLM in robot planning is sufficient for the robot to understand human language. Inspired by our observation, we propose a simple yet effective approach, mask-Oriented Context Cognition (TOCC), which decouples task comprehension from the planning decision decision-making process. Rather than designing a fully fledged solution, we intend to draw attention from the research community to this overlooked challenge and thereby motivate deep exploration.
Our contributions are threefold: (1) We systematically study how the instruction vagueness caused by REs impacts LLM-based robot task planners. To the best of our knowledge, this work provides the first systematic modeling of instruction vagueness in the context of robot task planning. (2) We develop REI-Bench by designing different levels of REs and context memory of human-robot dialogues, studying the success rate of robot tasks with vague instructions.
(3) We analyze the underlying reasons for performance degradation and propose a simple yet effective approach, TOCC, to enhance the robustness of planners. Extensive experiments on REI-Bench validate its effectiveness.
2 RELATED WORKS
Embodied Task Planning enables robots generate action sequences for various applications, including household and industrial tasks. Recent developments in LLM (Brown et al. 2020; Touvron
al. 2223) have led to language-driven planning methods (Kotb al.
2024 Driess et 2023]
Brohan et al. 2023 Chen et al. 2023; Liu et al., 2024a Li al.) 2023
Huang et al. 2023 such
as SayCan (Ahn et al., 2002). which leverage affordance functions
generate policies without
fine-tuning. However, evaluating these planners becomes difficult due to the high cost of real-world deployment, including expensive robotic hardware and resources. The early studies rely on human evaluation (Ahn et al. 2022 Huang et al. 2022b), which was subjective and inefficient, To address this, automated evaluation with simulators has emerged (Yin et [0044 Liu et al. 2024b). Methods such ProgPrompt (Singh et al. 2023) and LoTA-Bench (Choi et al. 2024) leverage datasets like VirtualHome (Puig et al. 2018) and A12-THOR (Kolve et al. 2017). Meanwhile, Embodied Agent Interface (Li al. 2024a) offers a general framework for evaluating L.MMs.
Linguistic Vagueness in LLMs has attracted attention among researchers in natural language processing (Liu et al. 22033a Ortega-Martín et al. (202)). The AmbiEnt benchmark (Liu al. (023a) evaluates the ability of LLMs to resolve linguistic ambiguities, revealing significant challenges even for advanced models such as GPT-4. To address these issues, APA (Kim al.) (024a) improves the management of vague queries by LLMs by leveraging their self aelf-assessment of vagueness. In the field of embodied AI, some prior works try to address these ambiguities by having robots ask clarifications or reason about uncertainty (Dogan et al. 2022, Park et al.) 2023 Ren et al. 2023). However, these works lack a systematic definition of linguistic vagueness, comprehensive evaluation of its impact on robotic performance, and effective methods to enhance the planner's 's own understanding capability. Building upon these developments, we further compare existing datasets and task planning benchmarks with linguistic ambiguity in Table [I
Table I: Comparison of BEI-Bench with existing datasets and benchmarks.
Benchmark/ / Method
Task
Planning Systematic
Multi-turn Context
Dataset / Size
Vagueness
REI-Bench (ours)
Evaluating coreferential vague in- structions (RE-bused) for robot task planning in AI2-THOR Ambiguous natural language task instructions for robot planning kitchen environment
REI--Diaset/ /
2.7k instructions (x vagueness levels)
AmbiK [vanova al. 2025
X
AmbiK / Ik anbiguous
instructions
CLARA Park al. 2023
Method for LLMs to classify whether the command is certain or not
X
SaGC/ 105 goals,
5222 tasks
KNOWNO Ren et 2023
Framework for measuring and aligning the uncertainty of LLM-- based planners
No dataset proposed
Dia
Questioner performer framework
53K task-relevant questions and answers
Gao et al. 2022
Asking Clantiiatioss
Clarification identification, clarifi- cution question generation, and swering for ambigunus language
X
CLAQUA 40K dialogue words
Xu et 2019
3

---

Published as a conference paper at ICLR 2026
REI Dataset HARDER
Seed Instruction
Introduce Noise and Information Lass
Context expanded Example
Simulator Scene
Replacement Examples
Explicit REs Short Context
Mixedd RE:
A Implicit REs
Description
Step1
Step3 Implicit
Introduced
Context Memary Generation
Explicit Noised Context
Mixed
Implicit
Ambiguous Names
Associated Name Background
Replacement
Noized Context
Naised Context
No Processed
Step2 Context Memory Processing
Replactable Explictt RE:
Explicit RES &
@ Mixed RE:
A Implicit REs Standard Contoxt
Standard
Replacement
Implicit REs Instruction
Implicit in Instruction and Context
Figure 2: Data curation pipeline of the REI dataset. From seed instruction, (1) generate context memory; (2) produce three context variants-Stanndaa Noised, Short; (3) replace explicit REs with implicit ones across varying degrees. This results in subsets reflecting nine levels of coreferential vagueness, determined by RE types (Explici/Mixee/IIppicc) and context variants.
3 APPROACH
Existing works Choi et al. (2024) typically evaluate L.L.-based robot task planners using clear instructions, whereas human instructions in real-world HRI scenarios often ambiguous. In natural language, instruction vagueness arises from the one-to-many relationship between a Signifier (the symbol itself) and its Signifieds (the entities it may represent in the physical world) (Hervaš & Finlayson 2010). For example, the signifier 'mouse' can refer to two signifieds: 'a rodent' or "a computer input device'. Pragmatics scholar Levinson et al. (Levinson) 1983) further distinguish vagueness into two types: Referential Expressions (REs) and Deictic Expressions (DEs). Humans can interpret REs through language context. For example, "Please bring me the mouse and keyboard" (referring to the computer input devices). In contrast, understanding DEs depends environmental context, such as time, space, and the speaker's s position, for example, "I want the book on the right side". Given our focus on LLM-based task planners, we confine our discussion to the impact of REs on task planning performance.
The objective of our work is to comprehensively evaluate and analyze how different levels of coreferential vagueness from implicit REs affect the planner's performance across diverse multi-turn dialogue contexts. To this end, we (1) systematically formalize REs in the HRI context (section 3.11 (2) establish the REI dataset and benchmark to evaluate planners on embodied tasks involving vague instructions (section) 3.2p, and (3) introduce simple yet effective solution (section 3.3
3.1 FORMALIZING VAGUENESS BY IMPLICIT RES AND HUMAN-ROBOR DIALOGUE CONTEXT
Levinson et al. (Levinson 1983) propose that understanding humans' intention depends both on Referving Expressions and Context Memory, where context memory refers to the previous dialogues which provide hints to determine the unique meaning of REs. Here, systematically model these two concepts into three levels to simulate varying degrees of vagueness.
Referring Expressions. In robot task planning, understanding REs is essential, as they provide key semantic cues for the goals of embodied tasks. Specifically, REs take various forms, including proper nouns (e.g., "apple"), ") definite noun phrases (e.g., "the apple"), indefinite noun phrases (e.g.. "an apple"), pronouns (e.g., "it", "them"), and attributive expressions (e.g., "sweet fruit"). The first three forms, known as explicit REs, have only one potential corresponding object and can be directly understood. In contrast, the latter two forms, known implicit REs, have multiple potential corresponding objects and should be identified by contextual inference. For example, the pronoun "it" simply indicates the referent is object rather than person, while "red fruit" only specifies the type and color of the item, without identifying which specific fruit it is.
To systematically model different forms of REs, we categorize them into three levels (Figure 2. At the "Explicit REs" level, all expressions are preserved as they appear in the original dataset. At the "Mixed" level, explicit REs in the instruction are replaced with implicit ones, while those in the context memory remain unchanged. The planner should refer to explicit REs to infer their implicit counterparts. At the "Implicit REs" level, all explicit REs replaced with implicit ones, except the
4

---

Published as a conference paper at ICLR 2026
first one in context memory, forcing the planner to rely on scene information to identify the referred objects. The latter two levels essentially simulate the vagueness introduced by implicit REs, which more common in daily human communication.
Context Memory. Linguists argue that the connection between words and objects is constructed by humans within specific contexts [Levinson] 1983). Analogously, different types of context can affect the reasoning capability of robot due to the inclusion of misleading cues or the omission of semantic information. One common source of misleading cues arises when a single signifier may plausibly refer to multiple, context-dependen entities, creating naming ambiguity. For example, the word "apple" may shift from denoting fruit to referring mobile phone brand when such mentions appear in the discourse. Additionally, semantic omissions can arise from the speaker's health issues or from different stages of cognitive development.
To simulate these scenarios. we deliberately introduce irrelevant information to the context while removing certain cues. Specifically, we define three types of context memory. In the "standard context", all information related to the task in context is provided. In "Noised Context", we introduce the Ambiguous Name noise, defined as a character or brand with a name intentionally resembling that of an object in the simulator scene. For example, "Apple" as a brand name is repeatedly mentioned in the dialogue, causing the planner to treat the fruit in the scene as the target. This noise reflects real-world cases where names are the same as objects, and tests the model's ability to correctly identify the intended referent. Moreover, the "Short Context" not only introduces noise but also omits partial task-relevant information, further increasing the difficulty of reasoning.
3.2 REI-BENCH DATASET
Existing vague expressions datasets (Marcus et al. 2011: Levesque al., 2012 Recasens et al.
2010), which are annotated by linguists, do not systematically formulate the position, frequency, and forms of REs, making infeasible to investigate their impact on task planning. Thus, establish comprehensive referring expressions instruction (REI) benchmark via an automatic pipeline to assess the effects of implicit REs on robotic planning tasks.
Specifically, we build the REI-Bench dataset upon ALFRED (Shridhar et al. 2020). a benchmark for embodied household tasks. From ALFRED, we select six tasks (Pick & Place, Stack & Place, Clean & Place, Heat & Place, Cool & Place, Examine in Light) and exclude the task of Pick Two & Place as it cannot be reliably completed by an embodied agent. Furthermore, since tasks that cannot be accomplished even with clear instructions fall outside our scope, we use "I.LaMA3. 1--BB + SayCan" to perform the six selected tasks in the A12-THOR (Kolve et al. 2017) simulator, keeping only the successfully executed tasks as seed instructions.
As shown in Figure 2 context memory is generated (Step 1). processed into three types (Step 2). and further transformed to the REI-Bench through implicit REs replacement (Step 3). In step 1, we extend the context of seed instructions by prompting GPT-4o-mini with a template that consists of context-expandee example and the text-based simulator scene description. The generated data consists of an instruction and a corresponding context memory. Typically, the generated instruction conveys the same task requirement as the seed instruction but in more human-like form. Meanwhile, the context memory captures the multi-turn human-robot dialogue before the instructions, which may include any objects present in the scene reflect the complexity of real-world human dialogue.
In step 2. we construct three types of context. The "standard context" retains the context memory generated in step I without further processing. For the "Noised Context" type. LLM is prompted to repeatedly insert an ambiguous name into the dialogue without altering its meaning. The ambiguous name is derived by slightly modifying the name of a simulator object (c.g., Rose - Mrs. Rose). To ensure natural adaptation, the LLM is given brief background prompt for the name (c.g.. "Rose is a family member"). In the "short context" setting, partial noun phrases are randomly removed from the context, including those containing task-relevant explicit REs. In Step 3, explicit REs are replaced with implicit ones following the rules outlined in Section 3.1 We adopt a CoT approach to determine which explicit REs can be substituted. Substitution examples are selected from OntoNotes (Pradhan et al. 2013), ensuring that the implicit REs are consistent with natural language usage. To ensure consistency in the number of explicit REs across tasks, define counting based rule for each level of implicit REs and discard any data that violates the rule. Consequently, the REI- Eel-Bench consists of 7,700 examples spanning nine difficulty levels, defined by combinations of varying RE vagueness and
5

---

Published as a conference paper at ICLR 2026
Standard Context & Explicit REs
Standard Context & Mixed REs
Human: cook cooked potato,
cooked potato,
Robot: Understood, Human. help with that. Human: Now, please put a heated potato in the sink.
Robot: Understood, Human. help with that.
Human: Now, please put the heated one the sink.
Rotato
potato AP
plate
plate
Success
plate
Falurre
Robot:
Robot:
find potato, pick potato, find microwave
find plate, pick plate, find a microwave
Reflection under Human Prompting
Human:
What does "the heated refer here?
Robot: this context, 'the heated one" likely refers to potato that has been heated cooked.
Task-Oriented Context Cognition (TOCC)
Aware Prompt (AP)
Chain-of- Thouught (CoT)
In-Context Learning (ICL)
Context Memary & Instruction
Context Memory & Instruction
Context Memory & Instruction
Context Memory & Instruction
AP: Instruction may include REs...
CoT Prompt: Clarify
In-Context Exa ssamples& & Explanation: [Examples] In this example, the heated refers potato
TOCC Prompt: Understand and rewrite the instruction with clear REs
vague REs
(Activated
only when implicit are detected)
Think Process: The heated potato
Cognition: Please put a heated potato in the sink
Input to Robots
Input to Robots
Input Robots
Input to Robots
Figure 3: Addressing implicit referring expressions in task planning. Top row: LLM succeeds with explicit REs ("potato"),, but misidentifies the object with implicit REs ("the heated one"). Middle row: reflection prompt from humans can guide the LLM to resolve the implicit REs and identify the correct object. Bottom row: Comparison among different prompting methods, including aware prompt (AP), chain-of-thous of-thought (CoT), in-context learning (ICL), and our task-oriented context cognition (TOCC).
context memory conditions. Please refer Appendix [ for detailed prompts of context expansion (B.I. context process (B.2), explicit REs replacement (B. and the counting- rule (B.4).
3.3
CONTEXT COGNITION
Based on the evaluation of multiple L.LM-based robot planners in REI-Bench, we find that most failures result from object omissions, as illustrated in the top row of Figure 3 When explicit REs are presented in the instruction, the robot correctly identifies the task target "potato" and successfully completes the planning. In contrast, when implicit REs, such as "the heated one", " are used, robots fail to identify "potato" but a iiss-irreeevant object "plate" instead. Furthermore, we find that LLMs resolve implicit REs when prompted explicitly (shown in the middle row of Figure 33. This suggests that LLMs inherently can interpret implicit REs, yet this ability can not fully manifest during planning and requires explicit prompting. As result, the idealized expectation that embedding LLM into embodied agents guarantees full comprehension of human language has been challenged.
To this end, we propose inject explicit prompts into the LLM-based robot planners to alleviate the coreferential vagueness. Specifically, we first evaluate three conventional prompting methods: (1) aware prompt (AP) (Gao et al., 2024a). which explicitly adds a prompt to guide the planner detect potential REs, (2) Chain-of- ofain-of-Thoughh Thought (CoT) (Wei et al. 2022). which guides the planner to resolve REs step by step before planning, and (3) In-Context Learning (ICL) (Brown et al. 2020), which provides examples of how to infer implicit REs from context. However, AP remains insufficient for handling implicit REs, as the prompt signals vagueness does not prompt deeper reasoning that LLMs struggle with during planning. Meanwhile, both CoT and ICL substantially lengthen prompts, hindering language understanding during planning, particularly when onboard computing resources limited and only small language model is available.
Consequently, we propose a simple yet effective mitigation strategy, Task-Oriented Context Cogmition (TOCC), to tackle the challenges posed by REs. As shown in the bottom row of Figure 3 TOCC decouples the REs interpretation step from the planning process, avoiding the LLM from devoting excessive attention planning within single generation step. By resolving the vague REs and

---

Published as a conference paper at ICLR 2026
GPT-4o-min
Framework: SayCan
Model: LLMMA3.1-88
Framework: SayCan
DeepSeekMath-P2
Framework: SayCan
Short. Context
44.40
23.30
21.30
Success Pate (96)
Context
50.30
28.80
20.80
Success Rate (96)
Context
29.50
17.40
12.10
Success Rate (%6
Noised Context
76.20
40
Noised Context
29.70
Noised
19.40
40
Contest
Standard Context
45,00
25.90
24.30
Standard
46.90
30.10
22.10
Standard
27.00
19.80
14.70
Explicit
Mixed
Impliit
Explicit
Mixed
Implicit
Explicit
Mixed
Implicit
Model: GPT-4a-m 20T-4a-mim
Framework: DAG
Model:
LiaMA3.1-8
Framework: DAG Plan
Model: DeepSeekdVath Framework: DAG Plan
Short Context
40.00
19.70
15.40
Short Contaxt
37.40
31.70
21.80
Short Context
15.70
11.00
5.50
60
60
60
Succers Rote
Noised Context
40.30
20.60
15.90
-10
Successs
Context
36.80
29.50
22.60
Success
Context
12.30
9.60
5.10
Standard Context
20.60
Standard
Standard
14.20
Context
Context
Explicit REs
Mixed
Implicit REs
Explicit REs
Mixed
Implicit REs
Explict REs
Mixed
Implicit
REs
Model:
GPT-4o-mini Framework:
Model:
LLaMA3.1-88
Framework:
Model: DeepSeekMath-78
Framework:
80
80
Context
57.20
17.10
13.90
60
Success Rate (86)
Context
59.80
21.10
16.70
60
Context
34.20
13.90
10.00
Successs Rate (%)
Success Rate
Noised Context
16.70
Nolsed Context
15.20
40
Naised Context
Standard Context
$6.50
18.60
14.20
Standard Context
58.40
22.00
15.30
20
Standard Context
33.70
15.60
10:10
Explicit
Mixed
Implicit REs
Explicit
Implicit Mixed
Explicit RES
Mixed
Implicit
RES
Model: GPT-4o Framework:: SayCan
Human Baseline the Subset (300 Samples]
100
Short.
73.10
61.80
52.90
Short
98.0
91.0
79.0
60
80
Success Rate (%)
Successs Rate
Noised Context
61.20
40
Noised
Context
Standard Context
73,50
61.60
52.60
28
Standard Context
97.0
91.0
86.0
20
Explicit
Mixed
Implicit
Explicit
Mixed
Figure 4: Success rate (%) of three task planner frameworks, SayCan, DAG-Plan, and HPE, using three LLMSs (GPT-4o-mini, LLLMA3.1-8BB DeepSeekMath-7B)) together with additional "GPT-40 + SayCan" planner and a human baseline on the REI dataset. Explicit, Mixed, and Implicit REs denote three levels of implicit REs in human instructions, and Standard, Noised, and Short Contexts represent three context memory types.
rephrasing the human instruction in more concise form before planning, TOCC demonstrates superior performance compared to existing methods. Implementation details and the exact variants used for AP and CoT provided in Appendix
EXPERIMENT
4.1 EXPERIMENTAL SETUP
We evaluate four state-of-the-an -ased embodied task planning frameworks, SayCan Ahn
al. 2022). DAG-Plan (Gao et al. 2024b), hierarchical task planning and execution (HPE) (Han et al. 2025), and LLM+P (Liu al. 2023b) on our REI-Bench dataset. Due to the deployment requirements on mobile robots, which must lightweight and open-source for on-robot adaptation, we focus solely on relatively small language models. For each task planner, we evaluate six LLMs, including GPT-4o-minn mini (Achiam et al. 2023). LLaMA3.1 1-8B (Grattafiori et al.. 2024), Ministral- (Shao et al. 2024). and
8B Jiang al. 2024). Gemma2-99 9B (Team al. 2024), DeepSeek-Mathh Qwen2.5-7B (Bai et al. 2023), which form a comprehensive benchmark consisting of 12 planners in total. To ensure balanced task coverage during evaluation, first construct 1,000-task subset
7

---

Published as a conference paper at ICLR 2026
Explicit Context
Explicit RES
Short Context
Mbxd REs & Standard Contect
Explicit REs & Noised Context
Mixed RES, Context
Explich REs
Noised Context
Mixed RES & Noleed Context
Mixed
20
Noised Conteett
Explicit Standard Context
Explicit REs
Standard Context
Mixed REs & Context
Mixed REs Context
implict RE: Context
Implictt REs & Short Context
Implicht RE: Standard
Implicit RE:
UaMA
Implict REs & Standard Context
Implicit
Own
AP
Notsed Context
Nalsed Context
ICL
TOOC
TOCC
LLMMAS.1-B-SSe
Qwen2 5-7-5yyonn
Contoo
Figure 5: Success rates (%) of various prompting methods applied to LLaMA 3.1-8B and Qwen2.5- 7B models with the SayCan framework the REI dataset.
of REI-Bench via stratified sampling across task categories and use it for all planner evaluations. To further compare the performance of LLM-based task planners with humans, invite human volunteers to conduct the same planning tasks on a randomly sampled subset of the REI-Bench.
4.2 BENCHMARK RESULTS OF B.M-BASED TASK PLANNERS
We evaluate the planning performance of all benchmark models the REI dataset. Due to page limitations, we present only six benchmark results in Figure Additional results and detailed comparisons are provided in Appendix A.3
LLM-based planners struggle to handle embodied tasks in multi-turn dialogue. We use the instruction portion (excluding the context memory) of the "Explicit REs + Standard Context" type of data as a baseline. for which the "LLaMA3. -8B+SayyCar planner achieves 57.7% success rate. However, as shown in the middle of the top row Figure multi-turn dialogues in "Standard Context" cause the success rate of "I.LaMA3.1 -ayB-SayCann model to drop significantly from 57.7 7 to 46.90% even without implicit REs. This performance gap highlights the limitations of existing LLM-based planners in handling natural, multi-turn human conversations.
The performance of LLM-based planners consistently decreases as implicit REs inerease, while remaining less affected by context memory noise. With the increase of implicit REs, all benchmark planners demonstrate consistent performance degradation across "Standard", "Noised", and "Short" example,
context memory. Take "LLaMA3. 1.E.AA..------S 1-8B+-ayCan'' (middle of the top row in Figure 4 the success rate drops 16.8% 15.5% 21.5% at the "Mixed REs" level and further decreases 8.0% / 8.5% 8.0 % at the "implicit REs" level. The consistent declines in LLMs' ability demonstrate that existing LLMM-based planners cannot effectively handle the vagueness of implicit REs within human instructions for embodied tasks. In addition, compared introducing multi-turn dialogues, adding naming ambiguity noise ("Noised Context") and further omitting partial task-relevant information ("Short Context") has little impact on performance. These observations suggest that existing LLM- based planners perform poorly when faced with multi-turn dialogues and implicit REs. However, such challenges are ubiquitous in human- robot interaction, especially when engaging with the elderly or children, and thus must be addressed.
4.3 ABLATION STUDY ON DIFFERENT PROMPTING METHODS
We compare four prompting methods that can mitigate the impact of implicit REs for LLM-basee task planners: AP, COT, ICL, and TOCC. As shown in Figure 5] by directly prompting the L.MM-based planners that human instructions contain potential vagueness, AP improves performance in most scenarios. However, in some of the "Explicit REs", the performance decreases inversely. We deduce that these APs may lead the planner to hallucinate and detect vagueness even when instructions clear. Meanwhile, ICL provides examples help the planner infer the meaning of implicit REs from context. However, ICL leads to performance degradation in almost all categories. We deduce that the small onboarding M-bs-a planners possess limited capability in learning from the
8

---

Published as a conference paper at ICLR 2026
Human previbus inguiry (Not Required Exccutey: Can you check we have any bread? want to make sandwiches for lunch. Maybe! could add some cabbaga them even some slices?
Human pending instruction: Now please ouser layer sandwich put
Target Object
Distracted Object
1:
ldge
2:open the Friddg
Fnd bread
4:plck up the bread
Object
Target Objjct
5:ffud
6:plck up
7=pul
the
Bxptck up
the bread
Distracted
Distracted error put down
error Bread is not visible because Lt DiningTabla
Object
error Robot is Object
Failed
Terget Objeet
Target Object
Distracted
Object
Target Object
Figure 6: Failure example on "Mixed REs Short Context", using LLaMAA.I-SB+SSaCCA 1-8B+SaCann. Due to the distracted object, the planner mistakenly put down the target object.
provided examples. Furthermore, CoT first guides the planner to autonomously analyse whether the instructions contain implicit REs and then perform planning based on the analysis, resulting in greater improvement. Building on the autonomous analysis from CoT, TOCC enables the planner provide more refined and task-relevant instructions by correcting implicit REs and reorganizing language. Planning is then performed based on the enhanced instruction. By decoupling RE resolving from planning. TOCC obtains the best performance, with an average success rate improvement of 6.5% on -LLaMA3.18BB-Say
We also provide results using only instructions as input. Under the Explicit REs type, the planner achieved performance comparable TOCC. However, under the Mixed REs and Implicit REs types, planner performance dropped sharply. Compared to the original instructions, the enhanced instructions from TOCC achieve improved planning performance. Moreover, the results are consistent with the pragmatic theory that context is indispensable for interpreting implicit REs. Please refer to appendix A.4 for completed ablation results.
4.4 ANALYSIS OF LLM-BASED PLANNER ERRORS
In this section, we review еπrо cases made by LLM- basdd planners in processing implicit REs. As shown Figure 6 the planner was uncertain whether "outer layer of the sandwich" referred bread or lettuce, revealing limitation of existing planners. Additional cases are provided in appendix A.6
Table Error rates (%) for the object omission and execution error types different benchmark models. Results under the "Standard Context" for three types of implicit REs reported.
Error Type
Model
Implicit REs Level
Overall Error Rate
Object Omission Rate Execution Error Rate
Explicit REs
7.1
47.9
55.0 74.1 (+19.1)
GPT-4o-mini
Mixed
37.0 (+-29.9)
37.1 (-10.8)
Implicit REs
46.2 (4-39.1)
29.5 (-18.4)
75.7 (+:20.7)
Explicit REs
22.6
30.5
53.1
LLaMA3.1-8B
Mixed
(+11.22
31.1 (-0.6)
(+16.8)
Implicit REs
(4-31.3)
(-.0()6.3)
(+2448)
Explicit REs
28.6
44.4
73.0
Deepseek-8B
Mixed
(+12.2)
(-5.0)
80,2 (-7.2)
Implicit REs
57.8 (+29.22)
27.5 (-16.9)
85.3 (+12.3)
For an in-depth analysis, we divide the task planning enrors into two categories: object omission and execution error. Object omission refers to the planner not correctly identifying the target object in human instructions. As shown in Figure 3 the planner wrongly identifies the referring expression "one" to "plate" as a typical object. In addition, an execution error occurs when the planner identifies the correct object but cannot generate the complete sequence of actions to achieve the goal. We summarize the eтror rate related to object omission and execution error for different benchmark models in Table 2 For simplicity, only the results under the "Standard Context" are reported. Please

---

Published as a conference paper at ICLR 2026
Table 3: Error rates (%) for the object omission and exccution types under different prompting methods (for LLaMA3.1-8B with "Standard Context").
Error Type
Method
Implicit REs Level
Overall Error Rate
Object Omission Rate Execution Error Rate
Explicit REs
22.6
30.5
53.1
LLaMA3.1-8B
Mixed
38.8
31.1
69.9
Implicit REs
53.9
24.0
77.9
Explicit REs
22.7 (+-0.))
30.5
53.2 (-0.1)
+ AP
Mixed
(-7.5)
39.7
71.0 (-1.1)
Implicit RFs
4-.9
27.4
77.3 (-0.6)
Explicit REs
22.5(-0.1)
30.2
(2.7(-0.4)
+ CoT
Mixed
34.9(-3.9)
34.2
(9.1
Implicit REs
47.6(-6.3)
30.3
77.9
Explicit REs
22.7(+
38.1
(-7.
+ ICI
Mixed
32.7(-6.1)
39.0
(-.7(-
Implicit REs
(-.9
28.7
(-.6
Explicit REs
16.8(-588)
24.2
4-.0(122.1)
TOCC
Mixed
28.5 (-103))
37.9
(-.4))
Implicit REs
40.1 (-1388)
30.6
70.7 (-7.2)
Explicit REs
(-5.4)
25.1
4 (-1.8)
Conntext
Mixed
(+42.88
5.3
(+17)
Implicit REs
85.1 (+31.2)
5.5
90.6 (+12.7)
refer to the appendix A.5 for more results under other context memory types. can be seen that the overall error rate increases the level of implicit REs grows. However, the error rates for object omission and execution error show opposite trends: the former increases significantly, while the latter decreases the level of implicit REs grows. This indicates that the main cause of task planning errors is that the implicit REs induce the task planner to overlook target objects in HRI, thus making unable to generate task plan correctly. Furthermore, shown in Table 3 our TOCC effectively reduces both the overall error rate and the object omission error rate across all three levels of implicit REs. These results demonstrate that TOCC effectively guides task planners to focus on target objects in human instructions, thereby enhancing robustness to finish instructions with vague REs.
5 CONCLUSIONS
We study how coreferential vagueness in human instruction affects robot task planning. By REI- Bench, we systematically simulate real-world language vagueness by categorizing REs and context memory. Extensive experiments show that implicit REs significantly reduce planning success rates. We explore the underlying reason and introduce the TOCC method, which effectively mitigates the negative effect of coreferential vagueness on robot task planner performance.
While our work discusses the impact of coreferential vagueness, human language vagueness is pervasive, and other forms of linguistic vagueness, such as deictic expressions, syntactic vagueness, and scopal vagueness, remain largely unexplored in the context robot task planning. Furthermore, to isolate the effect of REs. we filter the dataset by selecting tasks that LLMs can complete under clear instructions. As a result, the dataset consists of simple, short-horizon, single-objective tasks. As future L.LM-based planners become capable of solving more complex, clear- iiear-isttuuction tasks, we plan extend our analysis long-horizon scenarios. Moreover, experiments in the A12-THOR simulator provide initial results of the planner's semantic understanding capabilities. They do not capture multimodal information, including visual and spatial perception, which is required for robots to interpret other types of vague instructions. Thus, our future work will focus investigating the impact of deictic expressions on VLM-based task planners and validating the findings through experiments with physical robots.
ACKNOWLEDGMENTE
This work is supported by MOE Singapore Tier Grant RG83/25, RS36/24, and a Start-up Grant from Nanyang Technological University.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 Technical Report, 2023, arXiv preprint arXiv:2303.08774
Constructions Aeronautiques, Adele Howe, Craig Knoblock, ISI Drew McDermott, Ashwin Ram, Manuela Veloso, Daniel Weld, David Wilkins Sri, Anthony Barrett, Dave Christianson, et al. Pddl-the planning domain definition language. Technical Report., 1998.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as can, not as say: Grounding language in robotic affordances, 2022. arXiv preprint arXiv:2204001611
Mubarak Alkhatnai. Vague language and its social role. Theory and Practice in Language Studies, (2):122-127, 2017.
Tuo An, Yunjiao Zhou, Han Zou, and Jianfei Yang. lot-Ilm: framework for enhancing large language model reasoning from real-world sensor data, 2024. URL
02429
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen Technical Report, 2023. arXiv preprint arXiv:2309. 16609.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et Rt-2: Vision-language-action models transfer web knowledge to robotic control, 2023. arXiv preprint MtXiv:2307.1581
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1900 2020.
Guillaume Chaslot, Sander Bakkes, Istvan Szita, and Pieter Spronck. Monte-carlo tree search: A new framework for game ai. In AAAI, pp. 216-217, 2008
Yaran Chen, Wenbo Cui, Yuanwen Chen, Mining Tan, Xinyao Zhang, Dongbin Zhao, and He Wang. Robogpt: An intelligent agent for making embodied long-term decisions for daily instruction tasks, 2023. arxiv preprint Xix:v:231 1.15649.
Jae-Woo Choi, Youngwoo Yoon, Hyobin Ong, Jachong Kim, and Minsu Jang. Lota-bench: Benchmarking language-orienter task planners for embodied agents, 2024. arXiv preprint arXiv:2402.08178.
Herbert H Clark. Bridging. Theoretical issues in natural language processing/Assocciation for Computing Machinery, 1975.
Fethiye Irmak Doğan, Ilaria Torre, and Iolanda Leite. Asking follow-up clarifications to resolve ambiguities in human-robot conversation. In 2022 17th ACM/IEEE International Conference on Hlmman-Robo Interaction (IRII), pp. 461-469. IEEE, 2022.
Neil Drave. Vaguely speaking: A corpus approach to vague language in intercultural conversations. In New frontiers of corpus research, pp. 25-40. Brill, 2002.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid. Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model, 2023.
Jin Gao, Lei Gan, Yuankai Li, Yixin Ye, and Dequan Wang. Dissecting dissonance: Benchmarking large multimodal models against self-contradictory instructions. ECCV, pp. 404-420. Springer, 2024a.
Xiaofeng Gao, Qiaozi Gao, Ran Gong, Kaixiang Lin, Govind Thattai, and Gaurav S Sukhatme. Dialfred: Dialogue- Dialogue-enabled agents for embodied instruction following. JEEE Robotics and Automation Letters, 7(4): 10056, 2022.

---

Published as a conference paper at ICLR 2026
Zeyu Gao, Yao Mu, Jinye Qu, Mengkang Hu, Shijia Peng, Chengkai Hou, Lingyue Guo, Ping Luo, Generating directed acyelic dependency graphs for
Shanghang Zhang, and Yanfeng Lu. dual-arm cooperative planning, 2024b.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The LLaMA 3 Herd of Models, 2024. arXiv preprint arXiv:240.221783.
Songhao Han, Boxiang Qiu, Yue Liao, Siyuan Huang, Chen Gao, Shuicheng Yan, and Si Liu. Robocerebra: A large-scale benchmark for long-horizon robotic manipulation evaluation, 2025.
Peter E Hart, Nils Nilsson, and Bertram Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4(2):100-107, 1968.
Petra Hendriks, Charlotte Koster, and John CJ Hoeks. Referential choice across the lifespan: Why children and elderly adults produce ambiguous pronouns. Language, cognition and neuroscience, 29(4):391 407, 2014.
Raquel Hervaš and Mark Alan Finlayson. The prevalence of descriptive referring expressions in news and narrative. In ACL. @ Association for Computational Linguistics, 2010.
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world, 2023. arXiv preprint 1rXiv:231..12871.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In ICML, pp. 9118-9147, 2022a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng. Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models, 2022b.
Anastasia Ivanova, Bakaeva Eva, Zoya Volovikova, Alexey Kovalev, and Alcksandr Panov. Ambik:
Dataset of ambiguous tasks in kitchen environment. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 33216-33241, 2025.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts, 2024. arXiv preprint 22Xi:22401.04088
Andreas H Jucker, Sara W Smith, and Tanja Lüdge. Interactive aspects of vagueness in conversation. Journal of pragmatics, 35(12):1737-1765 2003.
Hyubng Joon Kim, Youna Kim, Cheonbok Park, Junyeob Kim, Choonghyun Park, Kang Min Yoo, Sang-goo goo Lee, and Taeuk Kim. Aligning Language Models Explicitly Handle Ambiguity, 2024a. arXiv preprint 1 1972.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model, 2024b.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai, 2017.
Mostafa Kotb, Cornelius Weber, Muhammad Burhan Hafez, and Stefan Wermter. Qt-tm:: Planning with transformer dynamics model and autoregressive q-learning. IEEE Robotics and Automation Letters, 2024.
Hector J. Levesque, Ernest Davis, and Leora Morgenstem. The winograd schema challenge. In Proceedings of the 13th International Conference on Principles of Knowledge Representation and Reasoning (KR), pp. 552-561, 2012.
12

---

Published as a conference paper at ICLR 2026
Stephen C Levinson. Pragmatics. Cambridge University Press, 1983.
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang. Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking Ilms for embodied decision making, 2024a. arXiv preprint arXiv:4410.07166.
Margaret Y Li, Alisa Liu, Zhaofeng Wu, and Noah A Smith. A Taxonomy of Ambiguity Types for NLP, 2024b. arXiv preprint MrXiv:2403.14072
Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators, 2023. arXiv preprint arXiv:2311.01378
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. We're afraid language models aren't modeling ambiguity, 2023a.
Bo Liu, Yugian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Lim+ p: Empowering large language models with optimal planning proficiency, 2023b.
Jiaming Liu, Mengzhen Liu, Zhenyu Wang, Lily Lee, Kaichen Zhou, Pengju An, Senqiao Yang, Renrui Zhang, Yandong Guo, and Shanghang Zhang. Robomamba: Multimodal state space model for efficient robot reasoning and manipulation, 2024a. arXiv preprint arXiv:2406.04339.
Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao. Mm-safetybench: A benchmark for safety evaluation of multimodal large language models. In ECCV, pp. 386-403. Springer, 2024b.
Mitchell P. Marcus, Martha Palmer, Lance A. Ramshaw, et al. Ontonotes: A large training corpus for enhanced processing. In Joseph Olive, Caitlin Christianson, and John McCary (eds.), Handbook of Natural Language Processing and Machine Translation: DARPA Global Autonomous Language Exploitation, pp. 27-48. Springer, 2011.
Miguel Ortega-Martín, Óscar Garefa-Sierra, Alfonso Ardoiz, Jorge Álvarez, Juan Carlos Ar- menteros, and Adrián Alonso. Linguistic ambiguity analysis in chatgpt, 2023. arXiv preprint arXiv:2302.06426.
Pierre-Henri Paris, Syrine El Aoud, and Fabian Suchanek. The vagueness of vagueness in noun phrases. In AKBC, 2021.
Jeongeun Park, Seungwon Lim, Joonhyung Lee, Sangbeom Park, Minsuk Chang, Youngjae Yu, and Sungjoon Choi. Clara: classifying and disambiguating user commands for reliable interactive robotic agents. IEEE Robotics and Automation Letters, 9(2):1059-106) 2023.
Megee Peter. Vague language as a means of avoiding controversy. Training, Language and Culture, 2(2):40-54, 2018.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. Towards robust linguistic analysis using ontonotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pp- 143-152, 2013.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang. Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In CVPR, pp. 8494-8502. 2018.
Marta Recasens, Lluís Mårquez, Emili Sapena, et al. Semeval-2010 task 1: Coreference resolution in multiple languages. In Proceedings of the 5th International Workshop Semantic Evaluation (SemEval 2010), pp. 1-8, 2010.
Allen Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners, 2023.
13

---

Published as a conference paper at ICLR 2026
EJ Robinson and Ian A Apperly. Children's difficulties with partial representations in ambiguous messages and referentially opaque contexts. Cognitive Development, 16(1):595-66 2001.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingehuan Zhang, YK Li, Y Wu, al. DeepSeekMath: Pushing the Limits of Mathemati- cal Reasoning in Open Language Models, 2024. arXiv preprint arXiv:2402.03300.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benehmark for interpreting grounded instructions for everyday tasks. In CVPR, pp. 10740-10749, 2020.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In ICRA, pp. 11523- 1133.. IEEE, 2023.
Wing Chee So, Özlem Ece Demir, and Susan Poldin-Meadow, When speech ambiguous. gesture steps in: Sensitivity to sescourse-prammati principles in early childhood. Applied psycholinguistics, 31(1):209-22) 2010.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving Open Language Models at a Practical Size, 2024. arXiv preprint arXiv:2408.001
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models, 2023. arXiv preprint atXiv:2302.1397.
Thomas Wasow, Amy Perfors, and David Beaver. The puzzle of ambiguity. Morphology and the web of grammar: Essays in memory of Steven G. Lapointe, pp. 282, 2005.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of Cfain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 25:24824-24837, 2022.
Lionel Wong, Jiayuan Mao, Pratyusha Sharma, Zachary S. Siegel, Jiahai Feng, Noa Korneev, Joshua B. Tenenbaum, and Jacob Andreas. Learning adaptive planning representations with natural language guidance, 2023.
Tao Wu, Chuhao Zhou, Yen Heng Wong, Lin Gu, and Jianfei Yang. Noisyeqa: Benchmarking embodied question answering against noisy queries, 2024.
Jingjing Xu, Yuechen Wang, Duyu Tang, Nan Duan, Pengcheng Yang, Qi Zeng, Ming Zhou, and Xu Sun. Asking clarification questions in knowledge-based question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1618- 1629, 2019.
Sheng Yin, Xianghe Pang, Yuanzhuo Ding, Menglan Chen, Yutong Bi, Yichen Xiong, Wenhao Huang, Zhen Xiang, Jing Shao, and Siheng Chen. Safeagentbench: A benchmark for safe task planning of embodied Ilm agents, 2024. arXiv preprint arXiv:2412.13178.
Chuhao Zhou and Jianfei Yang. Holollm: Multisensory foundation model for language- -nngudg--gruuned human sensing and reasoning, 2025.
14

---

Published as a conference paper at ICLR 2026
APPENDICES
Within this supplementary material, we elaborate on the following aspects:
Appendix A: Supplementary Experiment Results
A.1: Framework of Planners
A.2: Language Models List and Sampled-Subset Task-Type Distribution
A.3: Supplementary Benchmark Results
A.4: Supplementary Result of Ablation Study on Method
A.5: Supplementary Result of LLM-based Planner Errors
A.6: Supplementary of Task Planning Cases
Appendix B: REI Dataset Construction
B.1: Context Memory Generation
B.2: Context Memory Processing
B.3: Implicit REs Replacement
B.4: Data Filtering
Appendix C: Prompts and Implementation Details of Prompting Methods
C.1: AP and Gated AP Variant
C.2: Chain-of-Thought
C.3: In-Context Learning
C.4: Task-Oriented Context Cognition
Appendix D: Use of Large Language Models
15

---

Published as a conference paper at ICLR 2026
A SUPPLEMENTARY EXPERIMENT RESULTS
A.1 FRAMEWORK OF PLANNERS
SayCans (Ahn et al. 2022) is a method designed to help robots understand and carry out human instructions expressed in natural language. It breaks down complex instruction into smaller steps suggested by the LLM, and then evaluates whether each step is possible in the real world with an affordance- based value function (a separate model trained on data about how robots interact with their environment). With the emergence of more capable language tools, we employ Guidance (Choi al. 2024) to replace the affordance value function with L.L.-based feasibility assessment, which allows selecting skill in one generation pass and significantly reduces experiment time.
DAG-Plan (Gao et al., 2024b) is planning framework in which an LLM generates Directed Acyclic Graph (DAG) sub-tasks rather than a linear sequence. Each node represents a symbolic high-level action, and edges denote dependency relations. This structure makes the plan explicitly hierarchical and ensures that prerequisite conditions are satisfied before execution. By modeling sub-task dependencies, DAG-Plan improves robustness on multi-objer and multi-step tasks. As an official implementation of DAG-Plan has not yet been released, we re-implemented a DAG-Plan-stp architecture based on the publicly available descriptions of its model structure and prompting strategy, and adapted it to our experimental setting.
HPE (Han et al., 2025) is proposed as a unified evaluation architecture for ssiionlanguage-acttee (VLA) models, introducing hierarchical memory bank to ensure fair and structured long-horizon reasoning across different VI.A systems. Although HPE is designed for embodied VIAA models rather than LLM--asedd robotic task planners, we include it as representative framework that incorporates hicrarchical memory management. To enable comparison in our setting, we adapt the memory bank mechanism to LLM-based robot task planner, following the publicly released implementation.
LLM+P (Liu et al. 2023b) is hybrid framework that integrates large language models (LLMs) with classical symbolic planners to achieve robust and interpretable task planning from natural language instructions. The process typically involves three steps: First, the LLM translates high-level natural language commands into formal representations such as PDDL (Planning Domain Definition Language). Second, classical planner, such as Fast Downward, computes valid or optimal plan based on the generated problem and domain definitions. Finally, the LLM translated the resulting plan, sequence of low-level actions, back into natural language, making it more interpretable for users.
A.2 LANGUAGE MODELS LIST AND CAMPLED-SURSET TASK-TyPE DISTRIBUTION
Table 4: List of language models used in the experiments. Model names are either from the OpenAl API or the HuggingFace model hub.
Type
Model name
Model size Remark
Closed-source
GPT-4o-mini
Unknown
LLaMA3. 1-8B
Instruct
Gemma2-9B
Open-source
DeepSeckMath-77
Instruct
mistral-8B
Instruct
Qwen2.5-7B
Instruct
Table 5: Task-type distribution and average subtask steps in the 1,000-task sampled subset. The column "Original Proportion" corresponds to the original ALFRED dataset distribution, and our stratified sampling preserves this proportion. (GT denotes ground truth.)
Task Type
Original Proportion ( (%) Sampled Count
Avg. Subtask Steps (GT)
Cool & Place
16.8%
168
12
Heat & Place
16.8%
168
14
Clean & Place
16.2%
162
10
Examine Light
13.3%
Stack & Place
18.4%
184
Pick & Place
18.5%
185
Pick Two (Excluded)
Total Average
100%
1000
10
16

---

Published as a conference paper at ICLR 2026
The LLMs we used are listed in Table 4 To support the deployment of task planners edge devices, we focus on lightweight, open-source language models with relatively small parameter sizes. For each model series, we use the latest available base version at the start of our study. Since a compact version of DeepSeek-v3 was not yet available, we use the experimental DeepSeekMath model instead in our evaluation.
A.3
SUPPLEMENTARY BENCHMARK RESULTS
GPT-4o-mhi SayCan
GPT-4p Plan
GPT-4o-minl
mPT- LLM+
10
Short
23.3
21.3
50
Stori
19.7
Short
57.2
17.1
13.9
Sacum Sar 50
Shar
21.7
13.0
Sat
Quiten Sote
Conend
43.1
26.2
21.3
40
Contect
40.3
20.6
15.9
40
Contear
$5.0
16.7
13.3
Contrut
25.4
14.1
11.7
Surdarl Concoit
45.0
25.9
24.3
Stataad
41.7
20.5
14.7
Stardurd
Centee
56.5
18.5
14.2
Stoddad
23.2
15.3
10.6
Dipidt
Mied
Implice
Dyliet
Mbed
Implice
Dxplct
Miadd
Impict
Eodiet:
Miea
Insket
REx
RE:
11aMA3.1-88
San
114M63.1-88
DAG-Plan
1LaMAS.1-881 HPE
11.MAS.1-88
LIMIP
Contec
50.3
28.8
20.8
Contect
37.4
31.7
22.8
S Sor Pe)
Cantest
58.8
21.1
16.7
Contaxt
24.3
15.4
11.0
Su Neree
havee
45.2
29.7
21.2
S
hidued
36.8
29.5
22.6
Naloed
57.9
20.5
15.2
8
23.7
19.5
11.5
Conton
46,9
30.1
Content
38.7
30.0
Context
58.4
22,0
Contexl
18.6
Daplet
Miad
Implet
Dpllct
Maed
Implit
Miked
Implit
Ecalch
Mieed
Inskit
Gemma2-58 SayCan
Gemma2-38
DAG-Plan
Gemmaz298
Gemmaz-98 LLMI
30
50
50
Short
51.2
32.2
27.2
Stori
33.4
24.7
12.8
Stor
$4.0
33.4
24.1
Shart
9.9
5.2
3.8
10
Screet Rate 19
Sureel Roe-
Sureet Boe
Cuncot
48.1
35.2
Conteri
329
20.5
Conlex
52.2
32.8
8
Contaxt
5.8
Sundard Compe
49,2
35.7
28.9
Stancard
Comser
36.5
21.0
14.9
Stardard
Centest
52.4
37.8
21.6
Stundard
Contast
9.6
6.1
5.4
Puplitt RBS
Mied
Implie
Syplet 465
Mard
Impic
Expiit RES
Mived
Impkit RES
Peliet RES
Mied
Inake
Ministral-88 SayCan
Minisral-88 88 1 DAG-Plan
Miristral-88
HPE
Ministral-88
LIMIP
Cented
50.7
26.7
20.9
40
Contect
30.3
18.2
12.1
60
Contra
43.3
22.9
39.9
600
Contrul
31.7
19.9
13.7
Suere bame
Saree Rowe
Scere Bate
Surven Row
Nosel Concicl
46.7
30.6
22.4
15
Nikiel Conteel
32.5
17.8
10.5
Gorlz
Naseel
40.5
24.5
19.8
Nassod
30.0
23.3
13.9
Comton
29.4
Contect
19.1
Centext
Mundard
24.0
Eaplin Mied
impler
Pyplet
Wand
impivt
Expict
Mileed
Inpke
Mixd
Inoke
DeepSeskMath-7B SayCon
DeepSeeliMath-P0 7B DAG Han
DespoeekMMtch-70 +IPE
DempoeekMath-77
LLM+P
50
50
Store
29.5
17.4
12.1
Stor
15.7
11.0
5.5
Shart
34.2
13.9
10.0
Short
3.9
5.1
3.6
Cuncon
27.1
19.4
Conten
9.5
Contea
33.4
13.5
Context
6.9
Suaddrr Conteet
27.0
198
14.7
Stassan Consect
14.2
10.4
6.3
Staddet
Contrat
33.7
15.6
10.1
Staedard
Conteut
9.1
6.3
4.2
Diplit
Mied
impler
Pyplet
Mared
impire
Dipiit
Miled
Impke
Pelice
Micd
Inoke
Nb
tt:
kls
No
RE:
RE.
Owen2.5-78- + SayCan
Qwen25-78- + DAG-Plan
Owen2.5-77 +IPE
Owenn2.5-78
LLM++
Contoct
18.4
12.0
40 RL
Contooi
25.0
20.2
60 g
Contect
20.8
14.5
60
Contaxi
36.5
25.9
21.0
Guren 1am 301
Natued Conted"
18.5
13.1
9.1
40
Natund Cornect
26.7
20.3
13.6
40
Natted Contest
22.5
14.8
9.9
Nareed Contast
$1.4
24.8
186
Stardar
Stascand
21.6
Spardeed
Staedard
29.6
Explat
Mied
inglet
Spiet
Moed
Impiit
Surict
Mooo
brok
Scalc
Mical
Imaka
Figure 7: Success rate (%) of two task planner frameworks (SayCan, DAG-Plan, HPE, and LLM+P using three LLMS (GPT-40-mini, LLaMA3. 1-8B, DeepSeekMath-7B, Gemma2-9B, Ministral-SBB and Qwen2.5-7B) on REI dataset. Explicit, Mixed, and Implicit REs denote three levels of implicit REs in human instructions, and Standard, Noised, and Short Contexts represent three context memory types.
All experiments in this paper are conducted task instances sampled from REI-Bench. As shown in [ we construct a 1,000-task evaluation subset using stratified sampling that preserves the original AI. ALFRED task-type distribution to prevent partial bias.
17

---

Published as a conference paper at ICLR 2026
We observe that while modern planning paradigms introduce valuable structured memory symbolic constraints, navigating implicit referring expressions (REs) remains a shared challenge. As shown in Figure / all evaluated planners exhibit two consistent trends: (i) noticeable drop in success rates under multi-turn dialogue settings, and (ii) a steady degradation as the proportion of implicit REs increases.
Starting with the SayCan framework, performance varies across backbone LLMs but follows the same overall pattern: even the strongest variants (e.g., Gemma2-9B-SSyyan) experience a marked decrease when explicit REs are replaced with implicit ones, underscoring the intrinsic difficulty of vague instruction understanding.
Frameworks incorporating explicit structural priors encounter similar trade-offs. For instance, DAG-Plan suffers performance drops when implicit REs are introduced. This suggests that while dependency-grann srreeencyy-rra--beeee based hierarchical planning provides clear execution structures, adapting it to capture the highly nuanced context required for vague reference interpretation remains non-trivial. Similarly, HPE leads to declines at the "Mixed REs" level, followed by further degradation under "Implicit REs." This indicates that although manually structured memory banks are effective for organizing information, their strict formatting might inadvertently filter out the subtle contextual cues necessary for resolving implicit REs.
Expl REs Short Content
Explict
tEs
Explot
Miad Standard Contoct
Starda Contont
Explick Contest
Mileed RÉs Netued Connect
Expliit RES Standord Contect
Pxplict Br
Standard Contoct
Mixed
Stort
Insiet RS & Short Content
Standard Content
Impic RES
mplet
Uatt
+
Implet Ris Sowntard Connect
Qwer
+P
Nosued Connec
10kz
LL LLakMA:: Can
Owen2 7BHSwa
-erict
Tuplict REN Short Contett
Expliit &
Short Contect
Mlaed &t&
Explot
Conteut
Spansard Contest
Nobied Contest
Mard REs
Mildd REs Nedked Conteet
Malurd Cornect
Explict
Be REs Standard Contect
Short Comeet
Moxd REs
Implih Contect
Miead REs Shart Cortest
Implics Ris & Short Contoo
impilei RIs
Standars mopih
RE:
implet Rte Standard Conteett
Implice RB
+PP
Neaed banteot
Context
Gemmaz 9BrSayCan
-Comett
Explice Ris Short Contest
Standard
Explch
Explict RS
Standar Contoct
Mord RP
Shart Contont
Implict
Seepliak
Standard
Ira RE:
Nokoed Contoxt
TDGC
DensSeekkMath BaaSyyy
Figure 8: Success rates (%) of various prompting methods applied to LLaMA 3.1-8B, Gemma 2-9B, Ministral-SB, Qwen2.5-7B, and DeepSeekMath-775 models with SayCan framework on REI dataset.
Finally, LLM+P, which combines LLMs with a symbolic PDDL planner, typically performs lower than SayCan in these highly vague settings. While LLM+P has been shown to be highly effective for tasks with clear symbolic definitions, its reliance on a manually defined domain file and strict symbolic constraints presents challenges in multi-turn, context-dependent scenarios. Consequently, translating
18

---

Published as a conference paper at ICLR 2026
the broad commonsense reasoning of LLMs into rigid symbolic states for implicit instructions proves difficult. Overall, the results demonstrate that the vagueness challenges highlighted by REI-Bench not model-specific but constitute systematic hurdle for existing LLM--baee task planning frameworks.
A.4 SUPPLEMENTARY RESULT OF ABLATION STUDY ON METHOD
Table 6: Average token usage and inference latency per planning step across all planning methods.
Avg Output Tokens
Avg Input Tokens
Avg Total Tokens
Method
Latency (ms)
L SayCan
1822
45
1867
474.30
AP
1850
42
1892
492.75
Gated AP
1841
43
1884
487.07
917.92 (timeout
CoT
3485
128
3613
occurred)
Short CoT
3464
82
3546
705.64
946.36 (timeout
Segmented CoT
2595
137
2732
occurred)
ICL (2-shol)
3075
46
3121
514.65
TOCC (ours)
1894
62
1956
598.40
Figure 8 presents the success rates of five task planners after applying the three prompting meth- ods: AP. COT, and TOCC. "ILaMM3. 1.0.MAA..-8B.. 1-8B-SayCann., (CGemm2--B++SyCCam and "Ministral- 8B+SayCan" follow the general trend observed earlier. TOCC consistently yields the best perfor- mance, followed by CoT and AP. However, two planners, Qwen2.5-7B+SayCan and DeepSeckMath- 7B+SayCan, exhibit divergent behaviors. For Qwen2.557B+Sayyau although TOCC remains the most effective method, the application of CoT leads to substantial performance drop. This may be due to Qwen2.5-7B' limited ability to follow multi-step reasoning instructions embedded in CoT- style prompts. In the case of MeepSeekMatt-7BB-Saae performance. This is likely because DeepSeek Math--77 is an experimental model specifically trained for mathematical problem solving and has not undergone alignment with human preferences. Conse- quently, it exhibits the strongest hallucination tendencies and the weakest instruction woooooffloooingggo ability among all evaluated models. Given its similarly poor performance in the baseline planning tasks, whether DeepSeekMath- 7B is capable of supporting embodied intelligence remains questionable.
both CoT and TOCC result in decline in
Table further shows that TOCC introduces only modest overhead relative lightweight baselines while remaining substantially more efficient than CoT and ICL. Specifically, TOCC increases total token usage and latency by only 3.95% and 26. 18% over the vanilla model, and by 2.38% and 22.87 77% over AP, respectively. In contrast, TOCC reduces token usage and latency by 45.32% and 15.20% compared with Short CoT, and lowers token consumption by 38.41% % relative to standard ICL.
19

---

Published as a conference paper at ICLR 2026
A.5
SUPPLEMENTARY RESULT OF LLM-BASE BAMEDD PLANNER ERRORS
Table 7: Error rates (%) for the object omission and execution error types different benchmark models.
Context Memory Type
Implicit
Firor Type
Overall Error Rate
Model
REs Level
Object Omission Rate Execution Error Rate
Explicit REs
7.1
47.9
55.0
Standard
Mixed
37.0
37.1
74.1
Implicit REs
46.2
29.5
75.7
Explicit REs
7.5
49.4
56.9
GPT-4o-mini
Noised
Mixed
36.2
37.6
73.8
Implicit REs
50.6
28.1
78.7
Explicit REs
8.7
47.3
$6.0
Short
Mixed
47.5
29.2
76.7
Implicit REs
53.4
25.3
78.7
Explicit REs
22.6
30.5
53.1
Standard
Mixed
38.8
31.1
69.9
Implicit REs
53.9
24.0
77.9
Explicit REs
23.8
31.0
54.8
LLaMA3.1-8B
Noised
Mixed
39.3
70.3
Implicit REs
53.9
24.9
78.8
Explicit REs
22.6
49.7
Short
Mixed
45.4
25.8
71.2
Implicit RFs
57.9
21.3
79.2
Explicit REs
28.6
44.4
73.0
Standard
Mixed
40.8
39.4
80.2
Implicit REs
57.8
27.5
85.3
Explicit REs
31.1
41.8
72.9
Deepscek-8B
Noised
Mixed
44.9
35.7
80.6
Implicit RFs
61.6
26.0
87.6
Explicit
29.2
70,5
Short
Mixed
51.6
31.0
82.6
Implicit REs
61.9
26.0
87.9
20

---

Published as a conference paper at ICLR 2026
Table 8: Error rates (%) for the object omission and execution types under different prompting methods.
Context Memory Type
Implicit
Frror Type
Overall Error Rate
Method
REs Level
Object Omission
Rate Execution Error Rate
Explicit REs
22.6
30.5
53.1
Standard
Mixed
38.8
69.9
Implicit REs
53.9
77.9
Explicit
23,8
31.0
54.8
LLaMA3.1
Noised
Mixed
39.3
31.0
70.3
Implicit REs
53.9
24.9
78.8
Explicit REs
22.6
49.7
Short
Mixed
45.4
25.8
71.2
Implicit REs
57.9
21.3
79.2
Explicit Rhs
21.4
38.6
60.0
Standard
Mixed
31.7
39.3
71.0
Implicit REs
50.4
27.0
77.4
Explicit REs
21.1
37.3
58.4
AP
Noised
Mixed
32.0
69.6
Implicit REs
50.7
26.9
77.6
Explicit REs
35.4
$4.9
Short
Mixed
39,9
31.9
71.8
Implicit REs
51.9
25.6
77.5
Explicit REs
22.5
30.2
$2.7
Standard
Mixed
34.9
34.2
69.1
Implicit REs
47.6
30.3
77.9
Explicit REs
22.4
29.7
52.1
CoT
Noised
Mixed
35.2
34.7
69.9
Implicit RFs
48.5
29.9
78.4
Explicit
22.9
$1.8
Short
Mixed
41.4
31.2
72.6
Implicit REs
51.8
27.2
79.0
Explicit REs
22.7
38.1
60.8
Standard
Mixed
32.7
39.0
71.7
Implicit REs
49.9
28.7
78.6
Explicit REs
21.7
37.1
58.8
ICL
Noised
Mixed
30.6
38.8
69.4
Implicit REs
50.5
79.0
Explicit
20.8
58.7
Short
Mixed
33.4
37.8
71.2
Implicit REs
51.7
28.2
79.9
Explicit REs
16.8
24.2
41.0
Standard
Mixed
28.5
37.9
66.4
Implicit REs
40.1
30.6
70.7
Explicit REs
16.1
24.9
41.0
+ TOCC
Noised
Mixed
28.4
38.5
66.9
Implicit REs
42.9
27.7
70.6
Explicit REs
17.1
27.1
44.2
Short
Mixed
22.8
69.6
Implicit REs
43.5
31.4
74.9
Explicit REs
17.2
25.1
42.3
Standard
Mixed
81.6
5.3
86.9
Implicit REs
85.1
5.5
90.6
Explicit REs
16.8
25.4
42,2
Coonextt
Noised
Mixed
82.2
4.8
87.0
Implicit REs
84.3
90.5
Explicit REs
17.5
24.4
41.9
Short
Mixed
80.2
83.8
Implicit REs
87.6
92.2
21

---

Published as a conference paper at ICLR 2026
A.6 SUPPLEMENTARY OF TASK PLANNING CASES
Hunan previous inquiry(Net Required to Execute): Hey there, can you help ne with sonething?
want nake the roon feel cozson, so uas thinking about adjusting the brightness of the lamp- previous inquiry(Not Required Execute):
that
lemp is set just right for reading. also need to find good pen for ny notes.
Hunan previous inquiry(Net Required to Exacute): usually like blue ink, but any pen will do just
bright enough
clearly.
Ilunan previous inquiry(Not Required to Execute): That sounds perfect! want ensure can see everything well. Now, can you also nake sure the is within reach while I focus on ny notes?
pending holding pen.
No,
covers everything.
turn on lamp while
tifted
Riolek ite pen
3rflnd
deck Lanp
Hunan previous inquiry(Not Required to Exocuto): : Hey thare, can you help ne with sonethingi thinking about adjusting the brightness of the larp-
went make the roon feel cozier, previous inquiry(Not Required to Execute): Yes, please lighting is set just right for reading, I also need to find a good writing instrunent for ny notes.
anything will do for now. I just want rake sure the illurination is bright enough to see my notes clearly. Hunan previous inquiry(ket Required to Execute): That sounds perfect! want to ensure can everything well. Now, you also focus ny notes?
previous inqutry(Not Required to Exocuta): usually
blue
writing instrunent is within
while
Hunan pending instruction: No, think that covers everything. Please turn on the light while holding writing instrument.
desk
Boplck
3:find
dopick the
becogss
la shil
le In shalt
Expiek pen Antar Bos
Pofind
Rrplik up pen Bes
Refind
becgne It
Vialbla
it
DBrpick the
Hasiind deek Larp
ta Stolt's
Figure 9: Success case on "Explicit REs & Noised Context" (top) and failure case on "Implicit REs & Noised Context" (bottom), both using 5
22

---

Published as a conference paper at ICLR 2026
Human previous inquiry(Hot Required to Execute): Hey thene, you help with dinner potato tonight,
preparations? decided
right ingredients ready.
Huran previous inquiny(Hitt Required to Execute): definitely Manz the potato to be fluffy and any leftover butter, that would
great adding flaver you potato ready.
well-seasoned.
Huran previous inquiry(Hot (Not Required to Execute); Perfect! And while you're at it, can you also if have herbs spices to enhance the flavor of the potate? pending instruction:
exasine the pantryy
great
everything set,
potato is cooked, please reterber to serve it warn. Now, put heated potato in the sink.
pptats
tcp poteto
Wltrovave
Scopen stcrowere
Sapot doe the
Anclose the
Fiturs on the
Brtern off the
Scopen the sfcrowere
teiftad potato
12ipick the pstata
Miscose the
pappott doell the potaato
previous iNouiry(Not Required Execute): Hey there, can you proparations? I've dacided to nake baked potato tonight, but I need to nake sure have the right ingredients ready.
dinner
Huran pravious (tot Requiieed Execute): definitely
fluffy
seasored. Also, you could check the microwave for any leftover butter, that would be great for adding flavor once it's ready.
Huran previous inquiry(hitt Required Execute): Perfect! And while you're have any herbs spiccss enhance Huran pending instruction: That sounds like a great plan! Once everything is set, and it's cooked, please renember to serve it Name. Now, put heated one the sink.
also
pantry see
L:Find plate
Zppice up the plate
Ssflnnd microvave
Acopen nicrowers
Scput does the plate
6:Fund plate
fipick the plate
8:f und sta
Sopet drmn the plate
Find
previous (nquiry(Mot Required Execute): Hey there, you with dinner preparations? I've decided to nake baked potato tonight, but I need to nake sure have the right ingredients ready.
Huran previous inquiry(Nor (Not Required Execute): definitely
fluffy and well-
seasoned. Also, you for adding flavor once it's ready.
interouae any leftover butter, that
great
Haran previous Inquiny(Not Required to Execute): Perfect! And while you're at it, pantry see if have herbs spices Human pending instruction: That sounds a great planl Once everything is set, it's cooked, please renember to serve it Nome. Now, put heated one the sink.
also
the
Strund ntcrovave
Zccpen the sEcroware
Suget dem the
4:Find ntcrowaso
Figure 10: Success case on "Explicit REs & Short Context" (top), failure case on "Mixed REs & Short Context" (middle) and failure on "Implicit REs & Standard Context" (bottom), using ILLLAMA3--BB-SayCaa
23

---

Published as a conference paper at ICLR 2026
Hunan previous inquiry(Kot Required Execute): Hi, there, was thinking about rearranging the living roon bit. I noticed last night that the lighting was bit off, Can you carry out light analysis of the for previous inquiry(Not floor lamp next to the sefa, what do you think? It should provide better lighting for reading. Hunan previous (nqu Required to Execute): Gand, that's settled then. while we're at st, noticed purple pillow
Execute): thought
considering noving
edding some the bedroon. what if ue brought it into the living roon?
thinking
roon
Hunan provious inqutry(kot Required to Exacute): I's glad agree. So, the next time you're you bring careful, I don't want to get dirty.
bedroos,
purple pillow into living
llunan pending instruction: Yes, actually. Could you carry the purple pillow while turning on the larp?
how light plays off the color the pillow.
lifted plllow
Ziplck the piltow
3iftid desk Tanp
<itern the desk
Hunan previous inquiry(Not Required to Exacuto): Hi, thare, I was thinking about rearranging the living roos bit. I noticed last night that the lighting was bit off, Can you carry light analysis
Hunan previous inquiry(Not Required to Exacute): thought so too, was considering roving the floor lamp next to the sofa. what do think? It should provide better lighting for reading. settled
previous inquiry(Not Required Execute): I've been thinking about adding sone color to the roon as well. I noticed the purple pillow in the hedroon. what if brought it into the living room?
While
provious inquiry(Nct Required the bedroos, could you bring careful, I don't want it to get dirty. pending instruction: illurinating 1ight the cushion.
glad you agrae. purple pillow into living roos? remember to
Exacute):
actually. could you transport the lavender cushion while see lusinescence interacts the
i:fted pitlow
Z:plck up the pillow fnd desk lump
Actarn the dosk
3:fted
Fi
pel Tow down
ToorLanp
currently solding
S0:Ftad deak Tump
Secpot dons Ure
Fudd
denk Tupp
pkck da the
pillcw
dwn
Figure 11: Success case on "Explicit REs & Standard Context" (top) and failure case on "Implicit RES & Standard Context" (bottom), both using GGPT-4o-min++SayCan".
24

---

Published as a conference paper at ICLR 2026
llunan previous inquiry(Not Required to Execute); Hey there! I need to take closer look at lying around. you help by bringing the lamp table? You know how Cushion king always says, "Good lighting is the key to finding lost treasures!"
credit
to the
Hunan previous inquiry(Not Required to Execute): the card nunber carefully. It's crucial for ne to ensure everything is up to date and that the credit card is valid. Cushian King would definitely encourage a tidy workspace, especially if it helps avoid Ilunan previous inquiry(Not Required to Execute): Thanks for that, Additionally, could you nake sure there's no glare the surface where the credit card will be placed? Proper lighting is assential
want check
expiration date
corny
about clutter causing chaos!
snall details clearly. Cushion King isn't trying 'teach' the kids about organization while I'n looking!
Hunan pending instruction: I appreciate your help! Once everything is set please exanine credit card light tall larp.
card
lar
previous inquiry(Nct Required to Exacute): Hey, you with sonething inportant? nake sure the desk lamp turned on
find ny credit card, first, I and positioned conrectly to provide gond vistbility. You know how Cushion king always says, "Good lighting Is the key finding lost treasures!"
previous inquiry(Not Required also ensure that the area around the lamp is clear of any clutter? want to nake sure nothing is obstructing ny view when look for the credit card. Cushion King would definitely encourage helps avoid cormy science jokes about clutter causing
Execute): Exactly!
while you're it, could you
tidy workspace, espetially chaos!
Hunan previous inqutry(Kot Required to Exocute): Thanks! I apprectate that, Once the area is easter for spot the credit card
clear and the lanp is proparly 1tt, it will quickly and efficiently. Cushion King Isn't trying to "teach" the kids about organization while I'n looking!
pending instruction: Perfect! Now, you please check underneath tall illuninated
fixture for ny ni nisplaced card?
lifted credit card
Riptck up the creftt
siftrd floor Tamp
sipst dows the
Fetled
Siptck the card
sifted
9:flnd Floor
crodit card
card
deun
3Rrpet down the
Mislck to the
12ifted Floor Tamp
1ut Cowa the down
Micpckk up the
down
Fatled
Fultedl
5Sitted Floor
161 Mod
tred le card
cradit card
credit card Betor pit down
senar put down
Figure 12: Success case on "Explicit REs & Noised Context" (top) and failure case on "Mixed REs & Noised Context" (bottom), both using 1LLaMA3.1 I-8B+SayCan".
25

---

Published as a conference paper at ICLR 2026
Hunan previous Inquiry(Not Required to Execute): Hey, can you help ne find keys? need then to unlock the door before friend arrives. think they night be on the dining table. Ilunan previous inquiry(Not Required to Execute): Yes, definitely check around the floor lamp. leaving there. important
right spots.
Hunan provious inquiry(kot Required to Execute): That sounds good! If you find it, please exanine their condition too. Candle Glow would want to make everything is good shape. Once you
pending instruction: light the floor lamp? That will help no see then better, Candle Glow always arphasizes good Lighting for this.
keys, you
keys
lifted floes-lor
Bitarn the E100T
3:find dintng
4:find key chate
key
chale
inquiry(Not Required Exacuta): to unlock the door before ny friend arrives. I think It eight on dining table.
you find itea? need it
Hunan previcus inquiry(Net Required to Execute): Yes, definitely check around the tall fixture.
inportant look in the good! you exanine its condition too. Candle Glow would want to nake sure everything is in good shape.
romerbar leaving it there. previous inquiry(Net Required Exacute):
Ilunan pending instruction Lion: Great! Once find that ites, you exanine it under the light fixture? Lighting for this.
help ne better.
always exphasizes good
dialag
duplck 1P by
safa
4:plch up
takls
Floon
Exphik Lie
Po Rooe
Bract down
Flnil
chale
chaln
currendly tn loo
pat
Reychals
SPrplock
Baz Tear
B4rpst ley
chote
charin
erner put
Doifind
Floar Tump
doop lse
Brauut dous. bay
Find
Toar Lamp
BD lock the
chale
chain
ratled
Figure 13: Success case on "Explicit REs & Short Context" (top) and failure case on "Implicit REs & Short Context" (bottom), both using CAMMMM3.--BB-SaCCa.
26

---

Published as a conference paper at ICLR 2026
previous inquiry(Not Required Execute): Hey there, can you resind labels that can help ne organize the contents of the box? want everything to be neat before proceed.
any
previous inquiry(Not Required Execute):
ensure enough
1ight see Inside clearly. Can you adjust the 1ight source provide better illunination while I work?
pravious inquiry(Nct Required to Exacuta): Greatl alsa nake container clean before exanining then contents. you the spray bottlle and give 1t quick Mipe dow??
gather inquiry(Nct Required to Execute): Thank you! that's
need
the
categorize then properly. It's isportant keep
everything organized for future use.
Hunan pending instruction: Now that we're ready, please examine what's Inside under the light.
Hunan provious inquiry(Kot Required to Exocuta): Hey there, can you reslnnd me if have before
lebels help proceed.
box? everything
Ilunan previous inquiry(Net Required to Execute): Perfect! I also want to ensure have enough light inside clearly. can you adjust the light
provide better illumination
work?
Ilunan previous inquiry(Net Required to Execute): Great! also need to nake sure that the container clean before exanining then contents.
find the bottle and it
quick wipe down?
Itunnn previous inquiry(Not Required to Execute): Thank you! Once that's done, I'11 need to gather all the itees fron the container and categorize then properly. It's Ieportant keep
everything organized
use.
Hunan pending instruction: Now that we're ready, please exanine what's inside under the light.
1ifted saray
2:fted spray
Sifted sara
6:ftee spray Find
SprayRottta Find
SprayDutzta
Spr Spraybaente
Spray0otzta
spray
spray
spray
buttlu
ercer Casnot findd bottle
bottlv
bottle
bottll
arcer Cannot Flad Spresfottte
anoor Cannot Flad SoreyButtle
enven Cannot fited SornyBattte
Cannot find
Sprayouttt
30:Find срray buttle
Si:find spray
12:Find spray Sprayhutzle
13: spray
1i:Find spray SprayBiezte
Sp Spraytornte Fladd
Spi ppreytannte Fiaed
Spcayfattta
spray
speay
spray
speny
spray
Bottle
lottle
bottla
bottle
bottle
Speaysutzla
arrer Canrol Fied
arcin Cernot filed Sprayninzla
Spraynirile
Carnot fited
senion Gernot fited Sprayniesta
Sprayninnta
Cannal find
Figure 14: Success (top) and failure (bottom) cases the "Implicit REs & Standard Context" task using with TOCC applied the top case and omitted in the bottom case.
27

---

Published as a conference paper at ICLR 2026
B REI DATASET CONSTRUCTION
This section provides detailed description of REI Dataset automatic generation, with individual explanations for the three levels REs and three types context memory in REI-Bench dataset.
B.1 CONTEXT MEMORY GENERATION
For each seed instruction, we use an LLM identify the replaceable REs. The prompt is follows.
REs Identifying Prompt
I will input task, and you should output only the task objects mentioned in the task. There may be multiple task objects. If so, please separate them with commas.
Here are some examples:
Task: Place vase coffee table
Referring Expressions: vase
Task: Put the chilled sliced tomato in the microwave
Referring Expressions: tomato
Task: Pick up pillow and turn a lamp on
Referring Expressions: pillow, lamp
Task: {Seed Instuction]
Referring Expressions:
We use prompt including seed instruction, context-expanded example, and simulator scene description, and requirements guide GPT-4o-mini generating plausible context memory, which is shown below.
Context Memory Generation Prompt
Please integrate this sentence into script as dialogue: {Seed Instruction]
Scene description:
Alice and her home robot are at home, with only the following items in the environment: AlarmClock, Apple, BaseballBat, BasketBall, Bowl, GarbageCan, HousePlant, Laptop, Mug, RemoteControl, SprayBottle, Television, Vase, ArmChair, Bed, Book, Bottle, Box, But- terKnife, Candle, CD, CellPhone, Chair, CoffeeTable, Cup, DeskLamp Desk, DiningTable, Drawer, Dresser, FloorLamp. Fork, Newspaper, Painting, Pencil, Pen, PepperShaker, Pillow, Plate, Pot, SaltShaker, Shelf, SideTable, Sofa. Statue, TeddyBear, TennisRacket. TVStand, Watch.
Please do not mention any requirements outside the list.
Requirements:
1. The dialogue content of (Seed Instruction] should be included in Alice's final instruction. Please have Alice state the request mentioned above only in the last sentence and refrain from making any other requests. Before making this request, Alice should mention some other requirements related the {REs).
There should be six rounds of dialogue before this request. Each character's lines should contain no fewer than 20 words, and no actions should be included for any character.
Please do not output anything other than the dialogue. 5. Please try to retain the words of (REs} themselves in the conversation, rather than replacing them with pronouns like "it."
Below, I will give you an example: (Example)
We used this prompt expand seed instruction into full dialogue, which is shown below.
28

---

Published as a conference paper at ICLR 2026
Context Memory Generation Example
Seed Instruction: Put cooked tomato into the refrigerator.
Context Memory:
Human: Hey there, I've been thinking about what to do with the tomatoes we have. I really want make something fresh and delicious with them. Can you remind me what we can do with cooked tomatoes?
Robot: Absolutely, Alice! Cooked tomatoes can be used in variety of dishes, like sauces, soups, or even as topping for salads. They enhance flavors and add a nice texture to the meal.
Human:
That's exactly what I was thinking! I could use them to create savory sauce for pasta or maybe even a comforting tomato soup. Do we have everything we need to cook the tomatoes, though?
Let me check our supplies. We have pot for cooking, some spices like salt and pepper, and cutting board. We seem be well-equipped for cooking the tomatoes to perfection.
Robot:
Human: Perfect! want make sure the tomatoes are well-seasoned when cook them. Could you remind me how to properly prepare the tomatoes before cooking?
Robot: Certainly! First, we should wash the tomatoes thoroughly, then chop them into even pieces to ensure they cook evenly. After that, we can sauté them with the spices until they reach the desired consistency.
Instruction:
Great, that sounds easy enough! Let's get started with cooking the tomatoes. Once they're 'r cooked and ready, I'll need you to put cooked tomato into the refrigerator."
Human:
B.2 CONTEXT MEMORY PROCESSING
Noised Context Prompt
Here is the dialogue: (angguee;(Dillggnne DDalogue}
We are now going to adapt the previous dialogue.
Requirements:
Please add content only within the dialogue without deleting any existing content or changing the order of the dialogue. Please do not change the number of turns the dialogue. do not change the structure of the dialogue.
3. Please ensure the fluency of the dialogue. Please follow the requirements below for the adaptation.
Associated Name Background: There is another member (a human) of the family named (Ambiguous Name}. Please have Alice mention him 3 times when discussing anything related to THE REFERENCE, but without changing the existing meaning of the conversation. Some details of (Ambiguous Name): He is warm-hearted, a bit of a goofball, but deeply caring. He has natural curiosity and love for teaching, which often spills over into his family life. His humor is a bit cheesy, and he often makes corny science jokes, much to his kids' embarrassment. He is dedicated husband and father of two. He's the peacekeeper in the household and often plays the role of the mediator between his kids and his wife when disagreements arise.
In this step, we apply two types of modifications to the generated context, resulting in three distinct context memory types. To introduce noise, we incorporate naming ambiguity by randomly creating ambiguous names from objects in the simulator environment (e.g., Key Champ, Tomato King, and Mug Star). These ambiguous names span seven categories commonly found home environments, including persons (family members, friends, neighhors, colleagues, and strangers) and brands (elec-
29

---

Published as a conference paper at ICLR 2026
tronics brands and restaurant brands). In processing the context memory, the prompt includes the extended dialogue (generated in the previous step), the task requirements, and the ambiguous name.
Context Memory Processing
Hey there, I've been thinking about what to do with the tomatoes we have. Hreally want tonetttt--keeeeeeette fresh defiiios with them, you mee what we-ean do with cooked tomatoes? Robot:
Human:
Absolutely, Alice! Cooked tomatoes can be used a variety of dishes, like sauces, soups, or even topping for salads. They enhance flavors and add nice texture the meal.
Human: That's exactly what I was thinking! I could use them to create savory sauce for pasta maybe even a comforting tomato soup. know Mug Star would love to help with that! De-we-have everythinn we need eeek-eeketee tomatoes, though?
Robot: Let me check our supplies. We have pot for cooking, some spices like salt and pepper, and a cutting board. We seem to well-equipped for cooking the tomatoes to perfection.
Human: Perfect! I want te make sure the tomatoes are well seasoned when we week them. I bet Mug Star would add his own twist with some cheesy jokes while we cook. Could you remind me how to properly prepare the tomatoes before cooking?
Robot: Certainly! First, should wash the tomatoes thoroughly, then chop them into pieces to ensure they cook evenly. After that, we can sauté them with the spices until they reach the desired consistency.
Human: Great, that that- sounds easy enought Let's get started with cooking the tomatoes. Once they're re cooked and ready, I'll need you to put cooked tomato in the refrigerator. can already imagine Mug Star popping in with funny quip about how tomatoes are technically fruit!
B.3 IMPLICIT RES REPLACEMENT
To model various forms REs, we categorize them into three levels: Explicit REs, Mixed REs, and Implicit REs. We prompt including processed context memory, task requirements, and illustrative examples to guide GPT-4o-mini in replacing explicit REs with implicit ones in either the context memory the instruction.
Implicit REs Replacement Prompt
Please do not include the word "[REs)" in the sentence "(Seed Instruction)" but do not change the original meaning of this dialogue. You can use some descriptive language to replace the word (REs} itself.
Requirements:
Please output the whole new dialogue
2. You must output the whole new dialogue, including all the sentences from Alice and Robot Please retain every instance of "(REs}" in the previous text, except for replacing "(REs)"
in the last sentence spoken by Alice.
You need to output the complete multi-turn dialogue, including the multiple turns of language from both Alice and the robot.
Here is an example: (Example)
Here is the dialogue: (Dialogue}
Output:
30

---

Published as a conference paper at ICLR 2026
Context Memory Processing
Hey there, I've been thinking about what to do with the tomatoes we have. I really want to make something fresh and delicious with them. Can you remind me what we can do with them (tomatoes)?
Human:
Robot: Absolutely, Alice! They (Tomatoes) can be used in a variety of dishes, like sauces, soups, or even as a topping for salads. They enhance flavors and add a nice texture to the meal.
Human: That's exactly what was thinking! I could use them (tomatoes) to create savory sauce for pasta or maybe even comforting soup. Do we have everything we need to cook them (tomatoes), though?
Robot: Let me check our supplies. We have pot for cooking, some spices like salt and pepper, and a cutting board. We seem to be well-equipped for cooking them (tomatoes) to perfection.
Human: Perfect! want to make sure they're well- seasoned when we cook them (tomatoes). Could you remind me how to properly prepare the fruit (tomatoes) before cooking? Robot: Certainly! First, we should wash them (tomatoes) thoroughly, then chop them (tomatoes) into even pieces to ensure they cook evenly. After that, we can sauté them (tomatoes) with the spices until they reach the desired consistency.
Human: Great, that sounds casy enough! Let's get started with cooking the fruit (tomatocs).
Once they're re cooked and ready, I'll need you put them in the refrigerator.
This example demonstrates data for three levels: "explicit REs standard context," "mixed REs standard context," and "implicit REs & standard context." In the example, the red REs represent the implicit referring expressions used to replace the original REs in the instruction (with their explicit forms shown in parentheses). The orange REs denote the implicit referring expressions substituted within the context. The blue REs indicate the first referring expression introduced in the context, which is retained under the implicit REs category. However, in the implicit REs & short context setting, the sentence containing this RE will be removed as part of the contextual information.
B.4 DATA FILTERING
We counted the number of explicit and implicit REs in each data instance and retained only those that met the requirements listed the table below.
Table 9: Number of Explicit and Implicit REs in Each Data Sample
Data Types
Explicit REs
Implicit REs
Explicit REs in Instruction
Implicit REs
in Context
in Context
in Instruction
Memory
Memory
Explicit REs Types Mixed REs Types
3
2
Implicit REs Types
22
C PROMPTS AND IMPLEMENTATION DETAILS OF PROMPTING METHODS
C. AP AND GATED AP VARIANT
The Aware Prompt (AP) (Gao al.) 2024a) explicitly instructs the planner to detect and resolve potential referring expressions before generating task plan. While AP is effective when implicit REs are present, we observe that applying AP unconditionally may lead to unnecessary reference resolution, causing hallucinated substitutions even when the original instruction is fully explicit.
To address this issue, we adopt a gated AP variant that activates AP only when the input instruction contains patterns strongly indicative of implicit referring expressions. This gating mechanism prevents AP from being triggered on explicit instructions, thereby reducing false resolutions while retaining the benefits of AP when vagueness truly exists.
31

---

Published as a conference paper at ICLR 2026
Aware Prompt
I will check whether the "Human Pending Instruction" contains implicit or ambiguous references.
(Activated only when implicit RE patterns are detected; see below.)
I understand that "Human Pending Instruction" may include vague referring expressions, and
I can infer their meaning based on context and antecedents in the preceding dialogue.
C.2 CHAIN-OF-THOUGHT
The CoT prompting strategy (Wei et al. 2022) aims resolve implicit referring expressions by encouraging the model to perform step-by-step reasoning before generating plan. However, full CoT prompts substantially lengthen the input, increasing latency and inference cost- cutt--cuulllarly for onboard deployment scenarios that rely on small language models. We therefore adopt short CoT variant that preserves the key RE-resolution reasoning step while minimizing prompt length.
Chain-of-Thouge Prompt
The "Human Pending Instruction" may contain vague referring expressions. Before planning, I will first identify any referring expressions and reason about their intended objects based on the context below, and then restate the instruction with the resolved entities. [Context Memory + Instruction]
Step: Identify referring expressions infer their referents - rewrite the instruction with explicit object names.
C.3 IN-CONTEXT LEARNING
In-Context Learning (ICL) provides the model with several demonstration examples and relies on the model's is ability to infer the intended behavior by analogy. ICI. in our setting uses few-shot examples composed of (i) identifying and grounding vague referring expressions in the demonstrations, and (ii) target task rewritten without vagueness for the model to follow.
C.4 TASK-ORIENTED CONTEXT COGNITION
TOCC separates eeferring-expression resolution from planning by first rephrasing the human in- struction a concise, unambiguous form. The prompt used in our implementation shown below.
Task-Orie -riietrreet Context Cognition Prompt
Human pending instruction may contain vague referring expressions, such as "electronic devices". "beverages", "fruits", and "containers". which not specifie items. Use the previous context below to resolve the referring expressions:
[Context memory + instruction]
Do not add extra commentary or conversation to the whole plan; only output the clear instructions.
Algorithm Srise-rreeted Context Cognition (TOCC) for Step-Level Planning
promptroce ComposePrompt( /))
2: # Construct rewriling query
3: Lew M (rompprocc)
4: # Rewrite vague instruction
5: prompt ComposePromp(( (Taa Islea)
6: # Insert rewritten instruction
7: ConstrainedDecodell M. prompt
8: Decode constraint
9: return
32

---

Published as a conference paper at ICLR 2026
In TOCC, the LLM first interprets the user's intent and rewrites the original instruction into explicit and clear instruction Letea- The planner then relies solely on Letear to generate single executable action
D USE LARGE LANGUAGE MODEL
An LLM (ChatGPT) was used only for minor polishing of the paper's language. Additionally, as described in Section 3.2 of the main text, an LLM was utilized to assist generating part of the dataset. The LLM not used for the motivation, research methodology, experimental design. All research concepts, ideas, and analyses were conceived and performed exclusively by the authors.
33


Reviews:
[{"review_id": "jeAMd8jpal", "reviewer": "Reviewer_7zBe", "summary": "This paper introduces REI-Bench, a benchmark designed to isolate and analyze how vagueness in referring expressions within human instructions affects LLM-based robot task planners. The authors automatically construct the dataset based on ALFRED, defining three levels of referential-expression difficulty and three contextual conditions. They evaluate six 7B–8B LLMs using two task planners, SayCan and LLM+P, and further investigate the causes of performance degradation under vague instructions. To mitigate the issue, they propose a simple prompting strategy, task-oriented context cognition, which decouples instruction understanding from plan generation. Overall, REI-Bench frames embodied task planning as a language understanding challenge and provides a controlled, reproducible framework for studying it.", "questions": "Please refer to the weaknesses and address the issues.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Highly practical research: Because people naturally use referring expressions, robots and embodied agents that coexist with humans must be able to interpret and resolve such expressions to ensure safe, natural, and efficient collaboration. In this context, REI-Bench provides a crucial foundation for evaluating whether embodied agents can truly comprehend human-like referring language which is a core capability required for real-world human–robot interaction.\n* Appropriate benchmark design: The 3×3 structure of REI-Bench combining three levels of referring-expression difficulty with three context types enables a clear and fine-grained analysis of language understanding capabilities across varying degrees of linguistic vagueness, contextual noise, and dialogue-memory constraints.\n* Transparent descriptions: The authors provide comprehensive documentation of the dataset construction process, detailing all prompts, constraints, and post-processing rules in Appendix B. Their three-stage pipeline comprising context generation, context processing, and referring-expression replacement ensures both reproducibility and systematic control.\n* Multi-faceted experiments and analyses: The experiments and analyses are well structured, clearly showcasing the strengths of REI-Bench. By combining two planning frameworks and six LLMs, the study provides a comprehensive factorial analysis with detailed results for each condition and accompanying ablation studies. The findings are visualized across referring-expression levels and context variations, supplemented by human baselines and an interpretable taxonomy of errors, including object omission and execution failure.", "weaknesses": "* Potentially biased seed instructions: The dataset’s seed instructions were created only from tasks that LLaMA 3.1-8B + SayCan successfully executed in AI2-THOR. Tasks such as Pick Two & Place were simply excluded as \"not reliably completed.\" This means the benchmark originates exclusively from simpler, one-object, short-horizon tasks, producing an inherent bias toward language situations that small models already handle. Statistics of instructions per ALFRED task types need to be presented. \n* Outdated planners: Both SayCan and LLM+P represent early-generation LLM-based planning frameworks. The evaluation excludes more recent paradigms that incorporate elaborative reasoning and context management, which could have yielded more informative and comparative experimental results. Consequently, the paper’s conclusions about the “limits of LLM-based planning” may not fully generalize to modern planning agents.\n* Tested with small LLMs only: All models evaluated in the experiments are small-scale LLMs with 7B - 8B parameters. Such models are known to exhibit unstable discourse reasoning and weak coreference resolution. Consequently, the sharp performance drop from explicit to implicit referring expressions may reflect model capacity limitations rather than the intrinsic difficulty of the benchmark itself. The absence of recent larger models prevents any scaling analysis that could reveal capacity thresholds for robust language understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rZUYRJDwFh", "reviewer": "Reviewer_huym", "summary": "The paper shows that  referring expressions (REs),a source of coreferential vagueness would significantly degrade LLM-based embodied task planning. It introduces **REI-Bench**, a benchmark that systematically varies RE vagueness and dialogue context, and reports up to a 36.9% absolute drop in success for off-the-shelf planners (SayCan, LLM+P). The authors propose Task-Oriented Context Cognition (TOCC), a lightweight preprocessing step that resolves REs before planning, reducing object-omission errors across nine difficulty levels. The work surfaces an overlooked failure mode in HRI and offers both a rigorous evaluation protocol and a practical, immediately deployable mitigation.", "questions": "1. Please address the concerns summarized under Weakness 1.\n2. Can you provide a quantitative comparison table vs. prior ambiguity/RE/coreference datasets across Robotics/NLP/HRI ?A clear table and analysis would show REI-Bench fills a unique, previously uncovered evaluation axis (RE-driven vagueness in planning).\n3. Provide an explicit and detailed description of TOCC, such as pseudocode or  data-flow diagram. The pipeline should be precise enough to reimplement.\n4. Compared with stronger  variants of baselines with length comparisions, normalizing token cost/latency.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. Clear problem framing motivation. The paper isolates  coreferential vagueness from referring expressions (REs) as a specific, under-explored failure mode in embodied task planning, grounding the setup in linguistic theory (RE vs. DE, bridge inference).\n\n2. Salient empirical findings. The work documents  consistent, sizable drops in task success under implicit REs and multi-turn dialogue, and includes a human reference to highlight the performance gap in natural conversational settings.", "weaknesses": "1. I have substantial concerns about how the benchmark is constructed, this is the main issue, as the dataset’s design directly affects the trustworthiness of the empirical results.\n  * Sampling/selection bias: Filtering out Pick Two & Place and seeding only from tasks successfully executed by LLaMA3.1-8B+SayCan simplifies the benchmark and risks biasing task distribution; no unfiltered vs. filtered comparison is reported to rule out selection effects.\n  * The Ambiguous-Name perturbation (brand/person name collisions) is too narrow and misses common home scenarios (homophones,, family/pet/room names, and visual look-alikes like pot/pan/bowl), limiting generalization beyond the injected pattern. More ecologically valid generators of referential ambiguity would strengthen the dataset’s realism and external validity.\n  * Although the paper argues vagueness is more common for non-experts (elderly/children), the dataset does not model age-specific linguistic phenomena nor provide any evidence to test this claim.\n\n2. “First coreferential-vagueness planning benchmark” claim lacks quantitative positioning. The paper does not provide a systematic, quantitative comparison against prior ambiguity/RE/coreference datasets across robotics, NLP, and HRI, nor a detailed dataset-level contrast table.\n\n3. TOCC is under-specified at the algorithmic level. There is no precise interface or flow diagram: inputs/outputs, whether TOCC emits a disambiguated “clear instruction”, and whether it returns executable targets\n\n4. Confounds in empirical comparisons. Although §3.3 claims CoT/ICL substantially lengthen prompts, AP/CoT/ICL are compared to TOCC without length comparisions, normalizing token cost/latency. I also concers about the fairness for baselines, which are under-specified.\nBaseline improvements could be stronger with gated AP, short/segmented CoT, or ICL distillation; these are not reported, so TOCC’s margin may be overstated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CEvi2OBwit", "reviewer": "Reviewer_YMUn", "summary": "This paper is the first to identify and study the challenge of LLM-based robot task planning when vague human instructions arise from referring expressions (REs). The authors propose a novel benchmark, REI-Bench, to evaluate the impact of such vagueness and demonstrate that implicit REs can significantly degrade the performance of planners. To address this, they introduce TOCC, a simple yet effective approach to mitigate the effects of vagueness, achieving superior performance compared to existing baselines.", "questions": "1. Can the framework be extended to address other types of vagueness beyond coreferential vagueness? How easily can it be extended, and what would be the performance implications?\n2. Will modern, larger models be able to resolve the vagueness challenge more effectively?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a noval benchmark to simulate vagueness in human instructions which allows for a standardized evaluation of how vagueness impacts robot task planning.\n2. The problem of interpreting vague instructions is crucial for making robots more accessible. This work has practical implications in real-world environments.\n3. This paper conduct extensive experiments on several baseslines and demonstrates the proposed TOCC outperforms them.", "weaknesses": "1. The paper focuses primarily on a small subset of REs, specifically coreferential vagueness. Can TOCC be extended to address more descriptive phrases such as \"the heavy thing\"?\n2. The evaluation is limited to language models with relatively small model sizes. Testing on more powerful models as a point of comparison would help contextualize the results better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)