Pre-emptive Action Revision by Environmental Feedback for Embodied Instruction Following Agents
Authors: Jinyeon Kim
Cheolhong Min
Byeonghwi Kim
Jonghyun Choi
Venue: CoRL 2024 (2024)
Keywords: Replanning
Environmental Feedback
Brain plasticity
Embodied AI

Abstract: When we, humans, perform a task, we consider changes in environments such as objects' arrangement due to interactions with objects and other reasons; e.g., when we find a mug to clean, if it is already clean, we skip cleaning it. But even the state-of-the-art embodied agents often ignore changed environments when performing a task, leading to failure to complete the task, executing unnecessary actions, or fixing the mistake after it was made. Here, we propose Pre-emptive Action Revision by Environmental feeDback (PRED) that allows an embodied agent to revise their action in response to the perceived environmental status before it makes mistakes. We empirically validate PRED and observe that it outperforms the prior art on two challenging benchmarks in the virtual environment, TEACh and ALFRED, by noticeable margins in most metrics, including unseen success rates, with shorter execution time, implying an efficiently behaved agent. Furthermore, we demonstrate the effectiveness of the proposed method with real robot experiments.

Pre-emptive Action Revision by Environmental Feedback for Embodied Instruction Following A Agents
Jinyeon Kim 12, Cheolhong Min
Byeonghwi Kim Jonghyun Choil.t
'Sooul National University
Yonsei University
inneoonkim@oonsei... ac kr, {cheolhong min, trrooghwkkmm.fonglyuucchooo @snu kr
https//pred-agentt .tt.:/erdddaeett.gituu.....o. github.io
Goal: Cook two slices of potato and place them on a clean plate.
Attribute Feedback
Presence Feedback
Uhought knile only drawer,
The plate clean!
expected be dirty.
Glean plate
Pick up knife mraa-rrwwrr
Pick up a knife on a countertop
Skip cleaning plate
Appearance Feedback
Objects Relationship Feedback
was plate.
thought potato wouldn't be plate, but already plate.
But upon lock, pan.
Piek-up plate
Releeate potato
Put down and search for plate
Find another
Figure I: Overview of the proposed method (PRED). The agent needs adapt unexpected environmental state changes that are different plate, puts down pan when plate is needed, grabs a knife from the countertop without further searching, and ignores moving a potato that is already on its destination plate. By the proposed PRED powered by LLMs, the agent effectively achieves the goal of serving two slices of potato clean plate without making mistakes.
environment that initially conceived: skips cleaning an already clean
Abstract: When we, humans, perform a task, we consider changes in environments such objects' arrangement due to interactions with objects and other reasons; egg., when we find mug to clean, if it is already clean, we skip cleaning it. But even the state-of-the-art embodied agents often ignore changed environments when performing a task, leading to failure to complete the task, executing unnecessary actions, or fixing the mistake after was made. Here, we propose Pre-emptive Action Revision by Environmental feeDback (PRED) that allows an embodied agent to revise their action in response the perceived environmental status before it makes mistakes. We empirically validate PRED and observe that it outperforms the prior art on two challenging benchmarks in the virtual environment, TEACh and ALFRED, by noticeable margins in most metrics, including unseen success rates, with shorter execution time, implying an efficiently behaved agent. Furthermore, we demonstrate the effectiveness of the proposed method with real robot experiments.
Keywords: Replanning, Environmental Feedhack, Brain plasticity, Embodied AI
"qqual contribution. 'Corresponding author.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
Building robotic assistants that understand natural language and surroundings and perform tasks has long been an ambitious research goal. Recent advances in computer vision [1, 2, 3] and natural language processing [4, 5, 6, 7] have significantly contributed to the development of such agents, enabling them to carry out various tasks [8. 9, 10, 11, 12, 13, 14] in diverse environments [15, 16, 17].
Typically, agents plan actions based on initial environmental states [18, 19], but environmental discrepancies between expected and actual states often cause task failures due to misperception or incomplete environmental exploration. Unlike artificial agents, humans and animals adapt to environmental changes through brain plasticity [20, 21, 22], revising behaviors based on experience to avoid mistakes. In light of this, ask: Can embodied agents benefit from similar adaptabilitty
Drawing inspiration from neuroscience, we propose Pre-emptive Action Revision by Environmental feeDback (PRED), an instruction that follows the embodied agent that adjusts its behavior by perceiving environmental discrepancies such as environmental feedback and revising action plans using large language models (LL.Ms). For environmental discrepancies, considering that object perception plays an important role numerous embodied tasks, we particularly focus on four distinct environmental discrepancies related to objects: 1) object presence [23, 24], 2) object appearance [25], 3) object attributes [26, 27], and 4) object-object relationships [28, 29].
To address these discrepancies, we propose four components: 1) Dynamic Target Adaptation (DTA) that dynamically modifies navigation targets using object presence discrepancies, 2) Object Het- erogeneity Verification (OHV) that verifies whether an interacted object intended by examining object appearance discrepancies between the initial and subsequent perceptions of the object, 3) Muribute-Driver Plan Modification (APM) that modifies the original state-changing actions, such as cleaning an object, using object attribute discrepancy, and 4) Action Skipping by Relationship ( (ASR) that refrains agents from taking unintended actions using object relationship discrepancy. Previous approaches [26, 30] revise their original plans after taking actions and encountering failures. But failure may result in irreversible consequences e.g., splitting milk on the floor. Here, we propose preemptively action revision policy for the initially planned action sequences to avoid nonrecoverable failures such as irreversible state transitions in the mentioned example [12, 31J.
We empirically validate PRED in two challenging benchmarks, TEACh [13] and ALFRED [12], for embodied instruction following, as well as with real robot experiments. We observe that PRED outperforms prior arts notably in most metrics, including the main metric, the unseen success rate.
2 Related Work
Embodied instruction following agents. Developing agents that understand natural language to achieve goals is daunting challenge. To develop such agents, previous benchmarks [32, 33, 34] focus on language comprehension and navigation. For example, [32] tasks robot with inferring the next steps based on natural language, and [33] requires reaching a specified destination. However, these focus mainly on navigation without object interaction, limiting their application to more complex tasks (e.g., preparing breakfast using a toaster and coffee machine).
To tackle tasks beyond navigation, recent benchmarks [13, 14, 35, 36] include object interaction, which requires agents to understand language and manipulate objects. Early methods [37, 14, 38, 39, 40] learn to map multi-moda inputs (i.e., egocentric observation, and language) to the corresponding actions and object locations. However, these approaches often need large training datasets for high performance, which is expensive and sometimes unfeasible.
To address the data-scarcity issue, recent approaches [41, 42, 26, 30] use deterministic algorithms, such A FMM [43], for accurate obstacle-free navigation on semantic spatial maps [44, 18, 42, 30], significantly improving their performance. Inspired by this, we also exploit [43] for navigation.
Planning and revising using large language models. Recent work [30, 45, 46, 47] leverages LLMs to revise their plans after agents perform the actions and fail. However, some failures are

---

Input
Prompt
Instruction
Initial Action Plan
System Prompt
Revised Action Plan
Commander
Driver
(Pickup, Bow))
Comprsted
Action
Current Action Plan
Commander
There
(Put, SinkBasin) Cumret
(ToggleOn. Faucet)
Action
(Put, SinkBasin) (ToggloOn, Faucet)
(Pet SinkBesin)
Heggleonn; Fovees)
Hoogglofff
Driver Next?
LLM
(ToggleOff, Faucet)
(ToggleOft, Faucet)
D0PT-4
(Pickupp Bowll)
Commander
(Pickup, Bow))
(Pickup, Bowil
(Pour, HousePlant)
Water
(Pour, HousePlant)
(Pour, HousaPlant)
Driver
Action
Sune, see the plant the ratel
Feedback Information
RGB Image
Update Expected States
Environmental Feedback
Task is completed!
Appcarance Discrepancy
Attribute Discrepancy
Expected
Presence Discrepancy
Object
Semantic Data
Mask
Obj
(Pour, HousePlant)
Figure Workflow the proposed Pre-emptive Action Revision by Environmental feeDback (PRED), first generate an initial plan based language instruction using a L.LM and predict initial and target ohject states to achieve a goal. Simultancously, we predict the environmental states based on the predicted semantics. We then compare the anticipated and actual environmental states and maintain the discrepancies hetween them. If there discrepancies. we them environmental feedback, denoted by 'Feedback Information, to revise the original plan, denoted by 'Current Action Plan.'
irrecoverable (e.g., a broken plate), causing a task failure. Although these methods revise actions after the failure, we revise actions beforehand to avoid unnecessary steps.
Another line of work [48, 49] tries to solve task planning and failure correction by focusing on errors. However, approach explores the possibility of skipping unnecessary actions for efficiency. Our method enables agents to adapt their actions based the environment and improve efficiency like humans without needing detailed failure information.
To correct the plan by receiving environmental feedback, there some work that takes advantage of ground truth feedback [50, 51, 52, 53, 30). On the other hand, [30, 46) updates their plans, but does not focus object-centric representation that can lead interaction with unintended objects. [47, 54] try to address environmental discrepancies, but can only address limited discrepancy. For the sake of space, we elaborate the details of the mentioned works in the Appendix A.
3 Approach
Recently proposed embodied agents use LLMs [30, 46, 55] for commonsensical reasoning and semantic spatial maps [42, 44, 18] for path planning. However, these agents often encounter scenarios where their anticipated environmental states differ from the ones they actually perceive. These scenarios can result in unintended or incorrect actions and consequently lead to inefficiency or even failure in completing tasks. Inspired by humans and animals that adapt changing environments, we propose PRED that pre-emptively adjusts plans using environmental discrepancies as feedback. Figure 2 illustrates the workflow of PRED. Further details of PRED can be found in the Appendix C.
3.1 Revising Actions using Environmental Feedback
We aim to learn a policy that generates a modified action sequence for embodied agents [12, 13] by perceiving environmental changes. Specifically, we learn function V A A, that maps visual observation set, {um p- and an (N-length) action sequence, {a,, 1. 1n= into modified 1== [ V and A denotes all possible observation and action sets.
(K- length) action sequence, {ak
3

---

We consider an embodied agent, function, To V x X + A. that maps natural language instruction, x E X, and the current visual observation, f, E V, into an action, a, E A. We expect the generated action sequence, {at 1 to transform the initial environment state, 8; E S, to the desired state, 8 E S, where each X' and denotes a set of possible natural language instructions and environment states and T' a time budget. We detail our agent's architecture in the Appendix B.
In particular, when an agent faces unexpected scenarios caused by 'differences' between inferred and observed states, referred to as 'environmental discrepancies, PRED revises a original plan, {a,, } Jn by querying large language model, 4. with a prompt, P. P concatenates a system prompt, Ps. for a general description and guide of the task, the original plan, and a feedback prompt, Py, which describes the discrepancy encountered as environmental feedback generated by the LLM. Then L receives P and produces revised plan, {at as:
{d C(P) where P [Pas Pamiiin Pyl.
(1)
Additional details of revising actions are provided in the Appendix C.I. To build a feedback prompt, we consider four types of envirommental discrepancies caused by the presence, appearance, attributes, and relationships of objects based visual information that occupies a large proportion of sensory information perceived by humans [56, 57J. We propose four modules for respective discrepancies: Dynamic Target Adaptation (DTA), Object Heterogeneity Verification (OHV), Attribute- Attribute-rriven Plan Modification (APM), and Action Skipping by Relationship (ASR). We detail the proposed modules below and provide their examples of a system prompt, a current action plan, how the feedback prompt generated, and an LLM output in listings in the Appendix G.
3.1.1 Presence Discrepancy + Dynamic Target Adaptation
When agent is conducting a task, it may encounter scenarios where a target object is present in unexpected places. For example, in the task, 'boil a potato in the refrigerator using pot,' the agent may expect the refrigerator, possibly containing the potato, as its navigational target. However, the agent may find another potato on different objects, such as a table. In this case, revising the action plan by adapting to environments where target objects can be easily found may improve the efficiency and effectiveness of navigation. Here, we denote by an object presence discrepancy the difference between an expected location and an actual one for a target object to be found.
To address this, we propose Dynamic Target Adaptation (DTA) which detects object presence discrepancy and provides a feedback prompt describing it. For this, the agent first compares an inferred place of a target object, o, with one perceived and maintained in the agent's 's memory (see the Appendix B for more details), Ze, at the current time step, t. If the agent has previously observed the target object in a different place from the inferred one (i.e., 0 E Z₂) before reaching it, DTA returns the presence discrepancy as a feedback prompt, Py, indicating that the target object is not in the expected location (e.g., P "The target is either directly visible to the agent its information is already stored on the map.?").
3.1.2 Appearance Discrepancy -> Object Heterogeneity Verification
An agent often fails to interact with task-relevant objects due to misperceptions influenced by lighting, occlusions, and varying appearances from different viewpoints. This issue can be mitigated by examining objects from multiple perspectives. For example, an agent might mistake a cup for a mug due to object recognition errors from far distance.
To address this issue, we propose Object Heterogeneity Verification (OHV), which recognizes appear- ance discrepancies and provides feedback accordingly. Here, we define an appearance discrepancy as the difference in an object's 'ss predicted identities (i.e., classes) from the appearances observed in various viewpoints. The proposed verification requires the agent to pick up objects and change its view. That is, when the agent interacts with an object, verifies if the object intended by comparing its predicted classes that can be different due varying appearances from various viewpoints. These actions allow the agent to the object without occlusion, thus encouraging it to properly identify the object (i.e., mug) by the observations from various viewpoints.

---

Formally, let be a predicted object class from the appearance of a ith viewpoint of an object, where i E (0,...,T] denotes a viewpoint index among predefined viewpoints. = 0 is the viewpoint at the time of interaction (see the Appendix C.2 for more details). If the agent encounters a different predicted class (i.e., co + 0;) from any different viewpoint i > O, contrary to the expected class (i.e., Vi : co = q.), OHV detects an appearance discrepancy and specifies this in a feedback prompt, py, indicating that the object is not intended (e.g., P, "Aftter checking, the picked up object's 's mask is detected, but the object is not the desired one...").
3.1.3 Attribute Discrepancy - Attribute-Driven Plan Modification
Language instructions often lack detailed environmental descriptions, potentially causing the agent to do redundant actions or miss important actions due to unknown object attributes. Here, an attribute refers to the physical state of an object depending on its affordances' [13, 26J. We can mitigate redundant or missed actions from unknown attributes by checking the target object, o, its expected attribute, &.. and its actual observed attribute, &o.
For example, in a task, "clean mug and fill it with coffee," the agent expects that o (i.e., a mug) needs to be cleaned because it is dirty, so the expected attribute of the target can be represented as 6. = (Dirty). However, the agent might find a clean mug during exploration, where the detected target object's attribute is clean (i.e., t. = [Clean)). In this case, the agent can skip redundant actions in the original plan (i.e., cleaning the mug first), allowing efficient task completion. We denote an attribute discrepancy as the difference between the expected and observed attributes of an object.
For this, we propose Suribute-Driven Plan Modification (APM) to detect attribute discrepancies and provide environmental feedback. When the agent detects o, it captures a cropped image ex- clusively of the target from the current view. Then, we utilize an attribute detector, H. which takes the cropped image vn as input and predicts th (see the Appendix C.3 for more details). If Φo from H. does not match 6o (i.e., 6. * do), APM describes this attribute discrepancy as feedback prompt Py (e.g., "The target object in the test image has been either cleaned, as determined by the similarity check."').
3.1.4 Object-Object Relationship Discrepancy - Action Skipping by Relationship
Rearranging objects [10, 58] often poses challenges when the agent relocates multiple objects, potentially with the same look, of the same class, making it difficult for the agent to decide which object to move. To address this, the agent considers the current relationship, fos and the expected relationship, for between the target object, o, and its placement object, Op- Here, the relationship represents the spatial relationship between and Op- We write this r, = (o, Op).
For instance, if instructed to "place two pillows on the sofa," the agent assumes that o (i.e., a pillow) is not on the final place (i.e., sofa) and needs to be moved. Thus, the expected relationship can be defined as the pillows that are not on the sofa, represented as r. = -ppiloow, sofa). However, during exploration, the agent detects pillow on the sofa, the current relationship becomes ro (pillow, sofa), i.e., f. A To. and a relationship discrepancy occurs. If the agent ignores such discrepancy and continues interact with i.e., meaninglessly moving the pillow from the sofa to the sofa, it may not be able to complete the task.
To mitigate meaningless relocation, we propose Action Skipping by Relationship (ASR), which detects the relationship discrepancy and provides a corresponding feedback prompt. To obtain ra, we first predict the masks of objects from the current egocentrie view, {m, where me denotes the mask of We then find the most 'adjacent' mask, m,, of Op. mg and regard that is currently on Op (see the Appendix C.4 for more details). If the currently detected target object is already in the fi- nal place (i.e., F. # ra), ASR describes this relationship discrepancy in feedback prompt Pj, such as "After checking, it is found that the second object has already been placed in the desired location as indicated by the interaction mask, ...."
"We object attributes supported by A12-THOR [15) which evaluation benchmarks [12, 13]
5

---

Table 1: Comparison with the state of the arts in the TEACh benchmark. The path-lengtt-wweighted (PLW)
metrics are given in the parentheses each value. The highest and second highest values per fold and metric are shown in bold and underline, respectively.
TTD
EDH
Model
Unseen
Seen
Unseen
Seen
GC
GC
SK
ET [371
0.45 (0.12)
0.35 (0.59)
1.02 (0.17)
1.42 (1.82)
7.50 (0.90)
9.10 (1.70)
10.20 (1.70)
15.20 (1.10)
JARVIS [41]
1.50
3.10 (1.00)
5.40 (4.50)
15,80 (2.60)
16.00 (8.20)
15.30
22.60 (8-70)
FILM [42]
2,50
(1.00)
6.10 (2.50)
6.30 (2.60)
5.80 (11.60)
10:20 000
18.30 (2.70)
14.30 (2.10)
26-40
DANLI [26]
7.86 (3.20)
6.79 (6.57)
4.97 (1.80)
10.50 (10.27)
166.98 (7:24)
23.44 (19.90)
17.76 (9:228)
24.93 (22 200
HELPER [30]
13.73 (1.61)
11.17 (1.56)
12:15 (1.79)
18.02 (9,25)
17.40 (2.91)
25.86 (7.90)
18.39 (1.00)
32:09 (9.81)
PRED (Ours)
19.77 (5.16)
(8:31)
20.99 (4.64)
(11.03)
21.69 (4.44)
26.83 (7-46)
21.71 (4.62)
32.78 (10.39)
4
Experiments
We briefly explain the benchmarks, baselines, and evaluation metrics used for the experiments (see Appendix for details of benchmarks and baselines). We also validate our method in both simulation and real-world manipulation experiments. The experiment setup is detailed in Appendix E.
Benchmarks. We employ two challenging long- horizho instruction following benchmarks for embodied agents, TEACh [13] and ALFRED [12]. In TEACh, we evaluate our PRED in two sub- benchmarks, Trajectory from Dialog (TYD) and Execution from Dialog History (EDH). TfD requires the agent to solve long-horizon household tasks by understanding dialogs, while EDH requires performing session-specifie portion of the TfD tasks. ALFRED provides declarative instructions consisting of goal statement and step-by-step instructions that describe how to complete task.
Baselines. We compare PRED with recent state-of state-of-the-art methods as baselines for both benchmarks. For the TEACh benchmark, we compare ours with FILM [42]. DANLI [26], and HELPER [30]. For the ALFRED benchmark, we adopt HLSM [44], FILM [59]. and CAPEAM [18] as baselines.
Metrics. The primary metric is the success rate (SR), the ratio of the completed tasks. The goal condition success rate (GC) denotes the ratio of the satisfied goal conditions. To measure efficiency, the path-length--weighee (PLW) score penalizes SR and GC by the length of the actions taken.
4.1 Comparison with State of the Arts
We compare PRED with prior state-of-the-ee art methods on the TEACh and ALFRED benchmarks summarized in Table and Table 2, respectively. Additionally, we provide the results of a new valida- tion and test set in EDH, used exclusively by DANLI [26] and re-split from the original validation set, to ensure a fair comparison (see the result table in the Appendix E.2). PRED outperforms other baseline models in both benchmarks by noticeable margins in SR and GC, demonstrating its efficacy.
In the TEACh benchmark, in TfD and EDH setups, we observe that PRED outperforms the previous methods in unseen/seen environments for SR and GC, which implics the effectiveness of our proposed PRED. In addition, our model shows a larger performance gap between ours and prior art on TYD, more changeable setup than on EDH. This may be because TíD requires more interactions with objects than EDH, which performs specific portion of the TfD tasks. Thus, our proposed methods have more opportunities to be applied, leading to greater improvement in TID compared to EDH.
We observe that DANLI [26] achieves better PLW scores in the EDH setup. We believe that its 31) map tracking each instance's 's location, including height, eliminates the need for vertical scanning in navigation, unlike our top-down 2D map. In addition, its recovery plans' effectiveness is based on a lot of human-defined plans for all exceptions, improving PLW scores.
Table 2 shows the prior arts and PRED's performance in the ALFRED benchmark with few different settings. We include the 'Reproduced' section because the reproduced results of previous methods differ slightly from the originally reported ones. As shown in Table 1, we observe that our method outperforms the prior arts all metrics, implying the effectiveness of the proposed components.

---

Diverse initial states. In the AL- FRED benchmark, all objects' states are static at the beginning of every task, indicating the initial states of ob- jects are always fixed once a task for the agent perform is given. For ex- ample, in the task of moving cleaned mug, the initial states of all mugs in the environment are always set to be "dirty.' However, this evaluation does not fully address environmental dis- crepancies caused by different initial states of objects (e.g. cleaning a mug that is already clean), potentially re- sulting in unexpected scenarios.
Table 2: Comparison with the state of the arts in the ALFRED benchmark. The path-length-weiie weighted (PLW) metries are given in the parentheses for each value. The highest and second high- est values per fold and metric are shown in bold and underline, respectively. The 'Reported' and 'Reproduced' sections show the methods' performances from the paper and our reproduction results. 'Reproduced w/ Diverse Initial States' shows performances in di- verse initial states (Section 4.1).
Model
Test Unseen
Test Seen
SR
GC
Reported
[44]
(5.55)
30.31 (9.99)
29.94 (8.74)
41.21 (14.58)
FILM [59]
24.46(9.67)
34.75 (13.13)
25.77 (10,39)
36.15 (14.17)
CAPEAM (18)
43.69
(22.76)
47.36 (19.03)
54.38 (23.78)
Reproduced
HI SM [44]
21.32 (5.89)
31.09 (10.39)
31.90 (9.75)
43.22 (15.30)
FILM [59]
23.61 (15.10)
36.90 (12.99)
25.77 (10.58)
35.43 (14.62)
CAPEAM (18)
41.79 (18.07)
53.93 (23:41)
45.14 (18.79)
52.82 (23.25)
PRED (Ours)
46.96 (20.58)
57.35 (24.72)
51.40 (21.14)
59.04 (25.52)
To further address these discrepancies in the ALFRED benchmark, we in- tentionally modify the initial states of objects to have diverse ones and de- note these modified ones as 'diverse initial states.' For example, in the task of moving cleaned mug above, the agent may encounter already cleaned mugs and the agent can revise its plan for efficient and effective task completion. We observe that our PRED outperforms all prior approaches in this setting, implying that our PRED has better capability to adapt to environments with objects' diverse initial states.
Reproduced w/ Diverse Initial States
TILSM [44]
19.03 (5.60)
28.21 (9.64)
23.27 (7.59)
31.56 (12.60)
FILM 1591
(7.43)
(12.14)
13.89 (5.68)
(10.26)
CAPEAM (18)
24.00 (9.39)
31.92 (14.85)
25.18 (10,78)
32,97 (15.96)
(Ours)
32.31 (12.48)
42.62 (17.87)
35.09 (15.02)
43.27 (19.95)
4.2 Ablation Study
To assess the impact of each proposed component-DTA, OHV, APM, and ASR -we conduct ablation studies and summarize the results in Table 3. Additional ablation results, including other splits in TEACh and ALFRED, are provided in Appendix F. In our experiments, no simultaneous envirommental discrepancies were detected, but our PRED is designed to handle multiple discrepancies at once. Here, '(x: vs. (y), denotes a comparison between the x and y rows in Table 3.
No DTA. We observe that ablating DTA ((a) vs. (b)) leads to noticeable drops (up to 2.68% in ALFRED test unseen) in all metrics for TfD, EDH, and ALFRED. We believe that our agent without DTA does not consider object presence discrepancies, causing repeated unnecessary object interaction. This can increase the chance of interaction failure and thus, reduce task success rates.
No OHV. Second, we ablate OHV ((a) vs. (c)) to assess the impact of considering differences in object appearances. Ablating OHV results in significant performance drops, with a 4.57% SR decrease in the TfD valid unseen and a 3.01% SR decrease in the ALFRED test unseen. We empirically observe that the agent suffers from misperceptions such light reflection and occlusion, causing it to interact with irrelevant objects. Unlike the agent with OHV, the ablated one continues without verifying whether it interacted with the correct object, leading to task failures due to misperceptions.
No APM. When APM is absent from PRED ((a) vs. (d)), the agent may struggle to detect changes in the attributes of the object, leading to unnecessary actions or the omission of key actions. This results in performance drops in all metrics (e.g.. 7.98% SR in ALFRED test unseen). The oversight in object attributes can cause the agent to unnecessarily change an object's state (e.g., cleaning an already cleaned object) or attempt infeasible actions to the objects that are not in an available state for interaction. This increases failures and time steps, reducing the likelihood of task completion.
No ASR. We ablate ASR from our agent ((a) vs. (e)). Without ASR, the agent cannot comprehend the relationships between objects, which makes it attempt move the objects again that already satisfy the goal relationship (e.g., Plate should be in Sink). This may lead to achieving insufficient goal states the agent moved 'two' objects but actually, it did only 'one" object. Due to this, we observe that 2. 2.61% SR drop in TíD valid unseen and a 4.06% SR drop in ALFRED test unseen.
7

---

Goal Place blue in each of the yellow cupboards
Initial Plan |Pickup, BlueCan), iPut, YellowCupboara), (Pickup, BlueCan), (Put, YellowCupboard)
BlueCanzia BlueCansi
Euucan
RED w/o ASR
Kin Cuposandaaue
VAlTA
1=2
t
X
t=7
Detection
Pick Place
Detection
Pick Place
Fail
BluaCar
Daue
BlueCan VollowCupboard
RED ASR
Captasanioor Cuptrandiass
Revised Plan
(Pickup, BlueCan) (Put, YellowCupboard)
t=1
E=
Detection Replan
Pick Place
Success
Figure 3: Qualitative analysis of ASR in real-robot experiments. the leftmost figure, the agent initially checks the environment. Without the ASR method (top row), the agent follows the initial plan and repeatedly moves blue cupboards contain blue can. In contrast, with the ASR method (bottom row), the agent detects discrepancy the object-object relationship, recognizing that a can is already placed the right cupboard. The agent then moves the left blue can to the empty cupboard, successfully completing the task.
the yellow cupboard based
high confidence
However, the task fails not all
No PRED. Finally, we conduct abla- tion study when all the methods are ex- cluded ((a) vs. (f)).
Table Ablation study in TEACh and ALFRED for each proposed component. The path-length-weightee (PLW) metrics given in the parentheses for each value. (b) (e) show the performances of PRED without each component.
TDD
EDH
ALFRED
Model
Valid Unseen
Test Unseen
We observe a substan-
SR
GC
SR
GC
SR
GC
(6.16)
(8:31)
21.00 (4.44)
(7.46)
32.31 (12..48)
(17.8((7787)
tial performance gap across all splits includ-
(b) w/o DTA
17:65 (1.63)
13.39 (6.63)
20.67 (3.52)
25.74 (6.62)
29.63 (11.79)
39.19 (16.66)
OHV
15.20 (4.22)
12.58 (6.57)
20.35 (3.44)
22.80 (6.06)
29.30 (11.33)
38.95 (16.54)
ing valid unseen in TEACh and test un- seen in ALFRED, which demonstrates that PRED greatly contributes to solving the tasks.
w/o APM
(6.46)
19.80
(0.85)
32.11 (13.98)
(c) wio ASR
(4.44)
15.30 (7.41)
20.40 (4.06)
28.00 (7.42)
28.25 (1 1.22)
38.98 (17.52)
12.91 (2.91)
10.86 (5.62)
18.65 (3.09)
19.06 (5.80)
23.35 (8.72) 31,25 (14.61)
4.3 Qualitative Analysis
As shown in the top row of Figure (without ASR), the task fails despite completing the initial plan. In contrast, seen in the bottom row with ASR, considering object-object relationships enables faster and more successful task completion. We also provide qualitative analyzes of DTA, OHV, APM, and ASR including simulated environments and real robot experiments in Appendix G.
Conclusion
We propose PRED that revises their behaviors based on perceived environmental discrepancies inspired by brain plasticity of humans and animals before failure. Given perceived environmental discrepancies, PRED builds prompts, and queries LLMs to generate revised plan. To address these environmental discrepancies, we propose DTA, OHV, APM, and ASR for replanning for effective and efficient task completion. We observe that our PRED outperforms previous methods notably in two challenging embodied instruction following benchmarks, TEACh and ALFRED.
Limitations and future work. The environmental discrepancies are perceived based semantic information (e.g., object masks, semantic spatial maps, etc) predicted from single egocentric observation and therefore may not be accurate, possibly leading to inaccurate plan modification. A promising future direction is to modify a plan even with these possibly inaccurate discrepancies.

---

Acknowledgments
This work was partly supported by the IITP grants (No. ww..R--0022112200777 No.RS--02221122011 13, No.RS-2022-11220955 No. ww..S-2022-11208877
No.RS-2020-11201361 (Yonsei AT), No.RS-2021-
II21 1343 (SNU AI), No.RS-2021 RRR-2021-1122266 (AI Innov. Hub), No.RS-2022-1122 funded by the Korea government (MSIT). Authors thank to Seungmin Kim and Prof. Sungjoon Choi at Korea University for helping us about real robot experiments.
References
[1] K. He, X. Zhang. S. Ren, and Sun. Deep residual learning for image recognition. In CVPR. 2016.
[2] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[3] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in supervisdd vision transformers. In ICCV, 2021.
[4] S. Hochreiter and J. Schmidhuber. Long short-term memory. In Neural Computation, 1997.
[5] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014. 2014.
[6] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[7] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
[8] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and
A. van den Hengel. Vision-anddlaaguuage navigation: Interpreting visually-grounded navigation instructions real environments. In CVPR, 2018.
[9] Krantz, E. Wijmans, Majumdar, D. Batra, and S. Lee. Beyond the nav-graph: rision-and- language navigation in continuous environments. In ECCV, 2020.
[10] L. Weihs, M. Deitke, A. Kembhavi, and R. Mottaghi. Visual room rearrangement. In CVPR, 2021.
[11] K. Ehsani, W. Han, A. Herrasti, E. VanderBilt, L. Weihs, E. Kolve, A. Kembhavi, and R. Mot- taghi. Manipulathor: A framework for visual object manipulation. In CVPR, 2021.
[12] M. Shridhar, J. Thomason, Gordon, Y. Bisk, W. Han, Mottaghi, Zettlemoyer, and D. Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In CVPR, 2020.
[13] A. Padmakumar, J. Thomason, A. Shrivastava, P. Lange, A. Narayan-Chen, S. Gella, R. Pira- muthu, G. Tur, and D. Hakkami- Teach: Task-driven embodied agents that chat. In AAAI, 2022.
[14] X. Gao, Q Gao, R. Gong, K. Lin, G. Thattai, and G. Sukhatme. Dialfred: Dialogue-enabled agents for embodied instruction following. IEEE RA-L, 2022.
[15] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv, 2017.

---

[16] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: A Platform for Embodied Al Research. In ICCV, 2019.
[17] M. Deitke, W. Han, A. Herrasti, A. Kembhavi, E. Kolve, R. Mottaghi, J. Salvador, D. Schwenk, E. VanderBilt, M. Wallingford, et al. Robothor: An open simulation-to-real embodied ai platform. In CVPR, 2020.
[18] B. Kim, J. Kim, Y. Kim, C. Min, and J. Choi. Context-aware planning and environment-awan memory for instruction following embodied agents. In ICCV, 2023.
[19] W. Huang, Abbeel, D. Pathak, and Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In ICMI., 2022.
[20] B. Draganski, C. Gaser, V. Busch, G. Schuierer, U. Bogdahn, and A. May. Changes in grey matter induced by training. In Nature, 2004.
[21] R. J. Zatorre, R. D. Fields, and H. Johansen- -Berg. Plasticity in gray and white: neuroimaging changes in brain structure during learning. In Nature neuroscience, 2012.
[22] A. Gutchess. Plasticity of the aging brain: new directions in cognitive neuroscience. In Science, 2014.
[23] Yang, Z. Ren, M. Xu, X. Chen, D. J. Crandall, D. Parikh, and D. Batra. Embodied amodal recognition: Learning move to perceive objects. In ICCV, 2019.
[24] K. Kotar and R. Mottaghi. Interactron: Embodied adaptive object detection. In CVPR, 2022.
[25] A. Inceoglu, E. E. Aksoy, A. C. Ak, and S. Sariel. Fino-net: A deep multimodal sensor fusion framework for manipulation failure detection. In IROS, 2021.
[26] Y. Zhang, J. Yang, J. Pan, S. Storks, N. Devraj, Z. Ma, K. Yu, Y. Bao, and J. Chai. DANLI: Deliberative agent for following natural language instructions. In EMNLP, 2022.
[27] Zhou, P. Yin, and Neubig. Hierarchical control situated agents through natural language. In NACCLW, 2022.
[28] W. Yang. X. Wang, A. Farhadi, A. Gupta, and R. Mottaghi. Visual semantic navigation using scene priors. In ICLR, 2019.
[29] S. Y. Gadre, K. Ehsani, S. Song, and R. Mottaghi. Continuous scene representations for embodied ai. In CVPR, 2022.
[30] G. Sarch, Y. Wu, J. Tarr, and K. Fragkiadaki. Open-ended instructable embodied agents with memory-augmented large language models. EMNLP Findings, 2023.
[31] M. Salem, G. Lakatos, Amirabdollahian, and K. Dautenhahn. Would you trust a (faulty) robot? effects of error, task type and personality on human-robot cooperation and trust. In HRI, 2015.
[32] Thomason, M. Murray, M. Cakmak, and Zettlemoyer. Vision-and-dialog navigation. In CoRL, 2020.
[33] H. De Vries, K. Shuster, D. Batra, D. Parikh, J. Weston, and D. Kiela. Talk the walk: Navigating new york city through grounded dialogue. arXiv, 2018
[34] K. Nguyen, D. Dey, C. Brockett, and B. Dolan. Vision-based navigation with language-base based assistance via imitation learning with indirect intervention. In CVPR, 2019.
[35] B. Kim, M. Seo, and J. Choi. Online continual learning for interactive instruction following agents. In ICLR, 2024.
10

---

[36] T. Kim, C. Min, B. Kim, J. Kim, W. Jeung. and J. Choi. Realfred: An embodied instruction following benchmark in photo-realistic environment. In ECCV, 2024.
[37] A. Pashevich, C. Schmid, and C. Sun. Episodic transformer for vision-and-language navigation. In ICCV, 2021.
[38] K. Singh, S. Bhambri, Kim, R. Mottaghi, and Choi. Factorizing perception and policy for interactive instruction following. ICCV, 2021.
[39] B. Kim, S. Bhambri, K. P. Singh, R. Mottaghi, and J. Choi. Agent with the big picture: Perceiving surroundings for interactive instruction following. In Embodied Al Workshop CVPR, 2021.
[40] S. Bhambri, B. Kim, and J. Choi. Multi-level compositional reasoning for interactive instruction following. In AAAI, 2023.
[41] K. Zheng, K. Zhou, Gu, Fan, Wang. Z. Di, X. He, and X. E. Wang. Jarvis: A neuro- symbolic commonsense reasoning framework for conversational embodied agents. arXiv, 2022.
[42] S. Min, Zhu, Salakhutdinov, and Bisk. Don't copy the teacher: Data and model challenges in embodied dialogue. In EMNLP, 2022.
[43] J. A. Sethian. A fast marching level set method for monotonically advancing fronts. In PNAS, 1996.
[44] V. Blukis, Paxton, D. Fox, A. Garg, and Y. Artzi. A persistent spatial semantic representation for high-level natural language instruction execution. In CoRI., 2022.
[45] Z. Liu, A. Bahety, and S. Song. Reflect: Summarizing robot experiences for failure explanation and correction. In CoRI. 2023.
[46] C. H. Song, J. Wu, C. Washington, B. M. Sadler, W.-L. Chao, and Y. Su. LIm-planner: Few-shot grounded planning for embodied agents with large language models. In ICCV, 2023.
[47] A. Prasad, A. Koller, M. Hartmann, P. Clark, A. Sabharwal, M. Bansal, and T. Khot. Adapt: As- needed decomposition and planning with language models. arXiv preprint a:X1722311.05772, 2023.
[48] S. S. Raman, V. Cohen, E. Rosen, I. Idrees, D. Paulius, and S. Tellex. Planning with large language models via corrective re-prompting. In Foundation Models for Decision Making Workshop @ NeurlPS, 2022.
[49] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. In ICRA, 2023.
[50] N. Shinn, B. Labash, and A. Gopinath. Reflexion: an autonomous agent with dynamic memory and self-refiection. arXiv, 2023.
[51] M. Skreta, N. Yoshikawa, S. Aiellano-Rubache Z. Ji, L. B. Kristensen, K. Darvish, A. Aspuru- Guzik, F. Shkurti, and A. Garg. Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting. arXiv, 2023.
[52] T. Silver, S. Dan, K. Srinivas, J. B. Tenenbaum, L. Kaelbling, and M. Katz. Generalized planning in pddl domains with pretrained large language models. In AAAI, 2024.
[53] W. Huang, F. Xia, T. Xiao, H. Chan, Liang, P. Florence, A. Zeng. J. Tompson, Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. In CoRL, 2022.
11

---

[54] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv22210.03629, 2022.
[55] Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In ICRA, 2023.
[56] G. Zimbardo and I.. Ruch. Psychology and life. Scott, Foresman, 1975.
[57] F. Hutmacher. Why is there much more research on vision than on any other sensory modality? In Frontiers in psychology, 2019.
[58] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martín-Martinn C. Wang, G. Levine, M. Lingelbach, J. Sun, M. Anvari, M. Hwang, M. Sharma, A. Aydin, D. Bansal, S. Hunter, K.-Y. Kim, A. Lou, C. R. Matthews, I. Villa-Renteria, J. H. Tang, C. Tang, F. Xia, S. Savarese, H. Gweon, K. Liu, J. Wu, and L. Fei-Fei. BEHAVIOR-IK, Ik: A benchmark for embodied Al with 1,000 everyday activities and realistic simulation. In CoRL, 2022.
[59] S. Y. Min, D. S. Chaplot, P. Ravikumar, Y. Bisk, and R. Salakhutdinov. Film: Following instructions in language with modular methods. In ICLR, 2022.
[60] K. He, G. Gkioxari, P. Dollár, and Girshick. Mask r-cmn. ICCV, 2017.
[61] M. Bain and C. Sammut. A framework for behavioural cloning. In Mach. Intell. 15, 1995.
[62] A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. 2020.
[63] Ren, S. Liu, A. Zeng. J. Lin, K. Li, Cao, J. Chen, X. Huang. Chen, Yan, Z. Zeng, H. Zhang, F. Li, J. Yang. Li, Q. Jiang, and L. Zhang. Grounded sam: Assembling open- world models for diverse visual tasks. arXiv, 2024.
12

---

Note: Orange denotes reference to the main paper.
A
Additional Details of Related Work
Many recent researches try to use large language models (LLMs) to make their plan and revise it. For example, [30] retrieves the most relevant top-K examples of error correction, while [45] condenses robot's previous experiences to identify failures and devises new plans. [47] plans sequence of subgoals and revises its plan upon failure.
[54] mitigates this by reasoning and proposing the next subgoal after completing the previous one, implicitly considering failure in predicting the next step, but still does not account for failures during subgoal execution.
[48] proposes correcting the error or preventing the failure using LLMs with predefined preconditions. [49] proposes to generate complete plans using structured programming language prompt, with each action considering conditions for error-free execution.
[50,51,52] uses LLMs for self-corrective ability by receiving environmental feedback, but depends on ground-trae -ooun feedback from environments associated with a small number of actions. Recent approaches [53, 30] sometimes exploit ground-truth information (e.g., detection results, oracle infor- mation such as, "what causes task failure?," etc) for replanning, For instance, recent approach [53] exploits either ground truth or expensive human feedback.
[30] updates plans based on full RGB egocentric observations (i.e., images), which could make the model sensitive irrelevant factors like background changes. [46] adjusts plans using detected objects from the current observation as feedback after navigation failure but overlooks visual details such appearance, attributes, and relationships.
[47,54] limitedly address environmental discrepancies. They identify whether target object is at the expected destination and propose new target candidates if is not (i.e., object presence discrepancy). However, they do not address the other discrepancies (i.e., appearance, attribute, and relationship discrepancies), potentially reducing their effectiveness and efficiency.
B Additional Details of Agent Architecture
Inspired by the recent success of the SLAM-based approaches [42, 44, 18), we adopt an architecture that 1) plans a sequence of subgoal actions from natural language instruction, 2) maintains the
agent's memory in the form of a semantic spatial map built by an observation history with depth and masks predicted by pretrained perception models [60], [44], and 3) uses a deterministic algorithm [43] over the semantic spatial map for effective obstacle-fre path planning. Inspired by the recent success of the SLAM-based approaches [42, 44, 18], we adopt an architecture that 1) plans a sequence of subgoal actions from natural language instruction, 2) maintains the agent's 's memory in the form of semantic spatial map built by an observation history with depth and masks predicted by pretrained perception models [60], [44], and 3) uses deterministic algorithm [43] over the semantic spatial map for effective obstacle-free path planning.
Semantic spatial map. During exploration, the agent takes as input the current egocentric RGB observation, Ut, for each time step, t, and predicts its semantic information (e.g., object masks, {m)) depth map, de spatial semantic map, Z,, etc). Using {m)) the agent obtains the cropped images of objects, {uf [u J==o- from et Here, N denotes the number of all detected objects in Vt. and of an object's 's image crop with the masked-out background. Z, denotes 2D top-down map predicted by merging three inputs and is generated by the semantic map generator, G. as follows:
Z, G(de, (m=:): Zz-1).
G transforms each pixel of de and Noo into 3D point with its semantic label. These points are then summed along the gravitational axis to generate the current tow-down semantic spatial map
13

---

containing only current information. Finally, G' accumulate the obtained top-down map on Zt-1. resulting in an updated semantic spatial map, Z4-
Navigation policy. To interact with object, an agent must first reach the object in its close vicinity. For this, previous approaches [37] often use behavior cloning [61] to let the agent mimic the navigational behavior of a vision-language navigation expert. However, such behavior cloning requires a large number training trajectories and natural language annotations for satisfactory performance, but collecting these may not be trivial due to high computational costs and time.
To address this issue, recent approaches [41, 42, 26, 30] instead incorporate deterministic algorithms (e.g., A FMM [43], etc) to plan obstacle-free paths and observe significant improvement of navigational performance while alleviating the data collection burden. Inspired by this improvement, we also adopt a deterministic policy [43] to plan navigation routes.
C Additional Details of PRED
We provide more details of PRED. We further detail how we revise an action plan using large language models (LLMSs) based perceived environmental feedback, illustrated in Figure 4.
Additionally, we explain the AApeeaancce Detector Module' of OHV, which takes egocentric views in different views as input and predicts the class of the picked-up object; the 'Attribute Detector Module' APM, which takes the cropped image as input and predicts the object's attributes; and the 'Relationship Detector Module' of ASR, which takes the predicted masks as input and predicts the relationship between the target object and where can be placed.
C.1 Revising Actions by Environmental Feedback with LLMs
Subgoal planning. Given a natural language instruction, X, such as dialogs or directives, large language model, L. predicts the task-relevant contexts, E, such as task type, target object, its expected location, etc. Then, they are integrated into an LLM-ggenrated high-level action plan, T, for the corresponding task type through the integration process, 'Intg.'
Here, T is created to satisfy the desired states of the task type. For example, if the task type is "CLEAN ALL X" and X is an object, the desired state is defined object = [clean]. ). To create the plan, we assume that the initial state is the opposite of the desired one, such as object = [dirty), J, and then create a plan to clean the object.
Integration of T and resembles context-aware planning [18], but we use an unfine-tuned LLM for context prediction instead of trained model with benchmark data. Specific information obtained from the instruction can be incorporated into the action plan. For example, if the instruction indicates the mug is inside the fridge, action to open the fridge is added before picking up the mug.
The generated action plan consisting of high-level actions, (= in triplet format is systemati- cally converted to the executable action plan with low-level actions, {a', N <==1 in a tuple format by rule-based function, fcr following the prior work [18]. This process is expressed as in Equation 3:
È = L(X),
{a J. Int)(T,E), {ar - = fe(ta J
Revising actions. While performing the task, the agent may encounter unexpected environmental discrepancies (see Section 3.1). To translate the discrepancies detected by each module into natural language, we use an LLM to generate feedback prompt that describes the discrepancies. We provide the LLM with the Python codes corresponding to the detecting discrepancy and prompt explaining its role. Then, LLM generates the feedback prompt such as "The target is either directly visible to the agent or its information is already stored on the map.""
Once we obtain environmental feedback (i.e., environmental discrepancy) based on visual input from the LLM, we build prompt, P with this feedback to query L for plan revision. Specifically, build the prompt with 1) system prompt, P.. 2) the original plan (i.e., current action plan in Figure 4), {a, J. (4 1 or {a/, M and 3) the feedback prompt, Py.
14

---

Input
Prompt
Instruction
Task-Relevant Contexts
System Prompt
Revised Action
Commander
Task type: Water Plant
Role:
Driver
"Bowi": "eovw":("Toceeee ("Tocation" 'Table")
need
(Put, Sinkkeein))
There
tabik.
1097-40
whether to revise the action plan...
iToggloOn, Faucet)
Commander
Retrieve LLM- Generated Action Pian and Integrate Task Relevant Contexts
FfeggleGHf, Feveet
(Pickup, Bowll
Commander
Into
Our purpose is to pertorm the task officiently
(Pour, HousePlant)
Water the plant.
Driver
Initial
Sune, lsee the plntt the tabie!
RGB Image
(Pickup, Bowil
Completed
Current Action Plan
LAM
(Put, SinkBasin) -umune (ToggleOn, Faucet)
[Put, SinkBasin) e Action
Task completed!
(ToggleOrt, Faucet)
[TaggleOn, Faucet)
(Pickup, Bowl) (Pour, HausePlant)
(ToggleOff, Faucet)
(Pickup, Bowl)
[Pour, HousePlant)
Update Expected States
Environmental Feedback
Feedback Information
Appearance Diserspancy
Semantic Data
Discrspancy
The bowl is already filled.
Depth
Oij
Sam
Destripancy
Presence
Distrepancyy
0DR
(Pour, HousePlant)
Figure 4: Overall Process Revising Actions by LLMs, We extract task-relevant contexts from the dialog using an LLM. These contexts are integrated into a retrieved action plan generated by the LLM to form an initial action plan. Using semantic data from an RGB image and expected states from the initial plan, PRED verifies discrepancies environmental feedback. The prompt including the system prompt explaining the role and guide for the LLM, the current action, and the feedback information is given to the LLM to generate a revised action plan. The bottom right image shows the task is completed along with the efficiently revised plan.
The format of the current action plan, {a can be the sequence of the actions either in an executable tuple (low-level action), a',,- or a triplet (high-level action), a A triplet action is predefined and consists of a unit of action that includes several executable actions. For example, ('Move', 'Target', 'Parent') signifies the combination of ('Target', 'PickupObject') and ('Parent', 'PutObject'). We operate with both types of action plans simultaneously. For instance, the action plan is input in triplet form if the actions be revised the triplet level, while in tuple form if the revision is required at the detailed executable action level. If the output is in triplet form, it is systematically translated into executable tuple form by fc. This revision process continues whenever the environmental discrepancy is detected until the agent successfully completes the task.
for L varies according the type of discrepancy and given action plan consisting of various objects and actions. We provide several examples of the P including P.. a current action plan, Py, and the corresponding output of the L in Listing 1 to 5. We provide the prompt to extract & in Listing 6 and one to generate revised action plan in Listing 7.
C.2 Appearance Detector Module
The agent often fails at object interaction due to misperception caused by lighting, occlusion, etc. To address this issue, we propose to verify that an interacted object is the intended one. The intuition is that correct perception will yield consistent perception results.
Specifically, verify that predicted classes, co and c;, of the target object, o identical. Here, co is the predicted class of from the th viewpoint at the time of interaction, and c; is that of from the th viewpoint obtained from different pose from the one where the 0th viewpoint is obtained. The agent predicts co and c from egocentric images of the object at the 9th and jth viewpoints with a pretrained perception model. If on # c; the agent revises its actions accordingly.
There are various ways to obtain different viewpoints of an object, such as grasping it from different directions or rotating it. Here, simply lift lower the interacted object implemented by appropriate actions (here, LOOKUP UP and LOOKDowN in the Appendix D) supported by the agent's embodiment.
15

---

A better method of obtaining these viewpoints may improve our proposed module, but this is beyond our scope and thus, we leave this as future work.
C.3 Attribute Detector Module
We consider attributes related the desired state determine the success of the task. Furthermore. the expected attribute is one of the initial states related the attribute, and the initial states defined as the opposite of the desired state (detailed in the Appendix C.I).
We take a retrieval-based based approach to predict the attribute, 6.. of a detected target object, o. This approach is preferred over training a model because it requires neither extensive training nor large datasets. Additionally, if a new attribute is added, a training-basee -based model must be retrained or fine-tuned, whereas the retrieval-based approach only needs add new options for comparison.
To predict 6.. we retrieve the most similar image and assign the attribute of the object in the retrieved image to the detected target. To compare images focusing only on the object and excluding the background, we use the cropped image of the target object, 2n- and the cropped images, {uz 1 Ji=0+
from the training dataset. We compare these cropped images through cosine similarity, 'Cos', after extracting features processed by ViT-B/32 model [62]. pre-trained with CLIP weights [2], as:
argmax Cos(ur ex = 0, O.,,
where n/ and egat are the features of eg and eft, respectively. Then, we consider o as the attribute of the object in of having the highest cosine similarity score.
C.4 Relationship Detector Module
To determine the relationship between the target object, o, and the object placed on, Op. we need find the most 'adjacent' mask, m, of Op the target object's mask in all detected masks, { in the current egocentric view. Here, the mask i O is defined the mask of the target object, mo, of o. To find m, we dilate { and calculate the intersection over union (IoU) between the enlarged mask of the target object, my, and the enlarged masks of other objects, {m{], as follows:
argmax (m' m(); i 1.2........
(5)
We define m as the mask with the largest IoU score with my. considering it the 'adjacent' mask. The object represented by m referred to as the object op that the target object is placed on.
D Benchmark and Baseline Details
We validate PRED in two challenging benchmarks: TEACh [13], for dialog instruction following, and ALFRED [12], which provides declarative instructions, to assess generalization different task setup. We provide details for each benchmark and baselines used below.
D.I TEACh
The TEACh benchmark aims to allow agents to navigate and interact with objects based on instructions, with task completion achieved by meeting specified conditions, such as cleaning least one mug for the instruction "clean a mug".
Benchmark.
The instruction is a dialog in natural language, which is comprised of two components: the COM- MANDER that provides task-relevan information based oracle information about the task and the FOLLOWER that performs the task through the dialog. Upon receiving instructions, the FOLLOWER translates the natural language instructions and egocentric visual observations into executable actions. The executable actions are expected to succeed in the task.
The agent can take 16 different actions. Eight actions (FORWARD, BACKWARD, TURN LEFT, TURN RIGHT, LOOk UP, LOOK DOWN, STRAFE LEFT, STRAFE RIGHT) are designated for navigation,
16

---

and the other eight actions (PICKUP, PLACE, OPEN, CLOSE, TOGGLEON, TOGGLEOFF, SLICE, AND POUR) are for interaction. Navigation actions are discrete: head movements are by 30", turns are by 90º, and movements are in 0.25m increments. During interaction, the agent selects the object at coordinate (x, y) in its egocentrie view.
Additionally, the TEACh benchmark focuses on Execution from Dialogue History (EDH) and Trajectory from Dialogue (TfD). This benchmark is divided into train, validation, and test splits. Evaluation metrics encompass success rate (SR), goal coal-condition success rate (GC), and path-length weighted (PLW) scores.
State-of-the-art baseline models. We compare our PRED with the recently proposed state-of-the-
art methods: E.T. [37], JARVIS [41], FILM [42), DANLI [26], and HELPER [30]. E.T. learns a direct mapping from natural language dialog and an egocentric observation to a corresponding action and the position of an object to be interacted with. JARVIS employs an LLM trained on the TEACh dialog dataset to produce high-level subgoals, replicating the ones executed by human demonstrators. It utilizes a semantic map alongside the E.T. to locate objects. FILM enhances an LLM through fine-tuning to generate parameterized plan. Mirroring Jarvis, it leverages semantic map to execute subgoals and employs semantic policy for object search. DANLI fine-tunes an LLM for high-level subgoal prediction and employs symbolic planning with an object state and spatial map for execution plan formulation. incorporates an object search module and manual error correction mechanisms. HELPER utilizes Large Language Model (LLM) to generate initial high-level actions with additional data. When failure occurs, it predicts the error reasons through pretrained vision and language model, and revises the action using the LLM.
D.2 ALFRED
Benchmark. The ALFRED benchmark requires agents to complete long-horizon task by un- derstanding declarative natural language instructions with egocentric observations. The declarative instructions comprise two types of instruction: one is high-level description that provides a single sentence to complete the task and the other is a step-by-step instruction that details the process of performing a task. By following the instructions, the agent executes the two types of predefined actions. The navigation actions include MOVEAΠΣAD, ROTATERIGHI,, ROTATELEFT, LOOK UP, and LooKDowN. The interaction actions include PICKUPOBJECT, PUTOBJECT, OPENOBJECT, CLOSEOBJECT, TOGGLEOBJECTON, TOGGLEOBJECTÖFF, and SLICEOBJECT.
State-of-the baseline models. We compare our PRED with the recently proposed state-of-the- art methods: HLSM [44], FILM [57], and CAPEAM [18]. HLSM employs hierarchical controller to translate natural language instructions into actions the agent can execute. The high-level controller identifies the next subgoal based on the given instructions and map, while the low-level controller generates a sequence of actions to accomplish this subgoal. FILM uses a pre-constructed template as high-level action plan. It employs two BERT [6] classifier submodules identify the instruction type and determine the template arguments. It applies deterministic algorithm [43) to plan path without obstacles. CAPEAM employs context-aware planning devise sequence of subgoals and execute each subgoal using the appropriate detailed planners. It also utilizes extra memory to avoid interacting with unsuitable objects.
E
Additional Experiment
E.1
Real Robot Experiments
We conduct experiments in a real environment to evaluate the effectiveness of our proposed method. To perceive the environment and obtain feedback, we use grounded SAM [63]. Additionally, employed the UR5 manipulator for object interaction. First, we recognize the objects and environment from an egocentric view using a camera mounted on the back of the manipulator's gripper.
17

---

Table Alternative TEACh EDH evaluation split. The seatt-lenntt-weighhhe (PLW) metrics given in the parentheses for each value. The highest and second highest values per fold and metric are shown in bold and underline, respectively.
Validation
Test
Model
Unseen
Seen
Unseen
Seen
ET. [37]
8.35 (0.86) 6.34 (3.69)
8.28 (1.13) 8.72 (3.82)
7.38 (0.97)
6.06 (3.17)
8.82 (0.29) 9,46 (3:03)
DANLI [26]
17.25 (7.16)
23.88 (19.35)
16.89 (9.12)
25.10 (22.56)
16.71 (7.33)
23.00 (20.35)
18.63 (9.41)
24.77 (21.90)
HELPER [30]
17.25 (3.22)
25,21 (8 12)
19.21 (4.72)
33.54 (10.95)
17.55 (2.59)
26 19 (7.67)
17.97 (3.4g)
30,81 (8.93)
PRED (Ours)
21.52(4.64)
26.88 7. 25)
23.84(4.200
33.79(10.64)
22.04(4.24)
26.77(7.67)
(4.61))36)
(10. 31.86(10.199
Table 5:
Ablation study in TEACh for each proposed component. The path-length-weighcee (PLW) metries are given the parentheses for each value. (b) to (e) show the performances of PRED without each component.
TDD
EDH
Model
Unseen
Seen
Unseen
Seen
SR
GC
SR
GC
SR
GC
SR
GC
PRED
19.77 (5.16)
16.74 (8.31)
20.90 (4.64)
21.55 (11.03
21 (4.44)
26.83 (7.46)
21.71 (4.62)
32.78 (10.39)
(b) w/o DIA
17.65 (1.63)
13.39 (6.63)
16.02 (1.05)
16.31 13)
20.67 (3.52)
25.74 (6.6))
18.75 (3.59)
30.95 (8.69)
wlo OHV
15:20 (4.22)
12.58 (6.57)
16.02 (4.56)
17.46 (9
20.35 13.44)
22.89 (6.66)
18.75 (3.87)
29.87 (9.00)
(d) woo APM
16.18 (4.14)
13.66 (6.45)
14.36 (3.70)
18.59 (9.10)
19.80 (3.05)
24.33 (5.85)
19.98 (3.67)
29.42 (7.40)
wlo ASR
17.16 (4.44)
15.30 (7.41)
18.23 (5.11)
17.18 (8.92)
20.49 (4.06)
26.00 (7.42)
18.91 (2.53)
30.62 (9.38)
wlo all
12.91 (2.91)
10.56 (5.62)
(3.31
15.49 (7.41)
18.65 (3:09)
19.06 (5.8)
17.95 (3.77)
24.39 (8. (8.16)
Once the agent identifies the desired object using [63] and determines which object to interact with, it retrieves the detected mask of the object as input to select the correct object in the real world. The agent then receives the ground truth position of the selected object and proceeds with the interaction.
F.2 Results on New Validation Split
We conduct an additional experiment, exclusively done by DANLI [26] for fair comparison, in the TEACh bench- mark [13]. In this benchmark, we investigate the performance of our PRED in different splits provided on the TEACh GitHub and in [26). The leaderboard for EDH of the TEACh benchmark is unavailable, preventing the evaluation on its true test set. Thus,
Table 6: Ablation study in ALFRED for each proposed compo- nent. The path-length-weige parentheses for cach value. (b) to (c) show the performances of PRED without each component.
(PLW) metrics are given in the
Test Unseen
Test Seen
Model
SR
GC
SR
GC
PRED
32.31 (12.48)
42.62 (17.87)
35.09 (15.02)
43.27 (19.95)
(b) w/o DTA
29.63 (11.79)
39.49 (16.66)
33.59 (13.92)
(2.02(18.60)
(c) OIIV
29.30 (11.33)
(16.54)
33.53 (13.91)
(19.07)
(d) w/o APM
24.33 (9. 13)
32.11 (13.98)
24.66 (10.01)
32.42 (15.00)
(e) ASR
22)
38.98 (17.52)
(14.48)
(19.63)
(1) wlo all
23.35 (8.72)
31.25 (14.61)
24.27 (9.95)
32.24 (15.49)
we leveraged the original validation
splits for seen/unseen scenarios, aligning with the approach taken in most prior studies [30, 41, 42],
In Table we present the alternative validation and test splits. We observe that our method out- performs others in the new EDH split, achieving improvements with notable margins in SR and GC, similar to the results in the original split. In the seen environment, the agent encounters fewer misperceptions and navigation errors than in the unseen environment, making our PRED less effective.
F Additional Ablation Study
We provide the results of ablation studies in all splits including 'Seen' environments in TEACh and ALFRED benchmark in Table 5 and 6.
G Qualitative Analysis
DTA. First, Figure Sa describes two different simulation scenarios: PRED without (top) and with (bottom) DTA in Section 3.1.1. An agent generates an initial plan considering the information mentioned in the instruction ("mug, potentially in a cabinet") that it predicts that the mug is in the cabinet and has plan of opening the cabinet first, instead of picking up the mug. Thus, in the upper
18

---

scenario (w/ DTA), the agent keeps its original plan to open the cabinet even if the usable mug is observed in its sight. On the contrary, the agent with DTA changes the plan to skip the action with the cabinet and to pick up the mug right away after perceiving the mug. As a result, can finish the task efficiently as adapts to the discrepancy of the target object's presence in the environment.
Figure 6a shows two real robot scenarios: agent forms an initial plan based on the instruction ("Pepsi, possibly in red cup"), assuming the Pepsi is in the red cup and planning to move the cup before picking up the Pepsi. In the scenario without DTA (top), the agent follows the initial plan and moves the red cup, even though another Pepsi is visible. In contrast, with DTA (bottom), the agent adjusts the plan, skipping the red cup and picking up the Pepsi immediately after detecting it. This adaptation allows the agent to complete the task more efficiently by adapting to the actual environment.
without DTA (top) and one with DTA (bottom). The
OHV. Figure 5b illustrates the difference with and without OHV in the simulation, as described in Section 3.1.2. The first image shows the agent needing to pick up a 'Mug' but mistakenly picking up 'Cup' instead by misperception. The agents without OHV will not try to correct the wrong action since they do not consider the appearance discrepancy after the action is done. In contrast, agent with OHV verifies whether the interacted object aligns with the desired one by examining its appearance from multiple angles. If it detects the discrepancy, the agent rectifies its mistake by revising actions. As a result, OHV helps the agent to prevent the failure that comes from the interaction with the wrong object.
Figure 6b demonstrates the impact of OHV in real robot experiments. t 1, with the scenarios with OHV (top row) and without OHV (bottom row), the agents experience misperception because the color and design of the cans look almost identical from the top-down view. As result, t=2, the agent picks the can with the higher confidence score but mistakenly grabs drink other than the desired Fanta. In the scenario without OHV (top row), the agent sticks to the original plan, ignoring the mistake, and delivers the wrong item to the destination. However, with OHV (bottom row), the agent views the drink from the front, identifies the label, and realizes it has picked the wrong item, detecting the appearance discrepancy. The agent then puts down the incorrect item, updates the plan, and successfully completes the task.
APM. Next, we investigate the benefit of APM (in Section 3.1.3) depicted in Figure 5c with the scenarios where agents want to open a microwave, The agent without APM attempts to open as supposes that the microwave is not operating, leading to an interaction failure. In contrast, the agent with APM considers the attribute discrepancy coming from the difference in the microwave's s state. With this, it adds actions that toggle it off first future actions, enabling successful interaction.
ASR. Finally, we elucidate the advantage of ASR in Section 3.1.4. Figure 5d describes the benefit of considering relationship discrepancy. We refer to the relationship corresponding the goal condition in the instruction as the goal relationship. An agent without ASR may pick up remote and put it down with higher confidence score, even though already satisfies the goal relationship, resulting in only one remote on the table. If this were the second interaction (picking it up and putting it down on the table), the agent would think that it completed the task since it completed interactions twice. However, the task would fail since there are no two remotes on the table. In contrast, the agent with ASR considers relationship discrepancy and interacts with a remote not in the goal relationship instead of the one already goal relationship.
19

---

Goal: Find a mug, potentially cabinet, and clean it.
mug.
RED w/o DTA
RED wio
progreasing the Open a Cablnct to find a Mug
Fail
Detect another Mug but pass through
Head to a Cabinet
Pick up a "Cupp delected "Mag"
Do not examine and just Cup
Interact with a Cup (should be Mugi
Mugi
Mac
RED w/ DTA
Cubinet!
RED w/ OHV
Success
Thought pocked
Success
Detect a Mug and intermediatall plck up
Put a Mug i Sink and clean it
Washing the Mug is done!
Pick "Cup"
Examine and put down
Examine picced mug and keep
detected as "Mug"
Objects
goal
Object unmenfooee goal
Task irolevant action
Undestred Gbject
Destred Object
(a) Dynamic Target Adaptation (DTA)
(b) Object Heterogeneity Verification (OHV)
Goal: Put microwave.
Goal: Pul two remotes on the table.
Time
Fail
RED w/o APM
RED w/o ASR
Object: Microwere
Fail
Microwave
Attempt to open operating Microwave
Attempt open
Detoct Romotes
Pick the Remote with highest confidence score
Only one Remote on the Table
EU Staral
in TU Stand
Success
RED w/ APM
RED w/ ASR
2n FAB
Alt tottteeee parnanng
should offil before opening
Success
with a Remsse aieady
Attemott
Detect Remotes with relationships
Pick the Remate the TV stand
Two Remotes on the Table
attributes
Microwave
inactivating Microwavo
Detectied object
N.e Datected cbjject with Attribute
Considined
Oojects
relstionshpp
Resationhilp
(c) Attribute-Drivee -riven Plan Modification (APM)
(d) Action Skipping by Relationship (ASR)
Figure 5: Qualitative analysis of benefit of DTA, OHV, APM, and ASR.
20

---

Goal: Place Pepsi, possibly in red cup, the yellow cupboard
Initial Plan (Pickup, RedCup), (Put, Table), (Pickup, Papsi), (Put, YellowCupboard)
Reatas
RED w/o DTA DTA
PwdCCpp
Oupasard 2.W
2=2
T=5
Detection
Pick Place
Detection
Pick & Place
Success but time-consuming
Velw GPrT
Diecrpancy Detectio:: The object is found another place not the receptacles
RED w DTA
RedCop
Captaardio
1=2
Revised Plan
(Pickup, Pepsi) YollowCCpboard)
1+4
Detection & Replan
Pick & Place
Success
Dynamic Target Adaptation (DTA)
Goal: : Move Fanta on the yellow cupboard
Initial Plan: (Pickuo, Fanta), (Put, YellowCupboard)
RED w/o OHV
VAlTA
1=
Detection
Fail undesired object moved
ne Siscrepancy Detectior:
up
RED w/ OHV
object was
Cupbornd
t=2
Revised Plan
(Put, Table)
(Pickup, Fanta)
Put wwi.soocppooord
t+6
Detection
Pick & Detection & Place
Pick Place
Success
(b) Action Skipping by Relationship (OHV)
Figure 6: Qualitative analysis of benefit DTA and OHV
21

---

##* INPUT
System Prompt
You need to determine whether to revise the action sequence to solve task considering 'INFO' and then write down the final sequence actions if needed.
This action sequence is made for solving household task. Each action 's format is tuple.
The possible action spaces [ Target', "Pickupobject'), ['Target', 'PutObject '], ['Target', OpenObject'), ['Target', 'Closedbject'), ['Target] Toggle0bjectOn '], ('Target', 'TogglebbjectOfr 'I, ['Target', PourObject ['Target', 'Slice(bject For example, ['Target', 'PickupObject '1 command to pick up the target.
follows.
Modify the action sequence by adding or subtracting action to suit the situation if needed. If it is not needed to be revised, just write down given action sequence.
this context, 'action' refers the list(action sequence).
each element tuple format in
You can add or subtract an action at the very first part of the action sequence if needed. Do not modify the actions that follow, and write them down originally provided.
Do not add additional explannation. Just write the final action sequence in the right format (tuples in list).
GUIDE:
When you want pick up object, it sometimes located in receptacles that should opened pick up the object.
In this case, action sequence contains the sequence of open, pickup slice), close.
Your objective is just picking uor slicing) the object.
Current Action Plan
[
"Recep",
"OpenObject"
1,
[
"Target",
"Pickupdbject"
1.
[
"Recep",
"CloseObject"
1.
"Apple",
"SliceObject"
}.
[
"CounterTop",
"PutObject"
Feedback Information
The target is either directly visible to the agent its information is already stored the map
### OUTPUT
[
22

---

"Target", "PickupObject"
1.
"Apple", "SliceObject"
1.
"CounterTop", "PutObject"
Listing I: Example of the input(i.e. Prompt) and the output (i..,, Revised Action Plan) of LLM. This example describes the result where DTA is applied. Compared to the current action plan, the action plan is revised skip the actions (i.e. open and close the receptacle) make plan efficient. Target and Recep will be replaced with each corresponding object based on the context.
23

---

##* INPUT
System Prompt
You need to determine whether to revise the action sequence to solve task considering 'INFO' and then write down the final sequence actions if needed.
This action sequence is made for solving household task. Each action 's format is tuple.
The possible action spaces [ Target', "Pickupobject'), ['Target', 'PutObject '], ['Target', OpenObject'), ['Target', 'Closedbject'), ['Target] Toggle0bjectOn '], ('Target', 'Toggle0bjectOff 'I, ['Target', Pour(bject ['Target', 'Slice(bject For example, ['Target', 'PickupObject '1 command to pick up the target.
follows.
Modify the action sequence by adding or subtracting action to suit the situation if needed. If it is not needed to be revised, just write down given action sequence.
this context, 'action' refers the list sequence).
each element tuple format in
You can add or subtract an action at the very first part of the action sequence if needed. Do not modify the actions that follow, and write them down originally provided.
Do not add additional explannation. Just write the final action sequence in the right format (tuples in list).
GUIDE: When picking up object, it may not always be the intended object.
But you can only handle one object in your hand which means you have to put the object and repick the object if you want to pick another object.
When you
where put the object just use 'aaeet the place you put on. (In the action, ['A,, 'PutObject'], 2 should be the location to place the object on, not the object that you are holding.)
not
Let 's assume that you have done the first action. In this case, do not revise the first action, but you think actions should be added, do it right after the first action.
Current Action Plan
"Target", "PickupObject"
1.
[
"Bed", "PutObject"
Feedback Information
After checking, the picked object's 's mask is detected, but the object is not the desired one its center coordinates do not fall within the specified range (width between 100 and 200, height between 100 and 230).
### Output
"Target", "PickupObject"
24

---

],
[
"Parent",
"PutObject"
1.
[
"Target", "PickupObject"
1.
[
"Bed", "PutObject"
Listing 2: Example of the input(iee. Prompt) and the output (f.e., Revised Action Plan) of LLM. This example describes the result where OHV is applied. Compared to current action plan, the actions (i.e., put the target on the parent and pick up the target again) are inserted to put down the wrong object and pick up an appropriate one. Target and Parent will be replaced with each corresponding ohject based on the context.
25

---

##* INPUT
System Prompt
You need to determine whether to revise the action sequence to solve task considering 'INFO' and then write down the final sequence actions if needed.
This action sequence is made for solving household task. Each action 's format is tuple.
The possible action spaces [ Target', "Pickupobject'), ['Target', 'PutObject '], ['Target', OpenObject'), ['Target', 'Closedbject'), ['Target] Toggle0bjectOn '], ('Target', 'TogglebbjectOfr 'I 'I, ['Target', PourObject ['Target', 'Slice(bject For example, ['Target', 'PickupObject '1 command to pick up the target.
follows.
Modify the action sequence by adding or subtracting action to suit the situation if needed. If it is not needed to be revised, just write down given action sequence.
this context, 'action' refers the list(action sequence).
each element tuple format in
You can add or subtract an action at the very first part of the action sequence if needed. Do not modify the actions that follow, and write them down originally provided.
Do not add additional explannation. Just write the final action sequence in the right format (tuples in list).
GUIDE: Microwave cannot be opened if the microwave is toggled
Current Action Plan
[
"Microwave",
"OpenObject"
1,
[
"Mug",
"PickupObject"
1.
[
"Microwave",
"CloseObject"
1.
[
"SinkBasin",
"PutObject"
"Faucet",
"Toggle0bjectOn"
}.
"Faucet", "ToggleDbjectDfff
Feedback Information
After checking, the microvave's 's image is confirmed to be in the ToggleOn " state.
### OUTPUT
26

---

[
"Microwave",
"TogglebbeectOff"
1,
[
"Microwave",
"OpenObject"
1.
[
"Mug",
"Pickupdbject"
J.
"Microwave",
"CloseObject"
1.
"SinkBasin",
"PutObject"
1.
"Faucet",
"ToggleObjectOn"
1.
[
"Faucet", "TogglebbjectOff"
Listing Example the input (ie., Prompt) output (i.e,, Revised Action Plan) LLM. This example describes the result where APM is applied. Compared to the current action plan, the action (i.e. toggle off the microwave) is added in the revised action plan to avoid interaction failure.
27

---

#** INPUT
System Prompt You need to determine whether to revise the action sequence to solve the task considering 'GUIDE' and then write down the final sequence of actions if needed.
This action sequence is made for solving household task. Each action 's format is a triplet.
There are two options.
First, if you think that the first action is not needed,
that
action (triplet) and write down the actions after that. You can only revise the first action; do not modify the actions (from the second actions) that follow, and write them doun as originally provided.
In this context, 'action' refers each element triplet format the list list(action sequence).
Second, if you still think the first action is necessary, write it down exactly received.
Do not add additional explanation. Just write the final action sequence
right format (triplets in the list).
GUIDE: Our purpose is to perform the task efficiently.
Current Action Plan
"Clan"" "Tareet" "None" "
1,
[ "Move"
"Bread"
"Toaster"
1,
[
"Move"
"Bread",
"Toaster"
1.
[
"ToggleOn",
"Toaster",
"None"
1.
[
"Move"
"Bread"
"Plate"
1.
[
"Move",
"Bread",
"Plate"
Feedback Information
The target object in the test image has been either cleaned, as determined by the similarity check.
OUTPUT
[
"Move",
28

---

"Bread",
"Toaster"
1.
"Move",
"Bread",
"Toaster"
1,
[
"ToggleOn",
"Toaster",
"None" "
1,
[
"Move".
"Bread"
"Plate"
1.
[
"Move",
"Bread",
"Plate"
Listing 4: Example the input (f.e., Prompt) and the output (i.e., Revised Action Plan) of LLM. This example describes the result where APM applied. Compared to the current action plan, the action (i.e, clean the target) is deleted to skip unnecessary action. A Target will be replaced with the corresponding object based on the context.
29

---

#** INPUT
System Prompt You need to determine whether to revise the action sequence to solve the task considering 'GUIDE' and then write down the final sequence of actions if needed.
This action sequence is made for solving household task. Each action 's format is a triplet.
There are two options.
First, if you think that the first action is needed,
that
action (triplet) and write down the actions after that. You can only revise the first action; do not modify the actions (from the second actions) that follow, and write them doun as originally provided.
In this context, 'action' refers each element triplet format the list list(action sequence).
Second, if you still think the first action is necessary, write it down exactly received.
Do not add additional explanation. Just write the final action sequence
right format (triplets in the list).
GUIDE:
There can be some objects that are already located in the desired destination.
If you think executing the following action should be avoided it no longer needed, add a Pass action in triplet form (same as given action) with Pass for action, None for Target and Parent, before the given action (including given action) without further explanation. you think the following actions still needed, repeat the given actions.
Current Action Plan
Move", "Target", "Parent"
Feedback Information
After checking, it is found that the second object has already been placed in the desired location indicated by the interaction mask, so no further interaction is needed.
### OUTPUT
"aass"
"None"
"None"
}.
"Move"
"Target",
"Parent"
Listing 5: Example of the input (i.e., Prompt) and the output (f.e,, Revised Action Plan) of LLM. This example describes the result where ASR is applied. Given the current action plan (i.e., move target to recep), LLM adds 'Pass' action in triplet which leads skipping the ongoing interaction hefore the given action. Then agent not interact with the target since is already located the desired place as explained GUIDE, Target and Parent will be replaced with each corresponding object based on the context.
30

---

Driver tries solve the task. Commander gives information helpful have to get information through the dialog.
solve the task. Find the initial states of the objects and summarize them into a dictionary. If you cannot find proper information in the dialog, you should answer 'X'.
Just write dictionary without giving an additional explanation. In dictionary, fix the keys same with the example anawers. Each of the keys has properties (keys) e.g., "location" which contains the initial location of the object. In asked find multiple objects of one kind. In those cases, if object is on Cabinet and another object is on CounterTop, you should output ('Cabinet', 'CounterTop"]. (If you cannot know where the potato is initially, answer (X'.), "receptacle" which is the place that the object should be placed quantity" which represents the number of the objecta, guantity_of_elices which represents the number of the object's slices. Some objects (e.g., plate, mug) have a key i.e., "Cleaned" which represents whether the object should be cleaned (then write "T") not (then write "F"). If you cannot find the proper information, just write 'x'.
tasks, the driver may be
in ultimately.
Dialog: {DIALOG}
Answer:
Listing 6: Prompt for Extracting Task- Reek-Relevant Contexts from the Dialog, () denotes the section in the prompt that is replaced to each corresponding data. This prompt is designed to extract the useful information from the dialog use when making initial action plan.
31

---

System Prompt
You need to determine whether to revise the action sequence to solve the task considering 'INFO' and then write down the final sequence of actions if needed.
This action sequence is made for solving household task. Each action 's format tuple.
The possible action spaces are as follows.
[ ('Trgg'',, 'PickupObject"), ['Target', 'PutObject '], ['Target', OpenObject'], ['Target', 'CloseObject'), ['Target', ToggleObjectOn '], ['Target', 'Toggle0bjectDff" 'I, ['Target', PourObject ['Target', "Slicebbject'] For example, ('Target', "PickupObject"] is a command to pick up the target.
Modify the action sequence adding subtracting action to suit the situation if needed. If it is not needed be revised, just write down given action sequence.
In this context, 'action' refers to each element in tuple format in the (action sequence).
You add subtract
action the very first part of the action sequence if needed. Do not modify
actions that follow, and write them doun as originally provided.
Do not add additional explannation. Just write the final action sequence in the right format (tuples in list).
{GUIDE}
Current Action Plan [CURRENTACTIONPLAN)
Feedback Information (FEEDBACK)
Listing Prompt Revising Action Plan. (} denotes the sections prompt that are replaced to each corresponding data. The prompt is designed to envelope the system prompt which gives the overall guideline for the L.IM's task, the current action plan which is the source plan, and the feedback information which contains the feedhack from the environmental discrepancy.
32

---

##* INPUT
System Prompt
You need to interpret conditional statements and write the corresponding situations natural language prompts.
Based on the given code, describe is when the conditional statement is true. Do not start the sentence with 'The situation is
sentence what the situation
Code
test_ing a target object
If Cleaned is true, it means that the target object is already cleaned.
Cleaned = self check_similarity (testt_ttttee_tnt ing, "Clean", obj_name= planner Saapntrrninutaiteeeee inputa ['sts of actions '] [self, pointer [0]) Cleaned 1:
### OUTPUT
The target object in the teat image has been either cleaned, as determined by the similarity check.
Listing Example Generated Feedback Prompt. This example shows generated feedback prompt by LLM. It is instructed to output the feedback prompt by providing a code used to detect discrepancy with description. Output is the feedback prompt.
33


Reviews:
[{"review_id": "vDkmVUZ5Vy", "reviewer": "Reviewer_M1LD", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper studies the problem in the instruction following, in which the agent is tasked with the instruction and is required to complete the task. The paper focuses on how the agent can react to unexpected conditions in the environment, and adaptively change the plan. In the paper, there are four unexpected conditions -- object presence, object appearance, object attributes, and object-object relationships. The paper proposes several methods to deal with these cases. The method is being verified in several embodied AI tasks (TEACh and ALFRED), and results indicate the performance is good.", "summary_of_recommendation": "The quality of the paper is good, and the easy to read. The method seems to be novel.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "1. One question I have is that how do you get the images of the plate before doing the task. I can imagine that you hardcode the image of the plate for each task, and assume that this is known. However, if the agent does not have such information before hand, how do you detect the the difference?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JY8VTapJmX", "reviewer": "Reviewer_nukf", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper, the authors propose a RED framework which checks four distinct environmental discrepancies concerning objects to generate/revise actions for the agent to complete an embodied instruction following task.", "summary_of_recommendation": "There is no real-world evaluation and the novelty of the proposed method is a little weak.", "technical_quality": 2, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "- Pretrained perception model limitation: Based on the paper, the environmental discrepancies are all predicted by some pretrained perception model, mostly relying on some vision models. However, there are attributes, appearance, etc, which might be difficult to recognize with only visual information. For example, how could the perception model recognize if an apple is cooked or not. Also, is there any mechanism to deal with the situation where the pretrained perception model fails.\n\n- Difference and novelty of the proposed method with other methods: There are already a lot of works that try to revise or replanning action sequence based on large language models. What is the specific differences between the proposed method and other methods, such as (just name a few):\n[1] ReAct: Synergizing Reasoning and Acting in Language Models\n[2] ADaPT: As-Needed Decomposition and Planning with Language Models\nThe environmental feedback seems also included in other methods. I suggest a more thorough investigation of similar work and clarify the novelty of the proposed method. I am also curious about the performance when using a vision-language model for the task.\n\n- Real-world evaluation: It will be interesting to discuss the performance of the proposed method when it is applied in the real world. A real world demonstration will be a big plus to this work.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uE0LCjaZEh", "reviewer": "Reviewer_Wspw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes an approach for embodied instruction following to dynamically update the plan based on environment feedback, using LLMs. They look at 4 types of discrepancies, and propose respective modules to fix each one.", "summary_of_recommendation": "The paper proposes a novel approach in embodied AI but the approach faces scalability concerns due to specificity in handling discrepancies and potential dependencies on simulator-specific modules, raising questions on robustness and broader applicability. EDIT: The authors applied the approach on other baselines during the rebuttal period, and have shown improvements, addressing my concerns about it being too specific.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 1, "questions_for_rebuttal": "- Line 126: I’m not sure I understand the following paragraph: does “o” map to potato, “Z_t” to table in the example?\n\n- Line 170: How is the prompt e.g. “After checking, it appears that the target has already been cleaned” generated? Section B.3 in the appendix only describes how the attribute is predicted, but how is that information translated into language?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission introduces an approach for dynamic replanning for an agent's actions based on real-time environmental feedback. The proposed method reacts adaptively when discrepancies, such as object presence, appearance, attributes, and relationships, between the planned actions and the actual environment are detected. It is shown to be quite effective in 2 benchmarks (ALFRED and TEACh). On the other hand, the reviewers were concerned on the scalability and real-world applicability. The proposed method is only tested on simulated environments, which are even with relatively weak physical simulation, leaving questions about its effectiveness in complex scenarios and real world.

While it is indeed true that there is a need for further investigation, I recommend accept with the novelty of the proposed approach and its significant improvement. The learning based dynamic replanning, especially preventive one, is timely a good topic to discuss in the community. I however strongly suggest some real world demo in this case, which would enhance the paper quality and impact.
Decision: Accept