Pre-emptive Action Revision by Environmental Feedback for Embodied Instruction Following Agents

Authors: jinyeon  kim
cheolhong  min
byeonghwi  kim
jonghyun  choi
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-22 22:22:55

Abstract: When we, humans, perform a task, we consider changes in environments such as objects' arrangement due to interactions with objects and other reasons; e.g., when we find a mug to clean, if it is already clean, we skip cleaning it. But even the state-of-the-art embodied agents often ignore changed environments when performing a task, leading to failure to complete the task, executing unnecessary actions, or fixing the mistake after it was made. Here, we propose Pre-emptive Action Revision by Environmental feeDback (PRED) that allows an embodied agent to revise their action in response to the perceived environmental status before it makes mistakes. We empirically validate PRED and observe that it outperforms the prior art on two challenging benchmarks in the virtual environment, TEACh and ALFRED, by noticeable margins in most metrics, including unseen success rates, with shorter execution time, implying an efficiently behaved agent. Furthermore, we demonstrate the effectiveness of the proposed method with real robot experiments.

Main paper: 
 introduction: 
 Building robotic assistants that understand natural language and surroundings and perform tasks has long been an ambitious research goal. Recent advances in computer vision [1, 2, 3] and natural language processing [4, 5, 6, 7] have significantly contributed to the development of such agents, enabling them to carry out various tasks [8, 9, 10, 11, 12, 13, 14] in diverse environments [15, 16, 17] .Typically, agents plan actions based on initial environmental states [18, 19] , but environmental discrepancies between expected and actual states often cause task failures due to misperception or incomplete environmental exploration. Unlike artificial agents, humans and animals adapt to environmental changes through brain plasticity [20, 21, 22] , revising behaviors based on experience to avoid mistakes. In light of this, we ask: Can embodied agents benefit from similar adaptability?Drawing inspiration from neuroscience, we propose Pre-emptive Action Revision by Environmental feeDback (PRED), an instruction that follows the embodied agent that adjusts its behavior by perceiving environmental discrepancies such as environmental feedback and revising action plans using large language models (LLMs). For environmental discrepancies, considering that object perception plays an important role in numerous embodied tasks, we particularly focus on four distinct environmental discrepancies related to objects: 1) object presence [23, 24] , 2) object appearance [25] , 3) object attributes [26, 27] , and 4) object-object relationships [28, 29] .To address these discrepancies, we propose four components: 1) Dynamic Target Adaptation (DTA) that dynamically modifies navigation targets using object presence discrepancies, 2) Object Heterogeneity Verification (OHV) that verifies whether an interacted object is intended by examining object appearance discrepancies between the initial and subsequent perceptions of the object, 3) Attribute-Driven Plan Modification (APM) that modifies the original state-changing actions, such as cleaning an object, using object attribute discrepancy, and 4) Action Skipping by Relationship (ASR) that refrains agents from taking unintended actions using object relationship discrepancy. Previous approaches [26, 30] revise their original plans after taking actions and encountering failures. But failure may result in irreversible consequences e.g., splitting milk on the floor. Here, we propose preemptively action revision policy for the initially planned action sequences to avoid nonrecoverable failures such as irreversible state transitions as in the mentioned example [12, 13, 31] .We empirically validate PRED in two challenging benchmarks, TEACh [13] and ALFRED [12] , for embodied instruction following, as well as with real robot experiments. We observe that PRED outperforms prior arts notably in most metrics, including the main metric, the unseen success rate. 
related work: 
 Embodied instruction following agents. Developing agents that understand natural language to achieve goals is a daunting challenge. To develop such agents, previous benchmarks [32, 33, 34] focus on language comprehension and navigation. For example, [32] tasks a robot with inferring the next steps based on natural language, and [33] requires reaching a specified destination. However, these focus mainly on navigation without object interaction, limiting their application to more complex tasks (e.g., preparing breakfast using a toaster and coffee machine).To tackle tasks beyond navigation, recent benchmarks [13, 14, 35, 36] include object interaction, which requires agents to understand language and manipulate objects. Early methods [37, 14, 38, 39, 40] learn to map multi-modal inputs (i.e., egocentric observation, and language) to the corresponding actions and object locations. However, these approaches often need large training datasets for high performance, which is expensive and sometimes unfeasible.To address the data-scarcity issue, recent approaches [41, 42, 26, 30] use deterministic algorithms, such as A * or FMM [43] , for accurate obstacle-free navigation on semantic spatial maps [44, 18, 42, 30] , significantly improving their performance. Inspired by this, we also exploit [43] for navigation.Planning and revising using large language models. Recent work [30, 45, 46, 47] leverages LLMs to revise their plans after agents perform the actions and fail. However, some failures are We first generate an initial plan based on the language instruction using a LLM and predict initial and target object states to achieve a goal. Simultaneously, we predict the environmental states based on the predicted semantics. We then compare the anticipated and actual environmental states and maintain the discrepancies between them. If there are discrepancies, we use them as environmental feedback, denoted by 'Feedback Information,' to revise the original plan, denoted by 'Current Action Plan.' irrecoverable (e.g., a broken plate), causing a task failure. Although these methods revise actions after the failure, we revise actions beforehand to avoid unnecessary steps.Another line of work [48, 49] tries to solve task planning and failure correction by focusing on errors. However, no approach explores the possibility of skipping unnecessary actions for efficiency. Our method enables agents to adapt their actions based on the environment and improve efficiency like humans without needing detailed failure information.To correct the plan by receiving environmental feedback, there is some work that takes advantage of ground truth feedback [50, 51, 52, 53, 30] . On the other hand, [30, 46] updates their plans, but does not focus on object-centric representation that can lead to interaction with unintended objects. [47, 54] try to address environmental discrepancies, but can only address a limited discrepancy. For the sake of space, we elaborate the details of the mentioned works in the Appendix A. 
approach: 
 Recently proposed embodied agents use LLMs [30, 46, 55] for commonsensical reasoning and semantic spatial maps [42, 44, 18] for path planning. However, these agents often encounter scenarios where their anticipated environmental states differ from the ones they actually perceive. These scenarios can result in unintended or incorrect actions and consequently lead to inefficiency or even failure in completing tasks. Inspired by humans and animals that adapt to changing environments, we propose PRED that pre-emptively adjusts plans using environmental discrepancies as feedback. Figure 2 illustrates the workflow of PRED. Further details of PRED can be found in the Appendix C.We aim to learn a policy that generates a modified action sequence for embodied agents [12, 13] by perceiving environmental changes. Specifically, we learn a function f s : V × A -→ A, that maps a visual observation set, {v m } M m=1 , and an (N -length) action sequence, {a n } N n=1 , into a modified (K-length) action sequence, {a ′ k } K k=1 . V and A denotes all possible observation and action sets.We consider an embodied agent, a function, π θ : V × X -→ A, that maps a natural language instruction, x ∈ X , and the current visual observation, v t ∈ V, into an action, a t ∈ A. We expect the generated action sequence, {a t } T t=1 , to transform the initial environment state, s i ∈ S, to the desired state, s f ∈ S, where each X and S denotes a set of all possible natural language instructions and environment states and T a time budget. We detail our agent's architecture in the Appendix B.In particular, when an agent faces unexpected scenarios caused by 'differences' between inferred and observed states, referred to as 'environmental discrepancies,' PRED revises a original plan, {a n } N n=1 , by querying a large language model, L, with a prompt, P. P concatenates a system prompt, P s , for a general description and guide of the task, the original plan, and a feedback prompt, P f , which describes the discrepancy encountered as environmental feedback generated by the LLM. Then L receives P and produces a revised plan, {a ′ k } K k=1 as:EQUATIONAdditional details of revising actions are provided in the Appendix C.1. To build a feedback prompt, we consider four types of environmental discrepancies caused by the presence, appearance, attributes, and relationships of objects based on visual information that occupies a large proportion of sensory information perceived by humans [56, 57] . We propose four modules for respective discrepancies: Dynamic Target Adaptation (DTA), Object Heterogeneity Verification (OHV), Attribute-Driven Plan Modification (APM), and Action Skipping by Relationship (ASR). We detail the proposed modules below and provide their examples of a system prompt, a current action plan, how the feedback prompt is generated, and an LLM output in listings in the Appendix G.When an agent is conducting a task, it may encounter scenarios where a target object is present in unexpected places. For example, in the task, 'boil a potato in the refrigerator using a pot,' the agent may expect the refrigerator, possibly containing the potato, as its navigational target. However, the agent may find another potato on different objects, such as a table. In this case, revising the action plan by adapting to environments where target objects can be easily found may improve the efficiency and effectiveness of navigation. Here, we denote by an object presence discrepancy the difference between an expected location and an actual one for a target object to be found.To address this, we propose Dynamic Target Adaptation (DTA) which detects an object presence discrepancy and provides a feedback prompt describing it. For this, the agent first compares an inferred place of a target object, o, with one perceived and maintained in the agent's memory (see the Appendix B for more details), Z t , at the current time step, t. If the agent has previously observed the target object in a different place from the inferred one (i.e., o ∈ Z t ) before reaching it, DTA returns the presence discrepancy as a feedback prompt, P f , indicating that the target object is not in the expected location (e.g., P f = "The target is either directly visible to the agent or its information is already stored on the map.").An agent often fails to interact with task-relevant objects due to misperceptions influenced by lighting, occlusions, and varying appearances from different viewpoints. This issue can be mitigated by examining objects from multiple perspectives. For example, an agent might mistake a cup for a mug due to object recognition errors from a far distance.To address this issue, we propose Object Heterogeneity Verification (OHV), which recognizes appearance discrepancies and provides feedback accordingly. Here, we define an appearance discrepancy as the difference in an object's predicted identities (i.e., classes) from the appearances observed in various viewpoints. The proposed verification requires the agent to pick up objects and change its view. That is, when the agent interacts with an object, it verifies if the object is intended by comparing its predicted classes that can be different due to varying appearances from various viewpoints. These actions allow the agent to see the object without occlusion, thus encouraging it to properly identify the object (i.e., a mug) by the observations from various viewpoints.Formally, let c i be a predicted object class from the appearance of a i th viewpoint of an object, where i ∈ {0, • • • , I} denotes a viewpoint index among predefined I viewpoints. i = 0 is the viewpoint at the time of interaction (see the Appendix C.2 for more details). If the agent encounters a different predicted class (i.e., c 0 ̸ = c i ) from any different viewpoint i > 0, contrary to the expected class (i.e., ∀i > 0 : c 0 = c i ), OHV detects an appearance discrepancy and specifies this in a feedback prompt, P f , indicating that the object is not intended (e.g., P f = "After checking, the picked up object's mask is detected, but the object is not the desired one...").Language instructions often lack detailed environmental descriptions, potentially causing the agent to do redundant actions or miss important actions due to unknown object attributes. Here, an attribute refers to the physical state of an object depending on its affordances * [13, 26] . We can mitigate redundant or missed actions from unknown attributes by checking the target object, o, its expected attribute, φo , and its actual observed attribute, ϕ o .For example, in a task, "clean a mug and fill it with coffee," the agent expects that o (i.e., a mug) needs to be cleaned because it is dirty, so the expected attribute of the target can be represented as φo = {Dirty}. However, the agent might find a clean mug during exploration, where the detected target object's attribute is clean (i.e., ϕ o = {Clean}). In this case, the agent can skip redundant actions in the original plan (i.e., cleaning the mug first), allowing efficient task completion. We denote an attribute discrepancy as the difference between the expected and observed attributes of an object.For this, we propose Attribute-Driven Plan Modification (APM) to detect attribute discrepancies and provide environmental feedback. When the agent detects o, it captures a cropped image exclusively of the target from the current view. Then, we utilize an attribute detector, H, which takes the cropped image v o as input and predicts ϕ o (see the Appendix C.3 for more details). If ϕ o from H does not match φo (i.e., φo ̸ = ϕ o ), APM describes this attribute discrepancy as a feedback prompt P f (e.g., "The target object in the test image has been either cleaned, as determined by the similarity check.").Rearranging objects [10, 58] often poses challenges when the agent relocates multiple objects, potentially with the same look, of the same class, making it difficult for the agent to decide which object to move. To address this, the agent considers the current relationship, r o , and the expected relationship, ro , between the target object, o, and its placement object, o p . Here, the relationship represents the spatial relationship between o and o p . We write this as r o = (o, o p ).For instance, if instructed to "place two pillows on the sofa," the agent assumes that o (i.e., a pillow) is not on the final place (i.e., sofa) and needs to be moved. Thus, the expected relationship can be defined as the pillows that are not on the sofa, represented as ro = ¬(pillow, sof a). However, during exploration, if the agent detects a pillow on the sofa, the current relationship becomes r o = (pillow, sof a), i.e., ro ̸ = r o , and a relationship discrepancy occurs. If the agent ignores such discrepancy and continues to interact with o, i.e., meaninglessly moving the pillow from the sofa to the sofa, it may not be able to complete the task.To mitigate meaningless relocation, we propose Action Skipping by Relationship (ASR), which detects the relationship discrepancy and provides a corresponding feedback prompt. To obtain r o , we first predict the masks of objects from the current egocentric view, {m i } N i=0 , where m 0 denotes the mask of o. We then find the most 'adjacent' mask, m i , of o p , to m 0 and regard that o is currently on o p (see the Appendix C.4 for more details). If the currently detected target object is already in the final place (i.e., ro ̸ = r o ), ASR describes this relationship discrepancy in a feedback prompt P f , such as "After checking, it is found that the second object has already been placed in the desired location as indicated by the interaction mask, ..."  
experiments: 
 We briefly explain the benchmarks, baselines, and evaluation metrics used for the experiments (see Appendix D for details of benchmarks and baselines). We also validate our method in both simulation and real-world manipulation experiments. The experiment setup is detailed in Appendix E.1.Benchmarks. We employ two challenging long-horizon instruction following benchmarks for embodied agents, TEACh [13] and ALFRED [12] . In TEACh, we evaluate our PRED in two subbenchmarks, Trajectory from Dialog (TfD) and Execution from Dialog History (EDH). TfD requires the agent to solve long-horizon household tasks by understanding dialogs, while EDH requires performing a session-specific portion of the TfD tasks. ALFRED provides declarative instructions consisting of a goal statement and step-by-step instructions that describe how to complete a task.Baselines. We compare PRED with recent state-of-the-art methods as baselines for both benchmarks.For the TEACh benchmark, we compare ours with FILM [42] , DANLI [26] , and HELPER [30] . For the ALFRED benchmark, we adopt HLSM [44] , FILM [59] , and CAPEAM [18] as baselines.Metrics. The primary metric is the success rate (SR), the ratio of the completed tasks. The goal condition success rate (GC) denotes the ratio of the satisfied goal conditions. To measure efficiency, the path-length-weighted (PLW) score penalizes SR and GC by the length of the actions taken.We compare PRED with prior state-of-the-art methods on the TEACh and ALFRED benchmarks summarized in Table 1 and Table 2 , respectively. Additionally, we provide the results of a new validation and test set in EDH, used exclusively by DANLI [26] and re-split from the original validation set, to ensure a fair comparison (see the result table in the Appendix E.2). PRED outperforms other baseline models in both benchmarks by noticeable margins in SR and GC, demonstrating its efficacy.In the TEACh benchmark, in TfD and EDH setups, we observe that PRED outperforms the previous methods in unseen/seen environments for SR and GC, which implies the effectiveness of our proposed PRED. In addition, our model shows a larger performance gap between ours and prior art on TfD, a more changeable setup than on EDH. This may be because TfD requires more interactions with objects than EDH, which performs a specific portion of the TfD tasks. Thus, our proposed methods have more opportunities to be applied, leading to greater improvement in TfD compared to EDH.We observe that DANLI [26] achieves better PLW scores in the EDH setup. We believe that its 3D map tracking each instance's location, including height, eliminates the need for vertical scanning in navigation, unlike our top-down 2D map. In addition, its recovery plans' effectiveness is based on a lot of human-defined plans for all exceptions, improving PLW scores.Table 2 shows the prior arts and PRED's performance in the ALFRED benchmark with a few different settings. We include the 'Reproduced' section because the reproduced results of previous methods differ slightly from the originally reported ones. As shown in Table 1 , we observe that our method outperforms the prior arts in all metrics, implying the effectiveness of the proposed components. Diverse initial states. In the AL-FRED benchmark, all objects' states are static at the beginning of every task, indicating the initial states of objects are always fixed once a task for the agent to perform is given. For example, in the task of moving a cleaned mug, the initial states of all mugs in the environment are always set to be 'dirty.' However, this evaluation does not fully address environmental discrepancies caused by different initial states of objects (e.g., cleaning a mug that is already clean), potentially resulting in unexpected scenarios.To further address these discrepancies in the ALFRED benchmark, we intentionally modify the initial states of objects to have diverse ones and denote these modified ones as 'diverse initial states.' For example, in the task of moving a cleaned mug above, the agent may encounter already cleaned mugs and the agent can revise its plan for efficient and effective task completion. We observe that our PRED outperforms all prior approaches in this setting, implying that our PRED has better capability to adapt to environments with objects' diverse initial states.To assess the impact of each proposed component-DTA, OHV, APM, and ASR -we conduct ablation studies and summarize the results in Table 3 . Additional ablation results, including other splits in TEACh and ALFRED, are provided in Appendix F. In our experiments, no simultaneous environmental discrepancies were detected, but our PRED is designed to handle multiple discrepancies at once. Here, '(x) vs. (y),' denotes a comparison between the x and y rows in Table 3 .No DTA. We observe that ablating DTA ((a) vs. (b)) leads to noticeable drops (up to 2.68% in ALFRED test unseen) in all metrics for TfD, EDH, and ALFRED. We believe that our agent without DTA does not consider object presence discrepancies, causing repeated unnecessary object interaction. This can increase the chance of interaction failure and thus, reduce task success rates.No OHV. Second, we ablate OHV ((a) vs. (c)) to assess the impact of considering differences in object appearances. Ablating OHV results in significant performance drops, with a 4.57% SR decrease in the TfD valid unseen and a 3.01% SR decrease in the ALFRED test unseen. We empirically observe that the agent suffers from misperceptions such as light reflection and occlusion, causing it to interact with irrelevant objects. Unlike the agent with OHV, the ablated one continues without verifying whether it interacted with the correct object, leading to task failures due to misperceptions.No APM. When APM is absent from PRED ((a) vs. (d)), the agent may struggle to detect changes in the attributes of the object, leading to unnecessary actions or the omission of key actions. This results in performance drops in all metrics (e.g., 7.98% SR in ALFRED test unseen). The oversight in object attributes can cause the agent to unnecessarily change an object's state (e.g., cleaning an already cleaned object) or attempt infeasible actions to the objects that are not in an available state for interaction. This increases failures and time steps, reducing the likelihood of task completion.No ASR. We ablate ASR from our agent ((a) vs. (e)). Without ASR, the agent cannot comprehend the relationships between objects, which makes it attempt to move the objects again that already satisfy the goal relationship (e.g., a Plate should be in Sink). This may lead to achieving insufficient goal states as the agent moved 'two' objects but actually, it did only 'one' object. Due to this, we observe that a 2.61% SR drop in TfD valid unseen and a 4.06% SR drop in ALFRED test unseen. In the leftmost figure, the agent initially checks the environment. Without the ASR method (top row), the agent follows the initial plan and repeatedly moves a blue can to the yellow cupboard based on a high confidence score. However, the task fails as not all cupboards contain a blue can. In contrast, with the ASR method (bottom row), the agent detects a discrepancy in the object-object relationship, recognizing that a blue can is already placed in the right cupboard. The agent then moves the left blue can to the empty cupboard, successfully completing the task. No PRED. Finally, we conduct an ablation study when all the methods are excluded ((a) vs. (f)). We observe a substantial performance gap across all splits including valid unseen in TEACh and test unseen in ALFRED, which demonstrates that PRED greatly contributes to solving the tasks.As shown in the top row of Figure 3 (without ASR), the task fails despite completing the initial plan. In contrast, as seen in the bottom row with ASR, considering object-object relationships enables faster and more successful task completion. We also provide qualitative analyzes of DTA, OHV, APM, and ASR including simulated environments and real robot experiments in Appendix G. 
conclusion: 
 We propose PRED that revises their behaviors based on perceived environmental discrepancies inspired by brain plasticity of humans and animals before failure. Given perceived environmental discrepancies, PRED builds prompts, and queries LLMs to generate a revised plan. To address these environmental discrepancies, we propose DTA, OHV, APM, and ASR for replanning for effective and efficient task completion. We observe that our PRED outperforms previous methods notably in two challenging embodied instruction following benchmarks, TEACh and ALFRED.Limitations and future work. The environmental discrepancies are perceived based on semantic information (e.g., object masks, semantic spatial maps, etc) predicted from a single egocentric observation and therefore may not be accurate, possibly leading to inaccurate plan modification. A promising future direction is to modify a plan even with these possibly inaccurate discrepancies.Note: Orange denotes reference to the main paper. 
skip cleaning a plate: 
 Figure 1 : Overview of the proposed method (PRED). The agent needs to adapt to unexpected environmental state changes that are different from the environment that it initially conceived: it skips cleaning an already clean plate, puts down a pan when a plate is needed, grabs a knife from the countertop without further searching, and ignores moving a potato that is already on its destination plate. By the proposed PRED powered by LLMs, the agent effectively achieves the goal of serving two slices of potato on a clean plate without making mistakes.Abstract: When we, humans, perform a task, we consider changes in environments such as objects' arrangement due to interactions with objects and other reasons; e.g., when we find a mug to clean, if it is already clean, we skip cleaning it. But even the state-of-the-art embodied agents often ignore changed environments when performing a task, leading to failure to complete the task, executing unnecessary actions, or fixing the mistake after it was made. Here, we propose Pre-emptive 
 
 Appendix: Many recent researches try to use large language models (LLMs) to make their plan and revise it. For example, [30] retrieves the most relevant top-K examples of error correction, while [45] condenses a robot's previous experiences to identify failures and devises new plans. [47] plans a sequence of subgoals and revises its plan upon failure.[54] mitigates this by reasoning and proposing the next subgoal after completing the previous one, implicitly considering failure in predicting the next step, but it still does not account for failures during subgoal execution.[48] proposes correcting the error or preventing the failure using LLMs with predefined preconditions. [49] proposes to generate complete plans using a structured programming language prompt, with each action considering conditions for error-free execution.[50, 51, 52] uses LLMs for self-corrective ability by receiving environmental feedback, but depends on ground-truth feedback from environments associated with a small number of actions. Recent approaches [53, 30] sometimes exploit ground-truth information (e.g., detection results, oracle information such as, "what causes task failure?," etc) for replanning. For instance, a recent approach [53] exploits either ground truth or expensive human feedback.[30] updates plans based on full RGB egocentric observations (i.e., images), which could make the model sensitive to irrelevant factors like background changes. [46] adjusts plans using detected objects from the current observation as feedback after navigation failure but overlooks visual details such as appearance, attributes, and relationships. [47, 54] limitedly address environmental discrepancies. They identify whether a target object is at the expected destination and propose new target candidates if it is not (i.e., object presence discrepancy). However, they do not address the other discrepancies (i.e., appearance, attribute, and relationship discrepancies), potentially reducing their effectiveness and efficiency.Inspired by the recent success of the SLAM-based approaches [42, 44, 18] , we adopt an architecture that 1) plans a sequence of subgoal actions from a natural language instruction, 2) maintains the agent's memory in the form of a semantic spatial map built by an observation history with depth and masks predicted by pretrained perception models [60] , [44] , and 3) uses a deterministic algorithm [43] over the semantic spatial map for effective obstacle-free path planning. Inspired by the recent success of the SLAM-based approaches [42, 44, 18] , we adopt an architecture that 1) plans a sequence of subgoal actions from a natural language instruction, 2) maintains the agent's memory in the form of a semantic spatial map built by an observation history with depth and masks predicted by pretrained perception models [60] , [44] , and 3) uses a deterministic algorithm [43] over the semantic spatial map for effective obstacle-free path planning.Semantic spatial map. During exploration, the agent takes as input the current egocentric RGB observation, v t , for each time step, t, and predicts its semantic information (e.g., object masks, {m i } N i=0 , a depth map, d t , a spatial semantic map, Z t , etc). Using {m i } N i=0 , the agent obtains the cropped images of objects, {v c i } N i=0 , from v t . Here, N denotes the number of all detected objects in v t , and v c i an object's image crop with the masked-out background. Z t denotes a 2D top-down map predicted by merging three inputs and is generated by the semantic map generator, G, as follows:EQUATIONG transforms each pixel of d t and {m i } N i=0 into a 3D point with its semantic label. These points are then summed along the gravitational axis to generate the current top-down semantic spatial map containing only current information. Finally, G accumulate the obtained top-down map on Z t-1 , resulting in an updated semantic spatial map, Z t .Navigation policy. To interact with an object, an agent must first reach the object in its close vicinity. For this, previous approaches [37] often use behavior cloning [61] to let the agent mimic the navigational behavior of a vision-language navigation expert. However, such behavior cloning requires a large number of training trajectories and natural language annotations for satisfactory performance, but collecting these may not be trivial due to high computational costs and time.To address this issue, recent approaches [41, 42, 26, 30] instead incorporate deterministic algorithms (e.g., A*, FMM [43] , etc) to plan obstacle-free paths and observe significant improvement of navigational performance while alleviating the data collection burden. Inspired by this improvement, we also adopt a deterministic policy [43] to plan navigation routes.We provide more details of PRED. We further detail how we revise an action plan using large language models (LLMs) based on perceived environmental feedback, illustrated in Figure 4 .Additionally, we explain the 'Appearance Detector Module' of OHV, which takes egocentric views in different views as input and predicts the class of the picked-up object; the 'Attribute Detector Module' of APM, which takes the cropped image as input and predicts the object's attributes; and the 'Relationship Detector Module' of ASR, which takes the predicted masks as input and predicts the relationship between the target object and where it can be placed.Subgoal planning. Given a natural language instruction, X , such as dialogs or directives, a large language model, L, predicts the task-relevant contexts, E, such as task type, target object, its expected location, etc. Then, they are integrated into an LLM-generated high-level action plan, T , for the corresponding task type through the integration process, 'Intg.' Here, T is created to satisfy the desired states of the task type. For example, if the task type is "CLEAN ALL X" and X is an object, the desired state is defined as object = {clean}. To create the plan, we assume that the initial state is the opposite of the desired one, such as object = {dirty}, and then create a plan to clean the object.Integration of T and E resembles context-aware planning [18] , but we use an unfine-tuned LLM for context prediction instead of a trained model with benchmark data. Specific information obtained from the instruction can be incorporated into the action plan. For example, if the instruction indicates the mug is inside the fridge, an action to open the fridge is added before picking up the mug.The generated action plan consisting of high-level actions, {a h n } N h n=1 , in a triplet format is systematically converted to the executable action plan with low-level actions, {a l n } N l n=1 , in a tuple format by a rule-based function, f c , following the prior work [18] . This process is expressed as in Equation 3:EQUATIONRevising actions. While performing the task, the agent may encounter unexpected environmental discrepancies (see Section 3.1). To translate the discrepancies detected by each module into natural language, we use an LLM to generate a feedback prompt that describes the discrepancies. We provide the LLM with the Python codes corresponding to the detecting discrepancy and a prompt explaining its role. Then, LLM generates the feedback prompt such as "The target is either directly visible to the agent or its information is already stored on the map."Once we obtain environmental feedback (i.e., environmental discrepancy) based on visual input from the LLM, we build a prompt, P with this feedback to query L for plan revision. Specifically, we build the prompt with 1) a system prompt, P s , 2) the original plan (i.e., current action plan in Figure 4 ),{a n } N n=1 ({a h n } N h n=1 or {a l n } N l n=1), and 3) the feedback prompt, P f .Current Action Plan The bowl is already filled. We first extract task-relevant contexts from the dialog using an LLM. These contexts are integrated into a retrieved action plan generated by the LLM to form an initial action plan. Using semantic data from an RGB image and expected states from the initial plan, PRED verifies discrepancies as environmental feedback. The prompt including the system prompt explaining the role and guide for the LLM, the current action, and the feedback information is given to the LLM to generate a revised action plan. The bottom right image shows the task is completed along with the efficiently revised plan.Q (Put, SinkBasin3 Q (ToggleOn, Faucet3 Q (ToggleOff, Faucet3 Q (Pickup, Bowl3 Q (Pour, HousePlant)The format of the current action plan, {a n } N n=1 , can be the sequence of the actions either in an executable tuple (low-level action), a l n l , or a triplet (high-level action), a h n h . A triplet action is predefined and consists of a unit of action that includes several executable actions. For example, ('Move', 'Target', 'Parent') signifies the combination of ('Target', 'PickupObject') and ('Parent', 'PutObject'). We operate with both types of action plans simultaneously. For instance, the action plan is input in triplet form if the actions can be revised at the triplet level, while in tuple form if the revision is required at the detailed executable action level. If the output is in triplet form, it is systematically translated into executable tuple form by f c . This revision process continues whenever the environmental discrepancy is detected until the agent successfully completes the task.P for L varies according to the type of discrepancy and given action plan consisting of various objects and actions. We provide several examples of the P including P s , a current action plan, P f , and the corresponding output of the L in Listing 1 to 5. We provide the prompt to extract E in Listing 6 and one to generate a revised action plan in Listing 7.The agent often fails at object interaction due to misperception caused by lighting, occlusion, etc. To address this issue, we propose to verify that an interacted object is the intended one. The intuition is that correct perception will yield consistent perception results. Specifically, we verify that two predicted classes, c 0 and c i , of the target object, o are identical. Here, c 0 is the predicted class of o from the 0 th viewpoint at the time of interaction, and c i is that of o from the i th viewpoint obtained from a different pose from the one where the 0 th viewpoint is obtained. The agent predicts c 0 and c i from egocentric images of the object at the 0 th and i th viewpoints with a pretrained perception model. If c 0 ̸ = c i , the agent revises its actions accordingly.There are various ways to obtain different viewpoints of an object, such as grasping it from different directions or rotating it. Here, we simply lift or lower the interacted object implemented by appropriate actions (here, LOOKUP and LOOKDOWN in the Appendix D) supported by the agent's embodiment.A better method of obtaining these viewpoints may improve our proposed module, but this is beyond our scope and thus, we leave this as future work.We consider attributes related to the desired state to determine the success of the task. Furthermore, the expected attribute is one of the initial states related to the attribute, and the initial states are defined as the opposite of the desired state (detailed in the Appendix C.1).We take a retrieval-based approach to predict the attribute, φo , of a detected target object, o. This approach is preferred over training a model because it requires neither extensive training nor large datasets. Additionally, if a new attribute is added, a training-based model must be retrained or fine-tuned, whereas the retrieval-based approach only needs to add new options for comparison.To predict φo , we retrieve the most similar image and assign the attribute of the object in the retrieved image to the detected target. To compare images focusing only on the object and excluding the background, we use the cropped image of the target object, v c o , and the cropped images, {v ct i } N i=0 , from the training dataset. We compare these cropped images through cosine similarity, 'Cos', after extracting features processed by a ViT-B/32 model [62] , pre-trained with CLIP weights [2] , as:EQUATIONwhere v ′c o and v ′ct i are the features of v c o and v ct i , respectively. Then, we consider φo as the attribute of the object in v ′ct i having the highest cosine similarity score.To determine the relationship between the target object, o, and the object it is placed on, o p , we need to find the most 'adjacent' mask, m i , of o p to the target object's mask in all detected masks, {m i } N i=0 , in the current egocentric view. Here, the mask at i = 0 is defined as the mask of the target object, m 0 , of o. To find m i , we dilate {m i } N i=0 and calculate the intersection over union (IoU) between the enlarged mask of the target object, m ′ 0 , and the enlarged masks of other objects, {m ′ i } N i=1 as follows:EQUATIONWe define m ′ i as the mask with the largest IoU score with m ′ 0 , considering it the 'adjacent' mask. The object represented by m ′ i is referred to as the object o p that the target object is placed on.We validate PRED in two challenging benchmarks: TEACh [13] , for dialog instruction following, and ALFRED [12] , which provides declarative instructions, to assess generalization in different task setup. We provide details for each benchmark and baselines used below.Benchmark. The TEACh benchmark aims to allow agents to navigate and interact with objects based on instructions, with task completion achieved by meeting specified conditions, such as cleaning at least one mug for the instruction "clean a mug".The instruction is a dialog in natural language, which is comprised of two components: the COM-MANDER that provides task-relevant information based on oracle information about the task and the FOLLOWER that performs the task through the dialog. Upon receiving instructions, the FOLLOWER translates the natural language instructions and egocentric visual observations into executable actions.The executable actions are expected to succeed in the task. Additionally, the TEACh benchmark focuses on Execution from Dialogue History (EDH) and Trajectory from Dialogue (TfD). This benchmark is divided into train, validation, and test splits. Evaluation metrics encompass success rate (SR), goal-condition success rate (GC), and path-lengthweighted (PLW) scores.State-of-the-art baseline models. We compare our PRED with the recently proposed state-of-theart methods: E.T. [37] , JARVIS [41] , FILM [42] , DANLI [26] , and HELPER [30] . E.T. learns a direct mapping from a natural language dialog and an egocentric observation to a corresponding action and the position of an object to be interacted with. JARVIS employs an LLM trained on the TEACh dialog dataset to produce high-level subgoals, replicating the ones executed by human demonstrators. It utilizes a semantic map alongside the E.T. to locate objects. FILM enhances an LLM through fine-tuning to generate a parameterized plan. Mirroring Jarvis, it leverages a semantic map to execute subgoals and employs a semantic policy for object search. DANLI fine-tunes an LLM for high-level subgoal prediction and employs symbolic planning with an object state and spatial map for execution plan formulation. It incorporates an object search module and manual error correction mechanisms. HELPER utilizes a Large Language Model (LLM) to generate initial high-level actions with additional data. When a failure occurs, it predicts the error reasons through a pretrained vision and language model, and revises the action using the LLM.Benchmark. The ALFRED benchmark requires agents to complete a long-horizon task by understanding declarative natural language instructions with egocentric observations. The declarative instructions comprise two types of instruction: one is a high-level description that provides a single sentence to complete the task and the other is a step-by-step instruction that details the process of performing a task. By following the instructions, the agent executes the two types of predefined actions. The navigation actions include MOVEAHEAD, ROTATERIGHT, ROTATELEFT, LOOKUP, and LOOKDOWN. The interaction actions include PICKUPOBJECT, PUTOBJECT, OPENOBJECT, CLOSEOBJECT, TOGGLEOBJECTON, TOGGLEOBJECTOFF, and SLICEOBJECT.State-of-the-art baseline models. We compare our PRED with the recently proposed state-of-theart methods: HLSM [44] , FILM [57] , and CAPEAM [18] . HLSM employs a hierarchical controller to translate natural language instructions into actions the agent can execute. The high-level controller identifies the next subgoal based on the given instructions and map, while the low-level controller generates a sequence of actions to accomplish this subgoal. FILM uses a pre-constructed template as a high-level action plan. It employs two BERT [6] classifier submodules to identify the instruction type and determine the template arguments. It applies a deterministic algorithm [43] to plan a path without obstacles. CAPEAM employs context-aware planning to devise a sequence of subgoals and execute each subgoal using the appropriate detailed planners. It also utilizes extra memory to avoid interacting with unsuitable objects.We conduct experiments in a real environment to evaluate the effectiveness of our proposed method.To perceive the environment and obtain feedback, we use grounded SAM [63] . Additionally, we employed the UR5 manipulator for object interaction. First, we recognize the objects and environment from an egocentric view using a camera mounted on the back of the manipulator's gripper. Once the agent identifies the desired object using [63] and determines which object to interact with, it retrieves the detected mask of the object as input to select the correct object in the real world. The agent then receives the ground truth position of the selected object and proceeds with the interaction. We conduct an additional experiment, exclusively done by DANLI [26] for fair comparison, in the TEACh benchmark [13] . In this benchmark, we investigate the performance of our PRED in different splits provided on the TEACh GitHub and in [26] . The leaderboard for EDH of the TEACh benchmark is unavailable, preventing the evaluation on its true test set. Thus, we leveraged the original validation splits for seen/unseen scenarios, aligning with the approach taken in most prior studies [30, 41, 42] .In Table 4 , we present the alternative validation and test splits. We observe that our method outperforms others in the new EDH split, achieving improvements with notable margins in SR and GC, similar to the results in the original split. In the seen environment, the agent encounters fewer misperceptions and navigation errors than in the unseen environment, making our PRED less effective.We provide the results of ablation studies in all splits including 'Seen' environments in TEACh and ALFRED benchmark in Table 5 and 6 .G Qualitative Analysis DTA. First, Figure 5a describes two different simulation scenarios: PRED without (top) and with (bottom) DTA in Section 3.1.1. An agent generates an initial plan considering the information mentioned in the instruction ("mug, potentially in a cabinet") so that it predicts that the mug is in the cabinet and has a plan of opening the cabinet first, instead of picking up the mug. Thus, in the upper scenario (w/o DTA), the agent keeps its original plan to open the cabinet even if the usable mug is observed in its sight. On the contrary, the agent with DTA changes the plan to skip the action with the cabinet and to pick up the mug right away after perceiving the mug. As a result, it can finish the task efficiently as it adapts to the discrepancy of the target object's presence in the environment.Figure 6a shows two real robot scenarios: one without DTA (top) and one with DTA (bottom). The agent forms an initial plan based on the instruction ("Pepsi, possibly in a red cup"), assuming the Pepsi is in the red cup and planning to move the cup before picking up the Pepsi. In the scenario without DTA (top), the agent follows the initial plan and moves the red cup, even though another Pepsi is visible. In contrast, with DTA (bottom), the agent adjusts the plan, skipping the red cup and picking up the Pepsi immediately after detecting it. This adaptation allows the agent to complete the task more efficiently by adapting to the actual environment.OHV. Figure 5b illustrates the difference with and without OHV in the simulation, as described in Section 3.1.2. The first image shows the agent needing to pick up a 'Mug' but mistakenly picking up a 'Cup' instead by misperception. The agents without OHV will not try to correct the wrong action since they do not consider the appearance discrepancy after the action is done. In contrast, our agent with OHV verifies whether the interacted object aligns with the desired one by examining its appearance from multiple angles. If it detects the discrepancy, the agent rectifies its mistake by revising actions. As a result, OHV helps the agent to prevent the failure that comes from the interaction with the wrong object.Figure 6b demonstrates the impact of OHV in real robot experiments. At t = 1, with the scenarios with OHV (top row) and without OHV (bottom row), the agents experience misperception because the color and design of the cans look almost identical from the top-down view. As a result, at t=2, the agent picks the can with the higher confidence score but mistakenly grabs a drink other than the desired Fanta. In the scenario without OHV (top row), the agent sticks to the original plan, ignoring the mistake, and delivers the wrong item to the destination. However, with OHV (bottom row), the agent views the drink from the front, identifies the label, and realizes it has picked the wrong item, detecting the appearance discrepancy. The agent then puts down the incorrect item, updates the plan, and successfully completes the task.APM. Next, we investigate the benefit of APM (in Section 3.1.3) depicted in Figure 5c with the scenarios where agents want to open a microwave. The agent without APM attempts to open it as it supposes that the microwave is not operating, leading to an interaction failure. In contrast, the agent with APM considers the attribute discrepancy coming from the difference in the microwave's state. With this, it adds actions that toggle it off first to future actions, enabling successful interaction.ASR. Finally, we elucidate the advantage of ASR in Section 3.1.4. Figure 5d describes the benefit of considering relationship discrepancy. We refer to the relationship corresponding to the goal condition in the instruction as the goal relationship. An agent without ASR may pick up a remote and put it down with a higher confidence score, even though it already satisfies the goal relationship, resulting in only one remote on the table. If this were the second interaction (picking it up and putting it down on the table), the agent would think that it completed the task since it completed interactions twice. However, the task would fail since there are no two remotes on the table. In contrast, the agent with ASR considers relationship discrepancy and interacts with a remote not in the goal relationship instead of the one already in a goal relationship. Listing 3: Example of the input (i.e., Prompt) and the output (i.e., Revised Action Plan) of the LLM. This example describes the result where APM is applied. Compared to the current action plan, the action (i.e., toggle off the microwave) is added in the revised action plan to avoid interaction failure.# System Prompt You need to determine whether to revise the action sequence to solve the task considering ' GUIDE ' and then write down the final sequence of actions if needed . This action sequence is made for solving household task . Each action ' s format is a triplet . There are two options . First , if you think that the first action is not needed , remove that action ( triplet ) and write down the actions after that . You can only revise the first action ; do not modify the actions ( from the second actions ) that follow , and write them down as originally provided . In this context , ' action ' refers to each element in triplet format in the list ( action sequence ) . Second , if you still think the first action is necessary , write it down exactly as received . Do not add additional explanation . Just write the final action sequence in the right format ( triplets in the list ) .GUIDE : Our purpose is to perform the task efficiently . Listing 4: Example of the input (i.e., Prompt) and the output (i.e., Revised Action Plan) of LLM. This example describes the result where APM is applied. Compared to the current action plan, the action (i.e., clean the target) is deleted to skip an unnecessary action. A Target will be replaced with the corresponding object based on the context.# System Prompt You need to determine whether to revise the action sequence to solve the task considering ' GUIDE ' and then write down the final sequence of actions if needed . This action sequence is made for solving household task . Each action ' s format is a triplet . There are two options . First , if you think that the first action is not needed , remove that action ( triplet ) and write down the actions after that . You can only revise the first action ; do not modify the actions ( from the second actions ) that follow , and write them down as originally provided . In this context , ' action ' refers to each element in triplet format in the list ( action sequence ) . Second , if you still think the first action is necessary , write it down exactly as received . Do not add additional explanation . Just write the final action sequence in the right format ( triplets in the list ) .GUIDE : There can be some objects that are already located in the desired destination . If you think executing the following action should be avoided as it is no longer needed , add a Pass action in triplet form ( same as given action ) with Pass for action , None for Target and Parent , before the given action ( including given action ) without further explanation . If you think the following actions are still needed , repeat the given actions . # Current Action Plan [ " Move " , " Target " , " Parent " ] # Feedback Information After checking , it is found that the second object has already been placed in the desired location as indicated by the interaction mask , so no further interaction is needed .### OUTPUT [ " Pass " , " None " , " None " ] , [ " Move " , " Target " , " Parent " ] Listing 5: Example of the input (i.e., Prompt) and the output (i.e., Revised Action Plan) of LLM. This example describes the result where ASR is applied. Given the current action plan (i.e., move a target to a recep), LLM adds 'Pass' action in triplet which leads to skipping the ongoing interaction before the given action. Then the agent will not interact with the target since it is already located in the desired place as explained in the GUIDE. Target and Parent will be replaced with each corresponding object based on the context. Driver tries to solve the task . Commander gives information helpful to solve the task . You have to get information through the dialog . Find the initial states of the objects and summarize them into a dictionary . If you cannot find proper information in the dialog , you should answer 'X '.Just write a dictionary without giving an additional explanation . In a dictionary , fix the keys same with the example answers . Each of the keys has properties ( keys ) e . g . , " location " which contains the initial location of the object . In some tasks , the driver may be asked to find multiple objects of one kind . In those cases , if an object is on Cabinet and another object is on CounterTop , you should output [ ' Cabinet ' , ' CounterTop ']. ( If you cannot know where the potato is initially , answer 'X '.) , " receptacle " which is the place that the object should be placed on or in ultimately . " quantity " which represents the number of the objects , " quantity_of_slices " which represents the number of the object ' s slices . Some objects ( e . g . , plate , mug ) have a key i . e . , " Cleaned " which represents whether the object should be cleaned ( then write " T ") or not ( then write " F ") . If you cannot find the proper information , just write 'X '.Dialog : { DIALOG } Answer :Listing 6: Prompt for Extracting Task-Relevant Contexts from the Dialog. {} denotes the section in the prompt that is replaced to each corresponding data. This prompt is designed to extract the useful information from the dialog to use it when making an initial action plan.* We use object attributes supported by AI2-THOR[15] on which our evaluation benchmarks[12,13] run. 


Decision: The submission introduces an approach for dynamic replanning for an agent's actions based on real-time environmental feedback. The proposed method reacts adaptively when discrepancies, such as object presence, appearance, attributes, and relationships, between the planned actions and the actual environment are detected. It is shown to be quite effective in 2 benchmarks (ALFRED and TEACh). On the other hand, the reviewers were concerned on the scalability and real-world applicability. The proposed method is only tested on simulated environments, which are even with relatively weak physical simulation, leaving questions about its effectiveness in complex scenarios and real world.

While it is indeed true that there is a need for further investigation, I recommend accept with the novelty of the proposed approach and its significant improvement. The learning based dynamic replanning, especially preventive one, is timely a good topic to discuss in the community. I however strongly suggest some real world demo in this case, which would enhance the paper quality and impact.

Reviews: review: title: title: Good paper to read, with a lot of experiments
paper_summary: paper_summary: The paper studies the problem in the instruction following, in which the agent is tasked with the instruction and is required to complete the task. The paper focuses on how the agent can react to unexpected conditions in the environment, and adaptively change the plan. In the paper, there are four unexpected conditions -- object presence, object appearance, object attributes, and object-object relationships. The paper proposes several methods to deal with these cases. The method is being verified in several embodied AI tasks (TEACh and ALFRED), and results indicate the performance is good.
main_review: main_review: Strengths:
1. Clarity and Readability: The paper excels in its clarity and ease of understanding. The writing is straightforward and well-structured. The logical flow of the content ensures that key concepts are effectively communicated.
2. Innovative Methodology: The methodology presented in the paper is notably novel and introduces fresh approaches to the field. I did not see any paper in the field that talks about how to deal with such cases. Most of the papers focus on planning, not re-planning.
3. Comprehensive Experiments: The experiments conducted are thorough and cover a broad spectrum of scenarios. The detailed experimental setup and rigorous testing contribute to the robustness of the research findings. 

Weaknesses:
1. Lack of Real-World Robotics Experiments: One notable limitation of the paper is the absence of real-world robotics experiments. While the simulation-based experiments provide valuable insights, the lack of practical application in real-world settings means that the findings may not fully address the challenges encountered in actual robotic environments. For instance, how to deal with the conditions besides the four listed in the paper.
questions: questions: 1. One question I have is that how do you get the images of the plate before doing the task. I can imagine that you hardcode the image of the plate for each task, and assume that this is known. However, if the agent does not have such information before hand, how do you detect the the difference?
limitations: limitations: 3
review_summary: review_summary: The quality of the paper is good, and the easy to read. The method seems to be novel.
title: title: review for submission 58
paper_summary: paper_summary: In this paper, the authors propose a RED framework which checks four distinct environmental discrepancies concerning objects to generate/revise actions for the agent to complete an embodied instruction following task.
main_review: main_review: In this paper, the authors propose a RED framework which checks four distinct environmental discrepancies concerning objects to generate/revise actions for the agent to complete an embodied instruction following task. The logic of the paper is clear and easy to follow. Here are some concerns.

- Based on the paper, the environmental discrepancies are all predicted by some pretrained perception model, mostly relying on some vision models. However, there are attributes, appearance, etc, which might be difficult to recognize with only visual information. For example, how could the perception model recognize if an apple is cooked or not. Also, is there any mechanism to deal with the situation where the pretrained perception model fails.

- There are already a lot of works that try to revise or replanning action sequence based on large language models. What is the specific differences between the proposed method and other methods, such as (just name a few):
[1] ReAct: Synergizing Reasoning and Acting in Language Models
[2] ADaPT: As-Needed Decomposition and Planning with Language Models
The environmental feedback seems also included in other methods. I suggest a more thorough investigation of similar work and clarify the novelty of the proposed method. I am also curious about the performance when using a vision-language model for the task.

- It will be interesting to discuss the performance of the proposed method when it is applied in the real world. A real world demonstration will be a big plus to this work.
questions: questions: - Pretrained perception model limitation: Based on the paper, the environmental discrepancies are all predicted by some pretrained perception model, mostly relying on some vision models. However, there are attributes, appearance, etc, which might be difficult to recognize with only visual information. For example, how could the perception model recognize if an apple is cooked or not. Also, is there any mechanism to deal with the situation where the pretrained perception model fails.

- Difference and novelty of the proposed method with other methods: There are already a lot of works that try to revise or replanning action sequence based on large language models. What is the specific differences between the proposed method and other methods, such as (just name a few):
[1] ReAct: Synergizing Reasoning and Acting in Language Models
[2] ADaPT: As-Needed Decomposition and Planning with Language Models
The environmental feedback seems also included in other methods. I suggest a more thorough investigation of similar work and clarify the novelty of the proposed method. I am also curious about the performance when using a vision-language model for the task.

- Real-world evaluation: It will be interesting to discuss the performance of the proposed method when it is applied in the real world. A real world demonstration will be a big plus to this work.
limitations: limitations: 2
review_summary: review_summary: There is no real-world evaluation and the novelty of the proposed method is a little weak.
title: title: Well-motivated paper, but the approach seems too specific. EDIT: The rebuttal addresses my concerns.
paper_summary: paper_summary: This paper proposes an approach for embodied instruction following to dynamically update the plan based on environment feedback, using LLMs. They look at 4 types of discrepancies, and propose respective modules to fix each one.
main_review: main_review: **Strengths:**

* The paper addresses an important problem in embodied AI, namely, adapting an agent’s plan to remove discrepancy between the plan and the environment, and proposes a novel approach for it.
* Experimental evaluation shows significant improvement over 2 benchmarks — ALFRED and TEACh.


**Limitations:**

The main limitation of the approach seems to be that it is too specific in multiple ways:

* The authors propose 4 types of discrepancies, and a corresponding module to fix that problem. Thus, to make the system robust to errors that are not handled by the current approach would require defining more discrepancy classes and modules, which may not be scalable (particularly as these discrepancies and modules might interact with each other).
* Line 120 (main paper): In this example, the agent could find another potato before checking the refrigerator and complete the task faster, but isn’t this incorrect if the user wanted to use the one in the refrigerator? This may complete the task faster, improving results on the benchmarks, but does not solve the desired task.
* Lines 75-82 in the appendix describe the appearance detector module. This seems to be very simulator-specific — if the i_0 viewpoint is not a top-down view, would this step be needed?


**Other questions/issues:**


* What happens if the new modules proposed in this approach are not very good? E.g. the object attribute detector gives an incorrect prediction. Can this introduce errors into an already correct plan, instead of fixing them?
* The proposed approach should depend on how the initial plan was generated. Presumably, for any base approach that generates a plan, this approach can be applied on top to improve it. It would be good to see how the base approach used in this work performs without any of these modifications (the ablations in Table 3 addresses this to some extent, but it would be good to see the numbers without any of the additional modules). Further, if this approach was applied to other baselines, would that lead to further gains?



**Typos:**

Appendix Line 100: adjust → adjacent


EDIT:
In the rebuttal period, the authors applied the approach to 2 other baselines, and showed improvements on both. This addresses my concern that the approach might be too specific, and I've updated my rating from Weak Reject to Weak Accept.
questions: questions: - Line 126: I’m not sure I understand the following paragraph: does “o” map to potato, “Z_t” to table in the example?

- Line 170: How is the prompt e.g. “After checking, it appears that the target has already been cleaned” generated? Section B.3 in the appendix only describes how the attribute is predicted, but how is that information translated into language?
limitations: limitations: 1
review_summary: review_summary: The paper proposes a novel approach in embodied AI but the approach faces scalability concerns due to specificity in handling discrepancies and potential dependencies on simulator-specific modules, raising questions on robustness and broader applicability. EDIT: The authors applied the approach on other baselines during the rebuttal period, and have shown improvements, addressing my concerns about it being too specific.