REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction
Authors: Zeyi Liu
Arpit Bahety
Shuran Song
Venue: CoRL 2023 Poster (2023)
Keywords: Large Language Model
Explainable AI
Task Planning

Abstract: The ability to detect and analyze failed executions automatically is crucial for an explainable and robust robotic system. Recently, Large Language Models (LLMs) have demonstrated strong reasoning abilities on textual inputs. To leverage the power of LLMs for robot failure explanation, we introduce REFLECT, a framework which queries LLM for failure reasoning based on a hierarchical summary of robot past experiences generated from multisensory observations. The failure explanation can further guide a language-based planner to correct the failure and complete the task. To systematically evaluate the framework, we create the RoboFail dataset with a variety of tasks and failure scenarios. We demonstrate that the LLM-based framework is able to generate informative failure explanations that assist successful correction planning.

REFLECT: Summarizing Robot Experiences for Failure Explanation and CorrecTion
Zeyi Liu
Arpit Bahety" Shuran Song
Columbia University, New York, NY, United States
http://mbbot-rfllcc.giihhbi
The ability to detect and analyze failed executions automatically is crucial for an explainable and robust robotic system. Recently, Large Language Models (LLMs) have demonstrated strong reasoning abilities on textual inputs. To leverage the power of LLMs for robot failure explanation, introduce REFLECT, framework which queries LLM for failure reasoning based on a hierarchical summary of robot past experiences generated from multisensory observations. The failure explanation can further guide a sinuuage-based planner to correct the failure and complete the task. systematically evaluate the framework, we create the RoboFail dataset with variety of tasks and failure scenarios. We demonstrate that the LLM-based framework is able to generate informative failure explanations that assist successful correction planning.
Abstract:
Keywords: Large Language Model, Explainable AI, Task Planning
REFLECT Framework
RoboFail Examples
Multisensory Hierarchical Robot Summary
pot
cup coffee
Task Heat potato
Fellure: Miseed slep
Carrol out
machine
Action: Pick bow
Tesk: Store in fridge
Failed
Failure: Chose the wrong bowl
LLM
pick
LLM
Failure Explanation: turn microwave when door open
Failure Correction:
Microwave Door 2. Tumn on Microwave
Execution Failures
Planning Failures
Fig 1: A framework for robot failure explanation and correction. On the left, we show the REFLECT framework that converts multisensory observations (RGB-D, audio, robot states) to a hierarchical summary of robot experiences. The summary is then used to query Large Language Model (LLM) for failure explanation and correction. The right shows few example failure cases in the RoboFail dataset.
1 Introduction
With the increasing expectations for robots work on long-horizon tasks in complex environments, failures are inevitable. It is thus an essential capability for robotic system to reflect on its past experiences and explain its failures in natural language. The failure explanations can either help a human user to debug the robotic system without having to read through the tedious execution logs, or guide the robot to correct the failure by itself.
We hypothesize that an effective failure reasoning framework requires several key components: first, component to summarize "what happened" by converting unstructured, multimodal robot sensory data into a structured, unified format; second, component to reason "what was wrong" by inferring from the summary whether expected outcomes the robot plan were achieved; and finally, the ability to plan "what to do" based on the failure reasoning to correct the failure and complete the task.
indicates equal contribution
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Recently, Large Language Models (LLMs) 2, 3] have been shown to exhibit strong reasoning [4, 5, 6] and planning [7, 8, 9] capabilities, making a promising component for explainable and robust robotic sys- tems. However, the remaining challenges lie in how to generate a textual summary of robot sensory data and systematically query LLMs for failure reasoning. In other words, how do we transform the robot failure rea- soning task into a language reasoning task? We observe two important attributes of good robot summary:
Multisensory. The summary should cover all sensory modalities the robot has access to, such as visual, audio, contact, etc. This is because certain failures can be more easily identified through one type of sensory data than another. For example, it is casier determine if an object is dropped or if water is running from the faucet using auditory cues rather than visual ones.
Hierarchical. To support effective failure explanations, the robot summary requires multiple levels of abstraction: to quickly localize the failure, the highest summary level should focus on identifying misalignment between the robot high-level plan and execution outcomes; while the lower summary levels should maintain enough environmental context for LLMs to generate an informative explanation that useful for correction planning.
Based on these observations, we introduce REFLECT, a framework that summarizes robot experiences for failure explanation and correction. The framework first processes post-execution robot observations to generate a hierarchical summary with three levels of abstraction. Equipped with this summary, we propose progressive failure explanation algorithm for failure reasoning. Through our experiments, demonstrate that the framework is able to generate informative failure explanations as assessed by human evaluators, and also guide language- based planner to generate correction plans for several failure scenarios.
To systematically evaluate REFL EEF,, we also create the RoboFail dataset, which includes 100 failure demonstrations generated in the AI2THOR simulation [10] and 30 real-world failure demonstrations collected with UR5e robot arm. We hope the dataset will encourage development of more explainable and robust embodied AI systems.
2 Related Work
Dense Video Captioning for human activity videos has been a challenging task in computer vision. Recent video captioning methods typically train transformer-barte models to jointly localize and caption events in videos [11, 12, 13, 14, 15). Yet remains a challenge caption robot videos due to lack of data. With the emergence of large foundation models, zero-shot video captioning is made possible [16, 17, 18). These works combine VLMs and LLMs to caption egocentric human activity videos in a zero-shot manner. Extend- ing upon prior works, our approach generates captions that are task-centric and action-centric for temporal robot sensory data in a zero-shot manner, which helps downstream tasks such as robot behavior analysis [19].
Robot Failure Explanation an important task long studied in the HRI community to increase human trust in robotic systems [20, 21] and allow non-expert users to better assist robots under failure scenarios [22, 23J. However, prior works are limited as each only address specific set of failure scenarios: Das and Chernova [23] specifically study failure cases in picking, Diehl and Aamirez-Amaro [24] study two pick-and-place tasks, Song et al. [25] focus on object navigation failures whereas Inceoglu et al. [26] focus on detecting failures in a few short-horizon object manipulation tasks. By leveraging the advanced reasoning ability of LLM,, our framework is able to detect and explain a wide range of failure scenarios without assumptions on the task configuration or failure type.
Task Planning with Large Language Models Large Language Models can be leveraged to decompose high-level, abstract task instructions into low-level step-by-step actions executable by agents [7, 9, 27, 28]. Recent works have also demonstrated the self-reflective and self-corrective ability of LLMs based on environment feedback [8, 29, 30, 31, 32]. However, they all assume ground truth environment feedback associated with one or few actions. this work, we explore the reflective ability of L.LMs directly on mul- tisensory robot observations and its temporal reasoning ability on long-horizon robot task executions. Our framework is able to directly operate on real-world robot task execution data with various failure scenarios.

---

L2 Subgoal Summary
00:30. Geat pot sink. Visual: snkk fauuet (tumed of (impte) dish sponge nothing insice robut
Coption (00.00)
Caption (00/50)
Capdion (00-40)
mude
right
f L1 Wvent-llased Summary
00:40, Goat Toggle faucet, Vaual pot (illed with
voite ), fauret, sink Auditor foucet tumns
Captinn
Caption (90:20)
Caprioon (00/30)
Capricn (00:40)
[D0:15]
Helding
Object state
Oity o0) rolations
4
nomiing
Biy robost relations
LO Sensory- Input Summary
XBK
por
OnTTopOf
Task-in informoooddd
t Audio
Audio: Faucet tums on
fied water
counterr
RGB Visual
Aggregated sernantcc cloud
Pot Toggle
Fig Hierarchical robot summary is composed of: a) a sensory-input summary that converts multisensory robot observations (RGB-D, D, sound, robot states) into task inffomefomed scene graphs and audio summary; b) an event-based summary that generates captions for key event frames; c) subgoal boggall-bssed summary that contains the end frame of each subgoal.
3 Method: the REFLECT Framework
The REFLECT framework summarizes robot's past experiences for failure explanation and correction. It contains three modules: hierarchical robot summary module that summarizes multisensory robot data with three levels of abstraction (53,1), progressive failure explanation module that queries L.LM to detect and explain the failure ($3.2), and finally a failure correction planner that generates an executable correction plan ($3.3).
3.1 Hierarchical Robot Summary
To perform effective failure explanation and correction, we propose a hierarchical summary structure to aggregate and convert robot sensory data over time into unified structure: 2) summarize the robot experiences for efficient failure localization and explanation. The hierarchical summary structure contains three levels: sensory-input summary, event-based summary, and subgoal- -bseed summary.
3.1.1 Sensory-Input Summary
The sensory-input summary processes unstructured, multimodal robot sensory observations into unified structure that stores necessary information for failure explanation.
Visual summary with task-informed scene graphs. To understand a robot's interactions with its surrounding environment, is important to extract inter-object relations, robot-object relations, and object state information from the observations. Given the RGB-D observation at timestep f, we run object detection on the RGB image to obtain a semantic segmentation I and project to a 3D semantic point cloud using the observed depth. In addition, for objects that can change states (e.g. microwave can be turned and off), crop the image based the object's detected bounding box and compute the cosine similarity of the CLIP embedding [33] between the cropped image and a pre-defined list of object state labels (see details in $B.3). Given the semantic point cloud, heuristics are applied to compute inter-object spatial relations for commonly-usee spatial relations': inside, on top of, on the left, on the right, above, below, occluding, and near. We also infer the robot-object relation from gripper state and object detection results. To aggregate the 3D point cloud over time frames, we use similar approach as Li al. [34] align the newly observed point cloud pt with the accumulated point cloud from all previous time steps P-1 using heuristic operations: add, update, replace, and delete.'
Once having the aggregated point cloud, we construct scene graph G, (N + (Nrobot }, E}, which describes the object nodes (N) and their spatial relations (F)). Each node is defined as n, = (c,,8t), where e; the object class, 8; is the object state if any. Each edge contains the spatial relation between the two objects. We add the robot as an additional node Nyyobor and special relation "inside robot gripper". To make
Details of all heuristics can be found in appendix.
3

---

summary succinct and reduce computation and memory cost, we only consider objects that are relevant to the task (as extracted from the original rohot plan), and objects spatially related to the task-relevant objects.
Audio summary. Audio is a useful modality for failure reasoning as it provides immediate feedback of failure events (e.g. something drops from gripper to the ground) and detects state changes when visual cues are limited (e.g. stove burner turns on but occluded by an object on top).
Given input audio stream, we first segment the whole audio clip into several sub-clips by filtering
out ranges where the volume is below a certain threshold Then for each sub-clip s, we compute its
audio-lingual embedding with a pre-trained lanio-ln model (e.g. AudioCLIP [35], Wav2CLIP [36]). We calculate the cosine similarity between the audio embedding and the CLIP embeddings for a list of can- didate audio event labels L, where the highest-scoring label I* is selected: I" = gr fx(e)-((()) where fa is the embedding function for audio, f2 is the embedding function for text. [2)0011
C=
3.1.2 Event-Based Summary
Given that the censory-ipput summary ($3.1.1) computes scene graph for each frame and thus contains redundant information, the goal of the event-based summary is to select key framess and generate text captions from the corresponding scene graphs.
We design key frame selection mechanism based on visual, audio, and robot states. More specifically, frame is selected if it satisfies any of the conditions below: 1) The task-informed scene graph of the current frame G, is different from the previous frame G,-1. 2) The frame is the start or end of an audio event. 3) The frame marks the end of a subgoal execution.
For each key frame, we convert the scene graph into text with the following format.2 When constructing the visual observation, we only consider objects that are visible in the current frame.
[timestep] Action: [robot action]
Visual observation: objectl [state], object2, object3 [state] # objects and states
objectl is (spatial relation] object2 ... inter-ohject relations
object3 is inside robot gripper. # robot-object relations
Auditory observation: [audio summary).
3.1.3 Subgoal-Based Summary
The event-based summary (in $3.1.2) stores environment observations throughout the robot task execution. However, it's hard for LLM to infer the expected outcomes and identify failures for every one of the low-level actions. As a result, we introduce the subgoal-based summary, which consists of observations at the end each subgoal, for LLM to identify misalignment between the robot execution outcomes and its high-level plan (e.g. move to the toaster, put bread slice in the toaster). The subgoal-based summary enables the failure explanation module to quickly process the robot experience summary by checking whether each subgoal is satisfied while ignoring low-level execution details. Once failure is detected, relevant environment information stored in the event-based summary or sensory-input summary can be retrieved for detailed failure explanation.
3.2 Progressive Failure Explanation
The failure explanation algorithm should handle both execution and planning failures, where the former requires action-level observation details and the latter requires task-level information such as task description and robot plan. To do so, the algorithm first identifies the type of failure and then retrieves relevant information from the hierarchical summary construct the query to LLM. As shown in Fig. the algorithm first iterates through the subgoals and verifies success using the following prompt: 3
The robot subgoal is [robot subgoal at time Given (subgoal-based summary at time Q: Is the subgoal satisfied? A: Yes
"text color: blue: visual, green: audio light blue: contact, yellow: summary, orange: final state, failure explanation, brown: timestep, task name, robot subgoal, original robot plan, goal state, blue highlight: L.M output Examples of full prompts shown in the appendix.

---

Succesa Verification subgoal-based summary
Plan Analysis (w. plan final state)
Failure Explanation with LLM
Stove micnurch
The robot task <boil water>
A: The coboc placed the pat the fourth slove humer but tumeil the sasored elove burner, mismatch between the
Visual summary
Auditory summary
pess verifications
Given crabot plano-ffinal states
Q: What's wrong with the robot plan?
lecation the nctive burner.
Q: Is the subgoal satisfied?
Execution Analysia (w. event-based summary)
dropped
Answer LLM: /No
Else
A failure is identified at -00-4>
Given cevent summary up to 00:44>
dropped pot filled with water at 00:36 while moving to the fourth stove hurner.
Repeat all subgoala
Q: Explain the failure.
Fig 3: Progressive failure explamation verifies success for cach subgoal. If a subgoal fails, the algorithm enters the execution analysis stage for detailed explanation. If all subgoals are satisfied, the algorithm enters planning analysis stage to check errors the robot plan.
The LLM is instructed to output "Yes' or 'No'. If subgoal is not achieved, then we retrieve history observations stored in event-based summary for failure explanation as follows:
The robot task is to Itask name]. A failure is identified at t. Given (event- based summary up to t]
Q: Briefly explain what happened at and what caused the failure?
A: At 00:44, the robot attempled to put the pot on the fourth stove burner, but the pot was not in its gripper. The
failure was caused by the robot dropping the pot filled with water at 00:36 while moving to the fourth stove burner.
In case all subgoals are achieved but the task still failed, then it's likely that the plan itself was incorrect. We use the robot original plan and the final state of the environment to identify errors in the robot plan. The final state is obtained from the scene graph generated from the aggregated semantic point cloud in the last time step without view-specific relations (on the left, on the right, occluding).
The robot task is to [task name]. The task is successful if [goal state].
The robot plan is [original robot plan]. Given [final state]
Q: What's wrong with the robot plan that caused the robot to fail?
A: The robot placed the pot on the fourth stove burerr but turned on the second stove burner, causing a mismatch
between the pot's sot's location and the active burner.
Q: Which time step most relevant the above failure?
A: 00:49
3.3
Failure Correction Planner
A failure correction planner should generate an executable plan for the robot to correct the failure and complete the task, starting from the final state of the original task execution. Prior work [22] has shown that good failure explanations help non-expert users better understand the failure and assist the robot. Analogously, we hypothesize that the failure explanation can also guide a language planner to generate a high-level correction plan that leads to task success. The prompt is formatted as below:?
The robot task to [task name]. The original robot plan is [original robot plan]. Given [failure explanation] [final state] and [goal state] Conection plan: toggle off (stovehurer-22) toggle on (stovehumer-4)
To make sure the plan generated by the language model is executable in the environment, adopt the idea of Huang et al. [28] to map each LLM LLM-generated action to its closest executable action in the task environment using large pre-trained sentence embedding model.
4 The RoboFail Dataset
In simulation, we generate task execution data in AIZTHOR and manually inject failures. The dataset contains a total of 100 failure scenarios, with 10 cases for each of the 10 tasks (see details in ŞB.1). We store the RGB-D D observations, sound (20 classes in total), robot state data, as well as ground truth metadata obtained from simulation. The real-world dataset is collected by human teleoperation of a URSe robot arm in a toy kitchen environment. The dataset contains 11 tasks with total of 30 failure scenarios. We store the RGB-D observations (with Intel RealSense D415), recorded sound (with RØDE VideoMic Pro+). and robot proprioception data, A taxonomy of failure scenarios are visualized in Fig. 4.
5

---

Turn microwave before closing its door
Wrong action order (11%)
Unexpected dynamics (11%)
Object dropped
The plan does not specify which stove burner to use
Ambiguous reference (5%)
Falled execution:
Cannot pick knife due to a pot blocking
environment (13%)
Planning Fallures 65%
Execution Fallures 48%
Attempt place aon-milrrowavable inside the microwaae
Wrong actions (21%)
Failed
Cannot toggle faucet successfully
low-level (15%)
Never opened the fridge pick up the egg inside
Missing
actions (18%)
Perception error (6%)
Identify pan as pot
Fig 4: RoboFail Failure Taxonomy
5
Evaluation
We systematically evaluate the ability of REFLECT to localize, explain and correct robot failures. In AI2THOR simulation, the agent interacts with the environment through action primitives, such as pick up, toggle on, move left. We assume the framework has access to ground truth object detection and state detection in simulation. We also evaluate the ability of the framework to summarize real-world robot sensory data. The real-world failure data is collected by human telcoperating a URSe robot arm to mimic robot policies according to a provided high-level plan. We use MDETR [37] for object detection, CLIP [33] for object state detection, and AudioCLIP [35] for sound detection. We use GPT-4 [38] as the LLM. The below metrics evaluated in experiments:
Exp (explanation); percentage of predicted failure explanations that are correct and informative as determined by human evaluators*.
Loc (localization): percentage of predicted failure time that align with actual failure time. A predicted time is considered aligned it falls within the annotated failure time ranges in the dataset.
Co-plan (correttin planning success rate): percentage tasks that succeed after executing the cor- rection plan. Task success is determined by comparing the final state and the specified goal condition.
To demonstrate advantages of our framework, we compare with the following baselines/ablations::
BLIP2 caption: caption key frames with BLIP2, a state-of tfitee-oftheee the-art image caption model [39].
w/o sound: our approach without the audio modality.
w/o progressive: similar to open-ended Q&A in Socratic Models [16], directly query LLM for failure explanation given the robot summary without progressive failure explanation.
Subgoal only: using only subgoal-base summary for failure explanation.
LLM summary: convert all censory-input summary to text and prompt LLM to summarize the text for failure explanation.
w/o explanation: query LLM for a correction plan without failure explanation.
By evaluating REFLECT on the RoboFail dataset and comparing with the above baselines/ablations, have the below findings:
Execution failure
Planning failure
Method Exp Loc Co-plan
Exp
Loc Co-plan
wo progressive 46.5 62.8
60.5
61.4 70.2
64.9
REFLECT is able to generate informative failure explanations that assist correction planning. Tab.
Subgoal only 76.7 74,4
51.2
71.9 73.7
75.4
LLM summary 55.8 67.4
65.1
57.9 54.4
66.7
w/o explanation
41.9
56.1
1 and 2 summarize the evaluation result, where RE- FLECT achieves the highest performance in explain- ing, localizing, and correcting failures in both simula- tion and the real world. The performance slightly decreases in real world due to perception emror. We find that localization is slightly harder for planning failures as the failure is usually not associated with single time step. In simulation, our framework achieves around 80% correction planning success rates for both failure types.
REFL 88.4 96.0
79.1
184.2 80.7
80.7
Table Result in Simulation Environments
Audio data is useful for failure explanation. As shown Tab. 2, explanation and localization accuracy for [w/o sound] both decrease around 20% on execution failures as compared REFLECT. This because some unexpected events (e.g. object dropping on floor) or object states hard identify using visual detectors (e.g. stove burner occluded by pot on top) easier be detected through audio.
Details be found in appendix.

---

Input: Execution
Output: Explanation
Correction
Correction Execution
Task: Make coffee
pickuup (oup)
put, (oup, countertop)
At 00:45, the robot attempted to put
3. pickuue (mug)
the mug into the coffee machine while there was alreadl a cup inside the machine, causing fatlure duae to
put In imug. coffee machine)
logple sn muchin))
logple, (oeflie
the occupled space.
Toast bread
The robot plan failed beciuse it tumned on the toaster before putting the bread slice inside resulling broad slice being placed the toaster instead of inside
toggli, (baater)) plok_ap (bread silce, toggle_sm (fowotie)
Task: Make salad
10
put_oon (pstato, countertop) up
The robot plan falled include the stop of sllicing the potato befora pulling in the bowd, resulling in an incomptete salad with an unsliced potato.
slice (potato)
put (trifis, coursettao)
slckk up (pctass slice)
10. put_in (potatoo alica, bows)
Fig 5: Qualitative results in simulation. Given failed robot task exccution, REFLECT is able to generate informative failure explanations for both execution and planning failures. Conditioned on the explanation, a language planner can generate a high-lovel plan for the robot to correct the failure and complete the task.
Task-relevant object spatial and state information crucial. As shown in Tab. 2, [BLIP2 caption] achieves the worst performance in all scenarios because the captions generated by BLIP2 lack necessary information for
failure explanation. In contrast, zero-shot caption method is designed to capture enviromment information such as object states and spatial relations, which are task-relevant and crucial for failure explanation. The figure on the right shows that REFLECT is able to summarize object states such as "fridge (with door open)" and spatial relations such as "apple inside white bowl", which are not present the BLIP2 caption.
BLIP2 caption: robot is hokiing a bowl a kitchen.
REFLECT: fridge (with door open), apple, while bowl,
bowl. applo is Inside while bowl. dark blue bowl is inside robot gripper.
Progressive failure explanation is important. Our pro- gressive failure explanation algorithm leverages the hi- erarchical summary to first identify the failure with the subgoal- based summary. and then query LLM for failure explanation accordingly. Comparing to [w/o progressive] in Tab. and 2, the progressive algorithm helps with more accurate localization and informative explanation. To better understand the difference, a qualitative example is shown in Fig. 6: although [w/ progressive] mentions that the task failed because the robot did not have egg in its gripper to put in the pan, it does not reason why the egg was not present. In contrast, REFLECT identifies failure in the "pick up egg" action and then queries the event-based summary to infer the exact failure cause- "fridge was closed"'
Exacution failure
Planning failure
Method Exp
Loc
Exp
Loc
BIIP caption 6.25
25.0
35.7
S7.1
who sound 50.0
68.8
78.6
78.6
w/o progressive 43.8
81.3
71.4
78.6
Subgnal only 56.3
62.5
71.4
78.6
LLM summary 37.5
75.0
64.3
71.4
REFLECT 68.8
93.8
78.6
78.6
Table 2: Result in Real-world Environments
Hierarchical structure is important. The per- formance decrease of [Subgoal only] shows the importance of event-based summary as it stores intermediate environment observations that are use- ful for failure explanation. Consider the scenario when the robot accidentally dropped the pot it was holding when moving to stove burner. [Subgoal only] only infers that the object was not in the robot gripper at the end of the subgoal execution. In contrast, the event-based summary stores audi- tory observation of "something drops on ground" and visual observation of "nothing is inside robot gripper", which helps identify that the pot was dropped and the exact time step the failure occurred.
Task: cook ego
fridge closed
aummary:
Summanc:
to pick up
prioper
01:52 The robot attempted crack the ugo, but the emply.
stil
01:53 The robot arid to put the crecked egg in the pan, but the
wia programsive: The robet Indedd to cook an ega
beca de old not have the egg in its gripper when stteroting
gripper remeired amppl.
pripper
Euraratoo:
was during these actiona.
The robot taled to cook an egg
up og2 gripper. resuting
because was unable
egg 00-14, notert attemppted pick
cxrodig
emety prippr throughout
The should hoow tidge before trying to plak the ogp.
and provert ig egg
Fig 6: [w/ progressive] vs. [LLMM summary] vs. Ours
7

---

Real World Task Execution
Scene Graph
Caption
Explanation
Task: secure objects
wiong placements
02:15
opeil
62:15, Action: Pur kentle in fridge Visuall observation: frigge (utth door opent, knite pest crutos (closed)) knrle moils fridge. natling inside robsot gripee ppae..
The robot plan fa besause it placed the kndff the fridge instreed moct outtalle containet
was ploced
nather fridge.
Taski saudé carrol
knitle dropped
01:44
01:44, Action: ssucepon, carrot obzervatio uoepen tuble. nothing insile rolont powe
D1:1118 the sobot sauod the failure caused
dropping the knafi 00:55 wril
atteeppting to siice the carort, leaving with empty pripper
Taske coffee
wrong cup
02:04
At 02:27 the robor failed serve coffee because it picked up the pirkk cup instead of the brue cup that
Taching
observation coffee machinc, blue table. blue inside coffee
62:54, up
the coffiee
mochine
the robot placting emppy cup
machine park insida robot gripper.
Fig 7: Qualitative results in real world. REFLECT able summarize and generate informative failure explanations for real-world rohot executions. The above figure shows three failed task exccutions on the left, the corresponding scene graph and caption for one key frame in the middle, and the M-geeerrrted failure explanation on the right.
In addition, the hierarchical structure is better way to condense the sensory-input summary for failure explanation. We implement alternative [LLM summary], which prompts LLM summarize the sensory-input summary. The performance decreases significantly in both simulation and real world as the LLM-generated summary loses information that is relevant for failure explanation. As shown in Fig. 6, the summary and explanation of [LLMM summary] only mention that the robot tried to pick up the egg and the gripper remained empty, but did not mention that the fridge was closed, which the actual failure cause.
Failure explanation helps correction. From Tab. we observe that the correction planning success rate significantly decreases for [w/o expla- nation]. This is because the failure explanation can guide LLM to generate a correction plan based on the failure cause. As shown in Fig. 8, given the failure reason that the mug cannot be put inside the coffee machine due to presence of a cup, RE- FLECT generates a plan to move the cup away and then proceed with the task. Whereas [w/o ex- planation] simply repeats the original plan without taking any actions address the cause of failure.
Explanation: At 00:48, the robot sttempted to put the mag into the coffee
w/o explanation:
machine while there was already cup inside the machins, causing fallure due to the occupted space.
1. pick_up (mug)
2. put (mug, sink basin)
3. toggle_on _n (faucet)
4. toggle_ off (faucet)
REFLECT:
5. pick_up (mug)
plck_up (cup)
6. pour _iq__liqi_____ from (mug, sink)
pour_liquid
put on (cup, countertop)
7. put_in (mug, coffee machine)
3. pick_up (mug)
8. toggle_on (coffee machine)
in (mug, coffee machine)
toggle on (caffee machine)
9. toggle_off (caffee machine)
6. toggle_oft off (caffee machine)
10. pick_up (mug)
pick_up (mug)
11. put_on on (mug. countertop)
put, on (mug, countertop)
Fig 8: Failure explanation helps correction planning.
I Limitations. There are few limitations in the method used to convert sensory data into textual summary. Even though the heuristics used to generate scene graphs is sufficient for scenarios studied in the paper, it may fall short in more complex environments. Either training a large spatial reasoning model or fine-tuning an existing model on robotics data could be a promising solution [40, 41, 42]. In addition, the object state detection method assumes a given list of candidate object states, which can be potentially relaxed by a method (e.g. prompting a LLM) that output possible states given the object category.
The framework also assumes the rest of the environment will remain static throughout the robot task execution. Finally, given the information (object detection, object states, spatial relations) that the robot summary contains, it is less effective for handling low-level control failures. Future work may consider developing better perception methods that capture more low-level state information.
Conclusion
We propose framework, REFLECT, which converts multisensory observations into a hierarchical sum- mary of robot past experiences and queries LLM progressively for failure explanation. The generated explanation can then guide a language planner to correct the failure and complete the task. To evaluate the framework, we create dataset of robot failed executions in both simulation and real world and show that REFLECT achieves better performance as compared several basclines and ablations. We encourage future work to extend upon the framework and explore more use cases of the robot summary.
8

---

Acknowledgments
We would like to thank Cheng Chi and Zhenjia Xu for their help in setting up real world experiments, and Huy Ha, Mandi Zhao, Samir Gadre, Mengda Xu, Dominik Bauer for valuable discussions and feedback. This work was supported in part by NSF Award #2143601, #2037101, and #2132519. We would like to thank Google for the URS robot hardware. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.
References
[1] T. Brown, B. Mann, N. Ryder, M. Subbiah. J. D. Kaplan, P. Dhariwal, A. Neclakantan, P. Shyam, Sastry, A. Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
[2] S. Bubeck, V. Chandrasekaran, R. Eldan, Gehrke, E. Horvitz, E. Kamar, P. Lee, Lee,, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint xr11v 2023.
[3] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[4] J. Wei, X. Wang. D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11900 2022.
[5] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205. /916, 2022.
[6] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
[7] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey,
S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu,
C. Parada, P. Pastor, Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan,
A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and Zeng. Do i can and not as say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.01691, 2022.
[8] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, Tompson, I. Mordatch, Y. Chebotar, Sermanet, N. Brown, T. Jackson, L. Luu, Levine, K. Hausman, and B. Ichter. Inner monologue: Embodied reasoning through planning with language models. In arXiv preprint mxiv:2207.05600. 2022.
[9] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox. J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:229.113002 2022.
[10] E. Kolve, R. Mottaghi, W. Han. E. VanderBilt, Weihs, A. Herrasti. M. Deitke, Ehsani, D. Gordon, Y. Zhu, al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint 1rXix:1712.054477 2017.
[11] T. Wang, R. Zhang, Z. Lu, F. Zheng, Cheng, and P. Luo. End-to-end dense video captioning with parallel decoding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6847-6857, 2021.
[12] C. Deng, S. Chen, D. Chen, Y. He, and Q. Wu. Sketch, ground, and refine: Top-down dense video captioning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 234-243, June 2021.

---

[13] Q. Zhang, Y. Song, and Q. Jin. Unitying event detection and captioning as sequence generation via pre-training. In Computer Vision-FCCCV 2022: 17th Europeau Conference, Tel Avi; Israel, October 23-27, 2022, Proceedings, Part XXXVI, pages 363-379. Springer, 2022.
[14] W. Zhu, B. Pang, A. V. Thapliyal, W. Y. Wang, and R. Soricut. End-to-end dense video captioning as sequence generation. In Proceedings of the 29th Internatinnal Conference on Computational Linguistics, pages 5651-5665, 2022.
[15] Yang, A. Nagrani, H. Seo, A. Miech, J. Pont-Tuset, I. Laptev, J. Sivic, and C. Schmid. Vid2seq: Large-scale pretraining of visual language model for dense video captioning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattemn Recognition, pages 1071 10714-1077) 10726, 2023.
[16] A. Zeng, M. Attarian, B. Ichter, K. Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke, and P. Florence. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv, 2022.
[17] Z. Wang, M. Li, R. Xu, L. Zhou, J. Lei, X. Lin, S. Wang, Z. Yang, C. Zhu, D. Hoiem, et al. Language models with image descriptors are strong few-shot lidoo-language learners. arXiv preprint arXiv:2205. 107477 2022.
[18] R.-G. Pasca, A. Gavryushin, Y.-.. Kuo, O. Hilliges, and X. Wang. Summarize the past to predict the future: Natural language descriptions of context boost multimodal object interaction. arXiv preprint arXiv:2301.0922 2023.
[19] C. DeChant and D. Bauer. Summarizing virtual robot's past actions in natural language. arXiv preprint 27X66:2003.0667,, 2022.
[20] P. Khanna, E. Yadollahi, M. Björkman, I. Leite, and C. Smith. User study exploring the role of explanation of failures by robots in human robot collaboration tasks. arXiv preprint 2600:23333.10100 2023.
[21] S. Ye. G. Neville, M. Schrum, M. Gombolay, S. Chernova, and A. Howard. Human trust after robot mistakes: Study of the effects of different forms of robot communication. 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pages 1-7, 2019, doi: LL09/OO-MAN/AAA
[22] Das, Banerjee, and S. Chernova. Explainable ai for robot failures: Generating explanations that improve user assistance in fault recovery. In Proceedings of the 2021 ACM/IEEE International Conference Human-Robbe Interaction, pages 351-360 2021.
[23] D. Das and S. Chernova. Semantic-based explainable ai: Leveraging semantic scene graphs and pairwise ranking to explain robot failures. In 202/ IEEE/RS International Conference on Intelligent Robots and Systems (IROS), pages 0034-3041 IEEE, 2021.
[24] M. Diehl and K. Ramirez-Aman Why did fail? a causal- method to find explanations for robot failures. IEEE Robotics and Automation Lettes, 7(4):8925-89) 8932, 2022.
[25] C. H. Song, J. Wu, C. Washington, B. M. Sadler, W.-.. Chao, and Y. Su. LIm-planner: Few-shot grounded planning for embodied agents with large language models. arXiv preprint arX0v:2212.040888 2022.
[26] Inceoglu, F. F. Aksoy, C. Ak, and S. Sariel. Fino-net: deep multimodal sensor fusion framework for manipulation failure detection. In 2021 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages
IEEE, 2021.
[27] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. arXiv preprint 220907200 2022.
10

---

[28] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118-9147. PMLR, 2022.
[29] S. S. Raman, V. Cohen, E. Rosen, I. Idrees, D. Paulius, and S. Tellex. Planning with large language models via corrective re- re-prompting. arXiv preprint arXiv:2211.09933 2022.
[30] N. Shinn, B. Labash, and A. Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303. 11366, 2023.
[31] M. Skreta, N. Yoshikawa, S. Arellano- Rubach, Z. Ji, L. B. Kristensen, K. Darvish, A. Sspuru-Guzii F. Shkurti, and Garg. Emors are useful prompts: Instruction guided task programming with verificr-assisted iterative prompting. arXiv preprint arXiv:2303. 2023.
[32] T. Silver, S. Dan, K. Srinivas, J. B. Tenenbaum, L. P. Kaelbling, and M. Katz. Generalized planning in pddl domains with pretrained large language models. arXiv preprint arXiv:2305. 1014, 2023.
[33] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. 8763. PMLR, 2021.
[34] X. Li, D. Guo, H. Liu, and F. Sun. Embodied semantic scene graph generation. A. Faust, D. Hsu, and G. Neumann, editors, Proceedings of the Sth Conference Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 1585-1594, PMLR, 08-11 Nov 2022.
[35] Guzhov, F. Raue, Hees, and A. Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 976 980. IEEE, 2022.
[36] H.-H. Wu, P. Seetharaman, K. Kumar, and P. Bello. Wav2elip: Learning robust audio representations from clip. ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4563-4567. IEEE, 2022.
[37] A. Kamath, M. Singh, Y. LeCun, G. Synnaeve, I. Misra, and N. Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
[38] OpenAL. Gpt-4 technical report, 2023.
[39] J. Li, D. Li, S. Savarese, and Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.1255 2023.
[40] Li, D. Guo, H. Liu, and F. Sun. Embodied semantic scene graph generation. In Conference on Robot Learning, pages 1585-1594, PMLR, 2022.
[41] F. Liu, G. Emerson, and N. Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 1:635-651, 2023.
[42] A. Kurenkov, M. Lingelbach, T. Agarwal, E. Jin, C. Li, R. Zhang, L. Fei-Fei, J. Wu, S. Savarese, and R. Martn-Martun Modeling dynamic environments with scene graph memory. In International Conference on Machine Learning, pages 17976-17993, PMLR, 2023.
11

---

Appendix
A
Method Details
A.1 Spatial Relation Heuristics
We implement total object spatial relation heuristics given the aggregated semantic point cloud and 3D object bounding boxes. We consider two in-contact relations when the minimum distance between the object point clouds is smaller than 5em:
1. Inside. Object A is considered inside object B if over 50% percent of object A's point cloud inside the convex hull of object B.
2. On top Object A is considered on top of object B if satisfies the following two conditions: a) over 70% of object A's point cloud lies within XY-plane projected from object B's 3D bounding box. b) over 70% of object A's points are above the upper Z bound of object B's bounding box.
the object point cloud distance larger than 5cm smaller than 40cm, we subtract the center points of object and B's 3D bounding box, transform back to camera coordinates (Y-up) and normalize get 3 dimensional unit vector. The following relations are considered:
1. Above & Below. the y-component of the vector is over 0.9, then object A is above object B. If the y- y-component smaller than -0.9, then object A is below object B.
2. On the left On the right. If the x-component the vector over 0.8, then object A is on the right of object B. the x-component smaller than -0.8, then object on the left of object B.
3. Occluding. If 90% of object A's points have smaller depth than object B's minimum depth and object A's projected 2D bounding box (projected in current image space from its aggregated 3D point cloud) overlaps more than 50% with that of object B's projected bounding box, then object B is considered occluding object A.
4. Near. If none of the above relations satisfy but the object distance is smaller than 10cm, then object
A is near object B.
A.2 Scene Graph Aggregation Heuristics
We aggregate the 3D point cloud over time frames with similar approach as Li et al. [34]. Consider the point cloud observed in the current time step ps and the accumulated point cloud from all previous time steps P2-1, we can obtain the accumulated point cloud P: up to time step / with operations: ADD, UPDATE, REPLACE, and DELETE.
1. ADD. If an object is observed in ps but not in P.-1, consider it a newly appeared object and add it as a new node. If the node is task-relevant, then its relations with all existing objects will be computed and added as edges. If the node not task-relevant, then its relations with existing task-relevant objects will be computed and added as edges.
2. UPDATE, If an object in both pa and Pt-1. and the object point cloud is aligned, we update the object point cloud by concatenating the newly observed points and updating its existing edges with other objects by recomputing the spatial relations.
3. REPLACE. If an object is in both Pt and Pt-1. but the misalignment of the object point clouds is larger than threshold d, indicating the object has moved. We remove the old node in P and add the object
node.
4. DELETE. If an object is in PA-1 but not in Pt, and the robot is interacting with the object, then remove the object from the accumulated point cloud since its location becomes unknown.
12

---

A.3
Example Prompts
A.3.1
Subgoal Verification
system prompt:
You are a success verifier that outputs 'Yes' or 'No' to indicate whether the robot goal is satisfied given the robot observations.
user prompt:
The robor goal is ISUBGOAL). Here are the robot observations after exccution: [OBSERVATION] Q: Is the goal satisfied?
A: Yes
The [SUBGOAL] and [OBSERVATION] entry will be filled in when prompting for each subgoal. Here's complete example of a failed execution during the task "boil water'. The robot accidentally dropped the pot when navigating to the stove burner and then attempted to put the pot on the stove burner:
The robot goal is to pick up pot. Here are the robot observations after exccution:
Visual observation: pot (empty and clean). pot inside robot gripper. Q: Is the goal satisficd?
The robot goal is put pot in sink. Here the robot observations after exccution: Visual observation: faucet (turned off), pot (empty and clean), sink. pot (empty and clean) is inside sink. pot (empty and clean) on the right soap bottle. nothing inside robot gripper. Q: Is the goal satisfied? Yes
The robot goal is toggle on faucet. Here are the robot observations after exccution: Visual observation: pot (filled with water and clcan), faucet (turned on), sink. pot (filled with water and clean) is inside sink. pot (filled with water clean) on right of soap bottle. nothing inside robot gripper. Auditory observation: water runs in sink.
Q: the goal satisfied? A: Yes
The robot goal is to toggle off faucet. Here are the robot observations after execution: Visual observation: pot (filled with water and clean), faucet (turned off), sink. pot (filled with water and clean) is inside sink. pot (filled with water and clean) is on the right of soap bottle. nothing is inside robot gripper. Q: the goal satisfied? A: Yes
The robot goal is to pick up pot. Here are the robot observations after execution: Visual observation: pot (filled with water and clean), faucet (turned off), sink. pot is inside robot gripper.
Q: Is the goal satisfied?
The robot goal is put pot on fourth stove burner. Here are the observations after execution:
Visual observation: second stove burner (turned off), first stove burner (turned off), third stove burnrr (tumed off). fourth stove burner (tureed off). nothing inside robot gripper.
Q: Is the goal satisfied? No
Here a subgoal not satisfied, the program enters the execution analysis mode: history observations stored in the event- event-based summary are retrieved query LLM for failure explanation.
13

---

A.3.2 Failure explanation: execution analysis
system prompt:
You are expected to provide explanation for a robot failure. You are given the robot actions and observations so far. Bricfly explain the failure in 1-2 sentence. Mention relevant time steps if possible.
user prompt:
The robot task is to boil water. At 00:44, a failure was identified.
[Robot actions and observations before 00:44]
00:01. Action: Move to pot. Visual observation: nothing is inside robot gripper.
00:11. Action: Move to pot. Visual ohservation: faucet (turned off). nothing is inside robot gripper.
00:15. Action; Move to pot. Visual observation: pot (empty and clean). pot (empty and clean) is on left of potato.
pot (empty and clean) is on top of third countertop. nothing is inside rohot gripper.
00:18. Action: Pick up pot. Visual observation: pot (empty and clean). pot is inside robot gripper.
00:21. Action: Move to sink. Visual observation: pot (empty and clean), faucet (turned off), sink. pot is inside robot
gripper.
00:22, Action: Move to sink. Visual observation: pot (empty and clean), faucct (turned off), sink, pot is inside robot
gripper.
00:25. Action: Put pot in sink. Visual observation: pot (empty and clean). faucet (turned off). sink. pot (empty and
clean) inside sink. pot (empty and clean) is on the right of soap bottle. nothing is inside robot gripper.
00:28. Action: Toggle on faucet. Visual observation: pot (filled with water and clean), faucet (turned on), sink, pot
(filled with water and clean) is inside sink. pot (filled with water and clean) is on the right of soap bottle. nothing is inside robot gripper. Auditory observation: water runs in sink.
00:31. Action: Toggle off faucet. Visual observation: pot (filled with water and clean), faucet (turned off). sink, pot (filled with water and clean) is inside sink. pot (filled with water and clean) on the right of soap bottle, nothing is inside robot gripper.
00:34. Action: Pick up pot. Visual observation: pot (filled with water and clean), faucet (tumed off). sink. pot is
inside robot gripper.
00:36. Action: Move to fourth stove burner. Visual observation: pot (empty and clean), faucet (turned off), sink, pot
(empty and clean) is on the right of potato. pot (empty and clean) is on top of third countertop. nothing is inside robot gripper. Auditory observation: something drops.
00:38. Action; Move to fourth stove burner. Visual observation; pot (empty and clean), faucet (turned off), sink.
nothing is inside robot gripper.
00:42. Action: Move to fourth stove burner. Visual observation: second stove burner (tured off), fourth stove burner
(turned off), third stove burner (turned off), first stove burner (tumed off). nothing is inside robot gripper.
00:43. Action: Move to fourth stove burner, Visual observation: second stove burner (tumed off), fourth stove burner
(turned off), third stove burner (turned off), first stove burner (tumed off). nothing is inside robot gripper.
[Observation at the end of 00:44]
Action: Put pot on fourth stove burner. Visual observation: second stove burner (turned off), fourth stove burner (turned off), third stove humer (turned off), first stove burner (tumed off). nothing is inside robot gripper.
Q: Infer from [Robot actions and observations before 00:44] or [Observation at the end of 00:44], briefly explain what happened at 00:44 and what caused the failure.
A: At 00:44, the robot attempted to put the pot on the fourth stove burner, but the pot was not in its gripper. The failure was caused by the robot dropping the pot filled with water at 00:36 while moving to the fourth stove burner.
The failure steps 00:36 and 00:44 can be extracted from the answer by prompting LLM to extract time steps from the output failure explanation.
A.3.3 Failure explanation: planning analysis
In case all subgoals are satisfied, then there's likely mistakes in the robot original plan. The program will enter planning analysis mode. Take the failure scenario when the robot plan is wrong during the task 'boil water' as the robot placed the pot on one stove burner but toggled on another:
system prompt:
14

---

You are expected to provide explanation robot failure. You are given the current robot state, goal condition, and the robot plan. Briefly explain what was wrong with the robot plan in 1-2 sentence.
user prompt:
The robot task is to boil water. The task considered successful if a pot filled with water, the pot is on top of a
stove burner that is tumed on.
Here's the robot observation at the end the task execution:
faucet (turned off), second stove burner (turned on), sink, pot (filled with water and clean), fourth stove burner (turned
off). third stove burner (turned off), first stove burner (turned off)). pot (filled with water and clean) is on top of fourth stove burner (turned off). nothing insidl robot gripper.
The robot plan is:
00:18. Goal: Pick up pot.
00:25. Goal: Put pot in sink.
00:28. Goal: Toggle faucet.
00:31. Goal: Toggle off faucct.
00:34. Goal: Pick up pot.
00:46. Goal: Put pot on stove burner.
00:49. Goal: Toggle on stove burner.
Q: Known that actions in the robot plan were executed successfully, what's wrong with the robot plan that caused
the robot to fail?
A: The robot placed the pot on the fourth stove burnrr but turned on the second stove burner, causing a mismatch between the pot's location and the active burner.
The failure time step be obtained by follow-up query to the LLM with the prompt below:
Which time step most relevant the above failure?
A: 00:49
A.3.4 Correction
Still take the failure scenario when the robot plan is wrong so that the robot placed the pot on one stove burner but toggled on another. A complete prompt for generating the failure correction plan is as follows:
system prompt:
Provide a plan with the available actions for the robot to recover from the failure and finish the task. Available actions: pick up. put in some container, put on some receptacle, open (e.g. fridge), close, toggle (e.g- faucet), toggle off. slice object, crack object (e.g. egg). pour (liquid) from A to B. The robot can only hold one object in its gripper. in other words, if there's object in the robot gripper, it longer pick up another object.
The plan should 1) not contain any if statements 2) contain only the available actions 3) resemble the format of the initial plan.
user prompt:
Task: boil water
Initial plan:
1. pick_up (pot)
2. put_in (pot, sink)
3. toggle_on (faucet)
4. toggle_off (faucet)
5. pick_up (pot)
6. put on (pot, stove burner)
7. toggle on (stove burner)
Failure reason: The robot placed the pot on the fourth stove humer but turned on the second stove burner, causing a
mismatch between the pot's location and the active burner.
Current state; sink, pot (filled with water and clean), fourth stove burner (turned off), third stove burner (turned off),
faucet (turned off), first stove burner (turned off), sccond stove burner (turned on). pot (filled with water and clcan) is on top of fourth stove burner (turned off), nothing inside robot gripper.
Success state: pot is filled with water, the pot is on top of a stove burner that is tumned on.
Conrection plan: toggle_off (stovebumer-2), toggle_on (stoveburner-4)
15

---

B Evaluation Details
B.1 Dataset Details
Descriptions for the 10 simulation tasks and 11 real-world tasks in the RoboFail dataset shown below.
Task
Task Description Goal State
boil water
A pot is filled with water, the pot is on top of stove burner that is turned on
toast bread
bread slice is inside toaster that is turned on
fry egg
A cracked egg is in pan, the pan is top stove burner that is turned
heat potato
A potato is on a plate and inside a microwave that is tumned on
serve coffee
clean mug is filled with coffee and on top of the countertop
store egg
A bowl with an egg is stored inside the fridge
make salad
bowl of sliced lettuce, tomato and potato is stored inside the fridge
water plant
The house plant filled with water
switch devices
Laptop is closed on the TV stand and the television is turned on
serve warm water
A mug of water is heated in the microwave and served on the dining table
Table 3: Simulation Tasks
Task
Task Description / Goal State
boil water
pot is filled with water, the pot is on top of stove burner that is turned on
sauté carrot
A sliced carrot is inside pan, the pan is top of stove burner that is tumned on
heat potato
A potato is heated in the microwave and then put on the countertop
serve coffee
A mug filled with coffee and on top of the countertop
store egg
A bowl with egg is stored inside the fridge
secure objects
Knife is stored in a drawer and pear is stored in the fridge
apple in bowl
Apple inside bowl
pear in drawer
Pear is inside a closed drawer
cut carrot
Carrot is sliced
fruits in bowl
All visible fruits are inside a bowl
heat pot
A pot is on top of a stove burner that is turned on
Table 4: Real-world Tasks
B.2 Comparison with GPT-3.5
We use GPT-4 for evaluation in experiments. Here we provide a comparison of performance with the more accessible GPT-3.5. We have observed that GPT-4 exceeds GPT-3.5 in terms of both failure reasoning and planning abilities. The failure localization accuracy decreases as GPT-3.5 is less capable than GPT-4 to process irrelevant information in the summary and thus often wrongly local- ize the failure. The correction planning success rate with GPT-3.5 decreases more significantly due to the combined factors of less accurate failure explanations and less planning ability even given correct failure explanations. We also show one qualitative example of the failure explanations generated by GPT-4 and GPT-3.5, which shows that GPT-4 is better at reasoning the root cause of the failure than GPT-3.5:
Execution failure
Planning failure
Method
Loc
Co-plan
Loc
Co-plan
GPT-3.5 47.8
30.4
51.9
40.7
GPT44 68.8
93.8
78.6
78.6
Table 5: Comparison with GPT-3.5
GPT-3.5: The robot plan failed because the robot did not successfully put the bread slice in the toaster, even though it successfully turned the toaster.
GPT-4: The robot plan failed because tumed on the toaster before putting the hread slice inside it, resulting in the bread slice being placed on top of toaster instead inside
16

---

B.3
Object states
We list the object state candidates considered for all objects that can change states in our experiments. In general, the states are assigned based on object properties.
Object Type
Actionable Properties
Simmlation States
Real-world States
Fox
Fillable, Diryable
tilled, empry, dirty, clean
filled with water, empty
Toggleahle
loggled on, toggled
loggled toggled
StoveBumer
Togeleable
loggled on, toggled off
togaled togeled
Bread
Sliceable
sliced, unsliced
NA
Toaster
Toggleable
Ioggled toggled off
NA
Fridge
Openable
open, closed
open, closed
Dirtyable
dirly,
Egg
Sliceable, Breakable
sliced, unsliced, cracked. uncracked
NA
Potato
Sliccable, Cookable
sliced, unsliced, cooked, uncooked
NA
Plate
Breakable (Some)) Dirtyable
broken, dirty, clean
NIA
Microwave
logacablc, Openable
loggled on, toggled off, open. closed
toggled on. toggled off, open, closed
Mug
Breokable, Dirtyable
enpty. broken, ditty,
emply
UnlfeeMachine
Inggeable
Ioggled rn, toggled off
loggled toggled off
HousePann
Fillable
walered, not watered
NA
Tomalo
Sliccable
slicel, unsiced
NA
Lettcee
Sliceable
sliced, unsliced
NA
Breakable (Some)) Dirtyable
emityy, broken, dirty,
Lapoop
Openahle, Toggleable Hreakable
open, closed, roggled on, logeled off, broken
NA
Television
Toggleable, Breukable
loggled on, loggled oll, broken
NA
Carrot
Sliccahle
NIA
sliced, unsliced
Drawer
Openable
NIA
open, closed
Table 6: Object States
B.4 Human Evaluation
Similar the approach for human evaluation in Ahn et al. [7], ask groups of users, 3 in each group compare the ground truth failure explanation labelled in the dataset and REFLECT-geneee explanation for each failure scenario. The failure scenarios are randomly shuffled in the questionnaires sent to the users. The users are instructed to score 0 if the predicted explanation is incorrect, if the predicted explanation is correct, and if they are unsure. The final score reflected in the tables are the majority vote without counting "unsure", there's a tie in the answers or more than one "unsure" given, we will ask the users to re-score the specific case.
failure
17


Reviews:
[{"review_id": "ugHTULZQ6kT", "reviewer": "Reviewer_LGvx", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "* The authors showcase REFLECT, a system for logging robot tasks, events, and sensory streams\n* In order to log sensory streams, the authors use an ensemble of models to generate a scene graph. Changes in the scene graph are then logged as events (along with the end of robot actions). The logs are also annotated with subtask goals\n* The scene graphs, the event annotations, and the subtask goals are then used to engineer prompts for an LLM to explain robot failures\n* The authors show that with sufficient prompt engineering from the annotated data, the LLM is also able to construct a plan, which when grounded to skills, can often recover the error(s)\n* Finally, and crucially for the community, the authors release a RoboFail dataset for the community to work on failure explanations and recovery. The dataset also includes an initial taxonomy of the failures.", "summary_of_recommendation": "I am strongly in favour of including this work in the technical program.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "None", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths**\n\n* The pipeline(s) for logging robot data and related events is impressive\n* The prompt generation scheme used by the authors makes sense\n* The ablation studies and the results adequately test the papers' claims on the systems contributed by the authors\n* The authors leverage the LLM's ability to localize to the right level of explanation for the failure, which has been a big problem in the field of XAI. To the best of my knowledge, this is the first paper to do that directly from raw robot streams.\n* The authors leverage the LLM to generate a high level plan for repair, which is impressive emergent behaviour\n* The authors are contributing a failure dataset\n\n**Scope for Improvement**\n\nAmong the limitations, the authors might do well to call out that their scenarios are all in static environments. Some aspects of their solution, particularly the scene graph generation, might be harder in dynamic environments.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "eAdfyE4kiO", "reviewer": "Reviewer_BfQU", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method to summarize robot failure modes and use this information to improve LLM planning performance. It presents a novel method to create hierarchical summary of both planning and execution failures, and progressively generate failure explanation using LLM. It then uses the failure correction planner to generate a corrective plan to complete the task. The paper also contributes a robot failure dataset for future benchmarking. ", "summary_of_recommendation": "I recommend Weak Reject since there are still some unaddressed questions in the paper; I would be happy to hear the author's thoughts and discuss more. It would be great if more information is given on 1) the scope of contribution of the paper 2) more details on the experiment setup 2) some clarifications of the questions above. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1. What is the low-level policy used for the real-robot experiments? Are they learned policy skills, or hardcoded pre-defined controllers? I did not find this information in the paper. \n\n2. After passing in robot failures explanations to LLM, the LLM will come up with better planning and hence improve task performance; however, many failures (especially on real robot) will be due to low-level control failures. Can these failures be improved from robot explanation? It looks like there isn't really a module to improve low-level control policy.\n\n3. For the failure correction planner: Is it replanning from the start, or continuing where the failure happened? If the robot do something irreversible and different from the previous plan (e.g. dropping a pan), can the planner come up with something corrective (e.g. picking up the pan) from there?\n\n4. I have some confusion regarding Table 1: \"Co-plan (correction planning success rate): percentage of tasks that succeed after correction\" \n\n     a) What does this metric mean? Do you actually execute the planned actions in the environment and check the actual policy performance, or check the correctness of the planner?\n\n     b) The left table is \"Execution Failure\" - Why would the column \"planning success rate\" still apply to execution failure (what does this mean)? And why would generating failure explanation benefit the planning success of execution failure (since failure explanation is only used for the high-level planner in 3.3, I don't see how this affects the execution failure?)\n\n5. The paper uses GPT4 as the LLM. How well do the other more accessible language models like GPT3.5 compare in terms of 1) original planning ability 2) correction after failure summarization? This is not really critical to the method itself, but curious how the method respond to different LLM - since GPT4 is not that accessible as of now and is very expensive, making reproducibility of the method hard. (again, it is not this paper's responsibility to show that all language models work well, but would be nice to give some knowledge of its sensibility. )", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper has several strengths:\n\n1. The problem of giving explanation to robot failures is useful and important - it is a crucial aspect of the feedback loop to both robot policy learning as well as LLM decision making. \n\n2. One good insight of this paper is how more structured, task-oriented summary (captioning) of the scene is useful for robotics tasks, as opposed to using out-of-the-box image captioning models that are more general. This extracts information more informative to robot planning goals.\n\n3. The creation of the robot failure dataset is a good contribution to the community in studying robot failures, and the taxonomy of failure classification is a good contribution as well.\n\nWeaknesses: \n\n1. The evaluation results involves some confusion; it seems that there are two levels of metrics that the paper wants to evaluate: 1) failure explanation correctness 2) how failure explanation contribute to task success. The 2) here is confusing as there have not been made clear whether it is the success of the planner, or the success of task execution. (See Questions for Rebuttal P4.) I think the paper's presentation of results can be made clearer so that the actual contribution is clear. \n\n2. Related to Weakness P1 - I'm a bit uncertain about how far is the contribution of the paper, in the sense that there isn't too much evidence of how the failure explanation can contribute to better improvement of the policy performance. It is clear that this method presents a way to summarize robot failures, but it is unclear of how this could be used or its potential impact. I would be happy to hear the authors' thoughts on this. \n\n3. In terms of the method, the scene graph of summarizing object relationships is useful, but it contains many hardcoded heuristics and seems a bit hacky, which cast doubts on the generalization of the method. \n\n4. In terms of the experiments: it is not so clear about what specific setup of the robot policy is used here - it seems to be high-level LLM planning + some low-level controllers, but I don't really see how they interface with each other. It would be very helpful to explain what kind of policy this method applies to (and what the policy does not).", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "aJz6I3RWwvR", "reviewer": "Reviewer_GeFi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper uses LLMs to detect, recover and explain errors that occur while executing the given plan. The primary objective is to convert the multimodal sensory data into a text summary that focuses on information about the objects relevant to the plan. Moreover, this approach introduces two additional abstractions over the summary derived from sensory inputs, refining it based on events and subgoal attainment. The subgoal-level summary serves as a means to query LLMs for error detection, while the event-based summary comes into play during the generation of explanations if any errors are detected. The failure explanation thus generated is used to correct the plan. ", "summary_of_recommendation": "Overall, the paper is well written. The idea of getting a unified hierarchical summary is interesting but the techniques (in particular, heuristics) used to get the summary is brittle. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "1) Please provide information about how the state of an object is determined. In particular, how do we know what text input needs to be given to the CLIP? \n2) Please improve on the limitation section. The limitations sections mostly outline the future directions of this work. However, it does not describe much about the limitations of the work in consideration. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths**\n\n1) The paper is well-written and easy to read.\n2) The idea of converting multimodal sensory inputs into a unified summary and introducing a hierarchical structure in the summary is quite interesting. \n\n **Weakness**\n1) The method used for extracting the visual and audio summary requires prior information (or a DSL) about the possible options for the spatial relations and the states of a given object. This kind of enumeration can be hard for a complex task. \n2) The paper seems to assume information about the states that are possible for a given object. For example, the faucet can be turned on or off; the pot can be empty, filled and clean or filled and not clean.  However, it doesn't explain how this information about the possible states was obtained. The paper could possibly include a table in the appendix listing all the possible states for each object. \n3) The heuristics used to determine the spatial relations may be good enough for the experiments made in the paper. However, they are brittle in general for objects with complex shapes. Such heuristics may provide a zero-shot generalization to novel scenes in a restricted environment, but I am concerned that the same may not hold true in a complex environment. Utilizing learning methods or pre-trained models to determine spatial relations appears to be a potential solution to address this concern.\n\n**subtle issues**\n1) The task-relevant scene graph is helpful for gathering the necessary information, allowing LLMs to focus on the relevant details when generating explanations and correcting the plan. However, what happens if the action failure affects other objects unrelated to the objects used in the plan? For example, a pot falling from the table may disconnect the power connection of the microwave. The robot needs to reconnect the power cable and turn on the microwave to cook a meal. However, there is no way the LLM knows if the power cable is connected or not; thus, getting a summary using task-relevant scene graphs cannot generate a feasible plan in such scenarios. In my opinion, predetermining a fixed task-specific subgraph may not be an ideal solution in complex scenarios.  ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "Z6mUicqJdM", "reviewer": "Reviewer_6L3t", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper describes an LLM-based plan execution monitoring approach, focusing on its ability to summarize and explain failures, and—because the LLM is also the planner— to generate new plans based on the failure explanation. Per-frame observations are formed by converting a scene graph representing objects and their spatial relationships into text and combining it with a caption of the audio observation and the current action in the plan. The LLM is given the subgoal observations and asked to verify that each subgoal has been achieved. If the subgoal hasn’t been achieved, the sequence of event-based observations is fed to the model and an explanation for a failure occurring within the observation window requested. Otherwise, an explanation for an issue with the plan is requested. These explanations serve as context for a request for a new plan. Experiments in a simulated environment and with a toy kitchen environment show that the approach outperforms several reasonable ablations.", "summary_of_recommendation": "This approach is interesting and makes improvements in a vein that many are exploring. It does not make sufficient connections to the existing methods it is recreating, nor critically evaluate the ways in which it does or does not surpass them. These issues are surmountable without major revision, but revision is necessary.\n\nUpdate after rebuttal: the authors have adjusted the framing of the paper to better position the method and its strengths against possible alternatives. I think readers may still find the experiments confusing (i.e. including an evaluation of the method as a re-planning mechanism, where the value proposition is weaker), however there are likely many readers who will be interested in the overall scheme that the paper contributes, so I have raised my score to weak accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "3.2 and Fig 3: Does subgoal verification happen after every subgoal during execution? It’s somewhat ambiguous, because it seems that it could also happen more frequently, as it would in a typical planning system. Otherwise, the plan will continue executing long past the moment of failure. However, if the system is prompting after every subgoal, during successful execution, then the plan analysis stage will still prompt for an explanation of what’s wrong with the plan. Would this cause problems?\n\nAs with conventional planning systems, a key challenge is digesting the world state into a symbolic representation and grounding a plan into actions on particular parts of the world state. This is aided by having relatively clean environments with few duplicated objects and no other activity occurring around the robot; when there are more objects in the scene, there are more chances for perception to fail, potentially causing the robot to act on the incorrect object, or to believe there is no object to act on. Unlike a conventional planning system, it isn’t immediately clear if the approach would still work well in the presence of more objects and other activity _even with perfect perception_; at what point (if any) does the size of the textual description of the scene cause the model to lose track? If additional activities around the robot create additional extraneous event-based summaries, at what point does the volume of these summaries become a problem? \n\nThe text of this paper has had its character spacing reduced making it difficult to read. Please fix your template.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work is interesting as it further considers how LLMs can be used as planners, but in a way that takes more advantage of having a textual representation of the plan; the generated explanations would presumably be much appreciated by human bystanders or robot supervisors. There is reference to a human evaluation of the generated explanations that shows that they are indeed useful. The presentation is also thorough in its inclusion of quantitative and qualitative results for many reasonable ablations.\n\nThe primary weaknesses of this work come from missed connections with existing planning work.\n\nThe structure of the approach is parallel with existing planning systems while making no reference to this fact. Section 3.2 “Progressive Failure Explanation” is more commonly called “Execution Monitoring”. 3.3 “Failure Correction Planner” is called replanning. There is no other planner, it’s the same planner asked to make a new plan under new circumstances. It’s an interesting and quaint limitation of the LLM that it lacks the ability to plan unless it’s told explicitly how it messed up previously (as indicated by the w/o explanation ablation replanning success rate in Table 1.), a problem that most algorithms which people would call “planners” do not suffer from. For a representative historical example of these terms in the planning context, see 10.1109/70.370506. For more recent examples, simply search for \"robot architecture\" or \"deliberative robot architecture.\"\n\nOther downsides of completing basic reasoning tasks with tools that are not designed to complete reasoning tasks are not interrogated in this paper; LLMs are “a promising component for explainable and robust robotic systems” and the only “remaining challenge lies in how to generate a ‘textual summary’ of raw robot sensory data that can be used in prompts and a systematic way to query LLMs for accurate failure explanation”. Of particular interest to roboticists; how many queries to the model are made during a typical plan execution? At what computational cost and wall-clock compute time? \n\nShort of crafting a baseline which more effectively couples the LLM with an existing planner (e.g. by producing PDDL from the summaries and invoking a planner), there needs to be more substantive discussion of the trade offs compared to using a conventional approach.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach to analyse plan errors during execution and provide explanations for why failures happen and how to correct them. The approach is situated as an offline method that analysis execution of a vision based task planner operating on a scene graph representation of the scene and uses pre-defined skills/policies that can be composed for a goal-directed plans. The approach enters around using LLMs for the failure explanation and correction task and tackles the issue of summarising experience (using multi-modal representations), how to generate task-relevant explanations using progressive interaction with an LLM and  how to ameliorate errors by replanning actions. 

The esteemed reviewers have provided detailed feedback on the paper suggesting sharpening of the core contributions in relation to closely related approaches, better clarifications on the assumptions/limitations and more clarity on the evaluation metrics. The authors engaged with the reviewers and updated text in the manuscript. 

Overall, the discussions lead to an observation that the positioning of the paper is somewhat in between two paradigms: is this a paper that explores if and to what extent LLMs can be used for error recovery planning from offline data or a paper that advances error recovery and re-planning by improving speed, robustness etc.  We would request the authors to deliberate carefully on this question and align the writing and experiments accordingly. If the paper is to be viewed as the former then the experiments with more LLMs would be appreciated guiding the community forward on how to use pre-trained LLMs for this (and closely related tasks). If the paper is positioned as the latter, then engagement with classic and contemporary work in plan recovery would be appreciated.
Decision: Accept (Poster)