VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought
Authors: Gabriel Herbert Sarch
Lawrence Jang
Michael J. Tarr
William W. Cohen
Kenneth Marino
Katerina Fragkiadaki
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Multimodal Agents
Multimodal Large Language Models
Visual Demonstrations
Human-In-The-Loop
Instruction Following
Autonomous Web Agents
Ego4D
Primary area: human-AI_interaction
TL;DR: ICAL lets LLMs and VLMs create effective prompts from sub-optimal demos and human feedback, improving decision-making and reducing reliance on expert examples. It outperforms state-of-the-art models in TEACh, VisualWebArena, and Ego4D.

Abstract: Large-scale generative language and vision-language models (LLMs and VLMs) excel in few-shot in-context learning for decision making and instruction following. However, they require high-quality exemplar demonstrations to be included in their context window. In this work, we ask: Can LLMs and VLMs generate their own examples from generic, sub-optimal demonstrations? We propose In-Context Abstraction Learning (ICAL), a method that builds a memory of multimodal experience from sub-optimal demonstrations and human feedback. Given a task demonstration that may contain inefficiencies or mistakes, a VLM abstracts the trajectory into a generalized program by correcting inefficient actions and annotating cognitive abstractions: causal relationships, object state changes, temporal subgoals, and task-relevant visual elements. These abstractions are iteratively improved and adapted through human feedback while the agent attempts to execute the trajectory in a similar environment. The resulting examples, when used as exemplars in the prompt, significantly improve decision-making in retrieval-augmented LLM and VLM agents. Moreover, as the agent's library of examples grows, it becomes more efficient, relying less on human feedback and requiring fewer environment interactions per demonstration. Our ICAL agent surpasses the state-of-the-art in dialogue-based instruction following in TEACh, multimodal web agents in VisualWebArena, and action anticipation in Ego4D. In TEACh, we achieve a 12.6% improvement in goal-condition success. In VisualWebArena, our task success rate improves over the SOTA from 14.3% to 22.7% using GPT4V. In Ego4D action forecasting, we improve over few-shot GPT-4V and remain competitive with supervised models. We show finetuning our retrieval-augmented in-context agent yields additional improvements. Our approach significantly reduces reliance on manual prompt engineering and consistently outperforms in-context learning from action plans that lack such abstractions.

VLM Agetts Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought
Gabriel Sarch'
Lawrence Jang'
Michael J. Tarr'
William W. Cohen
Kenneth Marino
Katerina Fragkiadaki
Carnegie Mellon University
"Googl DeepMind
http://ical-eeaminggihhe
Abstract
Large-scale generative language and rision-language models (LLMs and VLMs) excel in few-shot in-context learning for decision making and instruction following. However, they require high-quality exemplar demonstrations to be included in their context window. In this work, ask: Can L.MSs and VLMS generate their own examples from generic, sub-optimal demonstrations? We propose In-Context Abstraction Learning (ICAL), a method that builds memory of multimodal ex- perience from sub-optimal demonstrations and human feedback. Given a task demonstration that may contain inefficiencies mistakes, a VLM abstracts the tra- jectory into a generalized program by correcting inefficient actions and annotating cognitive abstractions: causal relationships, object state changes, temporal subgoals, and task-relevan, visual elements. These abstractions are iteratively improved and adapted through human feedback while the agent attempts execute the trajectory in similar environment. The resulting examples, when used as exemplars in the prompt, significantly improve decision-making retrieval-augmmeneed I.L.M and VLM agents. Moreover, as the agent's 's library of examples grows, it becomes more efficient, relying less on human feedback and requiring fewer environment interactions per demonstration. Our ICAL agent surpasses the tate-offthheart in dialogue-based instruction following in TEACh, multimodal web agents in Visu- al Weberraa, and action anticipation in Ego4D. In TEACh, we achieve a 12.6% improvement in goal-condition success. In VisualWebAremt our task success rate improves over the SOTA from 14.3% to 22.7% using GPT4V. In Ego4D action forecasting, we improve over few-shot GPT-4V and remain competitive with super- vised models. We show finetuning retrieval-augmenner in-context agent yields additional improvements. Our approach significantly reduces reliance on manual prompt engineering and consistently outperforms in-context learning from action plans that lack such abstractions.
Introduction
Humans exhibit remarkable few-shot learning capabilities, rapidly generalizing from a single task demonstration to related conditions by integrating the observed behavior with their internal world model. They discern what is relevant and irrelevant for success and anticipate potential failures. Through repeated practice and feedback, they quickly find the right abstraction that helps to imitate and adapt the task various situations. This process facilitates continuous refinement and transfer of knowledge across a diverse range of tasks and contexts.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Learn from noisy demonstrations
Learn from human feedback
User instruction + RGB video demonstration
Execute w/ human- in-the-llop Performing refined actions...
Leave a comment confirming the insect in the picture.
User feedback You not fill out the comments sections
Comments
Generated Abstractions
Abstraction generation
Revised failed abstractions
Revise
Abstracted State:
Ul Nut (Mrskettshe eol
actions...
Refining
ICAL
Revised Abstracted State:
Revising actions and language abstractions
with plawe Commmred
IN UMM Handnahed Picurtt thoeed, deociotarceen orem sttck heppe 227 INFUUT
INPU
Adding
Task Decomposition:
language
"Quardene" comtainng the vearts itanth: mest fillr and crement tus before
abstractions...
Revised Task Decompositions
Sammant iame cam wemeeennii the Quastionl the bed cantaainn barara
weeding
(sed to uppa "Querdocr" you sie keid which carespook to which corpmpard
Trajectory Image:
Trajectory Image:
inth choe scllrno that the common ter aud conton the sevetfc
beida Med, then dick "Seall
Abstraction Comments:
Revised Abstraction Comments:
the du maikk sancls weler nation cown croaree actaal
WWhe the deottti kalae cooment butte tik and crowne www.eeeeelllaeeeeoooooooooe
exinermplers
Revised State Changes:
State Changes:
Prostced Nevd Tame ven stat show comee socfier Morach Duterfy7 conheiing the Ireects
sectir "Oaerdoen" Me Seatly balon ausitiee the conennt.
siler.
Revised Optimized Actions:
Optimized Actions:
trpw LTE Jo WIT alrive Mobrrc Butterr
Now Kiweroon)
Figure 1: ICAL (In-Context Abstraction Learning) is a method for efficient agent learning from both noisy visual demonstrations and human feedback using large language vision models. Left: The agent can take in a video demonstration, and generate a refined example with language annotations to be used later by the VLM via in-context learning. Right: Humans provide feedback, correct errors and supply additional knowledge.
Recent research has explored the use of large language models (LLMs) and visual-language mod- els (VLMs) extract high-level insights from trajectories and experiences. These insights generated through the model's introspection and are used to enhance performance by ap- pending them to prompts, leveraging their strong in-context learning abilities [39, 70, 56, 60]. Existing methods often linguistically focus on task reward signals [70. 56, 76, 79], store hu- man corrections following failures [88. 15, 68], use domain experts to haid-write or hand-pick examples without introspection [68, 73], or utilize language shape policies [30, 74] and re- wards [61, 3, 27, 21, 26, 59, 74, 35, 54). Critically, these methods typically are text- text-based and do not incorporate any visuals cues demonstrations, use introspection only in case of failures, which is only one of several ways that humans and machines can consolidate experiences and extract insights.
In this work, we teach VLMs novel tasks by learning in-context experience abstractions given sub-optimal demonstrations and human natural language feedback. We present In-Context Abstraction Learning (ICAL), method that prompts VLMs to create multimodal abstractions for unfamiliar domains. Unlike previous works that only store and retrieve successful action plans or trajectories [68, 76, 44), our approach emphasizes learning abstractions that encapsulate the dynamics and critical knowledge of tasks, as illustrated in Figure 1. Specifically, ICAL tackles four types of cognitive abstractions: fask and causal relationships, which identify the fundamental principles or actions needed to achieve a goal and how elements are interconnected through cause and effect [75]; changes in object states, which describe the various forms or conditions an object will take [4]; temporal abstractions, which break down tasks into subgoals [6]; and task construals, which highlight critical visual details within task [31]. When provided with optimal or suboptimal demonstrations, ICAL prompts a VLM to transform these demonstrations into optimized trajectories
Throughout the remainder of the paper, we refer to multimodal large language models capable of processing both text and images (e.g.. GPT-4V) as 'VLMs'.
2

---

while also creating pertinent language and visual abstractions. These abstractions are then refined through executing the trajectory in the environment, guided by natural language feedback from humans. Each step of abstraction generation leverages previously derived abstractions, enabling the model to improve not only its execution but its abstraction capabilities as well. Collectively, the learned abstractions summarize crucial information about action sequences, state transitions, rules, and focus areas, articulated through free-form natural language and visual representations.
We present a comprehensive evaluation of our agent, equipped with the learned example abstrac- tions, across three benchmarks: TEACh [63] for dialogue- based instruction in household settings, Visual Area [37] for multimodal autonomous web tasks, and Ego4D for video action anticipa- tion [28]. In TEACh, our agent sets a new state-of-the-ant outperforming VLM agents reliant on raw demonstrations or extensive domain hand-written examples, demonstrating the effectiveness of ICAL learned abstractions for in-context learning. Specifically, our approach achieves 12.6% improvement in goal condition success compared to the previous SOTA, HELPER [68]. We show that this approach leads to increasing performance gains on unseen tasks as the external memory grows, and achieves 14.7% performance increase after only ten examples. Moreover, our agent becomes increasingly efficient over time by leveraging stored abstractions, requiring 38.8% fewer environment steps and 71.6% less human feedback per example in the latter half of demonstrations processed. Integrating our learned examples with LoRA-based tine-tuning of an LLM [32] further improves goal-condition performance by 4.9%. In the VisualWebArent. our agent surpasses the ofate-of-the-ard GPT4 + Set of Marks [37], improving from 14.3% to 22.7% using GPT4V and from 18.9% to 23.4% using GPT4o. In the Ego4D setting, ICAL outperforms few-shot GPT4V using chain of thought, reducing the noun and action edit distance by 6.4 and 1.7, respectively, and competes closely with fully supervised methods, despite using 639x less in-domain training data. Our approach significantly reduces reliance on expertly-crafted examples and consistently outperforms in-context learning from action plans or trajectories that lack such abstractions [68, 76, 44].
2 Related Work
VLM Agents LLMs and VLMs trained from large scale vision-language data have been adapted for task planning and decision making tasks through in-context prompt optimization or finetuning. VLMS have been used to plan over high-level actions or code [80, 76, 68, 44, 72], incorporate error feedback [52. 45, 88], and understanding game instruction manuals [83]. Some studies use VLMs for learning from human feedback through retrievable knowledge [88]. question asking [66, 15], or converting language to actions or rewards [49, 50, 36, 11, 14]. Our work utilizes noisy visual demonstrations, and integrates multiple types of mudti-modal abstractions during the learning process.
Instructable Interactive Agents Benchmarks for embodied instruction following include question answering [25, 16, 93, 18, 17, 23], navigation [42, 41, 10]. interactive dialogue, and instruction following [86, 71, 63, 22]. Virtual agent benchmarks focus on web tasks where agents navigate static [53, 19] and dynamic web environments [92, 37, 85, 38], covering personal shopping, travel assistance, software engineering, and operating system tasks [51, 34, 69, 47J. This includes visual grounding and multi-turn planning, with prior studies using finetuning or few-shot prompts. In agent-based domains, retrieval-aus -uggeetee prompting and prompt optimization have improved task planning in instructional contexts [73] and open- world gaming [79. 76, 56, 62]. Unlike studies that rely solely on static external memory or text-based prompting, our research demonstrates that mudti, generalizable abstractions learned from few noisy trajectories and human feedback via in-context learning or finetuning can significantly enhance instruction-folloviin performance.
3 In-Context Abstraction Learning (ICAL)
In-Context Abstraction Learning (ICAL) aims at automating the acquisition of generalizable examples and knowledge for in-context agents. ICAL operates by receiving language instruction I with a noisy trajectory of observations and actions, denoted Enoisy = fou, ao..... Or, or in new task domain D. A new domain D represents changes in task variables not captured in VLM pretraining, such as a different environment (e.g., kitchen #1 vs. kitchen #2), task (e.g., "add the cheapest red bike to my wish list"), or user preference (e-g.. "I prefer the red cup for coffee"). The core aim of ICAL is to abstract each noisy trajectory into single example e, which then forms part of memory set
3

---

Noisy Demo
VLM Abstraction
Human:In:The:Loop
Memory
Fixed actions Abstraction comments
Environment
Captinized
Episode Success
Multimodal
St-1
Example
Eneckyy
VLM
State changes
aT
Fallure
Summary & Subgoals Abstracted state
Revised Trajectory
Multimodal Example
VLM
Hla o
Raw Experience
ICAL Abstracted Experience
Instruction:
Instruction:
Today taraaa dean book th loll tidps thers bovl to left the stoee. you can br
Tada slin of fonute head. There shos to Fidggs unditern aleed
Abstracted state:
Full State:
all ("ueeiy Thaure" "helecoe' Patso, "strreff Tris, "Oeeel'' Patst,
"spanssu"; Crlabelt: "Stanse', "boleieg': False, "saparteeaa"""
Eieeeh "Moldip" Note, "sppprtted bytt ('Cunterrae")) "Balle 38 Placte "tade".
Ialan.
*ratiee"; Trat, "iitttipsie" "witt", "Sossrtec, aV"
"Trtaa". (alan, reirty's ralte, "estiaet; ralee, > "Bou drr biuaalls Saa
"tavet, "teldirg"i reise, "surtyti relses
"garts Felse, "caake"; Falee,
teeperivaeer. ("lasat: supperidater "oiriwy:
tepperted by: ("auaa1)
"Telitre"
"stueet" Housterferll
"aa ('laat)) "Hacer", Happlerry holle)
"xopprteilyyt" ("Gusterapp))
"Salisiaue 2.1 (tbeetertn
Task Decomposition and Subgoals:
"lalitaurr "telding:: sales, (stteeheree" "herl)
"sapertel by: ""oweterfe"") "operlaattell"" "satetog::
levt
"balddrg": ratee, suury::
Plas
"spperted Prrlege")) "haars Fn ("laaall; Theate',, "soleits;; ralse "stieed Tie, Vlate". "talding": falms "strty:: fales, teggd/tmi, by [erter")))
2 Wheth the bour the dicc and the to thiith water
Pour water trom howl spnatc the bowh state not
areaity)....
Go the tanure and slor ano seroe
Raw Trajectory:
carglt.oee c0052 2taranooeeeeee Toaate", eaject instarce "Tamte'l
ICAL
Abstraction comments:
The in aloae from the ridga to diee the tonors. effect le cler (iduc with water
an Joet sceeetuseeeen aall. esjet ta sloap)
pincing
Leys www.alee
Rine
Jearf
Euch samats dae placnd the Lufling servic fwe slos socasss
Legt fant 95.em
Optimized Actions State Changes:
teptt (tribe11h 9.10 m111
di003))
tee) Ietered Law.loete "Tou'"" ty de serc Leat now Tesl (IP) a Beel La to the left
/1 dijeciinteee "Mait")
10% batts peeracttenddeee Strdtases" abdect Shata Stechtt 00'1 o Stnk.
tevi chriechepp.... stale
Nourt,' falme) a
tptate seel's stare Led with
to
Trajectory Images:
tarpt Inth
bettisiacttiree s1ate
water
ta at 10 1115 01330
balleee anteeeetteeeeee Truats check of the teet is ritted
change monaa "alliiiee lpdeto bad'ss filloaade tire
Indectsoeal cads for treattyl....
ICAL transforms raw experience into useful abstractions for in-context learning. Top: Given a noisy trajectory, It prompts a VLM to optimize actions and add language annotations. The optimized trajectory is executed, incorporating human feedback on failures. Successful examples are stored for future VLM in-context action generation. Bottom: An example of the raw, noisy trajectory (left), and the final abstracted example after ICAL (right).
Figure 2:
M. Each example E M represents an optimized trajectory Eaptimised with generalizable language abstractions L. The objective to ensure that M collectively encapsulates examples that, when used in VIMM context window, increase the likelihood of successful task execution in the new domain, while also containing knowledge that is transferable across similar tasks and contexts. This can be encapsulated as:
max ER, M
where R is the return or the cumulative reward acquired by performing actions based on the instruction I, observation or, and in-context example memory set M. Rather than using reinforcement learning to optimize prompt examples through trial and ehrh would lead to a challenging search problem that myopically focuses on improving rewards for the current scene- -we leverage VL.S* knowledge for abstraction, which we elicit through prompting.

---

3.1 Overview
Figure 2 shows overview of ICAL. Each iteration starts with noisy trajectory. ICAL abstracts two phases: (1) abstraction phase (Fabstract), where a VLM corrects errors and enriches the sequence with language comments (Section 3.2). During this phase, VLM identifies and corrects errors within the sequence, as well as enriches it with natural language comments. (2) The human- in-the-loop phase, denoted Fhati, during which the sequence is executed within the environment and its abstraction is guided by human feedback conveyed in natural language (Section 3.3). Upon the successful execution of the trajectory, it is archived within a continually growing repository of examples. These examples serve as contextual references for the agent both during its learning phase and during inference for unseen instructions and environments.
3.2 VLM-driven Abstraction Generation
We address the challenge of learning from a diverse set of noisy trajectories Enoiey {oo, ,..,,,....,000 oT, ar), which may be sub-optimal due to several factors: demonstrations by human non-experts, errors in inferring actions from visual passive demonstrations, and generated paths that include exploration failures. Please see Section 4.1 for details noisy trajectory collection.
Abstracting noisy trajectory. Enoisy. involves transforming it into a more optimized sequence, Soptimized: and formulating relevant language abstractions, L, as shown in Figure 2. The abstraction function, Fabstract, modifies Snoisy by correcting actions and generating language abstractions that encapsulate general knowledge and task-specific insights. It is defined as:
F Fabstract (Snoisy, I, (e",....,;")
{**))*
(Eoptimizeds L.)
where Enoiay is the initial noisy trajectory, / is the task instruction, and {e",..., e*) are the top-k previous successful in-context examples. The output consists the optimized trajectory Eaptimized and language abstractions L.
Corrections during abstraction include action adjustments and generating annotations (L) for ab- stracting subgoals, causal relationships. state changes, and reasoning steps. These annotations are produced by prompting the VLM to output a specified type of abstraction. We prompt the VLM abstraction function, Fabstract (GPT4V in this work), to produce the abstractions detailed below. For the complete prompts, please refer to the Appendix.
Task and Causal Abstractions: Task and causal abstractions pinpoint the essential principles or actions required achieve a goal and explain how elements are interconnected through cause and effect. Task and causal abstractions have been shown to be helpful in improving LLM generaliza- tion [56), and play strong role in human communication and learning [75, 24). We prompt the VLM to add annotations of task and causal abstractions in the form of natural language comments. For example, it might add a note explaining unnecessary actions, such as "Since the box is already open, there is no need to close it after placing the watches inside, ensuring the task is completed efficiently."
2. State Changes: Understanding how one's actions will affect the form and conditions of elements in a scene is crucial for decision-makinn making [4]. The VLM is prompted to identify and predict state changes that occur during the demonstration. For instance, annotation might note the bowl becoming clean, clearly indicating an expected state transition.
Breaking down complex task into intermediate steps and subgoals is crucial for managing extended and variable sequences of lower-level actions. These temporal abstractions are important for human reasoning [6] and have been shown to improve LLM outputs [81]. We prompt the VLM to add 1) a step-by-step plan detailing the demonstration, and 2) a natural language summary of the actions.
3. Task Decomposition and Subgoals:
Useful representations do not simply mirror every aspect of the world; instead, they selectively capture a manageable subset of details relevant to specific purpose [31]. We focus on identifying and including only those state variables that are relevant to the task at hand. This achieved by (1) selecting parts of the state that were directly interacted with by the agent during the demonstration, and (2) prompting the VL.M to suggest additional state variables not explicitly included in the demonstrations but potentially relevant understanding the task.
State Abstraction:
5

---

3.3 Abstraction Verification with a Human-in-the-loop
In this phase, ICAL verifies the generated abstractions with a human- -oau--i--teee too-hhe-lop...... This involves executing the optimized trajectory, Saptimized within the actual task environment, under the watchful guidance of a live human observer. The procedure is:
1. Execution of optimized trajectory: optimized sequence of actions Coptamized from the abstraction phase.
The agent attempts to perform the task by following the
2. Monitoring and Intervention: As the agent executes Coptimized: a human observer monitors the process. If an action a; fails, denoted by F(ae) 1, the observer intervenes by providing natural language feedback (at,, This feedback is context-specific, addressing the observed failure directly (e.g., explaining that the Toaster is currently full and can only toast one slice of bread). We provide additional details on the noman-in-the-logg in the Appendix Section S5.1.3.
Upon receiving feedback H (a,, o,), the VLM provided with this input alongside the current state of Coptimized and any existing language annotations L. The VLM is prompted to revise Coptimized to address the failure, to update existing annotations L, based on the feedback, and add new annotations that capture insights from the feedback.
3. Feedback Integration and Trajectory Revision:
This process can be represented by an update function:
(EpplateEoptsmiiee: H (a,, 0t), L. 1, (e',...,e*)) ee))) + Eoptimized L'
where Eupdate denotes the update function that takes the current trajectory Eoptimized: human feed- back H(a, ot), and current annotations I., and outputs the revised trajectory Soptimized and updated annotations L'. For the complete prompts. please refer to the Appendix.
Environment Reset and Retrial: Following a failure and subsequent feedback, the environment is reset to suitable state for retrying the task. The agent then attempts the task again, utilizing the newly revised trajectory Soptimised-
5. Success Criteria and Feedback Limit: This interactive phase continues until the human observer deems the task execution successful, or until a predefined maximum number of feedback iterations, N feedbacks: has been reached.
6. Saving example: If successful, we store the revised trajectory Soptimized and language annotations
L to the memory set M. If unsuccessful after N Needbacks iterations, we do not store the example and move the next demonstration. We experiment with relabeling partially successful demonstrations in Section $4.5 of the appendix.
3.4 Retrieval Augmented Generation at Deployment
Given the learned example set M and new instruc- tion I, we prompt the VLM to carry out the instruc- tion by producing action sequences (ao.....ay E from an action API that describes the skills set A (c.g.. go (x), pickup (x)), by retrieving the top K examples from M to include in the prompt based on their textual and visual similarity with the current scene. The aggregated similarity score s for each example e reads:
New Instruction
Today we make sandwich. 2 slices treed need be
Buy cheapest color photo Eimiy's
Deploy w/ ICAL examples learned
Action: breadgo_to)
Example
CALL
VLM
Action: click [16]
Example
s= - Avextual textual Avisual visual (4)
After the ICAL examples have been learned, ICAL deployed for new tasks and environments using retrieval-augmented generation.
Figure 3:
where s glextual, and gvisual are the similarity scores for the input text instruction, textual state, and visual state, respectively, computed via cosine similarity using embeddings from OpenAl's text-embeddingg ada-002 model and CLIP ViT-B/32 model. The coefficients X, Avextuals and Avisnall are weighting hyperparameters chosen in each domain by held out validation set.
The VLM prompt contains the new instruction J, the current webpage image for web agents or 12 video frames for ego4D annotated with sat-of-marrss [84], textual state description x; describing

---

the objects and their attributes for embodied agents and HTML elements for web agents, the action API A, and the retrieved set of in-context examples e', .... ea M. An illustration of this process is shown in Figure 3. The deployment prompt is provided in the Appendix.
Implementation details We use GPT-4-1106-previerr [1] for text generation, unless otherwise stated, and text-embedding 0000 [29] for text embeddings. We use opt-4-1106-vision -priivieeeeeiew [1] for the text and image generation model. We use k 5 for example retrieval. We use a temperature of 0 for TEACh and Ego4D, and 0.2 for VisualWebAren,
4
Experiments
We test ICAL for task planning in TEACh [63] and VisualWebArena [37] and for action forecasting in Ego4D [28] benchmarks.
4.1 Environments
TEACh [63] The TEACh dataset comprises over 3,000 dialogue- based instructions for household tasks in A12TTHOR [40]. We use the Trajectory from Dialogue (TfD) tasks where agents convert dialogue instructions into action sequences, such as MAKE COFFEE. It includes training and validation splits (seen and unseen), the latter featuring new environments and instructions. Agents receive egocentric image inputs o, and perform actions like pickup (X) and turn Seamm__etttt Task fulfilling all instruction conditions. Utilizing HELPER's [68] perception,
success is contingent navigation, and manipulation modules, the system relies on RGB images, depth maps, object masks, and egomotion for 3D mapping and object recognition. We remove domain-specific checks from HELPER's 's modules to allow ICAL to learn them independently. Noisy Trajectories. We use 250 noisy trajectories from TEACh, omitting action labels but retaining language instructions and corresponding RGB videos. To label actions from RGB video, we trained an inverse dynamics model using a transformer encoder-decoder based on the DETR architecture [7] from a seperate 300 TEACh episodes. Model predictions and human errors, like unnecessary movements, cause action noise in these demonstrations. 122 examples were successfully abstracted by ICAL.
VisueeWebArena [37] TisualWehArent consists of 910 episodes across various web tasks (Classi- fieds, Shopping, Reddit) requiring visual comprehension and reasoning. Instructions may include text and reference images, like adding an item seen in an image to a wish list. Agents operate with instructions 1, current webpage images, and an API for actions like click (x), executing tasks to fulfill instruction conditions. Noisy Trajectories. From WebArra, 30 human demonstrations and 62 model trajectories from few-shot GPT4V were abstracted using ICAL. The process led an example set of 92 for evaluation.
Ego4D [28] This task involves anticipating actions from Ego4D RGB egocentric videos in daily scenarios. Models select from 115 verbs and 478 nouns for predicting actions. We evaluate using 200 unseen videos from ego4D validation, applying edit distance
performance metric. Input to models includes sequences of video frames annotated with set-of-marks [84] tracking [12] and label masks. The supervised baseline [28] (243 video hrs of Ego4D V2) uses SlowFast backbone with Transformer aggregator. Noisy Trajectories. Due to the passive nature of this task, ICAL proceeds without sumaan-n-the--oge verification during ICAL (only Section 3.2, VLM-driven Abstraction Generation). ICAL successfully abstracted 92/100 demonstrations taken from the Ego4D validation set (8 failed due to GPT filters) for evaluation.
4.2 ICAL beats written& & unchanged demonstrations in household instruction following
Table I presents our findings on the TEACh unseen validation set, assessing performance on new instructions, houses, and objects. ICAL and baselines use HELPER's navigation and manipulation modules [68]. We compare with these baselines: 1. Hand-wrirten examples from HELPER, the SOTA on the TEACh benchmark, with 19 expert-written examples for attrievll-uggmente prompting. 2. Zero-shot chain of thought, prompting the LLM to output step-by-step. 3. Raw Visual Demos, retrieving unchanged demonstrations labeled with the inverse dynamics model. 4. Raw Kinesthetic Demos, retrieving unchanged demonstrations with true actions. Our metrics are: 1. Task success rate (SR), the % of tasks completed successfully. 2. Goal condition success rute (GC), the % partial fulfillment rate across sessions.
7

---

As shown in Figure 4. ICAL revises noisy trajectories, enabling more successful tasks completed on training tasks than mimicking raw trajectories, with increases of 42 and 86 successful tasks for kinesthetic and visual demonstrations, respectively. This shows how ICAL not only adds useful abstractions but also corrects errors in the passive video demos, improving success in the original demo environment. Please see the Appendix Section $4.3 for additional analysis.
As shown in Table 1, on unseen tasks, ICAL out- performs unprocessed demonstrations as in-context examples, achieving 17.9% absolute improvement in SR over raw demos with predicted actions and 8.6% over those annotated with true actions. This underscores the effectiveness of our abstractions in improving the quality of examples for improved in- context learning, unlike previous works that primarily save and retrieve successful action plans or trajecto- ries without abstractions [68, 76, 44].
140
ICAL visual demas
raplay visual demes
Toddx Suctendilly Campleted
ICAL kinesthetic damos
roplay Kinesthenic demos
100
80
40
Additionally, surpass the handwritten examples of the previous SOTA HELPER [68] by 12.6% in GC and 0.6% SR, and by 2.2% (relative 26.5%) using estimated perception demonstrating our method's efficacy with less expert intervention, leveraging only visual demos and non-expert feedback. Unlike HELPER, which requires domain experts to write 48-107 lines of text for each example, ICAL does not rely on such extensive input from experts. Instead, it allows non-experts to provide up five natural lan- guage feedback corrections to the agent, significantly reducing the required effort and expertise per example.
x00
Number of Interactions
Figure 4: ICAL enables greater success on training tasks. Tasks successfully completed by ICAL over number of interactions when using the ICAL method with kinesthetic or vi- sual demonstrations, and when replaying the kinesthetic or visual demonstrations directly.
Table 2: Results in VisualWebArena. ICAL outperforms the prior best, GPT4o/V + Set of Marks. Ablation studies were conducted with GPT4V on subset of 257 episodes.
Table 1: Evaluation on TEACh unseen valida- tion set. All evaluations are done using GPT3.5- turbo-1106 unless otherwise noted. Visual De- mos demonstrations labeled with inverse dy- namics model. Kinesthetic Demos = demos la- beled with GT actions. GC goal-condition success
Seen Unseen Average
GPT40+SoM [37]
18.9
ICAL (ours)
32.3
22.3
23.4
Success GC
GPT4V+SoM [37]
16.3
14.1
14.3
ICAL (ours)
38.8
20.9
22.7
Ground truth segm, depth, attributes HELPER aandwwittenn [68]
Ablations
34.5
36.7
GPT4V+SoM [37]
11.5
12.9
12.7
Zero-Shot CoT [39]
11.8
24.6
ICAL (ours)
28.0
21.6
22.2
Raw Visual Demos
17.2
26.6
w/o image
28.0
17.3
19.0
Raw Kinesthetic Demos
26.5
29.5
w/ full text trajectory
57.7
21.6
25.5
ICAL (ours)
35.1
49.3
w/o abstraction phase
29.4
44.9
Table Evaluation on the Ego4D unseen
w/o human-in -eee-iiinttt-oooo thop
29.9
41.0
validation subset. ICAL outperforms few-shot
w/ retrieval re-ranking
35.3
51.7
GPT4V and matches supervised baselines using
w/ GPT4
41.7
63.6
639x less in-domain data.
ED@ (Z=20)
finetuned
23.2
40.3
finetuned + retrieval
35.8
54.2
Verb
Noun
Action
Estimated perception
Supervised [28]
0.7251 0.7393 0.9235
HELPER hand-written [68]
8.3
14.1
(639x more data)
ICAL (ours)
10.5
15.4
Zero-shot CoT [39]
0,8796 0.7930 0.9639
Few-shot CoT
0.7877 0.7575 0,9414
ICAL (ours)
0.7802 0.6934 0.9242
8

---

4.3 ICAL obtains state-of-the-art performance on visual web tasks
We evaluate our agent with learned ICAL examples on the WesuaeWa evaluation set. We partition this into episodes 'seen' by our model during learning, and those 'unseen' during learning.
Table 2 presents the results on VisualWebArena. Our model, ICAL, outperforms the previous state- of-the-art [37], which uses GPT4V with few-shot, hand-designed examples and set-of-marks image prompting [84]. ICAL achieves an absolute 8.4% (relative 58.7%) improvement in average success rate over GPT4V and shows 23.8% relative improvement in average success rate over GPT4o.
4.4 ICAL outperforms few-shot VLMs on egocentric video action forecasting
We test ICAL on video action forecasting without using human-in-the-loop abstraction verification due the passive nature of the task. As shown in Table 3, ICAL demonstrates superior few-shot performance on Ego4D action anticipation compared to hand-written few-shot GPT4V that uses chain of thought [81], improving by 6.4 noun and 1.7 action edit distance. ICAL also remains competitive with the fully supervised baseline [28] in noun and action prediction despite using 639x less in-domain training data. We find GPT4V video processing have the least improvements for verb action prediction, possibly due to its limited video understanding capabilities.
4.5 ICAL shows continual improvement with more examples
shows shows continual improvements TEACh validation unseen success rate with more examples learned, as shown in Figure 5. This is in contrast to the unchanged visual demos used for seeding ICAL learning, which show only marginal improvements. Importantly, throughout learning, ICAL does not need to worry about forgetting previously learned knowledge since the agent is expanding memory examples and testing with a frozen VLM via in-context learning. Also noteworthy, our method benefits from even small amount of examples learned, with improvement of absolute 14.7% success rate over zero-shot chain-of of-in-of-thought [39] prompting and 6.8% over the unchanged demonstrations (with 10x less data) with just 10 abstracted demonstrations, showing the efficiency of our method.
4.6 Example retrieval improves learning efficiency
Efficient learning systems benefit greatly from leveraging past knowledge, allowing them reduce the need for human inter- vention and environment interactions as they continue to process new data. Our agent becomes increasingly efficient over time, requiring less human feedback and fewer environment interactions it pro- cesses more examples. By retrieving past successful abstractions during the VI.M- abstraction making and human-in-the-logg phases, it uses previously stored knowledge to help abstract new examples. As shown in Figure 6, for the second half of exam- ples processed, the model requires signifi- cantly fewer environment steps (436#88 267=43, p=0.1143) and human feedbacks (0.74+0.17 0.21+0.08, p=0.0089) per example. This demonstrates that retriev- ing abstracted examples during abstraction learning reduces both human effort and environment interaction over time. Consequently, using previously stored ICAL examples not only improves performance but also accelerates learning for future examples.
45
Valication Success Rate
40
25
20
15
Examples Learned
Figure 5: TEACh validation unseen success rate for ICAL with increasing number of exemplars. ICAL continually learns without forgetting, significantly out- performing the unchanged visual demos used to seed ICAL learning. denotes task success, while denotes goal-condition success.

---

4.7 Fine-tuning helps
We finetune the GPT3.5 model on the learned ICAL examples in TEACh using LoRA [32] in the AzureAl interface (see the Appendix Sec- tion S5.4 for details). The training data include the 122 successfully abstracted examples by ICAL, which we randomly split into 99 training samples and 23 validation samples. This leads to an improvement of 11.4% task success and 15.7% goal-condition suc- cess for the GPT3.5 model. Combining the finetuned model with retrieval-augmennte generation using the ICAL examples led to an additional improvement of 0.7% task success and 4.9% goal-condition success over using
Rist Hal
Second lNx
Moan
generation without finetuning: our top-performing agent. This demon- strates that consolidating the ICAL learned abstrac- tions with weight fine-tuning helps performance.
-14
4.8 Ablations
Enviromenn Intersctions Human Feedbaccss
show each component of ICAL is important
ICAL improves learning effi- ciency as more examples are added to mem- ory. First half (blue) versus second half (orange) of ICAL learning across tasks (left) and each task type separately (right) in TEACh. The second half of ICAL learning requires significantly fewer environment steps (436188 vs. 267=43, p=0.0143) and human feedbacks per episode (0.7420.17 vs. 0.21 =0.08, p=0.0089). This indicates that retriev- ing ICAL examples during learning is heneficial, reducing both human effort and environment inter- action over time.
Figure 6:
We ablate the components of ICAL in TEACh in Table We conclude:
1. The abstraction phase significantly helps for refin- ing the trajectories and adding generalizable knowl- edge. We observe a decrease in 5.7% success rate and 4.4% in goal condition success rate when removing the abstraction phase.
The human-in-the-lopg phase important for fixing errors and incorporating feedback from the user. We observe decrease in 5.2% success rate and 8.3% in goal condition success rate when removing the human-in-tee in-the-loop phase.
3. Our examples demonstrate scalability with larger LLMs. GPT-4 showed a 6.6% absolute increase in task success and a 14.3% absolute rise in goal condition success compared to GPT-3.5. ICAL can be combined with advanced prompting and sampling methods. We test this using re-ranking [78], where the model generates three diverse outputs from different retrieved examples (e.g., top 1-5, 6-10, ...), self-evaluates, and selects the highest scoring output. Improvements are modest but notable: 0.2% in task success and 2.5% in goal condition success.
5 Conclusion
We presented ICAL, a method that improves in-context learning by learning to abstract noisy demon- strations into actionable insightful plans, that when used as in-context examples improve performance of VLM agents over in-context learning from raw examples. ICAL proposes abstracting in-context examples general form of quick learning from a handful demonstrations and human-feedback. It also reduces the need for expert examples, and enables more efficient learning. Tested in TEACh, performance, demonstrating adaptabil-
VisualWebArent and Ego4D, ICAL achieves state-of-the ity to new tasks and environments. There are several limitations and future research directions for ICAL. While ICAL can handle noisy demos, ICAL may not be able handle extremely misleading demonstrations or feedback, and relies on fixed action API which may restrict adaptability in dynamic environments. Additionally, GPT4V's visual grounding deficiencies [90, 82, 55, 9] cannot always be overcome by in-context learning, and more research is needed to address this.
10

---

Acknowledgements This material is based upon work supported by National Science Foundation grants GRF DGE1745016 & DGE2140739 (GS), ONR award N00014-23-1-244 AFOSR Grant FA9550-23-1-0257, and DARPA No. HR001 12490375 from the U.S. DARPA Friction for Account- ability in Conversational Transactions (FACT) program. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the United States Army, the National Science Foundation, or the United States Air Force.
This research project has benefitted from the Microsoft Accelerate Foundation Models Research (AFMR) grant program through which leading foundation models hosted by Microsoft Azure along with access to Azure credits were provided to conduct the research.
References
[1] Openai, gpt-4 technical report. arXiv preprint arxiv:2303.087. 0877748774 2023.
[2] Anas Awadalla, Irena Gao, Josh Gardner. Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.0/ 2023.
[3] Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. Learning to understand goal specifications by modelling reward. arXiv preprint arXiv: 1e06.099660999 2018.
[4] Lisa Feldman Barrett and Moshe Bar. See it with feeling: affective predictions during object perception. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(15)1): 1334, 2009.
[5] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302. 2023.
[6] Matthew M Botvinick, Yael Niv, and Andew G Barto. Hicrarchically organized behavior and its neural foundations: A reinforcement learning perspective. cognition, 113(3):262-280, 2009.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 21 213-229. Springer, 2020.
[8] Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhut- dinov. Learning to explore using active neural slam. In International Conference on Learning Representations, 2019.
[9] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. arXiv preprint 12X.:::4411.1266 2024. URL org/ arg/abs/2400 2tt.b./.......1.........
[10] Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicene Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In Computer ECCin 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pages 17-36. Springer, 2020.
[11] Ching-An Cheng, Andrey Kolobov, Dipendra Misra, Allen Nie, and Adith Swaminathan. LIf-bench: Benchmark for interactive learning from language feedback. arXiv preprint arXiv:2312.06853, 2023.
[12] Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon- Yon-Young Lee. Tracking anything with decoupled video segmentation. In ICCV, 2023.
[13] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-iman image learning, arXiv preprint ar ar21::211..07143, 2022.
11

---

[14] Yuchen Cui, Siddharth Karamcheti, Raj Palleti, Nidhya Shivakumar, Percy Liang, and Dorsa Sadigh. No, the right: Online language corrections for robotic manipulation via shared autonomy. In Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, pages 93-101, 2023.
[15] Yinpei Dai, Run Peng, Sikai Li, and Joyce Chai. Think, act, and ask: Open-world interactive personalized robot navigation. arXiv preprint arXiv:2310.07966 2023.
[16] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1-10, 2018.
[17] Abhishek Das, Federico Carnvale,, Hamza Merzie, Laura Rimell, Rosalia Schneider, Josh Abramson, Alden Hung, Arun Ahuja, Stephen Clark, Gregory Wayne, et al. Probing emergent semanties in predictive agents via question answering. In Proceedings of the 37th International Conference on Machine Learning, pages 2376-2391, 2020.
[18] Samyak Datta, Sameer Dharur, Vincent Cartillier, Ruta Desai, Mukul Khanna, Dhruv Batra, and Devi Parikh. Episodic memory question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19119-19128, 2022.
[19] Xiang Deng. Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023.
[20] Bin Dong, Fangao Zeng, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Solq: Segmenting objects by learning queries. Advances in Neural Information Processing Systems, 14:21898.
21909, 2021.
[21] Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision- instruction following. arXiv preprint arXiv:1902.007722.2000 2019.
[22] Qiaozi Gao, Govind Thattai, Xiaofeng Gao, Suhaila Shakiah, Shreyas Pansare, Vasu Sharma, Gaurav Sukhatme, Hangjee Shi, Bofei Yang, Desheng Zheng, et al. Alexa arena: A user-centric interactive platform for embodied ai. arXiv preprini M.Xiv:2303.01886 2023.
[23] Xiaofeng Gao, Qiaozi Gao, Ran Gong, Kaixiang Lin, Govind Thattai, and Gaurav S Sukhatme. DialFRED: Dialoguc-enabled agents for embodied instruction following. IEEE Robotics and Automation Letters, 7(4): 100110000900066.00-11111100 2022.
[24] Noah D Goodman and Michael C Frank. Pragmatic language interpretation as probabilistic inference. Trends in cognitive sciences, 20(1 (0(11)8818-4 829, 2016.
[25] Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering in interactive environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4089-4098, 2018
[26] Prasoon Goyal, Scott Nickum, and Raymond Mooney. Using natural language for reward shaping in reinforcement learning. arXiv preprint arXiv: 1903.v1903.022 02000,220 2019.
[27] Prasoon Goyal, Scott Niekum, and Raymond Mooney. Pix12r: Guiding reinforcement learning using natural language by mapping pixels to rewards. In Conference on Robot Learning, pages 485-497. PMLR, 2021.
[28] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
[29] Ryan Greene, Ted Sanders, Lilian Weng, and Arvind Neelakantan. Openai. new and improved embedding model. 2022.
[30] Brent Harrison, Upol Ehsan, and Mark O Riedl. Guiding reinforcement learning exploration using natural language. arXiv preprint arXiv:1007.08616 2017.
12

---

[31] Mark K Ho, David Abel, Carlos G Correa, Michael L Littman, Jonathan D Cohen, and Thomas L Griffiths. People construct simplified mental representations to plan. Nature, 606(7912): 129 136, 2022.
[32] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 206.:22006.0688,, 2021.
[33] Ayush Jain, Pushkal Katara, Nikolaos Gkanatsios, Adam W. Harley, Gabriel Sarch, Kriti Aggarwal, Vishrav Chaudhary, and Katerina Fragkiadaki. Odin: A single model for 2d and 3d perception, 2024.
[34] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2023.
[35] Pushkal Katara, Zhou Xian, and Katerina Fragkiadaki. Gen2sim: Scaling up robot learning in simulation with generative models, 2023.
[36] Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv preprint arX1v:2310.00166, 2023.
[37] Jing Yu Koh Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents realistic visual web tasks. arXiv preprint 2401 2024.
[38] Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents. arXiv preprint arXiv:2407.0 004444777776,, 2024.
[39] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-222111 2022.
[40] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D) Environment for Visual AI. arXiv, 2017.
[41] Jacob Krantz, Theophile Gervet, Karmesh Yadav, Austin Wang, Chris Paxton, Roozbeh Mot- taghi, Dhruv Batra, Jitendra Malik, Stefan Lee, and Devendra Singh Chaplot. Navigating to objects specified by images. arXiv preprint arXiv2200..01122 2023.
[42] Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across- room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4392 4412, 2020.
[43] Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly zero-shot classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages
October 2023.
[44] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209. 07753, 2022.
[45] Jacky Liang, Fei Xia, Wenhao Yu, Andy Zeng, Montserrat Gonzalez Arenas, Maria Attarian, Maria Bauza, Matthew Bennice, Alex Bewley, Adil Dostmohamed, al. Learning learn faster from human feedback with language model predictive control. arXiv preprint arXi::2402.11450 2024.
[46] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer VCCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
13

---

[47] Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang, Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations (ICLR), 2018. https: // www.waiwwiraa...e.ee.ooeeee.owoe 180..088000
[48] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023.
[49] Huihan Liu, Alice Chen, Yuke Zhu, Adith Swaminathan, Andrey Kolobov, and Ching-An Cheng. Interactive robot learning from verbal correction. arXiv preprint arXiv:2310.17555, 2023.
[50] Huihan Liu, Shivin Dass, Roberto Martín-Martin, and Yuke Zhu. Model-based runtime moni- toring with interactive imitation learning. arXiv preprint arXiv:2310. 17552, 2023.
[51] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating iims as agents, 2023.
[52] Zeyi Liu, Arpit Bahety, and Shuran Song. Reflect: Summarizing robot experiences for failure explanation and correction. arXiv preprint arXiv:2306. /5724, 2023.
[53] Xing Han Lü, Zdenčk Kasner, and Siva Reddy. multi-turn dialogue, 2024.
Weblinx: Real-world website navigation with
[54] Yecheng Jason Ma. William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Fureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. /2931, 2023.
[55] Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Sasha Sax, and Aravind Rajeswaran. Openeqa: Embodied question answering in the era of foundation models. In Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[56] Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Burch Balris and Peter Clark. Clin: A continually learning language agent for rapid task adaptation and generalization. arXiv, 2023.
[57] So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, and Ruslan Salakhut- dinov. Film: Following instructions language with modular methods, 2021.
[58] So Yeon Min, Hao Zhu, Ruslan Salakhutdinov, and Yonatan Bisk. Don't ton't copy the teacher: Data and model challenges embodied dialogue. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9361-9368, 2022.
[59] Suvir Mirchandani, Siddharth Karamcheti, and Dorsa Sadigh. Ella: Exploration through learned language abstraction. Advances in neural information processing systems, 34:29m9,4 34:29529-29540, 2021.
[60] Jesse Mu, Victor Zhong, Roberta Raileanu, Minqi Jiang, Noah Goodman, Tim Rocktäschel, and Edward Grefenstette. Improving intrinsic exploration with language abstractions (2022). URL http:://urxii. org/ubs/2002.08993.
[61] Jesse Mu, Victor Zhong, Roberta Raileanu, Minqi Jiang, Noah Goodman, Tim Rocktäschel, and Edward Grefenstette. Improving intrinsic exploration with language abstractions. Advances in Neural Information Processing Systems, 35:33947-33960, 2022.
[62] Kolby Nottingham, Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Sameer Singh, Peter Clark, and Roy Fox. Skill set optimization: Reinforcing language model behavior via transferable skills. arXiv, 2024. URL attps::/arre org/ a//aabb///000 03244.
14

---

[63] Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan- Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur, Teach: Task-driven embodied agents that chat, 2021.
[64] Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision-and- language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15942-15952, 2021.
[65] Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[66] Allen Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners. In Conference on Robot Learning, pages 661-682. PMLR, 2023.
[67] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 635-635. JMLR Workshop and Conference Proceedings, 2011.
[68] Gabriel Sarch, Yue Wu, Michael Tarr, and Katerina Fragkiadaki. Open-ended instructable embodied agents with meeorry-augmentaee large language models. Findings of the Association for Computational Linguistics: EMNLP 2023, 2023.
[69] Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In ICML, 2017.
[70] Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint wrXiv:2303.11366 2023.
[71] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10740-10749, 2020.
[72] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay. Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEFE International Conference on Robotics and Automation (ICRA), pages 1 11523-11 115301 IEEE, 2023.
[73] Chan Hee Song. Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models, 2023.
[74] Tasmia Tasrin, Md Sultan Al Nahian, Habarakadage Perera, and Brent Harrison. Influeneing reinforcement learning through natural language guidance. arXiv preprint 20Xiv:2104.01550 2021.
[75] Joshua B Tenenbaum, Charles Kemp, Thomas L Griffiths, and Noah D Goodman. How to grow a mind: Statistics, structure, and abstraction. science, (31 2011.
[76] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv: 2rxiv-2305.16291 2023.
[77] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2023.
15

---

[78] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consisteney improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. //171, 2022.
[79] Zihao Wang. Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, Xiaojian Ma, and Yitao Liang. Jarvis-I: Open- world multi-task agents with memory- auggentdd multimodal language models. arXiv preprint arXiv: 2311.05997, 2023.
[80] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprini ar 2332v22302.01566 2023.
[81] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le. and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 1900,, 2022.
[82] Penghao Wu and Saining Xie. V*: Guided visual search core mechanism in multimodal Ilms. arXiv preprint 1rXiv:2312.14135, 2023.
[83] Yue Wu, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, and Tom M Mitchell. Read and reap the rewards: Learning play atari with the help of instruction manuals. In NeurlPS, 2023.
[84] Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set- of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023. URL https: //arxiv.org wtt.:/ggg/aa......////// 11441.
[85] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents, 2023.
[86] Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung- Yen Yang, Vidhi Jain, Alexander William Clegg, John Turner, et al. Homerobot: Open vocenuooabulary mobile manipulation. arXiv preprint w3Xh:22306.1156, 2023.
[87] Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv preprint arXiv:2111.0022 2021.
[88] Lihan Zha, Yuchen Cui, Li-Heng Lin. Minae Kwon, Montserrat Gonzalez Arenas, Andy Zeng, Fei Xia, and Dorsa Sadigh. Distilling and retrieving generalizable knowledge for robot manipulation via language corrections. In 2nd Workshop on Language and Robot Learning: Language as Grounding, 2023.
[89] Yichi Zhang, Jianing Yang. Jiayi Pan, Shane Storks, Nikhil Devraj, Ziqiao Ma, Keunwoo Yu, Yuwei Bao, and Joyce Chai. Danli: Deliberative agent for following natural language instructions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1280-1298 1298, 2022.
[90] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.016// 2024.
[91] Kaizhi Zheng, Kaiwen Zhou, Jing Gu, Yue Fan, Jialu Wang, Zonglin Li, Xuehai He, and Xin Eric Wang. Jarvis: A neuro-symbolic commonsense reasoning framework for conversational embodied agents. 2022.
[92] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2023.
[93] Hao Zhu, Raghav Kapoor, So Yeon Min, Winson Han, Jiatai Li, Kaiwen Geng, Graham Neubig, Yonatan Bisk, Aniruddha Kembhavi, and Luca Weihs. Excalibur: Encouraging and evaluating embodied exploration. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14931-14942, 2023.
16

---

S1 Overview
The structure of this Appendix is as follows:
Section S2 contains negative potential impacts.
Section S4 contains additional experiments.
Section S5 contains additional methods details.
Section S6 contains additional details on the evaluation environments.
S2 Potential Negative Impact
The work introduced by ICAL for Al agents carries potential risks including the perpetuation of biases, privacy infringement, user dependency, economic displacement, security vulnerabilities, and the emergence of unintended behaviors due technical limitations. Mitigating these risks necessitates the development of mechanisms for bias correction, privacy preservation, ethical guidelines, and security protocols. Engaging with diverse range of stakeholders is imperative to ensure that the deployment of these technologies aligns with societal values and contributes positively to the realm of human-Al collaboration, fostering advancements that are both innovative and responsible.
S3
ICAL relation to dAgger
S3.0.1 Relation of Abstraction Verification to DAgger
The human-in-the-loop phase of ICAL bears a conceptual resemblance to the Dataset Aggregation (DAgger) algorithm [67]. as both methods involve iterative refinement an agent's policy through interaction with expert feedback. However, ICAL extends this framework by incorporating natu- ral language feedback, updating both actions and abstractions, and utilizing aarieval-augmented generation (RAG) with explicit memory of optimized examples for policy improvement.
In DAgger, the agent collects data by executing its current policy and then queries expert obtain the correct action for each encountered state. Specifically, at iteration t, the agent observes a state s and takes action de (a(s:) according to its policy me The expert provides the optimal action oj and the agent aggregates this data into a dataset D:
D DU ((st, a;)}.
(5)
The policy is then updated by minimizing loss function over D:
t++1 arg min
Σ L(T(ss),aa). (ED
(6)
Similarly, ICAL's human-in-the-loop phase Frin, agent refines its behavior based on human feedback. When the agent executes an optimized trajectory Eopimized and encounters failure at action a,, human observer provides natural language feedback H (at, 0z) concerning the action a, and the observation or, The agent integrates this feedback to update both the trajectory and the associated language abstractions:
(Sopimizeds L') = Eupdate (Soptimized: H(a, or), L, (e, (e.,..,,e"
e*)),
where L represents the current language annotations, I is the task instruction, and {et ..... e retrieved examples from memory. This updated trajectory Siptimized and abstractions L' are then added to an explicit memory ε, enhancing the agent's policy through enriched context:
= E {Coppimired L'))-
(8)
The agent's policy TICAL. is implicitly updated by conditioning this memory during action genera- tion:
TICAL (8,, E) VIMM(s,
where VLM denotes the Vision-Language Model used for in-context learning.
17

---

The similarities between ICAL and DAgger lie in their iterative approach to policy refinement using expert feedback. However, ICAL offers several key benefits:
Natural Language Feedback: Unlike DAgger, which requires the expert to provide explicit action corrections ai ICAL accepts natural language feedback (a,, oz). This allows the human convey richer information, including explanations, suggestions, and contextual details that can address not only the immediate failure but also underlying misconceptions.
Revision Actions and Abstractions: ICAL updates both the action sequence and the associated language abstractions L. By refining the abstractions, the agent enhances its understanding of task structures, causal relationships, and state changes, which promotes better generalization to new tasks and environments.
Policy Improvement via Retrieval-Augmenned Generation: ICAL maintains an explicit memory & of optimized examples and abstractions. During deployment, the agent retrieves relevant examples from based on similarity measures and uses them as context for action generation. This retrieval- augmented generation (RAG) approach allows the agent to leverage past experiences effectively, adapting its policy without explicit parameter updates.
In contrast, DAgger relies solely on aggregating state-action pairs and updating the policy through supervised learning, which may not capture higher-level task structures or facilitate transfer to new domains. ICAL's ability to process natural language feedback and to update both actions and abstractions provides more flexible and powerful framework for policy refinement, aligning more closely with human learning processes. ICAL extends the traditional imitation learning paradigm represented by DAgger, enabling more efficient and generalizable learning from human feedback.
S4
Additional Experiments
S4.1
Learning efficiency broken down by task type
In Section 4.6 of the main paper, we showed how ICAL enables fewer environment interactions and human feedbacks per example. We provide the learning efficiency between the first and second half' of demonstrations processed broken down by task type in Figure S1.
Hean Staps Before
Half Steps
Secood
Mean Swedhar Befane Afber Half Siwas
Feeetbacks
Second
Clean X
Water Plane
Bel
Al K. One,
H. Siicess Of 1n_r
Sanswwich
Coftee
Pot, All Xee
Cookeed Starr DE
Plase Of
Figure S1: First half (blue) versus second half (orange) of ICAL leariing across tasks (left) and for each task type separately (right) TEACh.
84.2 Experimenting with different types of in-context examples in VisualWebArena
We experiment with alternate way to provide the ICAL in-context examples the VLM. Instead of retrieving a single time step, we give the full trajectory of observations, abstractions, and actions in textual format (no images provided). We run this reduced subset of 239 VisualWebArena
18

---

Table S1: VisualWebArena performance for ICAL performance when using a single time step with image input for in-context examples versus providing all time steps, but without image inputs.
Seen Unseen Average
GPT4V+SoM [37]
11.5
12.9
12.7
ICAL text trajectory
57.7
21.6
25.5
ICAL + text + single time step
28.0
17.3
19.0
ICAL text + single time step image 28.0
21.6
22.2
Table S2: Tasks successfully completed after applying the ICAL method (out of 250). We compare ICAL using either visual demonstrations or kinesthetic demonstrations. Kinesth. = Kinesthetie; demos with GT actions. Visual action labeled from RGB frames with inverse dynamics model.
GPT-4
GPT-355
Visual Demos Kinesth. Demos
Visual Demos
Breakfast
Boil_X
5
3
Water_Plant
13
16
9
Sandwich
9.
8
0
PuLAILX OnlY
12
15
Plate Of Toast
17
14
N Slicess OF In_Y
11
13
Clean X
13
16
3
Put All X One _ne_Y
13
16
10
Coffee
15
N Cooked O
3
Total
122
142
52
tasks. The results are presented Table S1. We find that providing the full trajectory increases seen success rate, but does not improve unseen success rate. In our final evaluation, we utilize the retrieval of a single time step with image input, since expanding the context length through the full trajectory adds to the cost without significantly improving the success rate on unseen tasks.
S4.3 TEACh results on ICAL learning using trajectories with ground truth action labels and GPT3.5
We present tasks successfully completed for each task type in Table S2, comparing ICAL that uses visual demonstrations and kinesthetic demonstrations.
We see that using GPT3.5 for ICAL significantly reduces the number of successful tasks by over half compared to using GPT4 (52 versus 122 tasks successfully completed). We show in Section S4.5 of the main paper how relabeling unsuccessful tasks can help improve performance when using weaker models, such GPT 3.5.
S4.4 TEACh validation accuracy by task type
We present ICAL agent performance after learning the TEACh validation set for each task type in Table S3.
$4.5 Relabeling unsuccessful examples improves ICAL when using weaker models
Instead of removing unsuccessful examples, can instead relabel the examples by querying LLM to generate new task instruction, step-by-step plan, and summary the partial task completion. In Table $4, we show performance of ICAL with and without relabeling. Relabeling improves performance when using weaker model, GPT3.5 during ICAL learning, by an absolute 3.4% in success and 0.5% in GC.
19

---

Table S3: TEACh validation performance of ICAL after examples have been learned for each task type when evaluated using GPT3.5 GPT4.
GPT3.5
GPT4
(%) GC))
(5) (%)
Breakfast
2.3
37.5
2.3
56.6
Water_Pinm
73.0
73.0
61.9
61.9
Salad
31.9
61.7
40.4
74.6
Sandwich
2.1
45.4
2.1
56.0
PutAILXXOO
56.0
65.6
66.0
74,0
Toast
N N_Slices_ Of X I
48.1
57.6
63.0
72.2
Clean All
58.8
70.7
75.0
Put_iLE MLLAllXXLU X_nn_One__
62.5
72.0
66.7
72.0
Coffee
31.6
44.6
49.1
59.5
N. Cooked Slices Jn__
30.6
70.0
Average
35.1
49.3
41.7
63.6
Table S4: Relabeling experiments. Relabeling unsuccessful demonstrations improves performance when using weaker models, such GPT3.5, during the ICAL learning process.
Success GC
ICAL
22.4
36.9
+ relabeling 25.8
37.4
S4.6 Running ICAL from RGB-only input
We ICAL from RGB inputs. We the perception, navigation, and manipulation modules from HELPER [68], which uses SOLQ [20] for object detection and ZeoDepth [5] for depth estimation. HELPER initializes objects with default attributes based the domain, and uses domain-specific pre- condition checks and error correction. However, we wish to automate the learning of these modules, and thus remove them. Additionally, HELPER initializes a memory of examples hand-written by domain expert. We replace these with our ICAL examples. For inferring attributes of the objects in the abstracted state, we apply CLgVLM [77], an open-source visual language model, on the detected object crops, which we found to work best compared to other models for object attribute detection on separate dataset of cropped object images (see Appendix).
As shown in Table S5, we find that our ICAL agent obtains performance close to that HELPER, lagging behind 1.7% success and 3.2% goal condition success, despite not hand-desggning object attributes, pre-condition checks, етгог correction, and in-context examples. Additionally, when using the hand-written HELPER examples with the ICAL execution modules, find that the ICAL examples outperform the HELPER examples by 2.2% in task success and by 1.3% in goal-condition success, despite the ICAL examples being obtained without hand-writing from domain-expert. Additionally, when using the perception of ODIN [33], which utilizes multi-view images and a 3D bottleneck for semantic segmentation, ICAL obtains performance on-par with that HELPER.
S4.7 Benchmarking open-source LLMs for attribute detection in TEACh
In household instruction following, ICAL benefits from accurate object and attribute detection from sensory input for state inference. For benchmarking object attribute detection TEACh, we build dataset of 2581 object crops of clean viewpoints of the object by having the agent pick up the object and centering the object in view. We build second dataset of 661 from random viewpoints of the object in the TEACh training set with different objects varying in their "dirty" and "cooked" attributes. Clean viewpoints are always centered, unoccluded, and posed, while the random viewpoints are often occluded and show the object from different angles. Example crops for the datasets for 'dirty plate' shown in Figure $2. We test the following models: OPENCLIP CLIP-VIT--IGG--4441AA 1-11AGGGG--...11 39B-B B 60,[ OpenAl CLIP
X- VLM[87), Llava 1.5[48), VLM(77,
20

---

Table S5: TEACh validation set (seen) from RGB imput. Our ICAL agent obtains performance close to that of HELPER, despite not hand-designing object attributes, pre-condition checks, error correction, and in-context examples.
Success
GC
E.T. [64]
1.0
1.4
JARVIS [91]
1.7
5.4
FILM [57]
5.5
5.8
DANLI [89]
5.0
10.5
HELPER [68]
12.2
18.6
ICAL w/ HELPER examples
8.3
14.1
ICAL (ours)
10.5
15.4
HELPER [68] ODIN [33]
13.8
26.6
ICAL (ours) + ODIN
13.8
25.5
Table S6: Attribute detection accuracy in TEACh for different open-source VLMs, We find CogVLM currently outperforms the other open-source VLMs at posed and unposed attribute detection for object crops.
Clean Views Random Views
OpenCLIP CLP-VVT-ggG--44iiinn----99------ [13]
0.860
0.715
OpenAl CLIP clip-vit-bas--atthh22 [65]
0.758
X-VLM [87]
0.785
Llava 1,5 [48]
0.862
0.839
Cog [77]
0.898
0.857
Diffusion Classifier [43]
0.665
Open Flamingo [2]
0.530
diffusion classifier(43), Open Flamingo[2). We queried CLIP by taking the best match of the image encoding with [a photo of a (category) that is [wordl], ), photo of a [category] } that is [word2] }]. where wordl and word2 are opposite attributes. We queried diffusion classifier with blurry photo of a [wordl] } {category). )., a blurry photo of a (word2) [category] J.]. as per the paper. We queried CogV VLM, Llava, and Open Flamingo with the image crop and asked it Is this (category) (wordl) or [word2]? Provide a single word answer, either " "[word]] }" or "I word2)". " We show the results on our evaluation dataset in Table S6. We find CogVLM outperforms the other open-source VLMs at posed and unposed attribute detection for object crops. We use cog VLM for our estimated perception experiments for detecting object attributes.
S5
Additional Methods Details
S5.1
In-Context Abstraction Learning (ICAL) Algorithm
We present the method for In-Context Abstraction Learning (ICAL) for single trajectory in Algo- rithm S1. Given a noisy trajectory, the method proceeds by first abstracting the trajectory through function Fabstract, which leverages LLM or VLM to correct errors or inefficiencies in the trajectory and generates language abstractions that capture the essence the task, including subgoals, causal relationships, and state changes. This phase does not require interaction with the environment or humans.
Initialization sets up for the Human- In-The--Toopppp (HITL) phase by resetting the feedback count and success flag. The method then enters a feedback loop where the optimized trajectory is executed in the environment. If the task execution is successful, the loop breaks, and the method proceeds to update the example set with the abstracted trajectory and its associated language abstractions. Otherwise, human feedback is solicited at the point of failure to revise the trajectory and language
21

---

Figure S2:
An example image crop of the attribute dataset collected in TEACh of a dirty plate. A clean viewpoint example image shown on the left, and random viewpoint example image is shown on the right.
abstractions further, utilizing the VLM again. This feedback loop continues until either the task is successfully executed predefined maximum number of feedback iterations reached.
S5.1.1 Noisy Trajectories
We collect a noisy sequence of observations and actions, denoted as Enoisy = Og. ag OT, ar, which represents a trajectory for the language-defined task we aim for agent learn and adapt to.
The trajectory sequences can come from variety of sources, including unlabeled video sequences. We can also accommodate sub-optimal or unsuccessful attempts. In our work, we identify three scenarios in which given sequence, Enoieys might be inefficient or incorrect, and using them directly as in in-context examples for our LLM/VLM agents could result in poor performance:
We gather demonstrations from humans without commain-peecific expertise. Specifically, these demonstrations Ao..... ar are collected from human participants who are provided with a textual instruction and an RGB image at each time step, and are instructed to choose actions a E A to complete the instruction. These humans commit errors, choose sub-optimal actions, and may not complete the task to fullest extent. For instance, episode within TEACh [63] has participant who picks up knife and then looks up and down before placing the knife down again, a sub-optimal action sequence not required by the instruction [58].
Human Non-experts:
Here, the agent given a sequence of observations {= ={00, 01: ....... ) that lack corresponding action labels. We collect these visual demon- strations for TEACh only. We use the TEACh human demonstrations as described in the previous text and take the egocentric RGB images without actions as the observation se- quence. To infer the actions executed in these demonstrations, an inverse dynamics model is applied to consecutive pair of frames Fiam (Ot, 0++1), which predicts the action a, respon- sible for the state transition or + Or+1- Along with sub-optimal human trajectories, the inverse dynamics may make prediction errors. We trained a transformer encoder-decoder model based on the DETR [7] architecture on 300 TEACh episodes (see Section S5.3 for more details).
Visual Passive Demonstrations:
Agent Trajectories: In Visual Web Arena [37], we obtain additional demonstrations sourced from deploying in-context VLM new tasks. Specifically, we first run ICAL process on 30 human demonstrations collected by non-experts. We run the ICAL process to abstract the 30 human demonstrations and then deploy our ICAL agent using the learned examples as in-context examples. Using this ICAL agent, we collect an additional 62 new trajectories Visual Web Arena tasks and continue to run the ICAL learning these new trajectories collected by the model.
S5.1.2 Abstraction phase implementation details
We present prompt template for the VLM abstraction generation phase in Listing S5.
22

---

We iterate through each Python program demonstration labeled with the inverse dynamics model. Given the noisy Python program, instruction, action API, and object state, the abstraction phase proceeds by prompting the LLM to 1) revise the the code for maximal efficiency and fix mistakes in the code (abstracted trajectory), 2) provide a summary of the functionality of the script (Task Decomposition & Subgoals), 3) provide step-by-step plan of the steps of the script (Task Decomposition & Subgoals), 4) Add object attribute state changes to the Python program (State Changes), and 5) add abstraction comments (Task and Causal Abstractions). For state changes, we parameterize the state changes in TEACh, allowing the LLM to add a change_ thange_state() function to the actions to indicate change in state of the objects the agent is interacting with. Each step uses retrieved examples of successful examples previously saved in memory.
TEACh.
reh We perform the abstraction phase for each individual action taken a (e.g.. click(element), hover(element) in each noisy trajectory Enoisy obtained in VisualWebArena, Specifi- cally, for each action in Enoisy. we first prompt the VLM to optionally revise the action (optimized trajectory), and output a summary and step-by-step reasoning for the chosen action (Task Decomposi- tion & Subgoals), given the instruction, image observation, textual state description, previous actions taken, and proposed trajectory action for the current time step. Next, we prompt the VLM to output predicted next state (State Changes), given the instruction, current and next image observation, current and next textual state, and the action taken 0ς- We next prompt the VL.M to output the most relevant state elements for the task instruction (State Abstraction), given the instruction, image observation, and textual state.
Finally, we prompt the VLM to output set of abstraction comments, given the full sequence of abstracted actions, task decomposition and subgoals, state changes, and abstracted state descriptions.
Ego4D. We perform the abstraction phase for each full demonstration, consisting of 20 predicted future times steps of actions. We give the VLM 3 video frames annotated with set-of-marks, the GT actions, and the actions in the video, and prompt the VLM to annotate the four types of abstractions for each example.
S5.1.3 Human-in-the-loop phase implementation details
We present our prompt template for the human-in-thee in-the-loop phase revisions in Listing S6.
TEACh. The TEACh simulator enables fine- the-grained analysis of task progress. During the human- in-the-loop phase, formulate this task progress into natural language feedback for failed actions (e.g., "The "The Toaster is full right now:" or missed task steps (e.g., "Thh Pillow needs to be put onto a Sofa"). The natural language feedback, along with the instruction, object state, action API, and failed actions/code, are given to the LLM to revise the program and abstractions.
VisualWebArena. For 20 tasks for each website related to the tasks in the demonstrations collected, we develop an interface to provide natural language corrections to the model based on the predicted next action by the model. The humans are tasked to intervene and provide feedback when they deem an action predicted by the model sub-optimal. When an action is proposed, the humans are given the ability to accept the action or reject the action if it is sub-optimal. If sub-optimal, the humans can type in natural language feedback which will be sent to the VLM to revise the action and abstractions. We provide an example of the previous outputs, human feedback, and revised outputs in Listing SI.
Ego4D. Due to the passive nature of the Ego4D task, where there is no agent executing the actions predicted, no human in the loop phase is implemented. Ego4D only uses the abstraction phase.
23

---

Algorithm S1 In-Context Abstraction Learning (ICAL) method for a single trajectory
Require: Noisy trajectory Enoisy {oo, ao, ..., Or, ar Task instruction Maximum feedback iterations NJ feedbacks
Ensure: Updated example set M
Abstraction Phase:
Abstraction of the trajectory D Prompts VLM
2:
(Sabstracted; L) +- Fabstract (Snoiiy 1, 1, ,e.,,..,, e*)
3: Initialization:
Prepare for the HITL phase
4: feedback count <- 0
5: success +- False
6: while feedback_come
Nş feedbacks and not success do 7: Execute Eabstracted in the environment
Feedback loop D Attempt task execution
8:
if Task is successful then
9: success <- True
break
11: else
Obtain human feedback H
Get feedback failure point
13:
(Eabstracted: L) &- Eupdate (Eabetracted, H, 1, e e*) feedback ceeeback_ccuuttttt count ff feedback count
b> Prompts VLM
15:
end if end while
if success then
18:
Update example set M + M U {(E ((Cntstracted: L)}
b Store the abstracted example
19: end if
24

---

Listing S1: Example of human-in somaan-n--the-ooy thop revision for visual web arena.
Instruction:
Show me the nost recently posted blue iPhone.
Abstractions
Teedbac2:
Abstraction Conments:
Whan searching for the most recently poated itan of apecific color, ansure the listinga are sorted by the nevest first and visually inspect the images for the coler before clicking the correct listing.
iPhone, need the classifieds rebsite. The search input field is identified by (5) I will type filter listings iPhones. This will help find the most recently liated blue iPhone.
find
recently posted
search function
iPhone"
Sunnary:
Typing "blue iPhone" into the search field will filter the listings shov relevant items, alloving progrezs tovards identifying the mast racently posted blue
State: search, the next state will display search resulta related blue iPhones, potentially including lietinge with inages and descriptions the items.
typing
search
initiating
Action: In suanary, the next action I will perfern is etttype [6] [blue iPhana]
Bunan feedback: That search likely will not vork vell clicking the Cell phones button and searching for the iPhone there on this classifieds site.*
Abstractions action before human feedback:
Abstraction Comments:
When searching for the most recently posted iten of a specific color, ensure the listings sorted by the navest first visually inspect the inages the color before clicking on the correct listing. Nhen searching for specific narrov down the search by selecting the appropriate category before typing in the search term.
classifieds website,
effective
Plan:
Let's think step- byop-by-step... The objective is to find the most recently pasted blue iPhone. The feedback indicates that should first clicking on the 'Cell phones' category, which has the ID [41]. After selecting the category,
dovn catagory
that category.
Sunnary: : To find the most recently posted blue iPhone, I need to first narrow dovn the search selecting the 'Cell phones' category.
Predicted State:
phones' category, page diaplay listings under that category, allowing for more targeted search for a blue iPhone.
clicking
Revise Action: In summary, the next action will perforn is ctellck [41] '..
S5.1.4 Relation of Abstraction Verification to DAgger
The human-in-the-lonn phase of ICAL bears conceptual resemblance the Dataset Aggregation (DAgger) algorithm [67], as both methods involve iterative refinement an agent's 's policy through interaction with expert feedback. However, ICAL extends this framework by incorporating natu- ral language feedback, updating both actions and abstractions, and utilizing retrieval-augmennte generation (RAG) with explicit memory of optimized examples for policy improvement.
In DAgger, the agent collects data by executing its current policy and then queries an expert to obtain the correct action for each encountered state. Specifically, at iteration the agent observes state s and takes an action a, = (s,))) according to its policy π.. The expert provides the optimal action at, and the agent aggregates this data into dataset D:
D DU ((st, a;)}.
(10)
The policy is then updated by minimizing loss function over
t++1 = arg min
Σ L(T(s)),,,).
(11)
(SDA))D
25

---

Similarly, in ICAL's human in--iin--t--ee thop phase Fui,, the agent refines its behavior based on human feedback. When the agent executes an optimized trajectory Eoptimized and encounters failure at action dz, human observer provides natural language feedback H(a,, 0t) concerning the action a, and the observation or. The agent integrates this feedback to update both the trajectory and the associated language abstractions:
(Coptiized: I updaate (Sopúmized: (a,; 0z), T, (e,,.,.....
(12)
where T. represents the current language annotations, 7 is the task instruction, and (et, retrieved examples from memory. This updated trajectory Soptimizeed and abstractions T'' are then added to an explicit memory ε, enhancing the agent's policy through enriched context:
e*) are
= (Eopttmizzed: I/))-
(13)
The agent's policy TICal is implicitly updated by conditioning this memory during action genera- tion:
TICAL (st. 8) VLM E).
(14)
where VLM denotes the Vision- LanguueL Model used for in-context learning.
The similarities between ICAL and DAgger lie in their iterative approach to policy refinement using expert feedback. However, ICAL offers several key benefits:
Natural Language Feedback: Unlike DAgger, which requires the expert to provide explicit action corrections dr ICAL accepts natural language feedback H (a,, 0z). This allows the human convey richer information, including explanations, suggestions, and contextual details that can address not only the immediate failure but also underlying misconceptions.
Revision of Actions and Abstractions: ICAL updates both the action sequence and the associated language abstractions L. By refining the abstractions, the agent enhances its understanding of task structures, causal relationships, and state changes, which promotes better generalization to new tasks and environments.
Policy Improvement via Retrieval-Angmennee Generation: ICAL maintains an explicit memory of optimized examples and abstractions. During deployment, the agent retrieves relevant examples from based on similarity measures and uses them as context for action generation. This retrieval- augmented generation (RAG) approach allows the agent to leverage past experiences effectively, adapting its policy without explicit parameter updates.
In contrast, DAgger relies solely on aggregating state- state-action pairs and updating the policy through supervised learning. which may not capture higher-level task structures facilitate transfer to new domains. ICAL's ability to process natural language feedback and to update both actions and abstractions provides more flexible and powerful framework for policy refinement, aligning more closely with human learning processes. ICAL extends the traditional imitation learning paradigm represented by DAgger, enabling more efficient and generalizable learning from human feedback.
S5.2 Deploying the ICAL agent after the examples have been learned.
We present our algorithm for deploying our ICAL agent after the examples have been learned on new instructions in Algorithm S2. We additional present our prompt template for the VLM planning after examples have been learned in Listing S7.
S5.3 Inverse Dynamics Model
In this section, we provide implementation details for the inverse dynamics model used in TEACh. A high-level architecture diagram is shown in Figure S3
Backbone. Given input image pair Timages 2xx3xxoxWo Wo (2 frames and 3 color channels), CNN produce lower-resolution activation maps f e R2xC' HxW where Ho and Wo denote the original height and width, respectively, and H and W represent the dimensions of the resulting feature map.
Transformer Encoder. The spatial features are input into a transformer encoder, where they undergo self-attention. We reshape the spatial dimensions into single dimension, resulting in feature
26

---

9: end for Algorithm S2 Deploying the ICAL agent after the examples have been learned.
Require:
Predefined action API based on skill set A. Set of in-context examples M = {e! ,...,, ,..,, e
Language instruction I.
Initial observation oo-
Initial textual observation Z0.
Maximum steps T.
1: Initialize observation Or + t I0.
2: for = to T 1 do
3: Retrieve top K examples: (ej ef (etrievcTopK(z Or. I, M.).
4: a, < (1, ot, I, ... et e A) to generate an action or Python code.
5: Execute action de to receive new observation 0++1, 2++1-
6: if stop criteria met then
7: break
8: end
map with dimensions x HW. Each layer of the encoder consists of multi-head self-attention mechanism and a feed-forward network. Fixed spatial positional encodings and learned frame encodings added to the inputs at each attention layer. The transformer encoder comprises six self-attention layers, utilizes eight heads, has an embedding size of 384, and contains six encoder layers.
Transformer Decoder: The decoder incorporates cross-attention mechanisms for both object and action queries with respect to the encoder features. It processes N queries simultaneously across its layers. The embeddings, comprising action queries and object queries, add learned positional encodings at the input of each attention layer. Each layer of the decoder includes cross-attention from the queries to the encoder features, self-attention among the query features, and a feed-forward network. The transformer decoder consists of six self-attention layers, employs eight heads, has an embedding size of 384, and includes six encoder layers.
After the decoder, we use feed-forward network scalar value. These scalar values for actions and objects are concatenated, creating final action logits for each action and each object. The model is trained using cross-entropy loss for both actions and objects, such as 'pickup' and 'apple'. Additionally, we introduce an extra query for 'no object' to accommodate actions that do not involve manipulating an object (e.g., move_forward().
reduce each embedding from dimension d to
Dataset. We use a random subset of 649 training episodes from the TEACh training dataset and 181 validation episodes from the TEACh validation seen dataset, which do not overlap with the episodes used for the ICAL example learning. We use Each episode contains a trajectory of observations and actions {oo, 0p. ... OT, or,,rr,r aT). We use each pair of observations (0r, ot+1). and the action responsible for the state transition o: - 0 as training samples for the network.
Implementation details. We use a learning rate of 2e - 5, batch size of 64, step learning rate scheduler with 7 0.1 and step size 30 epochs. We use early stopping based on validation loss and train for 45 epochs. We use cross entropy loss with a manual class weight rescaling based on frequencies in the training set. Training and model implementation done in PyTorch.
Applying Inverse Dynamics Model on held-out demonstrations. On the held out 250 TEACh episodes used for ICAL example learning, we feed each pair of observations the trained inverse dynamics model to predict actions. We convert the sequence of predicted actions into a Python program based on the ICAL action API for TEACh. This involves converting each action into Python function and aggregating contiguous navigation actions (move_fore forward(). mn__ee(() into a single go_to() function in the program. An example of the predicted Python program is shown in Listing S2. We also provide the fully revised program after running ICAL learning on the predicted program in Listing S3.
27

---

Listing S2: Demonstration program of actions inferred from the inverse dynamics model for an episode of making Salad.
fridge
Intaractiinibjject ("Fridgg"" object_.nsstace "Fridge*)
target__riide.
serget__fride..pp open
Interactiondbjett (""ettucce" abject_instance "Lettuce") target_lettuce
target_countertop Intersattinndjeettccoome ("Conntrrrr"""" object_.nntance "CounterTop")
http=_leetuccpcccppadd__.. (target_counterto))
target__knifa
("tetccnnnnbcct"tktee""" objact_instance "Knife")
tarkes..nitre.pccupp)
varget__ettuuee.o..tb
sarret_jetruceealice))
target_breaad
InteractionOhjett( (*Bread,,
"Bread")
srrget.bread.go.toa
target_bread.clice))
tarree_ccontertop.po..taa
target_tonatto InberaccionDDeect ('Toaato', object_instance "Tonsto")
surgtt.tnife.pickupl
target_breea.go.tto)
target_bread.sllced
srrret_coantertopggo.tooo
target_breadsliced Interaction(bject ("BreadSliced", object_instance None,
parent_object "Bread")
target_plate Incrracctnnaaaaaceeeeeeeeeo (*Platt"" objact_ instance "Plate")
targes_Dreadslieed..teee
target..pot - Interactionubeee: ("PPtt, object, - "Pot")
http:aaisceeeppccpu.aadd....ll (target.plata)
target__apoon
InteractienOhject ("Spoon,, ibject_inatance "Spoon")
"srget_spooa.picxupp
carres_.couterroo.oo.ttt
InteracticaDbject
("LettuceSliced", object_instance
parent__bbectt "Lettuce") parent
barge__eetuceeeleeppiioogg (ickpl))pee.ppppp()
sarget.plate.go_tor
(lacc
target_ togttottonatt InteractioaDbjert ("a ("TomatoSliced', object_instance "TomatoSliced")
target_ targot_tomatoslicn sickup_and_pal
28

---

Listing S3: Revised demonstration program (revised from the program in Listing S1) after abstraction phase cleanup and human- numan-in-the-loww in-the-loop for an episode of making salad.
Initialize Interactionúbjett instances for the fridge, lettuce, knife, tonato, and plate
rridge intteractioonnbnfeettfffff objact_ inttet__intttcc=""riccc
lettuce Iateracttoonbbbeett ("Lettuce" ooject..uetano--"ttttu...._
Lettuce in the "jjettinstacce"""httt.a.71" Knife on the countertop the table tonato (taaracctoonbjjecttTTattt ofjert_ # Plate on the countertop
fridge
knife Interactionbjjer
plate - aateractioalbjeei((PPllte object_ "iaeti_iattance="llttt...
sink_baein Iatoractiona(ject ("Siabaaeii
faucet Interaction(bject ("Paucet", object_instance- Faucet_ 87") # Faucet
Go the fridgo and open it retrieve the lettuce tridga.go.... tridge-go_to
pridge.open.
Retrieve
fridge, slice then, create
sliced
lettuce
lettuce. go_to
lettuce. jettuce.pickupl (ickup
inifa.go_tor
kniie.. (ickkp))
cestuce.slice)
lottuco__llced__1
InteractionDbject ("LettuceSliced", parent_ obsett.object-letuue
instanc)) Initialize nev aliced object fron sliced parent
lettuce_sliced_ InteractionObject (rrrroaddbeee(otttueeeseeee parent_object-letuuee object_instanca) Initialize sliced object fron slicad parent
table,
sliced
conato.go_tor
comato..
comato_slicet InteractionDbjact ("TomatoSliced parent_objactstomtt obj.entttooooo.......o.... instance)
Initialize new sliced object from sliced parent
Check if the plate is dirty and clean it if necessary plate go
if atecc.ttrrrerebb ("ditty,, True):
cink__bssin.go.tol
faucet. faucat.go_to()
faucet. soucat.toggle_on(
plate
faucet. touget..oggle_off # Turn off the faucet after cleaning up
plate plate.pickup() plate.chango_dttee ("dirty", False) Update the plate's state to clean
plate
Place two slices of lettuce and slice of tonato the clean plate lettuca_alicad_11 [tt lettuce_eliced_11 plate.go__to[]
lettuco_eliced_.ppp
Settuce__ll.cdd_2...ccpp.
tomato_eliced.
comato_eliied..g__tel [o_ttceedgg_ot()
tomato_sliced. pttp::eeeiieeeeppupp.pppp
comato_sliced. place(plate)
fridge
retrieving
fridge.
S5.4 LLM finetuning details
We use the examples obtained from the ICAL method applied in TEACh, total of 122 examples. We split the dataset randomly into 99 training samples and 23 validation samples. Input tokens for training consist of each example (instruction, object state, and API) with the prompt template used for zero-shot prompting- Output tokens consist of the Python program with abstraction comments for each example The mean input token length per sample is 3145.17, while the mean output token length per sample 432.82, We use the Azure OpenAl Service for fine-tuning, which uses the next-token prediction objective and LoRA [32] for parfmeter-effficiet finetuning of gpt-35-turbo-L10 1106.
29

---

Object
Frame
Decoder
prediction
Encoder
Action
prediction
Frame 2
Action tokens
Object
tokens
Figure S3: Architecture of inverse dynamics model used for labeling TEACh trajectories.
S5.5
Compute Resources
We single Nvidia RTX A6000 for training the inverse dynamics model and running all evaluations. We use Azure for finetuning GPT-3.5-1 106 as mentioned in Section S5.4. We use Azure OpenAl APT for VLM inference.
S6
Additional implementation details
S6.1
TEACh
The TEACh dataset builds on the Ai2thor simulation environment [40]. At each time step the agent may choose from the following actions: ForwardO, Backward(). Turn Left(, Turn Right(), Look UpO, Look Down() Strafe Left(), Strafe Right(, Pickup(X), Place(X), Open(X), Close(X), ToggleOn(X), ToggleOff(X), Slice(X), and Pour(X), where X refers an object specified via a relative coordinate (zr, y) the egocentric RGB frame. Navigation actions move the agent discrete steps. We rotate in the yaw direction by 90 degrees, and rotate in the pitch direction by 30 degrees. The RGB and depth sensors are resolution of 480x480, field of view of 90 degrees, and lie height of 0.9015 meters. The agent's coordinates are parameterized by single (rr, y, coordinate triplet with æ and corresponding movement in the horizontal plane and reserved for the vertical direction. The TEACh benchmark allows a maximum of 1000 steps and 30 API failures per episode.
S6.1.1 Planning at test time
Given a new environment and instruction, ICAL first maps out the scene to build a navigation map and detect objects and their attributes (see next sections). ICAL then retrieves the top-k examples relevant to the instruction and object state (see Section 3.4). ICAL then obtains the abstracted object state give the LLM (see Section 15.1 ICAL then prompts the LLM, given the instruction, abstracted object state, and retrieved in-context examples, to output Python code to carry out the new instruction in the environment. execution error and ask the LLM to revise the code.
code execution failures occur, we re-prompt the LIM with the
$6.1.2 ICAL differences with HELPER
In TEACh, we build on HELPER [68] for program execution. Here, we give an account of HELPER. HELPER prompts an LLM, namely GPT-4 [1], to generate plans as Python programs. assumes that the agent has access to a set of action skills S (e.g., go to (X), pickup (X), etc.). We use reduced set of these skills (e.g., we remove the cook(), clean(), and toast() primitives as we wish for our model to learn these). HELPER generates code that decomposed into these action skills. Instead of
30

---

demoposing them into action primitives, we run the Python code generated from the LLM directly (i.e., using the "exee' function in Python). Each action skill comes with set of pre-engineered pre-condition checks, which we also remove. HELPER maintains a 3D semantic map for navigation and keeping track of objects (see next sections).
$6.1.3 Obstacle map
ICAL maintains 2D overhead occupancy map of its environment E RR that it updates each time step from the input RGB-D stream. The map is used for exploration and navigation in the environment. At every time step information of the camera to obtain a 3D occupancy map registered to the coordinate frame of the agent, similar earlier navigation agents [8]. The 2D overhead maps of obstacles and free space are computed by projecting the 3D occupancy along the height direction at multiple height levels and summing. For each input RGB image, we run a SOLQ object segmentor [20] (pretrained on COCO [46] then finetuned on TEACh rooms) to localize each of 116 semantic object categories. For failure detection, we use simple matching approach from [57] to compare RGB pixel values before and after taking an action. When using ground truth perception, we use ground truth semantic segmentation, depth maps, object attributes, and action failure detection.
unproject the input depth maps using intrinsic and extrinsic
S6.1.4 Object location and state tracking
We maintain object memory as a list of object detection 3D centroids and their predicted semantic labels {, Y. Z[); & E (1...N)), = 1..K], where is the number of objects detected thus far. The object centroids are expressed with respect to the coordinate system of the agent, and, similar to the semantic maps, updated over time using egomotion. We track previously detected objects by their 3D centroid C E R We estimate the centroid by taking the 3D point corresponding to the median depth within the segmentation mask and bring it to common coordinate frame. We do a simple form of con-maximum suppression the object memory. by comparing the euclidean distance of centroids in the memory to new detected centroids of the same category, and keep the one with the highest score if they fall within a distance threshold.
For each object in the object memory, we maintain an object state dictionary with pre-defined list of attributes. These attributes include: category label, centroid location, holding, detection score, use, sliced, toasted, clean, cooked. For the attributes, these are initialized by sending the detected object crops in the abstracted state, defined by the detector mask, to the VLM model, and asking "Is this (category) (word]) or [word2) ? Provide only your answer, either "(word1) )) or "(o"(won "(word2)". and taking the answer the output attribute.
S6.2 VisealWebArena
The WisueeWebArena [37] builds on Web Arena [92] contains 910 evaluation instructions with three interactive websites: Classifieds, Reddit, and Shopping. At each time step, the agent obtains Set of Marks annotated image and and the webpage content, in textual format listing the button text with their set of marks ID. The set of marks bounding boxes and textual state are extracted from the HTML code for the current webpage. At each time step, the agent must select an action to carry out the instruction. The instruction includes natural language description and potentially one more reference images. The action space is as follows:
click [elem] Click on element elem.
hover [elem] Hover on element elem.
type [elem] [text] Type text on element elem.
press [key comb] Press key combination.
new tab Open new tab.
tab focus [index] Focus the i-th tab.
tab close Close current tab.
goto [url] Open url.
go back Click the back button.
31

---

forward Click the forward button.
scroll [upldown] Scroll up or down the page.
stop [answer] End the task with an optional output
S6.3 Additional details on ICAL agent deployment in VisualWebArena
At each time step, we retrieve the top-5 examples and prompt the model with the 5 in-context examples. Each in-context example consists of the image input, abstracted textual state, summary, step-by-step reasoning, predicted next state, abstraction comments, and predicted action. We use the Set of Marks (SoM) [84] representation for image inputs, implemented in Wesurra by [37]. An example in-context example is shown in Listing $4.
32

---

Listing S4: In-context example used in Visual WebArena Note that the webpage screenshot with SoM annotations for the in-context example is also provided the VLM, but is not displayed.
Input:
OBJECTIVE: I recall seeing this exact iten pillozs Household section site, add a conmant on its listing with the title "Connentary" and text "oae funky
OBSERVATION:
[4] [A] [Publish Ad]
[ [StaticText] [ Search results: pillove]
[8] [INPUT] []
[] [StaticText] [min.]
[15] [A] [Housebold]
[18] [A] []
[ [StaticText]
[Listings]
O [StaticTezt]
[Xorth Potomac (Karylsad)]
[] [StaticText]
[15.00 $1
O [StaticText]
[Pattery Barn Natine Drape panel)
20.00
Household
Lake Barcroft (Virginia) Beaut]
2023/11/11
O [StaticText] (100.00 $)
[33] [IMG] [Cenforter with Pillows, description: green quilt and pillove on the ground,
url: oc-content/uploode htwpp///////////45.... 2aload//126//22Sshhuubbaas thwnbassssaaasllleeeeo..ao png]]
[ [StaticText] [Roanoke (Virginia)]
0 [StaticText] [Canforter with Shans and Throw Pillow 711 7'3]
PREVIOUS ACTIONS:
1. type [INPOT] [] [pillous]
2. hover [SELECT] [Select a categor/aatiquestplaaaaeeee o parteBeauty
partslonptteraaleectoneee trucksCell phonestollectsilecccoptt..
click [SELECT] [Select atgerAArgee/lllaaeeerrre craftatute partaBeauty
BealtaBLikeBBassBooseeaar + trucksCell pho.ee/oolleccibleecccupuuuu
parssoapuerslletroiiceera
4. click [A] [Houschold]
5. type [INPUT] [] [pillovs]
click [BUTTON] [Apply]
7. acroll [doun]
[25] [A]
scroll [doun]
[2]
11. click [A] [Household]
12. scroll [dovn]
click [A] [2]
Output:
pillous conment. I' already aearched for pillous, filtered the Household category, and navigated through the pages. The iten I'm looking for is not visible on the current continue acroll proceed to add the comment with the specified text "How funky....
need
exact
"Connentary"
find Once locate
Sumnary:
Scrolling doun will allov me to find the exact pillows listing where I need to add comment, progreasing tozards
complation.
Predicted Next State: The next state will dieplay additional lietinge household itens, including pillow options, content that wа previously not visible.
page
Action: In suamary, the next action I sill perform is ertscroll [domn] tie
Abstraction comments:
1. When the end of page is reached and the task requires vieving more itens, navigate further.
page instead sttenpting 2. When posting conment on a classifieds vebsite, ensure that all required fields, such submitting comment.
are filled 3. vhen searching for itens in specific location on classifieds sebsite, it is important use location filter narros down the aearch resulta to the before spplying the filter.
ensil
desired
4. Vhen a comment has been typed into the appropriate field, the next step is to submit connent using provided subniseion button, rather the conment.
retyping editing
specific price range, minimum and naximum price filters before applying the search to narrox down the results.
zearching
6. vhen tasked with selecting specific iten in sequence, folloved based on the given instructions. In this case, the iten must be selected completing
the correct order is
clicked
33

---

When the objective leave with both title and that both fields are complated before submitting the connent.
S6.4 Ego4D
Ego4D is daily life activity video dataset of hundreds of scenarios. We focus on the long-term action anticipation task predict the future user actions given an RGB egocentric video. Models must choose from 115 verbs and a set of 478 nouns for action predictions. For evaluation, we take 100 seen validation videos that come from the same videos used for ICAL example learning but at different, unseen location, and separate 200 completely unseen validation videos for evaluation. We follow previous work and use edit distance as a metric, which is computed as the distance over sequences of predictions of verbs, nouns and actions. The goal of this measure is to assess performance
way which robust to some error the predicted order of future actions. GPT4V evaluations give image inputs annotated with DEVA tracking masks [12] with Set-of-Marks labels [84]. For in-context examples to GPT4V, we concatenate uniformly spaced video frames and give it as single image input. For the input video to GPT4V, we take 12 video frames uniformly spaced and provide 4 images each with concatenated frames. The supervised baseline uses Slow backbone with Transformer aggregator and trains on Ego4D V2 (243 video hrs) [28].
S6.4.1 Noisy Trajectories
100 demonstrations from validation set were abstracted using ICAL. Due to the passive nature of this task, we perform ICAL without the abstraction verification with human-in-the-loop phase, and only perform the VLM-driven Abstraction Generation (Section 3.2). 92 demonstrations (8 failed due to GPT4V filters) were successfully abstracted by ICAL for example set size of 92 for evaluation.
34

---

Listing S5: Prompt template for VLM abstraction generation phase
**Objactive:** Aa helpful assistant with expartise in {DOMAIN), your task is to produce useful abstractions and language
help
perfora
IIformaaioon Provided:**
You will receive:
KINPUT INFORNATION)
-*Output Format:::
Summary:
summary Summary:" and limit it to single line, no more than 6 sentences.
perforuing
Abstracted State:
slements pertorning and are important for the task. Refer to the elements their object ID, and for each element, description the object and and relevant attributes. Start the list with "Abstracted State:', and put each element that you choose new line.
relevant
Step-by-step Reasoning: Explain
step each step. Mention specific object numerical IDs when referencing objects. Start "Step-hy-step Reasoning:"
demonstration
reasoning
this section than 6 sentences.
to single line, no
4. Predicted State Change: Provide in natural language any relevant state changes objects and visusl elements that will take place due future actions. Remenber focus on state changes that will help soneone else perform the task.
language abstraction commente. such as causal abstractions, task abstractions, and other abstractions that will help soneone learn the task. Put each abatraction on nex line. Mention spacific object IDs vhen referencing objects.
Comments: Provide
Optimized Demonstration Script: Present any optimized actions for completing the task efficiently the denonstration script is already optinelly efficient and no revisions are needed.
environment. It possible that the provided
x Actioon Space**
(ACTION API}
-I Contert Examples:** (EXAMPLES)
=+Gudeelines:**
Follay these strict guidelines:
1. Adhere to the previously defined output format without deviating. Refer the examples provided for proper fornst.
2. Reason through each step nethodically, as shovn in examples. your reasoning it's relevant.
Reference objact/part
4. Your primary focus abould be on generating useful comments that will help someone else accurately perform
35

---

Listing S6: Prompt template for human in -h-thp revisions based on human feedback
cemman-in-the-aeee
+*Objective You are an autonomous intelligent agent tasked with {DOMAIR). Your prinary webaite based feedback so that the action successfully makes progress towards completing the task
goal
language corrective
Infornation Provided:*s
Here the information you'll have: CINPUT INFORMATION)
**Output Format:**
Explain: Why
complete
imply
what revisiona ahould be made to fix the error? This should be single line, and at nost six sentences.
Sumnary: Single-ling cumnary of vhat the proposed action will carry and how it sill nake progress tovards the objective.
Abstracted State:
relevant perforning and are important for the task. Refer to the elements by their object ID,
elament, description
objact of the demonstration, the reasoning for each
attribatea.
4. Step-by-step Reasoning: Explain each
step, and vhy the revised action vould make the nest genee.
5. Predicted State Change: Predict vhat the state will look like after taking the proposed revized action.
Correction Abstraction:
Provide numberad list of useful language abstraction comments, such causal abstractiona, taak abatractions, , other abatractions that will help someone learn line. Mention specific object IDs when referencing objects. Also, incorporate the correction inte some generslizable knowledge about the
the tack. Put each abstraction
why is mistake, and hox fix 7, Revised Action: Output the revised action to take from the actiens provided belov.
>* eeAction Space** (ACTION API}
-IIn-Contee Examples:** (EXAMPLES)
GrGudellnne::*
Follay these strict guidelines:
1. Adhere to the previously defined output format without deviating. Refer the examples provided for proper fornat.
2. Reason through each step nethodically, as shovn in examples. it's relevant.
your reasoning 4. Your primary focus abould be on generating useful comments that will help someone else accurately perform
Reference objact/part
36

---

Listing S7: Prompt template for VLM planning after examples are learned
-+Objective Aa helpful assistant with expertise in (DUMAIN), your task is to (DOMAIN TASK)
IIformaation Provided:*s
You will receive:
KINPUT INFORNATION)
-*Output Fornat:**
Sunnary:
you perforuing Start and limit it to single line. no more than 6 sentences.
summary
SSunnary
Abstracted State:
scene by deacription and pertinent attributea for each. Start the 11at with Abstracted State:', and put each elenent that you choose on new line.
objects
numerical IDs, providing
3. Step-by-see Reasoning: Explain each step of the task and the reasoning for each step Nention apecific object numerical IDs when referencing objects. Start this section "Step-by-step Reasoning:"
single 4. Predicted State Change: Provide in natural language any relevant state changes that throughout
5. Abstraction Commente: Provide aumbered list of useful language abstraction commenta, such as causel abstractions, task abstractione, and other abstractions that vill help soneone learn predict the future actions fron the egocentric rideo. Put each abstraction on nev line. Mention specific object numerical IDs when referencing objects.
6. Predicted Actions: Present the actions the agent should take to carry out the task.
eetction Space:**
(ACTION API)
-IInConttett Examples:**
(RETRIBVED EXAMPLES)
>*Guidelinea=**
Follov these strict guidelines:
1. Adhere to the previously defined output format without deviating. Refer to the examples provided for praper fornat.
2. Reason through each atep nethodically, as ahoun in examples. object/part
your reasoning
relevant.
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction outline the proposed method (ICAL), its objec- tives, the types of abstractions it handles, and the benchmarks used for evaluation, all of which detailed further in the paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the conclusion and Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
38

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide anonymized code and detail all implementation in main paper and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
39

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide link anonymized code which has README instructions for running our models and experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
Poblli//guiee/Codeeeee
wwwssssssollllloooe onPolicyl icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / wttpp.ccpppblliiipppp wwtpccccpccccceeccccccccccccttlt cebollccggidess.
ceabbmmmsssonPPe ssiiisolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all data splits and hyperparameters in the main paper and Appendix. We additional provide all data splits in our code release.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the deterministic nature of our models and cxisting resource constraints, we do report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
40

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide compute resources used for training and evaluation in the Ap- pendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tttus:/neurips... cctppblic//eeeecccccccccc wtwpai//eeehcchieeeelelleeeee
Answer: [Yes]
Justification: We adhere the NeurlPS Code of Ethics guidelines, ensuring that our research and practices meet the ethical standards set forth by NeurIPS.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we include section on societal impacts of the work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
41

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We will add safeguard agreement to our github repository when the code is publicly released.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We are the original creators of the assets used and give credit to previous work when building upon others' code or using external code or data.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
42

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithcoe thcode. com/coo/////ooooo conddataeet has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Our accessible website (anonymized) offers easy access to our results and code. Additionally, provide anonymized code that is well-doeumented with README files for setting up the models and env environments.
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not use crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not crowdsourcing human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
43

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
44


Reviews:
[{"review_id": "iwLDhlHayD", "reviewer": "Reviewer_VRBk", "summary": "The goal of this paper is to teach VLMs novel tasks by prompting VLMs to create multimodal abstractions for unfamiliar domains. Given instructions paired with noisy demonstration trajectories, this paper proposes a method to encapsulate the information from these into examples consisting of optimized trajectories paired with generalizable language abstractions. A VLM is specifically prompted to produce abstractions such as essential task steps, cause and effect generalizations, expected state changes, a step by step plan, and relevant parts of state information. The abstracted example includes an executable trajectory which is executed and potentially refined with a human in the loop via natural language feedback. These examples are then used for in context of a VLM to improve task performance.\nThe paper includes experiments on 3 datasets - TEACh, VisualWebArena and Ego4D, comparing respectively to the HELPER model - a prior SoTA model for TEACh, and GPT4V for VisualWebArena and Ego4D. The most improvement in seen in VisualWebArena, followed by partial success on TEACh. On Ego4D, performance is on par with supervised learning methods that use more data but Zero shot CoT with GPT4V performs better.", "questions": "It is possible I missed this in the paper, but I how was the human in the loop feedback obtained for the TEACh experiment? Given that the checklist says no human subjects were recruited, I assume this feedback was provided by authors. Is this correct?", "limitations": "The authors include a discussion of limitations and potential negative social impact.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The overall goal in this paper of using VLMs to distill demonstrations into canonical examples is an interesting combination of currently popular foundation models with more classical ideas such as case based learning.\nThe paper includes experiments on multiple benchmarks with sufficiently different tasks to suggest generalizability of the proposed method. It is also refreshing to see that the authors do not hesitate to include the unusually good performance of Zero shot GPT4V on Ego4D which might be seen as a negative result.\nGiven the topic of the paper, a number of engineering details and information about prior work is needed to fully understand the paper and increase the likelihood of reproducibility of the work. The authors have made a very strong attempt at this with a detailed appendix.", "weaknesses": "While the appendix goes a long way towards reducing this, it is likely that the paper might still be difficult to follow for readers unfamiliar with the datasets and prior methods referenced in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**While the appendix goes a long way towards reducing this, it is likely that the paper might still be difficult to follow for readers unfamiliar with the datasets and prior methods referenced in this paper.** \n\nThank you for your feedback. We acknowledge that the paper may still be challenging for readers unfamiliar with the datasets and prior methods. To address this, we will provide more detailed explanations on dataset processing and implementation in the appendix in the camera-ready version and the README in the code repository.\n\n>**It is possible I missed this in the paper, but how was the human in the loop feedback obtained for the TEACh experiment? Given that the checklist says no human subjects were recruited, I assume this feedback was provided by authors. Is this correct?** \n\nYes, the authors and their lab colleagues provided the human-in-the-loop feedback in the paper. We did not recruit participants due to resource constraints, specifically the costs associated with recruiting participants for running ICAL and ablations. During this phase, we translated task progress into natural language feedback during failures, including failed actions (e.g., “The toaster is full right now.”) and missed task steps (e.g., “The pillow needs to be put onto the sofa”). The feedback provided did not require domain expertise, such as focusing on precise coding or low-level action changes. For example, typical feedback from VisualWebArena includes comments like, \"This does not have 2 upvotes and includes a meme of three Spider-Men, a flag of the Netherlands, and a flag of Croatia. You should scroll down to see more posts.\" In TEACh, typical feedback is of the form, \"The sink is full right now. Empty the sink before interacting with it.\" We will make these details more clear in the main paper.\n\n>**Performance comparison on Ego4D.** \n\nTo clarify, zero-shot CoT with GPT-4V shows strong results, but ICAL with GPT-4V outperforms it. Specifically, ICAL achieves lower edit distances relative to GPT-4V zero-shot COT: increases in performance by 9.9 for verbs, 10.0 for nouns, and 4.0 for actions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uoXYKvdeZ8", "reviewer": "Reviewer_ih5w", "summary": "The paper proposes ICAL, In-Context Abstraction Learning, which builds a memory of suboptimal experiences that are abstracted into states and plans, as well as correction and reflection from human feedback. The approach is based on extensive prompting to elicit structured representations from past experiences, and apply RAG at test time. ICAL exhibits improvement in success rates across virtual domains compared to raw VLMs and CoT methods.", "questions": "Could you further discuss the limitations of the work? I think currently the paper lacks thorough discussions on the limitations. For example, what are the common failure modes? And is there a viable and scalable solution to it potentially?", "limitations": "The overall prompting strategy is quite sensitive to the model performance. The fine-tuning improvement is quite limited as discussed in 4.6.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea of using extensive prompting to generate structured representation for RAG is clean and intuitive, and the authors clearly detail the steps involved. Using RAG enables the overall pipeline to improve continually.", "weaknesses": "My concern with the paper is the limited contribution in terms of the approach. It is a relatively simple adaptation of previous methods such as CoT, ReACT, Socratic Model [1] as well as RAG.\n\n[1] Zeng, Andy, et al. \"Socratic models: Composing zero-shot multimodal reasoning with language.\" arXiv preprint arXiv:2204.00598 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Clarifying the novel contributions and advancements beyond existing methods, such as CoT, ReACT, Socratic Model, or RAG.**\n\nWhile our framework incorporates chain of thought prompting from CoT, interleaves reasoning and acting as seen in ReACT, predicts plans over pretrained modules similar to Socratic models, and uses retrieval-augmented generation of action plans inspired by RAG methods, it is distinct in its primary contribution: optimizing examples for improved in-context learning. We leverage VLMs to create abstractions for examples, improving their utility as RAG inputs. To the best of our knowledge, this is the first work to do this. Specifically, we introduce a novel two-stage approach that refines each example and generates four types of generalizable knowledge, enabling rapid adaptation of multimodal agents with few demonstrations. Our results, in Tables 1-3, demonstrate that both stages are crucial for high performance, significantly outperforming CoT and RAG, which use raw or handwritten in-context examples. Additionally, Figure 5 illustrates our method's support for continual learning without forgetting.\n\nFurthermore, our optimized in-context examples are applicable to various multimodal agents across three complex domains: long-horizon instruction-following for household robots, visually-grounded web navigation, and action forecasting from real-world egocentric videos. Reviewers vg3K and VRBk note that these experiments on \"sufficiently different tasks... suggest the generalizability of the proposed method.\" Previous works have been limited to text-based environments, gaming scenarios, or single real-world domains. To our knowledge, we are the first to show robust results across these three diverse and complex domains.\n\nWe position our approach as a superset that incorporates the strengths of previous VLM agent works, while introducing a novel focus on optimizing in-context learning examples. As Reviewer VRBk puts it, ''the overall goal of using VLMs to distill demonstrations into canonical examples is an interesting combination of foundation models with classical ideas such as case-based learning.'' Our approach can be likened to in-context policy improvement with a multi-modal LLM actor, where instead of aiming to maximize rewards through specific reasoning strategies explored in previous works, it refines and optimizes in-context examples.\n\n>**The overall prompting strategy is quite sensitive to the model performance. The fine-tuning improvement is quite limited as discussed in 4.6.**\n\nWe apologize for not clarifying these points in our paper. Specifically, we show that fine-tuning CoT-prompted GPT-3.5 significantly improves performance, doubling its success rate from 11.8% to 23.2%. Additionally, incorporating retrieval-augmented generation (RAG) with fine-tuning (using ICAL examples as memory) provides further improvement, resulting in our best-performing agent. This agent outperforms the non-finetuned memory-augmented CoT-prompted GPT-3.5 by 4.9% in goal-condition success.\n\nWe view weights and external memory of abstractions as two forms of memory with distinct benefits. Weight fine-tuning requires many examples, while RAG can learn from a single example. In scenarios with limited data (150 examples or fewer), external memory updates and RAG use data more efficiently and remain competitive with weight fine-tuning. This is relevant to our study as our considered domains fit this category.\n\n>**Discussion on the limitations of our work.**\n\nWe will expand our limitation section per your request. Below, we include additional limitations and error modes of our agent with specific examples. We will include this discussion in the main paper.\n\n1. **Visual Grounding Limitations of GPT-4V** \nWhile ICAL improves performance in visually grounded tasks, errors persist due to the base VLM's limitations in visual grounding. For example, the agent fails to identify colors accurately, leading to errors like selecting the wrong item. This issue is evident in the \"reddit_2\" and \"reddit_9\" web tasks, where the agent navigates to incorrect posts, showing a failure to match images to webpage tabs. The agent also occasionally struggles with pre- and post-condition recognition. We found that in-context multimodal examples helped grounding in cases where grounding elements were unique, but easily identifiable but continue to fail in fine-grained cases. This grounding limitation has been noted in previous work as well [1]. We expect improvements with more data, added grounding objectives [2], and fine-grained image annotations [3]. In future work, we plan to extend ICAL for better fine-grain language grounding using multimodal retrieval, building on methods like ViperGPT [4].\n\n2. **Fine-grained in-context planning failures**\nDuring learning phases, ICAL may not fully acquire all necessary information for test time, resulting in failures. For instance, in the \"shopping_3\" web task, where the instruction is to display the most expensive red controller from the \"PS4 accessories\" category, the agent navigates to the PS4 category but fails to access the accessories subcategory. This demonstrates a lack of understanding of website structures and navigation. This limitation suggests that agents need to recognize when their knowledge base is insufficient and query the user for missing information. We plan to address active querying during testing in future work.\n\n**References**\n\n[1] Zheng et. al. GPT-4V(ision) is a Generalist Web Agent, if Grounded.\n\n[2] Ma et al. (2024). Groma: Localized Visual Tokenization. arXiv:2404.13013.\n\n[3] Garg et al. (2024). ImageInWords: Hyper-Detailed Image Descriptions. arXiv:2405.02793.\n\n[4] Suris et. al. (2023) ViperGPT: Visual Inference via Python Execution for Reasoning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KgvMxUaYcK", "reviewer": "Reviewer_vg3K", "summary": "The paper proposes a pipeline for Large Language and Vision-language models (LLMs and VLMs) to digest and learn from sub-optimal demonstrations and human feedback. The LLM/VLM, given sub-optimal task demonstrations, is prompted to produce abstractions of the trajectory (including task and causal abstractions, state changes, task decomposition and subgoals, and state abstractions), and possibly refine the given trajectory with insight from these abstractions. Optimally, these produced abstractions can be further refined by interactions with humans. Experiments show that the proposed framework greatly surpasses zero-shot CoT baselines and is competitive with/slightly better than having expert demonstations.", "questions": "- Can we easily design a reward function that specifies how sub-optimal a trajectory is? If we provide these sub-optimal trajectories and their corresponding rewards to the LLM/VLM, can it use this reward information to further refine the trajectories? The \"in-context RL\" capability of LLMs are previous studied in works like [1].\n- What is the difference between expert and non-expert feedback (mentioned in lines 281-282)? Examples with qualitative differences can be very informative.\n\n[1] Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf. https://arxiv.org/abs/2309.04658", "limitations": "The authors fail to include sufficient details on the human-in-the-loop fine-tuning phase. The evaluation might be biased if the human feedback provider is very familiar with the task and the LLM; they might be able to provide much more effective feedback than non-experts. If this is the case, then the amount of training required for the human to provide more effective feedback is worth investigating.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors provide sufficient details with their experiment environments and experiment setup, as well as the code, for readers to better understand and reproduce the results.\n- The paper proposes a novel method for LLMs to autonomously refine sub-optimal trajectories and build a high-level abstract understanding of the task based on sub-optimal trajectories. For very complex tasks, these high-level abstract descriptions have the potential to help humans better understand the task and the general strategies to accomplish the task.\n- The experiments span three different types of environments, demonstrating the generality of the proposed method. The proposed method also achieve good experimental results over the baseline methods.", "weaknesses": "- The experiment result focuses on the accuracy of the method; not enough comparison is provided in terms of the efficiency. For example, whether is might be more efficient for the human feedback provider to directly edit and improve the sub-optimal trajectories and provide the human-refined ones to the LLM/VLM?\n- The scaling capability of the proposed method is unclear. In the proposed method, each trajectory needs to go through the human-in-the-loop fine-tuning process which seems quite inefficient. For simple tasks in the TEACh benchmark the method needs ~100 trajectories to perform well according to Figure 5. For more complex tasks it might be less sample efficient and difficult to scale.\n- It also requires the environment to automatically reset itself, which is another limitation.\n- Some text in figures 1 and 2 are too small and difficult to read and understand. A more illustrative example of what the task is, what are the inputs/outputs, and what kind of information is included in the abstracted state is desired.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Efficiency of ICAL method?  For simple tasks in the TEACh benchmark the method needs ~100 trajectories to perform well according to Figure 5. Scaling capability of the human-in-the-loop phase?** \n\nActually, these roughly 100 trajectories are used for all tasks in TEACh combined. In fact, each task has on average 9.67 tractories in the memory after ICAL learning. The TEACh dataset includes 12 complex, long-horizon task types, averaging 67 steps for the shortest task (water plant) and 359 for the longest task (breakfast) for a human expert to complete. This complexity is significantly greater than other popular benchmarks like ALFRED, which averages only 50 steps across all tasks for experts. Despite the complexity, ICAL only requires a few *noisy* trajectories for each task. This is 6x less in-domain data than the strongest behavior cloning baseline (E.T. [1]), and achieves 21X the success rate, while also using noisy demonstrations. In VisualWebArena and Ego4D, which also represent high complexity, the number of demonstrations used for ICAL is of a similar magnitude (~100). \n\n**Importantly, our agent becomes increasingly efficient over time, requiring less human feedback and fewer environment interactions as it processes more examples.** By retrieving past successful abstractions during the VLM-abstraction making and human-in-the-loop phases, it uses previously stored knowledge to help abstract new examples. **As shown in Figure R1, for the second half of examples processed, the model requires significantly fewer environment steps (436±88 vs. 267±43, p=0.0143) and human feedbacks per episode (0.74±0.17 vs. 0.21±0.08, p=0.0089). This demonstrates that retrieving abstracted examples during abstraction learning reduces both human effort and environment interaction over time.** Consequently, using previously stored ICAL examples not only improves test performance but also accelerates learning for future examples.\n\nIt is essential to note that the human feedback is sparse, provided in natural language, and required only a few times for each task. In VisualWebArena, our agent needs an average of just 5.36 natural language feedbacks per example, with each feedback taking less than 15 seconds to create. In TEACh, our agent needs just 1.516 average feedbacks per episode with an average length of 18 words per natural language feedback. In fact, natural language feedback in ICAL could be easily communicated via speech by a human while observing the agent, whereas low-level coding and abstraction writing communication would be significantly more laborious. In order to hand-write GPT4-V-length abstractions, this would require 202.62 words on average for each example in VisualWebArena and 48-107 lines of text for each example in TEACh, including the precise programming of actions or code. \n\n>**Can we easily design a reward function that specifies how sub-optimal a trajectory is? Can the LLM/VLM use this reward information to further refine the trajectories?**\n\nThank you for this interesting suggestion. Based on your comment, we prompted GPT-4 with the sub-optimal trajectory and retrieved examples from memory, and asked the model to assign an \"optimality\" score from 1 to 5 with a reflection explaining the score. This score and reflection were then given to the VLM during the abstraction generation phase. After running this experiment for 40,000 steps, we observed no significant difference in the number of tasks successfully completed (22 with the score and 24 without the score).\n\nWe hypothesize that this is because the VLM already performs this evaluation implicitly. This is supported by Figure R1, where in-context examples reduce the need for human feedback and environment interaction over time. This demonstrates that the VLM effectively utilizes the provided examples during the learning phases to revise the trajectory, inferring sub-optimality without explicit scoring.\n\n>**What is the difference between expert and non-expert feedback? More details on the human-in-the-loop fine-tuning phase.**\n\nThe authors and their lab colleagues provided the human-in-the-loop feedback in the paper. We did not recruit participants due to the costs associated with recruiting participants for running ICAL and ablations. However, the authors have no more familiarity with the websites used in VisualWebArena than a typical web user and did not provide expert feedback, meaning it did not focus on precise coding or low-level changes. We will make these details more clear in the main paper. For example, typical feedback from VisualWebArena includes comments like, \"This does not have 2 upvotes and includes a meme of three Spider-Men, a flag of the Netherlands, and a flag of Croatia. You should scroll down to see more posts.\" In TEACh, typical feedback is of the form, \"The sink is full right now. Empty the sink before interacting with it.\" \n\n>**It also requires the environment to automatically reset itself.**\n\nICAL improves learning efficiency as more examples are learned, reducing human effort *and number of environment resets* over time (Figure R1). Our approach also minimizes the need for extensive resets required to train reinforcement learning (RL) or large data collection for behavior cloning. Despite this, some resetting is necessary and we will add this to the limitations of the method.\n\n>**Some text in figures 1 and 2 is too small and difficult to read and understand. A more illustrative example of the task, inputs/outputs, and abstracted state is desired.**\n\nThank you for the feedback. We will increase the font size in Figures 1 and 2. Additionally, we will include an extra figure to clearly illustrate the task, inputs, outputs, and the abstracted state. This figure will specifically show the exact inputs (e.g., images, text) for VisualWebArena and the corresponding outputs from the abstraction generation phase.\n\n**References**\n\n[1] Pashevich et. al. (2021). Episodic Transformer for Vision-and-Language Navigation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OVIneumPra", "reviewer": "Reviewer_foCb", "summary": "This work proposed ICAL, aims to improve decision-making in large language and vision-language models by generating optimized trajectories and language annotations from noisy demonstrations and human feedback. ICAL abstracts noisy trajectories into optimized sequences with language comments, refined through human feedback during execution. The experiment shows ICAL significantly improves performance in benchmarks like TEACh, VisualWebArena, and Ego4D, surpassing state-of-the-art methods.", "questions": "The VLM-driven Abstraction Generation component appears to heavily rely on the performance of GPT-4V. There is no ablation study for replacing GPT-4V. If such results exist and I missed them, please indicate where they can be found.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1、Improved Performance: ICAL significantly enhances decision-making and task success rates across various benchmarks, such as TEACh, VisualWebArena, and Ego4D.\n\n2、Reduced Reliance on Expert Examples: The method minimizes the need for expert-crafted examples by generating useful abstractions from sub-optimal demonstrations and human feedback.\n\n3、Versatility: ICAL is effective in multiple domains, including dialogue-based instruction following, multimodal web tasks, and video action anticipation.\n\n4、Human Feedback Integration: The approach incorporates human feedback to refine and adapt abstractions, improving the agent’s performance over time.", "weaknesses": "1、The method still relies on human feedback for refining abstractions, which may not always be feasible or scalable.\n\n2、The process of generating and refining abstractions heavily rely on GPT4-V\n\n3、The effectiveness of ICAL is constrained by the capabilities of the underlying Vision-Language Models (VLMs), such as GPT-4V.\n\n4、The method’s performance can be affected by the quality of the initial noisy demonstrations and the accuracy of human feedback.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**ICAL's effectiveness is constrained by the capabilities of VLMs like GPT-4V. The VLM-driven Abstraction Generation relies heavily on GPT-4V. Is there an ablation study for its replacement?**\n\nWe appreciate the comment regarding the reliance on GPT-4V for the VLM-driven Abstraction Generation component. We acknowledge the importance of evaluating alternative models and have addressed this in our work.\n\nIn Table S2 of the Appendix, we provided an ablation study using GPT-3.5 for the VLM-driven Abstraction Generation on the TEACh dataset, where visual experiences are converted to textual format. The results indicate a performance drop when using GPT-3.5 for ICAL Abstraction Generation, with the number of successfully revised trajectories reduced by more than half compared to GPT-4V (52 versus 122 tasks successfully completed). In Section S3.4 of the Appendix, we demonstrate how relabeling unsuccessfully refined trajectories can help bridge this performance gap when using weaker models like GPT-3.5, thereby reducing the dependence on GPT-4V. We will highlight these findings more prominently in the main paper to ensure their visibility. At the time of our experiments, no open-source VLMs with in-context multimodal generation capabilities comparable to GPT-4V were available. Per your comments, we implemented LLaVa-NeXT for the abstraction phase in VisualWebArena given multimodal inputs. As shown in Figure R2, we found that it failed to properly revise the abstractions to take into account the feedback. However, we are committed to continuing our experiments with emerging open-source VLMs as they become available. \n\n>**The method still relies on human feedback for refining abstractions, which may not always be feasible or scalable.**\n\nIt is essential to note that this feedback is sparse, provided in natural language, and required only a few times for each task. In the Visual Web Arena, our agent needs an average of just 5.36 natural language feedbacks per example, with each feedback taking less than 15 seconds to create. In TEACh, our agent needs just 1.516 average feedbacks per episode with an average length of 18 words per natural language feedback. In fact, natural language feedback in ICAL could be easily communicated via speech by a human while observing the agent, whereas low-level coding and abstraction writing communication would be significantly more laborious. In order to hand-write GPT4-V-length abstractions, this would require 202.62 words on average for each example, including the precise coding of actions. \n\n**Additionally, our agent becomes increasingly efficient over time, requiring less human feedback and fewer environment interactions as it processes more examples.** By retrieving past successful abstractions during the VLM-abstraction making and human-in-the-loop phases, it uses previously stored knowledge to help abstract new examples. **As shown in Figure R1, for the second half of examples processed, the model requires significantly fewer environment steps (436±88 vs. 267±43, p=0.0143) and human feedbacks (0.74±0.17 vs. 0.21±0.08, p=0.0089) per episode. This demonstrates that retrieving abstracted examples during abstraction learning reduces both human effort and environment interaction over time.** Consequently, using previously stored ICAL examples not only improves test performance but also accelerates learning for future examples.\n\nFurthermore, the feedback provided does not require specialized expertise. For example, typical feedback from VisualWebArena includes comments like, \"This does not have 2 upvotes and includes a meme of three Spider-Men, a flag of the Netherlands, and a flag of Croatia. You should scroll down to see more posts.\" In TEACh, feedback might be, \"The sink is full right now. Empty the sink before interacting with it.\" Such straightforward feedback significantly reduces the complexity and cost of data collection. It also often provides more context for the model to create generalizable abstractions. For example, the sink being full is not directly necessary for correcting the mistake, but will help the agent learn generalizable knowledge via the generated abstractions. \n\nOur approach additionally minimizes the need for detailed annotations or extensive interactions required to train reinforcement learning (RL) or behavior cloned agents for long-horizon, multimodal tasks. This method not only shortens the data collection process but also ensures minimal human expertise and interaction, making it a scalable and cost-effective alternative.\n\n>**The method’s performance can be affected by the quality of the initial noisy demonstrations and the accuracy of human feedback.**\n\nWe acknowledge the concern regarding the impact of the quality of initial noisy demonstrations and the accuracy of human feedback on our method. Our method is designed to handle and successfully revise even very noisy or incorrect demonstrations.\n\nFor instance, Listings S2 and S3 in the Appendix of the paper illustrates a very noisy code trajectory that was successfully corrected by ICAL. While it is true that ICAL may not always recover from extremely noisy demonstrations or feedback, such cases are typically filtered out or relabeled, as discussed in step 6 of Section 3.3 in the main paper.\n\nThis challenge is not unique to our method; gradient-based methods also suffer from inaccurate examples that can lead to incorrect model updates. As suggested by reviewer vg3K, one viable solution to this could be to utilize a reward model to identify and remove misleading or extremely noisy demonstrations before processing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers' positive feedback. Reviewer VRBk commended our innovative combination of foundation models with case-based learning, robust generalizability across benchmarks, and transparency in results. Reviewer ih5w highlighted ICAL's improved success rates across domains, and our clean, intuitive structured representation. Reviewer vg3K noted our framework's advancements over zero-shot CoT baselines, detailed experimental setup aiding reproducibility, and potential for helping humans understand complex tasks. Reviewer foCb noted ICAL's superior performance across various domains and continual integration of human feedback to improve over time.

We address overall concerns in this global response and specific reviewer concerns in their respective responses.

>@Reviewers vg3K, foCb: **How efficient is the human-in-the-loop phase? Is it scalable?** 

**Our agent becomes increasingly efficient over time, requiring less human feedback and fewer environment interactions as it processes more examples.** By retrieving past successful abstractions during the VLM-abstraction making and human-in-the-loop phases, it uses previously stored knowledge to help abstract new examples. **As shown in Figure R1, for the second half of examples processed, the model requires significantly fewer environment steps (436±88 vs. 267±43, p=0.0143) and human feedbacks (0.74±0.17 vs. 0.21±0.08, p=0.0089) per example. This demonstrates that retrieving abstracted examples during abstraction learning reduces both human effort and environment interaction over time.** Consequently, using previously stored ICAL examples not only improves test performance but also accelerates learning for future examples.

It is additionally essential to note that this feedback is sparse, provided in natural language, and required only a few times for each task. In VisualWebArena, our agent needs an average of just 5.36 natural language feedbacks per example, with each feedback taking less than 15 seconds to create. In TEACh, our agent needs just 1.516 average feedbacks per episode with an average length of 18 words per natural language feedback. In fact, natural language feedback in ICAL could be easily communicated via speech by a human while observing the agent, whereas low-level coding and abstraction writing communication would be significantly more laborious. In order to hand-write GPT4-V-length abstractions, this would require 202.62 words on average for each example in VisualWebArena and 48-107 lines of text for each example in TEACh, including the precise programming of actions or code. 

>@Reviewers foCb, ih5w: **Does the method rely on GPT4V? How sensitive is it to the prompting strategy?**

In Table S2 of the Appendix, we provided an ablation study using GPT-3.5 for the VLM-driven Abstraction Generation on the TEACh dataset, where visual experiences are converted to textual format. The results indicate a performance drop when using GPT-3.5 for ICAL Abstraction Generation compared to GPT4V (52 versus 122 tasks successfully completed). In Section S3.4 of the Appendix, **we demonstrate how relabeling unsuccessfully refined trajectories can help bridge this performance gap when using weaker models like GPT-3.5, thereby reducing the dependence on GPT-4V.** We will highlight these findings more prominently in the main paper to ensure their visibility. 

Per your comments, we implemented LLaVa-NeXT for the abstraction phase in VisualWebArena given multimodal inputs. As shown in Figure R2, we found that it failed to properly revise the abstractions to take into account the feedback. However, we are committed to continuing our experiments with emerging open-source VLMs as they become available.

We further show in Section 4.6 that fine-tuning CoT-prompted GPT-3.5 significantly improves performance, doubling its success rate from 11.8% to 23.2%. Additionally, incorporating retrieval-augmented generation (RAG) with fine-tuning (using ICAL examples as memory) provides further improvement, resulting in our best-performing agent. This agent outperforms the non-finetuned memory-augmented CoT-prompted GPT-3.5 by 4.9% in goal-condition success. In scenarios with limited data (150 examples or fewer), external memory updates and RAG use data more efficiently and remain competitive with weight fine-tuning.

>@Reviewers ih5w: **How does this differ from CoT, ReACT, Socratic Model, and RAG?**

While our framework incorporates CoT, interleaves reasoning and acting as seen in ReACT, predicts plans over pretrained modules similar to Socratic models, and uses RAG for inference, it is distinct in its primary contribution: optimizing examples for improved in-context learning. We introduce a novel two-stage approach that optimizes examples for improved in-context learning. To the best of our knowledge, this is the first work to do this. As Reviewer VRBk puts it, ''the overall goal of using VLMs to distill demonstrations into canonical examples is an interesting combination of foundation models with classical ideas such as case-based learning.'' Our results, in Tables 1-3, demonstrate that both stages are crucial for high performance, significantly outperforming CoT and RAG, which use raw or handwritten in-context examples. Additionally, Figure 5 illustrates our method's support for continual learning without forgetting.

Furthermore, our optimized in-context examples are applicable to various multimodal agents across three complex domains: long-horizon instruction-following for household robots, visually-grounded web navigation, and action forecasting from real-world egocentric videos. Reviewers vg3K and VRBk note that these experiments on "sufficiently different tasks... suggest the generalizability of the proposed method." Previous works have been limited to text-based environments, gaming scenarios, or single real-world domains. To our knowledge, we are the first to show robust results across these three diverse and complex domains.

Meta-review: This work proposes a technique that leverages large-scale models to generate experience abstractions (or canonical examples) from sub-optimal demos and feedback from humans, bringing ideas of case-based learning and foundational models together. This, in turn, helps adapt the agent to perform better across a range of benchmarks/tasks -- Teach, Visual Web Arena, and Ego4D.

The committee highlighted several strengths, particularly the novelty of the method. The committee found the paper effectively demonstrates broad utility, as empirically shown across three domains of tasks. This work has a high impact as an alternative pathway to the reliance on labels/demos from experts.

The discussion phase led to further clarifications that helped solidify the committee's positive outlook on the work. The AC concurs with the reviewers' positive assessment and recommends acceptance. The authors are encouraged to utilize their constructive and detailed discussion with reviewers to further raise the quality of this submission.
Decision: Accept (spotlight)