Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake?
Authors: Yuwei Bao
Keunwoo Peter Yu
Yichi Zhang
Shane Storks
Itamar Bar-Yossef
Alex de la Iglesia
Megan Su
Xiao Lin Zheng
Joyce Chai
Venue: EMNLP 2023 Findings (2023)
Keywords: Visual Task Guidance
Multimodal
Large Language Model
Foundation Model
TL;DR: A new benchmark dataset for multimodal task guidance generation with natural dialog and spontaneous mistakes

Abstract: Despite tremendous advances in AI, it remains a significant challenge to develop interactive task guidance systems that can offer situated, personalized guidance and assist humans in various tasks. 
These systems need to have a sophisticated understanding of the user as well as the environment, and make timely accurate decisions on when and what to say. To address this issue, we created a new multimodal benchmark dataset, Watch, Talk and Guide (WTaG) based on natural interaction between a human user and a human instructor. We further proposed two tasks: User and Environment Understanding, and Instructor Decision Making. We leveraged several foundation models to study to what extent these models can be quickly adapted to perceptually enabled task guidance. Our quantitative, qualitative, and human evaluation results show that these models can demonstrate fair performances in some cases with no task-specific training, but a fast and reliable adaptation remains a significant challenge. Our benchmark and baselines will provide a stepping stone for future work on situated task guidance.

Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake?
Yuwei Bao', Keunwoo Peter Yu', Yichi Zhang', Shane Storks!, Itamar Bar-Yossef', Alexander De La Iglesia', Megan Su', Xiao Lin : Chai TUniversity of Michigan, $Syracuse University
Abstract
that can be easily generalized any arbitrary task without prior exposure, given task manual or one demonstration. This requires the model to have a robust knowledge base and in-context learning abilities to easily pick up new task to guide the human through.
Despite tremendous advances in AI, it remains a significant challenge develop interactive task guidance systems that can offer situated, personalized guidance and assist humans in ious tasks. These systems need to have so- phisticated understanding the user as well the environment, and make timely accurate decisions on when and what to say. To ad- dress this issue, we created a new multimodal benchmark dataset, Watch, Talk and Guide (WTaG) based on natural interaction between human and human instructor. We fur- ther proposed two tasks: User and Environment Understanding, and Instructor Decision Mak- ing. We leveraged several foundation models to study to what extent these models can be quickly adapted to perceptually enabled task guidance. Our quantitative, qualitative, and human evaluation results show that these mod- els can demonstrate fair performances in some cases with no task-specific training, but fast and reliable adaptation remains a significant challenge. Our benchmark and baselines will provide stepping stone for future work on situated task guidance.
Traditional approaches to develop AI agent for interactive task guidance like this require large amount of task-specific training rules to recog- nize object states (Gao et al., 2016), mistakes in actions (Du al., 2023), and interact with hu- mans (Wu al., 2021), thus limiting their ability to generalize. However, the recent rise of foun- dation models trained on large amount of mul- timodal data from the web (Brown et al., 2020; Radford et al., 2021b; Alayrac et al., 2022; Li et al., 2022, 2023) creates new opportunities for develop- ing robust open-domain Al agents for this problem. These works have undergone paradigm shift and begun explore the zero- and few-shot adaptation of these models to various embodied Al problems (Khandelwal et al., 2022; Huang et al., 2022; Ahn et al., 2022; Kapelyukh et al., 2023).
In this work, we extend this paradigm shift by examining the application of recent state-of-the- art foundation models to situated interactive task guidance. We created Watch, Talk and Guide (WTaG), new multimodal benchmark dataset which includes richly annotated loman-hummn dia- log interactions, dialog intentions, steps, and mis- takes to support this effort. We define two tasks: (1) User and Environment Understanding, and (2) Instructor Decision Making to quantitatively and qualitatively evaluate models' task guidance per- formance. With the inherent complexity of the problem itself, this dataset can help rescarchers understand the various nuances of the problem in the most realistic human-human interaction set- ting. We used a large language model (L..M) as the backbone for guidance generation, and explored three different mutimodal methods to extract vi-
1 Introduction
You have probably watched a lot of YouTube videos on how to bake cheesecake, or how to change the car windshield wipers, but something always goes wrong and you just wish there is an expert right there to guide you through. Can we design an artificial intelligent system to watch, talk and guide humans step by step to complete given task?
Task guidance for human users is challeng- ing problem, as requires an interactive system to have a sophisticated understanding of what the user is doing, under the environment setup, and providing appropriate timely guidance. What is more challenging is if we could design a model
Work done during a summer internship at the University of Michigan.

---

sual and dialog context. Our empirical results have shown promising results with foundation models, and highlighted some challenges and exciting areas of improvement for future research. The dataset and code are available at https: /gi tattthu..co com/ sled-groupp group/WWacchTalkk aall--and-aa and-Guide
All Data
Data Storage
AR
Video, Audio
Audio
User
Instructor
Server
2
Related Work
Figure I: Data collection setup for WT&G. Human users follow simplified recipes while communicating with the human instructor through an augmented reality device. Human instructors watch the egocentrie view from the user and offer guidance based detailed recipes. All data streams are recorded for further analysis.
2.1 Task Guidance Systems
Traditional task guidance systems (Ockerman and Pritchett, 1998, 2000) focused on providing the user with pre-loaded saec-fpe information with- out tracking the current state of the environment, or being able to generalize to new tasks (Lecla- sawassuk et al., 2017; Reyes et al., 2020; Lu and Mayol-Cuevar. 2019; Wang al., 2016). The com- plexity of the problem comes from various aspects, such as environment understanding, object and ac- tion recognition, user's preference and mental state detection, real-time inference, etc (Manuvinakurike et al., 2018; Kim et al., 2022). In this work, we collected a multimodal dataset with real human- human task guidance interactions to better study the depth and breadth of the problem. We established strong zero-shot baseline without prior exposure of the given tasks and develop task guidance sys- tem with the help of the latest AR advancement to incorporates both users' perception and dialog.
2022; Moon et al., 2020) that can reason and gen- erate one modality type given another. While some proprietary models can be difficult or expensive to access (e.g. GPT-4v), other open source mul- timodal foundation models have been adapted problems related to task guidance systems, e.g.. action success detection (Du et al., 2023). this work, we leverage the large-pretrained world knowledge embedded in these foundation models, and LLMs' in context learning ability, to build a generalizable situated task guidance system.
A Dataset for Situated Task Guidance
In this work, we introduce Watch, Talk, and Guide (WTaG), new dataset for situated task guidance. WTaG includes nearly 10 hours of ego- centric videos of human users performing cooking tasks while guided by human instructors through live, natural interaction. Synchronized videos and audio transcripts in WTaG present variety of chal- lenging phenomena, including perceptual under- standing, communications, natural mistakes, and much more. We hope this dataset can serve as the starting point to dive into this complex problem.
2.2 Language and Multimodal Foundation Models
Large language foundation models (LLMs) such ChatGPT,' GPT-4 (OpenAI, 2023a), and Bard? have demonstrated a wide range of language gen- eration and reasoning capabilities. These models are not only equipped with huge knowledge bases through training on web-scale datasets, but also the in-context learning ability that allows them to learn new tasks from few examples without any parameter updates (Brown et al., 2020).
Egocentric video datasets (Table 1) have gar- nered much attention in the past decade thanks their potential application in interesting research ar- eas such as embodied AT and task guidance systems. Most of the large egocentric video datasets contain unscripted activities (Damen et al., 2018, 2020; Lee et al., 2012; Su and Grauman, 2016; Pirsiavash and Ramanan, 2012; Fathi et al., 2012; Grauman et al., 2022), while others have collected (semi-)scripted activities where camera wearers are asked fol- low certain instructions (Sigurdsson et al., 2018: Li et al., 2018; Sener et al., 2022). Mcanwhile, re- lated works in embodied AI have collected similar
Meanwhile, multimodal foundation models such as GPT-4v (OpenAI, 2023b), CLIP (Radford et al., 202 are also on the rise to incorporate large scale vision (Betker et al., 2023; Peng et al., 2023; Zhang et al., 2022; Li et al., 2022, 2023), audio (OpenAI, 2022), embodied (Brohan et al., 2023) and other in- put modalities (Zellers et al., 2021, 2022: Radford al., b021b; Li et al., 2022, 2023; Alayrac al.,
'https://openal httpppmmai..mmm/oog//haagggge /nttps://ai.google//sttae google/static/documee yoogle-about-oe babout-bard..

---

Statistics
Characteristics
Annotations
Task Sessions
Action
Dialog
Mistake
Dataset
Hours
Natural Interactive Mistakes
Descriptions
Intents Types
ADL (Pirsiavshh and Ramanan, 2012)
10
20
Charades- Ego (Sigurdsson al., 2018)
69
68.5K
EGTEA Gaze+ (Li al., 2018)
28
86
Epic- Spic-Kitehenn (Damen et al., 2018)
55
432
Ego4D (Grauman et al., 2022)
3.7K
931
Assembly101 (Sener al., 2022)
513
362
ALFRED (Shridhar et al., 2020)
8.1K
MindCraft (Bara et al., 2021)
12
100
TEACh (Padmakumar et al., 2022)
3.2K
WTaG (Ours)
10
56
Table I: Comparison of WTaG to past egocentric task-oriented video datasets. We compare datasets in terms of data statistics, dataset characteristics (whether videos are natural, involve dialog interaction, or have annotated mistakes). and types of annotation (action type narration, dialog act categorization, mistake details) available.
collaborative task completion datasets generated through virtual environment simulators (Shridhar et al., 2020; Bara et al., 2021; Padmakumar et al., 2022). They also target task-oriented dialog and mistakes but not natural nor can be general- ized the open world as can our work.
worn by the user. We use Microsoft Azure auto- matic speech recognition (ASR)² to convert au- dio recordings from videos into conversations tran- scripts, and manually corrected them needed. All data from both sides are synchronized and sent server for storage and future process- ing. Although not used in our experiments, we also collect 12 additional types of synchronized data using Microsoft Psi on top the egocen- tric RGB video, user and instructor audios for each recording. More details can be found at https: thub. com/ mi mwroooostss.ii....mmm.
While smaller than some existing datasets (Ta- ble 1), WTaG emphasizes the interactions between the user and the instructor, prioritizes depth over breadth with this uniquely rich dataset for situated task guidance, and is (to our knowledge) the first of its kind with natural, human-tor to-human interactive videos annotated with conversations texts, recipe steps, dialog intents, and mistakes.
A total of 56 recordings were collected from 17 subjects and 3 instructor subjects. All human subjects were over the age of 18, English-speaking college students with different genders and are from different cultural background with normal or
3.1 Data Collection
To collect data, two human subjects, an instructor and user, are paired up. The user is tasked with completing of 3 cooking recipes² while commu- nicating with instructor. To encourage natural mistakes and interaction between the instructor and user to occur, the instructor has access plete, detailed ground truth recipe, while the user only has a simplified version of it, with high-level directions and minimal details (Figure 9).
vision, recruited through messaging platforms. Members of the study team served as the human instructors.
Together, 4,233 English dialog utterances were collected, evenly distributed over the 3 recipes (19 pinwheels, 18 coffees, 19 cakes). The length of the videos range from to 18 minutes, with the median of 10 minutes. As shown in Figure 2, each video contains median of 31 instructor utterances and 35 user utterances. The instructors talk a bit longer than the users each utterance, with a median of 6 words per utterance versus 3 words for users. Instructors also speak faster than the users with a median speed of 398 ms/word versus 522 ms/word.
As shown in Figure 1, the instructor is separated from the user during task completion, watching and communicating with them through an egocen- tric camera view interface and external microphone connected to an augmented reality (AR) headser!
*Recipes include peanut butter and jelly pinwheels, pour- over coffee, a microwaved mug cake. *Wo use Microsoft HoloLens mi aiccooot.ccomeemne cww.coomo////eceeoco hets/holllens///. hut such interac- tion can be cnabled by any wearable audiovisual device.
(https://ww //www.
ihttps://azure. www.:a/www.mirroo.ttcooo wtwp:r/ssoott.omm/eeesss///rrrdt

---

10.0
#vids
10
10
7.5
10
2.5
0
0.0
0
20
40
60
50
100
200
400
0
200
400
Inst #Utterances/iid
User #Utterances/Vid
Inst #Words/idd
User #Words/id
1000
#Utterances
600
500
500
500
0
20
20
1000
2000
2000
4000
Inst #wd/Utterance
User #Wd/Utterance
Inst Talk Speed (ms/wd)
User Talk Speed (ms/wd)
Figure 2: WTaG dataset statistics histogram for videos dialog interactions.
3.2
Data Annotation
Mistake Types
User Intertion
To have in depth understanding how human instructors navigate this complex task and nuanced situations, and an easier way to evaluate models' performance on WTaG, we provide rich annota- tions on recordings. First, we annotate the time span of each recipe step the user performs in each video, facilitating user state detection. Secondly, we manually go through all the ASR results, cor- rect the transcripts and voice activity time span as needed, and filter out any potential harmful speech. Lastly, we categorize user and instructor utterances into of dialog intentions, together with mistake types if any. Details are as follows:
Oteer
Whong saats
Worl FRUNCT
20%
reStaton
en
Marp
Ceta
Hot
User utterances are categorized into 6 intents: Question, Answer, Confirmation, Self Description, Hesitation, and Other.
Instructor Intention
"Instruction" Type
Figure 3: User and Instructor Dialog Intention Type Distributions and Mistake Type Distributions of WTaG
User mistakes are categorized into 3 classes: Wrong Action, Wrong Object, and Wrong State (in- cluding measurement and intensity).
(c) GT no one said anything for 10 seconds
Each query point provides systems with the latest frame image, dialog history, the task recipe, and current elapsed time into the task. This creates variety of situations, where the instructor may may not need intervene and provide guidance.
Instructor utterances are categorized into the following coarse-grained intents: Instruction, Question, Answer, Confirmation, and Other. the instructor decided to issue an "instruction", the instructions are further classified into 4 types based what they inform users about: Mistake Correction, Current Step, Next Step, and Details.
More specifically, for each query time point f, given the user's ' egocentric video frame, and the chat history, we formulate the following two tasks for the models to predict.
4 Task Definitions
For benchmark evaluation, we extract query points from the WTaG dataset whenever one of the fol- lowing conditions is met:
User and Environment Understanding
1.
User intent prediction: Dialog intent of user's last utterance, if any (options).
(a) GT user said something
(b) GT instructor said something
Step detection: Current step (options).

---

Instruction
On you should first whisk the ingredients
Dialog
Prompt Template
(Mistake)
GT Recipe
Confirmation
Oh okay
lools: ..... Shapa:
Ingredents:
BLIP2: Scene Description
Dialog ASR
Self
Airight combining the ingredients...
Description
Chat History
Timo 4.1, User
Question
good enough?
Time 25.8 You
Time 63.3, User
Question
Are there any lumps?
Observation
Answer
Doesn't look very lumpy
ingredients
ingredients mbkin
table
(Opt The user is
Arawer
not, then looks good!
0
Object State Detection
EgoHOB ChatGPT:
The whisk dirty
interacting with whisk
User said: "he this good enough?
Video
Understanding
intercciing whtik. whiil dity..
Figure 4: Interactive Task Guidance Pipeline: Synchronized video and dialog transcripts inputs to the system. We annotated each utterance to reflect dialog intent. The dialog history is inserted into the template, and the latest utterance is part of the observation. To process the videos, each queried frame either goes through BLIP2 for scene description, goes through EgoHOS for object and state detection. Zero, one of the two video extraction output inserted into the prompt template. The prompts are sent to ChatGPT for instruction predictions.
3. Mistake Existence and Mistake Type: Did the user make a mistake at time (yes/no). If so, what type of mistake (options).
designed prompt template (Figure 4) that includes the ground truth recipe template, user-instructor chat history up to time point t, observations of the user and environment, as well as list of questions listed in Section 4. We then send the prompts to ChatGPT to answer questions related to the pro- posed tasks in each query time point. To enrich the prompts and offer more context, we explored the following three methods that translate multimodal precepts from the egocentric vidco into language: Language Only (Lan): As a baseline for the LLM, we extracted the ground truth user and in- structor dialog up until time t. All the past utter- ances are added to the prompt part of the interac- tion history, and only the most recent is added the observation to avoid model cheating. To enable temporal reasoning, observations include how long the user has been following the recipe so far. This method does not offer any visually- dependent information to the LLM backbone, and challenges the model to infer the context purely based on the conversation.
Instructor Decision Making
1. When to Talk: Should the instructor talk at time & (yes/no).
2. Instructor Intent: If yes to 1, instructor's dia- log intention (options).
3. Instruction Type: "Instruction", what type (options).
yes to and intent in is
4. Guidance generation: If yes to 1, what to say in natural language.
utterance
5 Methods
Given the WTaG dataset and the two tasks defined above, we explore the application of pre-trained large language and vision foundation models on this problem without task-specific training.
For each recording in WTaG, we feed the video frame by frame to our system, together with the synchronized dialog transcripts, and only query ChatGPT at the three query conditions (Section 4). The visual frames go through optional multimodal information extraction process detailed below, to translate the relevant visual context into natural language. The dialog transcripts are extracted at the first frame each utterance appears, and offer conversational context for model predictions. We
Scene Description (See): In this method, we generate free-text scene description by applying BLIP-2 (Li al., 2023) the latest frame image, and insert it into the prompt as part of the obser- vations. Depending on the prompts to BLIP-2, we ask generic questions such as "What is the user do-
htttps///opeaa..cc///ll///eeeeee com/blog/chatgpt: used Azure end- point for GPT335-turbo-000, which is trained on data from up to September 2021.

---

ing" "This is picture of"" to get the descriptions. While this approach is flexible and open-domain, there is no control of how much situation-specific the descriptions would be. Together with the time lapsed and the dialog history, this method offers more scene level visual context to the LLM.
Ob:
Sce
Lan
User Intention
Mistake?
Mistake Type
(a) User and Environment Understanding
Ub
Object and State Detection (Obj): In this method, we extract more fine- fine-grained scene infor- mation by detecting important objects in the frame image and their corresponding states, and inserting them into the prompt observations. At each query time t, we use EgoHOS (Zhang et al., 2022) to segment user's hands and the objects that they are interacting with. We then extract the object seg- ments and predict their most likely object names and their corresponding states using CLIP (Rad- ford et al., 2021b). To balance the performance and generalizability of CLIP predictions, first extracted list of most likely objects and their po- tential states from the recipe through separate prompt to the LLM backbone. The resulting list of objects and states go through a minor manual cleanup before being used by CLIP. This narrows down the scope of search for CLIP, and offers bet- ter targeted vision to language prediction, but is also casily generalizable open-world object and state detections.
Sce
Pimehee
Coffee
(b) Step Detection per Recipe
cbjj
When Talk
Instructar Internion
'instruction" Type
(c) Instructor Decision Making
Figure Interactive Task Guidance Micro Scores: For user and environment understand tasks, the models demonstrated well above random chance performance in user intention prediction and step detection, but struggle with mistake recognition. For Instructor decision predic- tion tasks, the models showed above random chance per- formance in predicting instructor's intention and instruc- tion types, but issued higher communication frequen- cies than human instructors. Across the three methods, Language Only (Lan) showed comparable performance even without any visual context.
All three methods were queried at the same time points extracted in Section 4 for fair compari- son. We reserved recordings (2 of each recipe) for hyperparameter and prompt tuning, and the rest for evaluation. Each method was repeated times. ChatGPT APIs configurations are: Max t00ens=100, temperaturc=0,, stop words= In \n oon "1. All experiments conducted on a sin- gle GPU NVIDIA RTX A6000 and Intel(R) Core(TM) 10-00XX CPU @ 3.70GHz.
6.1
User and Environment Understanding
As interactive task guidance agent, it is impor- tant for the model have comprehensive under- standing of the task's physical environment, well as users' mental and physical states, through their conversations as well as actions.
The overall user utterance intention prediction, step detection, mistake recognition and type pre- diction performances can be found in Figure 5a,5b, was observed that all three methods using zero- shot foundation models have demonstrated decent performance significantly above the random guess- ing (grey dash line) on user intention predictions and step recognition, but struggled with mistake detection. This is most likely due the limited visual context the models can offer accurately detect mistakes (More in Section 6.3). Out of the ones that the model did correctly predict that a mistake has happened, it displayed chance level performance.
6 Experimental Results
In this section, we evaluate the three methods above on the following two tasks: User and Environment Understanding, and Instructor Decision Making- Micro Fl scores reported for each classifica- tion task. We further break down the performance, and evaluate how accurately each vision extraction module can translate the scene into language, and how helpful or annoying the model's generated sentences are under a human evaluation.
Among the three methods, it is observed that with just the dialog context alone, the model was

---

able achieve comparable performance the other two. This shows that the conversations be- tween the human user and instructor disclose lot information about the user and the environment even without visual perception inputs. The Object and State Detection (Obj) method outperforms the Language Only method by small but significant margin on the user intention and step prediction tasks. Compared Obj, the Scene Description (Sce) showed worse and more inconsistent perfor- mance across the tasks. It is likely that the visual context this method extracts too generic or hallu- cinating, which on the contrary confuses the LLM predictions. We further evaluated how well these visual extraction modules perform in Section 6.3.
Cur carert
90%
Die
Anythi Rie
Dusalin
20%4
Minsion Camediion
Naxt Shas
Delatb
"Instructior" Type
Figure 6: Dialog Intention Prediction Distribution: Chat- GPT has strong tendency issue instructions, and pecially instructions about the current step to the users. Compared to human instructors (Figure 3), the guidance is less situated, personalized, natural.
Human language is rich and diverse and there are usually more than one acceptable way of guid- ing the users. Therefore, we further looked into the distributions of the model intention predictions. Comparing Figure 6 with the human intention dis- tribution in Figure 3, the LLM tends issue lot more instructions to users than humans do; human instructors offer more diverse responses, includ- ing more answers and confirmation. Among all the instructions, the models tend to describe the "Current Step" whereas human instructors describe more evenly distributed instruction types. With lim- ited user modeling and visual understanding, it is understandably harder for LLMs to offer situation- specific responses, and therefore resort to more generic instructions about the current step.
6.2 Instructor Decision Making
Given the context of what the user is doing and what the task environment is at each query point during task execution, the instructor needs to pro- vide situated guidance when to talk, and what to say in terms of intents and free-formed guidance.
We evaluate if the models can correctly predict "when to talk" at each query point, based on if there is ground truth instructor utterance in the next o seconds, where x median median((nStspeed x median meddian(Instud/wwtrr [[nsttd/uutt)) The model could decide if needs to talk triggered by user's utterance, previ- ous ground truth instructor's s comment, and visual context when no one talks. To evaluate models' de- cision making performance on "what to talk" (dia- log intention and prre-fored guidance), we collect models' predictions at each query condition (b), i.c. whenever the GT instructor talked. Note, the GT instructor utterances were not part of the prompt to prevent information leakage.
Lastly, we conducted a human evaluation on models' generated language guidance.
Situated interactive task guidance is personal- ized challenge. The guidance frequency and con- tent vary lot from user user, according to their familiarity with the task itself, their chattiness, men- tal states, etc. We broke down the performance based on the number of dialog utterances that oc- curred in recordings into three categories: short, mid, and long. All test recordings were evenly divided accordingly. In this section, we selected test recordings (2 per recipe), and asked 3 hu- man evaluators with different genders and cultural backgrounds rate the following for each output:
The overall instructor decision prediction can be found in Figure 5c. It was observed that all three methods predicted when talk with around chance performance. Going through the examples (Figure 10, 11), we have found that the ChatGPT has a stronger tendency to offer more frequent guidance when the ground truth humans don't. All three methods have significant above random chance aligning with the ground truth instructor and in- struction intentions. Similar to user and environ- ment understanding performances, the Language Only method demonstrated strong performance. especially in deciding the instructor's intention, whereas the Sce method fell short across all three subtasks.
1. How helpful do you find this instructor utter- ance is? Rate 1/2/3, Very Helpful
How annoying do you think it is? Rate 1/2/3.
3 Not Annoying
A total of 936 time points were evaluated, and each time point was rated by two annotators.
The aggregated results for the three methods are shown in Figure Overall, the average quality

---

Not
very Helpful
Annaying
Somewhat
Somoyyng
Helpful
Very
Not Helpful
Annoying
Short
Mid
Long
Short
Mid
Long
(a) Instruction Helpfulness
(b) Instruction Annoyance
Figure 7: Human Evaluation on Model Generated Guidance: Overall on average, most guidance are not considered as very helpful, and somewhat to very annoying by human evaluators. Although not significant, more guidance in shorter videos are found to be less helpful, and more in longer videos are labeled as very annoying. In general, task guidance can be very personalized experience.
of the generated instructions was not considered as very helpful by the human evaluators. They are somewhat to very annoying. This shows that situated natural language task guidance still has long way to go to be applicable even though they' 've shown above random chance level performances across most of the tasks evaluated. Among the three groups, although not significant, more guidance in the short videos were considered as "Not Helpful" by humans, whereas more guidance the long videos were thought as "Very Annoying".
that are not present or completely irrelevant to the scene. For example, "a person is putting ketchup on a plate" for the pinwheel recipe in the WTaG dataset is not truthful, this action is not part of any recording. This metric is more factual based than subjective, and all results below were evalu- ated by one person.
For the Scene Description evaluation (Figure 8a), we tried out 3 different prompts to the BLIP- 2 model: prompt (only the image), "Question: What is the user doing? Answer:", and "This is picture of". There wasn't a significant difference in truthfulness among the three prompts, and we went with no prompt for all the experiments. However, overall, it was observed that the truthfulness BLIP-2 is below 30%, which means most of the scene descriptions are actually hallucinating. This could cause huge confusions to the LLMs about exactly what is happening in the scene.
We also measured the inner annotator agreement these results, and calculated the Cohen's (Co- hen, 1960) value for helpfulness (0.14) and annoy- ance (0.02) ratings. This shows that the task guid- ance can be a very personal experience and that different users have different preferences and tol- erance on the guidance. While the methods we experimented with here did not inject user prefer- ences into the prompts, and the LLM was unable to detect nuanced user mental preferences through the dialog, future work may begin to study how foun- dation models can be used for these challenges.
For the Object and State Detection evaluation (Figure 8b), since EgoHOS outputs unstable pre- dictions from frame to frame, we conducted an ex- periment on detected object smoothing strategies. For sliding window of 10 frames, we tried out if the same object needs to occur from I to 5 times in order be included in the LLM prompts. For each prompt query, we evaluate 1) if any object and state were detected at all from the module, and 2) if the detection results were truthful. According to Fig- ure 8b, it was observed that as the required smooth occurrence goes higher, the same object needs to be detected more frequently be included in the output; while at the same time, fewer objects were detected throughout the recipe overall. We picked smooth rate of for all the experiments. As a result, the visual detection is about 70% accurate with the this method. is much higher than the BLIP-2
6.3 Perceptual Input Extraction
order for the LLM backbone model to have an accurate assessment of the user and the environ- ment, high quality visual perception extraction module is a necessity. In this experiment, we did an ablation study on how truthful the Scene De- scription (Sce) and the Object and State Detection (Obj) modules are while translating the perceptual inputs language. We conducted small scale hu- man evaluation on recordings of each recipe) and evaluated the truthfulness of the vision out- puts. The truthfulness is defined as whether the vision output of each scene contains information

---

Prompt
Truthfulness
and user's attention (e.g.. eye gaze), and other sen- sory inputs (c.g., sound) to achieve richer and more relevant user and environment descriptions.
None
26.2
"Question: What the user doing? Answer:"
25.4+
Second, the overall performance of each predic- tion task from the large language model is fairly above random guessing, especially in zero-shot setting with no task-specific finetuning, but are re- alistically too low to be useful in practice. There is much room for improvement, but also great po- tential to be generalized to more tasks, multi-user or multi-tasking environments.
"This is a picture of""
22.3
(a) BLIP2 Scene Description Evaluation
1.0
0.8
0.6
0.4
Detection Rate
0.2
Truthfulness
Third, there is a big difference between hands- on task guidance versus watching a "How-To" YouTube video. If we look closely, most of the model generated responses are instructions, instead of more situation-aware and personalized responses such question answering and confirmations. It is easy to repeat recipe instructions or perhaps offer little more detail leveraging the knowledge base, but harder to communicate with the user more situation-relevant way. Sometimes, a simple confir- mation can boost the user's confidence, and worth lot more than repeating what the user should do. Furthermore, besides user intent and hesitation, there are other types of user states that can be help- ful but are not currently modeled by our system. Are they familiar with the recipe? Do they look con- fused? Are they getting annoyed by all the instruc- tions? Are they emotionally stable? All of these can change the way the instructor should talk. It is hard to categorize and collect users' mental states at each time point throughout the task. Sometimes experienced human instructors can infer through actions and utterances, but the large language models have not demonstrated strong ability in that way in our experiments.
Smooth Occurrence Rate
(h) EggHOS Object and State Detection Evaluation
Figure 8: Vision to Language Translation Performance. (a) BLIP2 scene descriptions are reported to be only about 25% truthful. (b) A smooth occurrence rate of 2 chosen for the experiments and offers about 70% truthfulness of the extracted visual context.
Scene Description method, but still noisy enough to confuse the language model.
7 Conclusion and Discussions
this work, we explored how to leverage foun- dation models to guide human users step by step to complete task in a zero-shot setting given an arbitrary task recipe. We created new benchmark dataset, Watch, Talk and Guide (WTaG), with nat- ural human user and instructor interactions, and mistake guidance. Two tasks were proposed to evaluate model's s ability on user and environment understanding, as well as instructor decision mak- ing. We used a large language model as the back- bone for guidance generation, and compared three configurations of inputs incorporating language and visual context through dialog history, scene descrip- tion, and object and state detection with multimodal foundation models. We conducted quantitative, and human evaluations of the three methods on our dataset, and discovered several challenges for future work.
Lastly, on the model architecture side, we lever- aged some of the best performing multimodal and language models currently available, translated the rich audiovisual context collected in WTaG to lan- guage, and used LLM as the only reasoning engine to generate guidance. Language is too concise of a medium offer context, and sometimes a picture is worth thousand words. The current multimodal foundation models have limited reasoning capabili- ties to answer complicated questions and offer ac- curate timely guidance. This opens a wide variety of future opportunities for improving multimodal foundation models for situated task guidance.
First of all, vision to language translation is chal- lenging. Having an accurate and relevant descrip- tion of what the user is doing and the environment setup is non-trivial. Some of the methods we' ve experimented with often describe untruthful scenes objects, or offer true but generic or irrelevant information, e.g., "the user preparing food." For future work, would be helpful if these vision to language models could leverage recipe information,

---

Limitations
situated personalized guidance is yet to be ideal. About the experiment results, there is a fair amount of randomness depending on what prompt was used, which version of the model it is, how we set all the hyperparameters for each model, etc. We reduced the temperature to be zero minimize the randomness, and yet the exact same prompt can lead to different responses due to the inherent randomness of the LLM. An exhaustive prompt tuning and hyperparameter search is almost impos- sible and most of them can hardly be measured quantitatively. We reported several prompt and hy- perparameter tuning results in Section 6, and ran the same prompts through ChatGPT three times. We would try conduct larger scale tuning and evaluations in future works when resources are available.
As exciting this work is, there are several limi- tations we would like acknowledge and hope to improve for future works.
One of the biggest challenges we faced was eval- uation. There isn't I a great method to comprehen- sively evaluate generative language models in a scalable way. There are hardly many quantitative ways capture the syntax, semantics, informa- tiveness, relevancy, elc of the outputs, and human evaluation is costly and subjective. In this work, we tried to include both aspects, by evaluating models' performance on several classification tasks, as well as human evaluation on the guidance context. At the end of the day, we just need functioning robot that communicate with human, and guide us step by step to make cake. But to get there, we believe these are some of the milestones that the models should be able achieve.
Last but not the least, we experimented two vi- sion processing methods, one for frame image cap- tioning, and the other for object segmentation and object state detection. We chose these models they've 'e been reported demonstrate state of the art performance recently in related tasks, also with a minor consideration of the processing speed, since the task in nature is ideally real-time. Nevertheless, there are a lot more models out there that can ex- tract more fine- fine-grained information, such as hand gestures, object segmentation, object tracking, can take advantage of temporal video information instead of still frame inputs. We are planning on exploring more vision or other modality processing models for our task.
Another challenge related to evaluation was how to evaluate an interactive system in a real world set- ting. Every perception understanding can change models' decision making, and every decision pre- diction could completely change what would hap- pen next. In our work, we had focus on single step decision prediction given the ground truth in- teraction history. Our recorded dataset serves as resource to train/fine- twwe//nncootteeeee context learn/cvaluate models on these decision points. Once a decent per- forming model is ready, we would love to conduct real-time human-bot interaction evaluations.
The third challenge scale and resources. Our dataset contains 10 hours of recordings and 3 recipe tasks. more robust baseline would ideally include a more diverse pool users and instructors, more tasks, different environment setups, and etc. This would take collected effort and we are planning on expending our work in the future.
Ethies Statement
The Institutional Review Board (IRB) of our institu- tion approved this human subjects research before the start of the study. The WTTGG dataset contains identifiable data (audio), but no facial identifiable data as all videos are first person views of the task environment. Members of the study team served the human instructor, and recruited human subjects as the users. The human subjects were prepared the experiment setup, how to use the augmented reality headset, and potential risks before the ex- periment, and were debriefed after the recording. The consent to video and audio data for publica- tion is optional and will share the consented de-identified subset of the data (video and dialog ASR) when the paper published. All data are stored in password protected internal servers with restricted access to only the study team. The hu-
On the computation side, due to limited re- sources, we only ran the full experiments on the latest version of the GPT335--urbo-030 model in- stead of GPT4. Querying large language models can be costly especially when we are querying at a high frequency (every few seconds per video) with a growing sized prompt. Meanwhile, there will always be another model version update in few months that may achieve higher performance. How- ever, we believe lot of our observations still stand as the visual language translation is a tough bot- tleneck to offer appropriate context, hallucination is still a big challenge for generative models, and

---

man evaluations were conducted by 3 members of our study team, including two genders and two races. All collected data were filtered for content moderation during the ASR annotation stage. The generated utterances abide by the OpenAl content policy.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jian- feng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh, 2023. Improving image generation with bet- ter captions. https://cot http////ddooooniii...//a//a/a openai com/ com/papers/ dall-e-3. .df.. Accessed: 2023-10-21.
This work intends to have a positive impact on the society as the goal is to design system that can assist human to complete tasks in more per- sonalized way. The impact of this work on the final goal is limited as we generated pre-recorded videos to study the baselines of the challenge, but there is the potential for the future models to be misused, or offer ill ill-intended guidance to human. We hope future works take responsible Al into considera- tions while designing human centered interactive systems.
Anthony Brohan, Noah Brown, Justice Carbajal, Yev- gen Chebotar, Xi Chen, Krzysztof Choromanski. Tianli Ding. Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonza- lez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alex Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee. Sergey Levine, Yao Lu, Hen- ryk Michalewski, Igor Mordatch, Karl Pertsch, Kan- ishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Bri- anna Zitkovich. 2023. RI-2: Vision- Tision-angguage-ctioon models transfer web knowledge robotic control. In arXiv prepriat arXiv:2307. 158888
Acknowledgements
This work is supported by DARPA PTG program HR001 12220003. We would like to thank the anonymous reviewers for their valuable comments and suggestions.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-9911
References
Michael Ahn. Anthony Brohan, Noah Brown, Yev- gen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jes- month, Nikhil Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Kuang-Hue Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Pe- ter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes. Pierre Sermanet, Nico- las Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. 2022. Do i can and not as say: Grounding language in robotic af- fordances. In arXiv preprint 2xXi::2204.0169)
Jacob Cohen, 1960. A Coefficient Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1):37-46.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, al. 2018. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the Eurupean Conference on Computer Vision (ECCV), pages 720-736.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma. Da- vide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. 2020. Rescaling egocentric vision. arXiv preprint arXiv:2006. 13256.
Jean- cann-Bapistte Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech. lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736.
Yuqing Du, Ksenia Konyushkova. Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. 2023. Vision-language models as success detectors. arXiv preprint arXiv::3003 00728.
Cristian-Paul Bara, Sky CH-Wang, and Joyce Chai. 2021. MindCraft: Theory of mind modeling for situ- ated dialogue in collaborative tasks. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 1112-1125, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Alireza Fathi, Jessica K Hodgins, and James M Rehg. 2012. Social interactions: A first-person perspective. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 1226-1233. IEEE.
Qiaozi Gao, Malcolm Doering, Shaohua Yang, and Joyce Chai. 2016. Physical causality of action verbs

---

grounded language understanding. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics [Volume I: Long Papers), pages 1814-1824, Berlin, Germany, Association for Computational Linguistics.
Teesid Leelasawassuk, Dima Damen, and Walterio Mayol- Cuevas 2017. Automated capture and de- livery of assistive task guidance with an eyewear computer: the glaciar system. In Proceedings of the Sth Augmented Human International Conference, pages 1-9.
Kristen Grauman, Andrew Westbury, Eugene Bymee Zachary Chavis, Antomino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Nija Ra- dosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do Morrie Doulaty, Akshay Erapalli, Christoph Feicht- enhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis. Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li. Zhenqiang Li. Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro. Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ra- mazamova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi. Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Gio- vanni Maria Farinella, Bernard Ghanem, Vamsi Kr- ishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Ki- tami, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. 2022. Ego4d: Around the World in 3,000 Hours of Ego- centric Video. In IEEF/CVF Computer Vision and Pattern Recognition (CVPR).
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. arXiv preprini arX1v:2301.12597.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre- training for unified vision- -siiun-a understanding and generation. In International Conference on Ma- chine Learning, pages 12888-12999 12900. PMLR.
Yin Li, Miao Liu, and James M Rehg. 2018, In the eye of beholder: Joint learning of gaze and actions in first person video. In Proceedings of the European conference on computer vision (ECCV), pages 619- 635.
Yao Lu and Walterio Mayol-Cuevas. 2019. Higs: Hand interaction guidance system. In 2019 IEEE interna- rional symposium on mixed and augmented reality adjunct -Ajjuee,jj pages 376-381. IEEE.
Ramesh Manuvinakurike, Trung Bui, Walter Chang. and Kallirroi Georgila. 2018. Conversational im- age editing: Incremental intent identification in a new dialogue task. In Proceedings of the 19th An- nual SIGdial Meeting Discourse and Dialogue, pages 284-28822 294,299, Melbourne, Australia, Association for Computational Linguistics.
Seungwhan Moon, Satwik Kottur, Paul Crook, Ankita De, Shivani Poddar, Theodore Levin, David Whit- ney, Daniel Difranco, Ahmad Beirami, Eunjoon Cho. Rajen Subba, and Alborz Geramifard. 2020. Situ- ated and interactive multimodal conversations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1103-1121, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for em- bodied agents. In International Conference on Ma- chine Learning, pages 91 18-9147. PMLR.
Ivan Kapelyukh, Vitalis Vosylius, and Edward Johns. 2023. Dall-e-bot: Introducing web-scale diffusion models to robotics.
Jennifer Ockerman and Amy Pritchett. 2000. A review and reappraisal of task guidance: Aiding workers in procedure following. International Journal of Cognitive Ergonomics, 4(32:111212.
Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. 2022. Simple but effec- tive: Clip emheddings for embodied ai. In Proceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR).
Jennifer J Ockerman and Amy R Pritchett. 1998. Pre- liminary investigation of wearable computers for task guidance in aireraft inspection. In Digest of Papers. Second International Symposium on Wearable Com- puters (Cat. No. 98EX215), pages 33-40. IEEE
Hyounghun Kim, Doo Soon Kim, Seunghyun Yoon, Franck Dernnncour, Trung Bui, and Mohit Bansal. 2022. Caise: Conversational agent for image search and editing. Proceedings of the AAAI Conference on Artificial Intelligence, (0910))00003-099
OpenAI. 2022. Introducing whisper. / /openai.. homm/reseercch/hhiie Accessed: 2023-
Yong Jae Lee, Joydeep Ghosh, and Kristen Grauman. 2012. Discovering important people and objects for egocentric video summarization. In 2012 IEEE con- ference on computer vision and pattern recognition, pages 1346-1353. IEEE
OpenAI. 2023a. Gpt-4 technical report.
OpenAl. 2023b. Gpt-4v(ision) system card. https: //openai. wwwpseerchhgtt---yyttttt Accessed: 2023-10-21.

---

Aishwarya Padmakumar, Jesse Thomason, Ayush Shri- vastava, Patrick Lange, Anjali Narayan-Cher Span- dana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkami-Tur. 2022. Teach: Task-driven em- bodied agents that chat. Proceedings of the AAAI Conference on Artificial Intelligence, 36(2):2017- 2025.
Xuan Wang, SK Ong, and Andrew Yeh-Ching Nee. 2016. Multi-modal augmented reality assembly guid- ance based bare-hand interface. Advanced Engi- neering Informatics. 306):406-421.
Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, and Liang He. 2021. A survey of comaan-n--the-oope for machine learning. CoRR, 2as/2108.00941.
Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023. Kosmos-2: Grounding multimodal large language models to the world.
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu. Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusu- pati, Jack Hessel, Ali Farhadi, and Yejin Choi. 2022. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16375-16387.
Hamed Pirsiavash and Deva Ramanan. 2012. Detecting activitics of daily living in first-person camera views. 2012 IEEE conference on computer vision and pattern recognition, pages 2847-2854. IEEE.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- ur, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021a. la. Learn- ing transferable visual models from natural language supervision. CoRR, abs/2103.00002
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu. Jac Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 34:23634-23651.
Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi. 2022. Fine-grained egocentric hand- object segmentation: Dataset, model, and applica- tions. In European Conference on Computer Vision, pages 127-145. Springer.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- uy. Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021b. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-878 PMLR.
Arvin Christopher Reyes, Neil Patrick A Del Gallego, and Jordan Aiko Deja. 2020. Mixed reality guid- ance system for motherboard assembly using tangible augmented reality. In Proceedings of the 2020 4th International Conference on Virtual and Augmented Reality Sinulations, pages 1-6.
Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He. Dipika Singhania. Robert Wang, and Angela Yao. 2022. Assembly 101: A large-scale -ueti-view video dataset for understanding procedural activi- ties. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 11096-211066.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke 7ettlemyer, and Dieter Fox. 2020. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In The IEEE Conference on Computer Vision and Pallern Recognition (CVPR).
Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. 2018. Charades- ego: A large-scale dataset of paired third and first person videos. arXiv preprint Ar.ivv1804.0000 1804.0966..
Yu-Chuan Su and Kristen Grauman. 2016. Detecting engagement in egacentric video. In Computer Vision- ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part V 14, pages 471. Springer.

---

A
Appendix
A.1 Sample Prompt
You are instructor guiding user to complete the task of Pour-over Coffee
[Recipe Content]
Chat History:
Time: 34.6, User: I wanna make sure I'm puring
ten ounces
Time: 36.4, You: should be twelve ounces of
water
It has been 38.2 seconds into the recipe
Scene description: person preparing food
table
OR
The user interacting with Measuring cup
The Mcasuring cup is filled with water
User said oh twelve ounces ok
Answer the following questions:
1. What is their dialog intention? Choose among Question, Answer, Confirmation, Hesitation, Self
Description, and Other
2. Which step is the user at? For example Step
3. Should you say anything? Yes or no
3.1. If yes, what would you say?
3.2. yes, choose your dialog intention among Instruction, Confirmation, Question, Answer, or Other
3.3. If your dialog intention is Instruction, is it about current step, next step, details, or mistake correction
4. Did the user make mistake? Yes or No? If yes, choose among wrong object, wrong state, wrong
action
Answer:

---

Recipe: Pinwheels
Ingredients
8-inch flour tortilla
Jar of nut butter or allergy-friendly altermative (such as sunbutter, soy butter, or seed butter) Jar of jelly, jam, or fruit preserves
Tools and Utensils cutting board
butter knife
paper towel
toothpicks
12-inch strand of dental floss plate
Steps
1. Place tortilla on cutting board.
2. Use butter knife to scoop nut butter from the jar. Spread nut butter onto tortilla, leaving 122inch
uncovered at the edges.
3. Clean the knife by wiping with paper towel.
4. Use the knife to scoop jelly from the jar. Spread jelly over the nut butter.
5. Clean the knife by wiping with a paper towel.
6. Roll the tortilla from one end to the other into a log shape, about 1.5 inches thick. Roll tight enough
to prevent gaps, but not so tight that the filling leaks.
7. Secure the rolled tortilla by inserting 5 toothpicks about inch apart.
8. Trim the ends of the tortilla roll with the butter knife, leaving 1/2 inch margin between the last
toothpick and the end of the roll. Discard ends.
9. Slide floss under the tortilla, perpendicular to the length of the roll. Place the floss halfway between two toothpicks.
10. Cross the two ends of the floss over the top of the tortilla roll. Holding one end of the floss in each hand, pull the floss ends in opposite directions to slice.
11. Continue slicing with floss to create 5 pinwheels.
12. Place the pinwheels on a plate.
Recipe (Simplified): Pinwheels
Take out a tortilla
Scoop choice of spread. Spread the spread.
Clean the knife.
Scoop different spread. Spread. Clean the knife.
Roll the tortilla.
Insert toothpicks.
Trim the ends of tortilla
Cut tortilla roll Transfer pinwheels.
Figure 9: Sample Recipe and Its Simplified Version

---

Frames
User
GT Inst
Lan
Sce
Obj
Ok I don't drink so never prepared actuall....
Remind them of next which is step
Sure,
at all
Uh, fiter cone.
This fiter cone. It's part the dripper that holds fibar. okay
This is paper filter. You to fold it according 3.
Yeal filler the metal thing.
We'll move
Explain that instrument is the kette.
pace comfortable
with you.
Fold twice?
Please unfoid
fiher egain, and fold half
again
Yes you fold the paper fitter to create a cone.
are takiig out the coffee
chicle Place
Yeah.
again.
the fitter cone.
Like this?
paper fiber the After
folding it half, you
Yeah the coffee ground should be consistency of coarse sand.
again so it'll shape like cone.
Yeah. You
again.
In half okay
Fold in half to produce
dircle and half again quarter circle then the filter the difpper.
You should carefully follow the mentioned.
Fold the filter in half first and then again.
Uh...
You spread out the fiter the shape of the the coffee drip through into the mug.
Place the paper fitter cone and spreed open cone shape.
Yeah and now spread
Figure Sample Data from WTaG: Coffee

---

Frames
User
GT Inst
Lan
Sce
Obj
Spread butter to the leaving % inch uncovered edges.
Spread the nut lortllia, leaving is inch uncovered at the edges.
Oh you should place the tortilla cutting board.
Oh okay.
you place the tortilia cutting board.
You should place the tortilla the cutting board
You should place the tortila the cutting board
Oh you should knifa scoop nut butter from jar.
Place the tortilla the cutting board.
Clean the knife by wiping with paper towel.
place the the cutling board
Oh oh
butter knife to scoop the jar. Spread
Oh you should butter knile to scoop nut better from jar.
Secure the tortilla by inserting toothpicks about inch apart.
tortila,, leaving % inch uncovered at the edges.
knile to scoop the jar. Spread
Please clean the knife by wiping paper towel
torilla, leaving inch uncovered
knifle by wiping with paper towel.
It's the right with Ed.
at the edges.
Of oh yes.
You are daing remember to use knife scoop the nut butter from the jar.
You should tell the where the nut butter ia.
you have cuttigg board?
Figure 1I: Sample from WTaG: Pinwheel


Reviews:
[{"review_id": "xtI37yVUWB", "reviewer": "Reviewer_i9MT", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing this is a \"bold\" research, and that our dataset will be promising to inspire future work. We also appreciate the valuable feedback and questions which we will address below. We welcome more questions during the discussion period if any of our response is not clear.\n\n**[Problem Definition]:** At the high level, the goal is to design an interactive system that can offer timely personalized and situated guidance by watching and communicating with the user. In Section 4, we define the problem and key characteristics of it that we can evaluate models for. \n\nMore specifically, given inputs: 1) user's first person view video, and 2) user's utterances, both up until time point $t$, predict the followings:\n\n[User and Environment Understanding]\n1) User's utterance intention (if any), choosing from a set of choices\n2) Current step, choosing from recipe steps\n3) Is there a mistake at time $t$, yes or no. If yes, then predict the mistake type, choosing from a set of choices\n\n[Instructor Decision Making]\n1) Should the instructor talk at time $t$, yes or no\n2) If yes to 1), Instructor's utterance intention, choosing from a set of choices\n3) If yes to 1) and 2) type is \"Instruction\", Instructor's instruction type, choosing from a set of choices\n4) If yes to 1), the instructor's utterance\n\nWe will add mathematical definitions of the tasks and clarify this part in the revision.\n\n**[Technical Contribution]:** As the reviewers mentioned, the problem has numerous aspects to consider and can be challenging to find a starting point. Our main goal with this work is to collect a dataset with both human users and instructors as the ‘gold standard’ to study and understand the problem better. We hope this work could serve as a starting point in the most realistic setting (natural language conversations, natural mistakes, first person view, synchronized multimodal inputs, etc) for future work to improve upon.\n\n**[Baseline Comparisons]:** For baseline comparisons, we aim to construct a complete pipeline that takes multimodal inputs, and outputs language guidance given any arbitrary task. As observed through the experiments, a bigger challenge of leveraging LLMs as backbones is incorporating visual context, rather than generating guidance. We therefore focused on experimenting with three different methods of incorporating multimodal inputs for the LLM backbones. There are, however, indeed a lot of other angles that can be further investigated (e.g. different language models, different object and state detection methods, different multimodal fusion methods, contextual variable controls, user’s preferences, etc) and we plan on further studies in future works.\n\n**[Emergent Abilities]:** If this was in reference to Section 2.2 (Ln 104), we believe we were referring to the emergent abilities of LLMs there instead of multimodal large models, but we will rephrase this section to be clearer.\n\nWe will fix the typos and double check the rest in the revision. Thanks for pointing them out!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The task of building a zero-shot multimodal task guidance system by leveraging LLMs is quite interesting and bold.\n\n2. The authors propose a new dataset which is promising to facilitate this kind of research line.\n\n3. The experiment results can inspire more related researches.", "reasons_to_reject": "1. Unlike LLMs, in my opinion, multimodal large models have not been shown that they have the ‘emergent abilities’.\n\n2. The problem to resolve is not clearly defined. And the technical contribution is limited. As a benchmark experiments on a new dataset, comparison with baselines is missing, e.g., using various large models.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates a very challenging task of building a zero-shot guidance model by leveraging LLMs. The authors propose a new dataset WTaG which mimics the real-world situations to testify whether the LLMs can fulfill the above task. The task is very difficult, especially involving multimodal information, i.e., textual and visual data. And the experiments can show the promising performance of LLMs although there is still much room for improvement.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Some typos: Line 033, 109, 544."}, "extra_text": {}}, {"review_id": "GIvwjwWY17", "reviewer": "Reviewer_gqEJ", "summary": "", "questions": "- Can the narrations/generations be further used to improve the models rather than a zero shot inferencing approach? This could be a more interesting approach highlighting the need to understanding the domain. \n- LLMs are quite good at recommending recipes and guidance related to recipes. \n- How is the the time synchronization issues handled especially with lags in the generations and the time sensitiveness? ", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for recognizing the problem that this work is tackling is challenging and that this work promisingly identifies several limitations of the current methods and provides a valuable dataset. We also appreciate all the insightful suggestions and comments which we address below.\n\nFirst of all, we would like to clarify that our dataset contains three tasks instead of just cake making: pinwheels, pour-over coffee, and a microwaved mug cake. For example, A.4 is a sample prompt for coffee making, and Figure 12 is a recipe for pinwheels. We ultimately aim to design a task guidance system that has the generalizability to be extended to any given task manual/recipes (Section 2.1), more than just making cakes or within the cooking domain. We will clarify this in the revision.\n\n**[Challenging]:** We fully agree with the reviewer that situated task guidance is very challenging. There were a lot of angles to consider, and it was indeed difficult to find a niche point to start. That was exactly why we decided to collect this dataset with both human instructors and users as the ‘gold standard’ to study and understand the problem better. We want to mimic the task in the most realistic setting (natural language conversations, natural mistakes, synchronized multimodal inputs, an arbitrary task manual given to the LLM etc), which we haven’t found similar resources in the previous works, and we documented all the data along the way. For baseline method comparisons, we focused on one of the technical difficulties (multimodal context incorporation into LLMs) that we observed to experiment on, when there are indeed a lot of other nuances that deserve attention. While this effort lays a foundation, we hope that this work and dataset can enable future work to study the depth and breadth of the problem with all of its numerous complexities.\n\n**[Q1: Further Improvement with This Data instead of Zero-Shot]:** Absolutely! It’s possible that more tailored models are needed for complex tasks like this. For example, as observed in Section 6.2 human evaluations, task guidance can be a fairly personalized experience. Guidance generation can be further trained to accommodate situation specific, user specific adjustments. The dataset we collected allows us and others to further the studies on interesting directions like this in future works.\n\n**[Q2: LLMs are good]:** Yes they are. One big challenge that we observed with LLMs through our experiments is to offer situated and personalized guidance with multimodal context of what is happening, how is the user doing, etc. Our final goal is to design a fully interactive system that can accommodate this, and we hope this work serves as a starting point on what the current existing methods can achieve so far and to identify several key areas for future works to improve on.\n\n**[Q3: Time Synchronization]:** That is indeed a very important issue that we have been keeping in mind while designing the system with the tradeoff of model performance and model processing speed. About this work specifically, these issues are less pertinent as we only evaluate them based on collected data, with the observation that the LLM based methods still have a large room for improvement even without considering these efficiency challenges. However, these issues will indeed become significantly more important in the real-time interaction settings. As such, we hope this work and the dataset that we collected can help future model design, performance and efficiency comparisons.\n\nWe will clarify all the issues mentioned above and incorporate the references in the revision. Thanks a lot for all the suggestions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Task guidance is a challenging problem. The problem has multitude of complexities not limited to the capabilities of the model but also on the user perception (annoyance for instance). The work very promisingly identifies these limitations and provides a valuable dataset. \n", "reasons_to_reject": "- Task guidance in a situated setting has remained a challenge. (See for example: Conversational image editing: Incremental intent identification in a new dialogue task; Manuvinakurike et al., 2018;CAISE: Conversational Agent for Image Search and Editing; Kim et al. 2022). One of interesting aspect of real-time task guidance is that of real-time or incremental nature of the problem. Some steps in the guidance last a few seconds while a few last minutes. Contextual variables (type of dough or ingredients for instance), personal preferences etc. all play a role. The work doesn't highlight the nuances involved in these domains which could be more valuable for the community and potentially allow development of novel applications. \n- \"When to ask\" is a problem involving turn-taking elements which involve a lot of contextual understanding. The literature shows this to be a challenging problem and is rather a more involved process. \n", "questions_to_authors": "- Can the narrations/generations be further used to improve the models rather than a zero shot inferencing approach? This could be a more interesting approach highlighting the need to understanding the domain. \n- LLMs are quite good at recommending recipes and guidance related to recipes. \n- How is the the time synchronization issues handled especially with lags in the generations and the time sensitiveness? ", "paper_topic_and_main_contributions": "The paper explores utilizing foundational models for task guidance in a cake making scenario. The work contributes to the literature by developing a dataset consisting of conversational interactions between a guide and a task performer. The task pipeline consists of vision modules tasked with understanding the visual content and generating context for the LLMs which is later converted to a prompt to allow generation of utterances to be used for task guidance. The authors perform evaluations on the perceptions and the model performance for the pipeline and infer that, while the LLM show promise there is much left to be done on the multimodal front to perform task guidance (for cake making). There are also differences that exist in real-time task guidance when compared to performing more deterministic activity recognition. ", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VPpvQe7dSb", "reviewer": "Reviewer_nrzY", "summary": "", "questions": "A) Who are these subjects, how were they recruited, what are their demographics, and experience in the tasks (cooking).\n\nB) What is the language of the dataset? I assume it is English but it should be clarified.\n\nC) Have you tried training a model on this data instead of using a zero-shot setting?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for recognizing the novelty and annotation efforts of this work! We would also like to thank the reviewer for all the detailed reviews and helpful suggestions! Here are a few things that we can address here and we definitely welcome more questions during the discussion period if any questions remain.\n\n**[Train on This Data instead of Zero-Shot, Q-C]:**\n\nIt definitely came across our mind to see if we can train or fine-tune a model to achieve better performance. The main consideration that led to the decision was that we are aiming for the **generalizability** of the system which hopefully can be extended to any arbitrary given task manual or recipe, a large scope for which it is impossible to collect comprehensive training data.\n\nWith this goal in mind, a model would either have to learn a set of predefined tasks, or in our method, leverage the world knowledge of pre-trained large models (GPT, CLIP, etc) and in-context learning capabilities.\n\nTraining or fine-tuning on this dataset, on the other hand, would entail limiting the vocabulary size of object detections, tailoring the reasoning model towards a specific set of tasks, and overfitting the language generation model towards the set of human user preferences in our experiments. *It might improve the performance on this dataset, but might not be the answer to the ultimate goal that we strive for.*\n\nThat being said, training definitely has its merits and as observed, the current system has a lot to improve on. We hope this dataset can help with model improvements in various ways in future works.\n\nWe will clarify these in the revision.\n\n**[Human Subject Demographic, Q-A]:**\n\nOur IRB approval limits the types of subject data we can collect and share, but all subjects were over the age of 18, English-speaking college students from different cultural background with normal or corrected-to-normal vision, recruited through messaging platforms.\n\nWe did not explicitly survey their experience in cooking as one single self-identified value might not be as informative as we would hope for. Instead, all the subjects were given a simplified recipe which may not be something they are familiar with. For example, the cultural background of the subjects varies and an experienced cook for one cuisine doesn’t necessarily entail they are familiar with recipes from other regions; or the subject’s familiarity with individual recipe steps could be very different from their familiarity with the rest of the recipe and a regular coffee drinker might stumble into (how to use) a new food scale.\n\nWe will clarify this and offer as many details about the users as we could in the revision.\n\n**[More Examples and Analysis]:** We are happy to include more examples in the revision. The dataset and code will also be available upon publication. We were hoping Figure 2b offers some insights on the interactivity between users and instructors, but will reorganize it in a more intuitive way and include more comprehensive analysis.\n\n**[Q-B: Dataset Language]:** Sorry about that - all data collected is English.\n\nWe will fix all the typos listed, double check the rest, and include the additional references in the revision. Thanks a lot!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The dataset presented is novel and targets the interesting scenario of task guidance in a real setting.\n2. Annotation of both sides of the dialogue and classification of the main types of interactions and mistakes.", "reasons_to_reject": " 1. The approach presented relies on the foundational model’s ability to decipher and understand the input many times with insufficient context. It would have been good to have, as a baseline, a model trained on this data, given that all of the tasks are annotated.\n2. The paper does not mention the origin and experience of the subjects utilized in the data collection experiment.\n3. I would have liked to see more examples of the datasets' conversations and a more complete analysis of the comprehensiveness and interactivity between user and instructor.", "questions_to_authors": "A) Who are these subjects, how were they recruited, what are their demographics, and experience in the tasks (cooking).\n\nB) What is the language of the dataset? I assume it is English but it should be clarified.\n\nC) Have you tried training a model on this data instead of using a zero-shot setting?", "paper_topic_and_main_contributions": "In this paper is proposed a new dataset for multimodal task guidance called Watch, Talk and Guide (WtaG), as well as some baselines based on foundational models.\n\nThis dataset is composed of about 10 hours of egocentric videos of users performing 1 recipe out of 3 considered, while guided by a human instructor in a live setting.\nAccording to the authors, one of the things that differentiates this dataset from previous work is the interactivity and annotation of both dialog acts and mistakes.\nThe authors define two main tasks: 1) user environment understanding, and 2) instructor decision-making - casting most of these tasks into a classification task for easier evaluation.\n\nRegarding the methods used, the authors considered a foundational model (GPT-3.5) without task-specific fine-tuning. Given that the model only works with text, the authors have two additional input strategies. The first is based on scene description using BLIP-2, and the second is based on object and state detection using fine-grained scene information.\n\nThe results showed low accuracy attributed to the difficulty of the task, and it being a zero-shot setting. The results also showed that visual information was mostly unhelpful to the model, given that they were too generic or irrelevant, generally resulting in hallucinations.\nAn additional study also showed that the quality of task guidance is subjective, hinting at the possibility of future work in modeling the user as well.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Not a necessity but could have also cited: \"Situated and Interactive Multimodal Conversations\" – situated multimodal interactions in the fashion domain."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos and Grammar:\n\nLine 3 - assist humans in various\n\nLine 14 – we leverage a large language model backbone\n\nLine 37 – to guide the human through\n\nLine 66 – quantitatively\n\nLine 92 – review the last sentence\n\nLine 126 – videos\n\nLine 150 – review the sentence and “open-world”\n\nLine 187 – Figure 2 and in the Appendix (specify where in the appendix)\n\nLine 937 – “medium” – is this median or average\n\nLine 99 – ranges from \n\nLine 451 - To to"}}]

Author rebuttal:

Meta-review: In general, reviewers agreed that this paper provided sufficient support for its major claims, although some minor points could benefit from further support or clarification.  They exhibited mixed enthusiasm for seeing the paper at the conference, with excitement ratings ranging from mediocre to strong.  The authors provided additional support for many of the claims that reviewers found to be more weakly justified, and they promised to update the revised manuscript with this additional information.

**Summary of Reviewer Feedback and Discussion:**
- **Reviewer i9MT** found the proposed task to be interesting and bold, and they appreciated that the authors proposed a dataset to facilitate research towards that task.  They also felt that the experimental results may inspire additional related research.  However, they were unconvinced that multimodal models have demonstrated evidence of emergent abilities, and they felt that the paper's problem definition was unclear.  They also felt that the paper's technical contributions, including comparisons of different potential baselines for the new dataset, were limited.  The authors clarified their problem definition in the rebuttal and promised to formalize it more clearly in the revised manuscript.  They also clarified that their technical contribution is the new dataset, and agreed that there are many more model comparisons that they can perform in the future to study their problem from different angles.  As a point of clarification, the authors noted that they intended to refer to emergent abilities of LLMs, rather than large multimodal models.
- **Reviewer gqEJ** thought that the proposed task was challenging and that the provided dataset will be useful for investigating it further, but they felt that the paper didn't adequately highlight the nuances involved in associated problem domains.  They also felt that the authors may have missed some relevant literature, and they raised several clarifying questions and a clarifying comment.  In their rebuttal, the authors clarified the variety of tasks in their dataset and noted that they chose one specific technical challenge as the focus of their experiments but that they agree that there are many other nuances to explore as well.  They also responded to Reviewer gqEJ's other clarifying questions, and promised to clarify these points in the revised manuscript.
- **Reviewer nrzY** thought that the proposed dataset was novel, high-quality, and targeted an interesting scenario.  However, they wished that there had been a baseline trained on this data rather than only baselines using foundation models.  They also requested more details pertaining to both the dataset itself and the subjects who participated in its development.  In their rebuttal, the authors clarified that they hadn't used a baseline trained on their dataset because they were interested in generalizability beyond their dataset alone.  They also responded to Reviewer nrzY's other questions, provided additional details regarding their human subjects, and promised to include more examples and analyses in the revised manuscript.
Decision: Accept-Findings