Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?
Authors: Apratim Bhattacharyya
Bicheng Xu
Sanjay Haresh
Reza Pourreza
Litian Liu
Sunny Panchal
Leonid Sigal
Roland Memisevic
Venue: NeurIPS 2025 poster ()
Keywords: Multi-modal
Large language Models
Vision and Language
Primary area: applications
TL;DR: Current multi-modal LLMs struggles with live, step-by-step task guidance. We built Qualcomm Interactive Cooking (a new dataset with mistake videos and timed feedback) and LiveMamba (a streaming model) to enable better real-time interactive guidance.

Abstract: Multi-modal Large Language Models (LLM) have advanced conversational abilities but struggle with providing live, interactive step-by-step guidance, a key capability for future AI assistants. Effective guidance requires not only delivering instructions but also detecting their successful execution, as well as identifying and alerting users to mistakes, all of which has to happen in real-time. This requires models that are not turn-based, but that can react asynchronously to a video stream, as well as video data showing users performing tasks including mistakes and their corrections. To this end, we introduce Qualcomm Interactive Cooking, a new benchmark and dataset built upon CaptainCook4D, which contains user mistakes during task execution. Our dataset and benchmark features densely annotated, timed instructions and feedback messages, specifically including mistake alerts precisely timestamped to their visual occurrence in the video. We evaluate state-of-the-art multi-modal LLMs on the Qualcomm Interactive Cooking benchmark and introduce LiveMamba, a streaming multi-modal LLM designed for interactive instructional guidance. This work provides the first dedicated benchmark and a strong baseline for developing and evaluating on live, situated coaching.

Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?
Apratim Bhattacharyyal
Bicheng Xu
Sanjay Haresh' Leonid Sigal? University of British Columbia
Reza Pourreza'
Litian Liu'
Sunny Panchal'
Pulkit Madan' QQulloom AI Research
Roland Memisevic!
Abstract
Multi-modal Large Language Models (LLM) have advanced conversational abilities but struggle with providing live, interactive step-by-step guidance, key capability for future Al assistants, Effective guidance requires not only delivering instructions but also detecting their successful execution, as well as identifying and alerting users to mistakes, all of which has to happen in real-time. This requires models that are not turn-based, but that can react asynchronously to video stream, as well as video data showing users performing tasks including mistakes and their corrections. To this end, we introduce Qualcomm Interactive Cooking, new benchmark and dataset built upon CaptainCook4D, which contains user mistakes during task execution. Our dataset and benchmark features densely annotated, timed instructions and feedback messages, specifically including mistake alerts precisely timestamped their visual occurrence in the video. We evaluate state-of- the-art multi-modal LLMs on the Qualcomm Interactive Cooking benchmark and introduce LIVEMAMBA, streaming multi-moda LLM designed for interactive instructional guidance. This work provides the first dedicated benchmark and a strong baseline for developing and evaluating live, situated coaching.
1 Introduction
Multi-modal Large Language Models (LLMs) have recently advanced, enabling AI systems to interact with users more naturally, fluently, and in real-time by processing audio, speech, and visual inputs for conversations about images or videos.
However, to be useful as an AI assistant, multi- modal LLMs should be able to guide user through task "live" by providing interactive step-by-step instructions. This encompasses three key abilities, i) to provide the next instruction, ii) detect if the instruction has been successfully accomplished by the user, iii) if not, then detect any mistakes made by the user and alert the user as soon as possible. Consider example where multi-modal LLM guides user while making, e.g., Bruschetta in Fig. 1. At the stage where the tomatoes are being sliced, an instruction with the desired thickness the tomatoes needs to be provided. Then the model needs to detect whether the user has sliced the tomatoes to the desired thickness, and in the case it observes the mistake. This calls for the multi-mode LLM to be able react interactively to events in the video streams. However, current ofate-of-the-ann art multi-modal LLMs are still largely limited turn based interactions [26, 40, 55, 56, 64] they only produce responses when prompted by the user or to narration tasks [8, 35).
mistake, the model needs to alert the user as
To address the challenge of live, step-by-step coaching, we introduce the Qualcomm Interactive Cooking dataset and benchmark, as currently available large-scale languaee datasets and
"Equal contribution.
I Work done while employed Qualcomm Research. Qualccmm AI Research is initiative of Qualcomm Technologies. Inc.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Sreaacn the pietie bricks Sprrible mosoanlla chereo fomalo throughtu
Garn an sessorning
a difccie oil sedrwarrg olie oil stout Seleeption over the ertine platne.
Streaming Video
Feedback make single layer the thick slioes of tomatoes, not layer.
Instruction: Now one tomato into about 1/2 inch thick slices.
tomato into about 1/2 inch thick slices, but instead, slioed into targor slicss, about
thick slices of tomaloes on platter, ensuring they only make single layer.
Streaming Video
Instruction:
Feedback: garnished the platter with Italian seasoning property.
Instruction: Now add extra-virgin olive oll, about tabiespoon, over the entire platter.
Feedback: You virgin olive all, not vegetable oil.
Feedback: You added teaspoons instead of tablespoon.
garnish the platter with Italian seasoning.
Figure An overview of the step-by-step task guidance scenario in our Qualcomm Interactive Cooking benchmark and dataset, where the multi-modal LLM provides instructions and feedback that are sufficient guide the user towards the goal, e.g., making a tomato mozzarella salad (above).
benchmarks [12, 22, 52] inadequately capture such interactive scenarios. These existing datasets and benchmarks largely consist of participants recording their daily activities expert demonstrations, which are insufficient for effectively assessing multi-modal L.LMs' ability to provide step-by-step instructions, they lack the critical scenarios where users make mistakes diverges from the plan. Therefore, we construct our Qualcomm Interactive Cooking benchmark and dataset leveraging the videos from the CaptainCook4D dataset [46], specifically because they contain these vital instances of user errors. Each video within Qualcomm Interactive Cooking is accompanied by detailed step-by-stey plan along with instructions and feedback, timed appropriately according the recipe and any mistakes made by the user.
Our contributions in detail are, 1. We introduce the Qualcomm Interactive Cooking dataset and benchmark by extending the CaptainCook4D dataset [46], with timed instruction and feedback messages. The instruction messages deseribe the next recipe step to follow and the feedback messages are provided to acknowledge successful instruction completions or mistakes made by the user. They designed be sufficient independently guide the introduce the LIVEMAMBA model, an light-weight multi-modal LLM designed to provide interactive instructions and feedback in cooking scenarios. 3. We evaluate state of the multi-modal LLMs on our Qualcomm Interactive Cooking benchmark, highlighting the strong performance of our LIVEMAMBA model such interactive scenarios.
complete the given recipe. 2. We
2 Related Work
Datasets for Procedural Activities. We provide an overview of related datasets and benchmarks in Tab. The Epic-Kin Kitchess [12] dataset consists ego-centric videos that document unseripted, daily activities within a kitchen environment. The Ego-Exo4D [23] dataset incorporates a considerably diverse array human activities, as performed by subjects exhibiting with variety of skill levels. However, the videos in these datasets are not driven by specific (multi-step) goal (Tab. 1, col 3). Assembly-101 [49] features videos of people with diverse skill levels assembling and disassembling 101 "take-apart" toy vehicles. HowTo 100M [44] and COIN [52] provides narrated instructional videos, featuring variety of activities. The YouCook2 [66] dataset contains cooking videos of diverse recipes. However, the subjects in these datasets are usually experts and the videos do not
Click here to access the data and click here to the code.

---

Table 1: Comparison to our Qualcomm Interactive Cooking dataset and benchmark (Ours): where, Mult-step Goal Driven refers to whether the videos are driven by specific goal (e.g., cooking recipe); Step-by-Step Instructions: whether the videos contain subjects following a set of step-by-step instructions; and Timed feedback: whether the subjects receive timed feedback per step that are sufficient to guide the subjects the goal.
Multi-step Goal Driven
Step-by-Step Instructions
Timed Feedback
Dataset
Domain
Longth (hrs)
Epic-Kitchens [12]
Cooking
100
Ego-4D [22]
Daily-life
X
3670
Ego-Exo4D [23]
Daily-life
X
X
1422
Assembly-101 [49]
Toy Assembly
X
X
513
HoloAssist [57]
Obj. manip.
V
166
HowTo100M [44]
Diverse
V
134k
COIN [51]
Diverse
X
512
YouCookv2 [66]
Cooking
X
176
WTAG [5]
Cooking
X
10
QEVD [45]
Fitness
X
474
Ours
Cooking
94
contain mistakes, unlike our Qualcomm Interactive Cooking benchmark and dataset (Tab. 1, col 5). Finally, HoloAssist [57] and WTAG [5] include videos where subjects are provided live feedhack by tele-operator, that cover certain mistakes user queries. But unlike Qualcomm Interactive Cooking benchmark and dataset, these feedback are not sufficient to independently guide the subject towards completion of the high-level goal.
Multi-modal Large Language Models. Vision language models have seen amazing progress in recent years following the language modeling breakthroughs. Earlier efforts on contrastive learning of vision and language representations led to CLIP [48] like architectures [27, 30, 31]. However, more recent works have reformulated vision tasks as text generation tasks giving rise to a diverse array of large multi-modal language models [1, 4, 21. 54 54-56] building on the success of large language models (LLMs). Early works in this space like Flamingo [2] leveraged pre-trained language models and vision adapters align the language and visual representational spaces. Follow up works like Llava [39], InstructBLIP [11] instruction tuned these models to enable excellent mudti dialogue capabilities. These works were primarily applied image data which have since been generalized to videos [3, 28, 33, 36, 43, 63). While these models can take videos as input, they only enable turn based interaction, i.e,, the models usually answer question or narrate the whole video lacking the ability to interactively respond to events in the video unlike our LIVEMAMBA model.
Streaming Video Large Language Models. Many recent works have started to explore online video understanding with large multi-modal language models. framework to enable online dialogue over videos with LLMs and use train models narrate long streaming videos. ReKV [13] proposes a training free approach to enable existing video language models solve streaming visual question answering (Stream VQA). Similarly. TimeChat Online [61], Stream [41], LiveCC [9], Flash Vstream [65], StreamMind [14], LION-FS [35] introduce various innovations make multi-modal language models to work on streaming video tasks. This has also prompted development of many streaming video benchmarks [6. 37, 45, 58, 60). However, all of the previous benchmarks deal with different forms of VQA where the model is asked questions paired with streaming video in natural settings whereas we deal with multi-step interactive videos where the model needs responds to the visual input without being prompted in a goal-directed setting.
[8] proposes
3 Qualcomm Interactive Cooking Benchmark and Dataset
Here we introduce the Qualcomm Interactive Cooking benchmark and dataset to evaluate the ability of multi-modal LLMs to provide step-by-step instructions, focusing on the cooking domain. The Qualcomm Interactive Cooking benchmark and dataset uses the videos from the CaptainCook4DD [46] dataset as contains actions with mistakes. This allows us create a setup akin "non-reactive simulation", where we task the multi-moda. LLM to produce the right instruction and feedback at the

---

Table Dataset statistics: where, the Followed Feedback and Divergent Feedback indicate whether the action associated with the feedback follows the given instruction not.
Followed
Followed
Divergent
Divergent
Total Length (hours)
Number of Videos
Number of Instructions
Success Feedback
Mistake Feedback
Success
Mistake
Split
Feedback
Feedback
Main Set
Training Validation
52.4
213
2913
2394
686
15.7
62
861
659
257
Testing
26.4
109
1489
1135
445
Advanced Planning Set
Training Validation
51.5
209
2888
2119
423
244
229
14.3
57
781
524
133
78
84
Testing 7.8
481
123
144
115
119
appropriate time, but the subject is non-compliant. Such a setup still provides us with useful insight into ability of multi-modal LLMs provide step-by-step task guidance, as fully reactive setups are not possible with offline datasets. We provide statistics of the Qualcomm Interactive Cooking benchmark and dataset in Tab. 2, including the total length in hours, number of videos, and numbers of instructions and feedback messages.
Instruction and Feedback Protocol. The instructions and feedback are annotated using the following protocol, starting from step-by-step plan. The first instruction of the plan occurs at the beginning of the video. If the user makes a mistake while completing the instruction, our benchmark and dataset contains a corresponding feedback just after occurs. If an instruction completed successfully, it is acknowledged. As long as the subject tries to complete the given instruction, irrespective of mistakes, we provide the instruction for the next step. If the user performs actions which are not aligned with the instruction, e.g., performing recipe steps out of order, we update the step-by-step plan, e.g., by repeating the instruction (see advanced planning set). Once all the steps are completed, acknowledge the completion for the whole plan.
Annotation Process. To generate instructions and feedback we leverage the annotations for temporal action segments with action descriptions in the CaptainCook4D dataset. We also leverage the mistake descriptions for the actions containing mistakes. First, we begin by removing noisy mistake annotations and then we annotate the timestamps where those mistakes occur. This allows us to annotate timely feedback for each mistake. We do this for all types of mistakes except order error and missing steps. Those mistakes reflect the cases where the subject ignores the provided instruction, i.e., they perform a recipe step out of order or ignore the recipe step completely. Such cases require more complex reasoning from the multi-modal LLM, the model needs to reason using the information of the instructions completed in the past and the remaining future steps. Thus, propose two sets of our Qualcomm Interactive Cooking benchmark and dataset: Main Set and Advanced Planning Set. The training, validation, and testing splits within each set follow the original video recoding split from CaptainCook4D., and the test splits correspond to the Qualcomm Interactive Cooking benchmark.
Main Set. This set reflect the case where the user largely follows the given instruction. To build step-by-step plan for this set we leverage the order of the actions the video. First we sort the actions by their start time. Normally one action forms one step in the plan. In cases where actions are performed in parallel (compound actions), we group those actions into the same step. In this set, instructions are to be given according to their order in the plan, thus eliminating the need for complex reasoning to provide the next instruction.
Advanced Planning Set. This set targets scenarios where the user diverges from the prescribed sequence, such as performing steps out of order. To construct it, we use the graph-structured recipe for each CaptainCook4D video to determine the correct step order. When the user does not follow the given instruction, we notify them and identify the action being performed based on the initial plan. The future instruction sequence is then updated to reflect this action (examples in the supplementary material). The test split includes only videos where the user diverges from the initial plan at least once. To reduce complexity, we further restrict the test set to vidcos without compound actions.

---

Gumta the plater winh ltakinn scasening
Si Add diicic oxtra ligin olive oll about leblospoor, over the eriie plotee
Streaming Video
IntaaVi (MFook)n))
1000 QOC
$100 OOC
DOJOCCOr
O-Former
O-Foree
Q-Former
Q-Foreer
O-Foree
Q-Forner
(K tokens)
f
0i in0
Language Backbone (Mamba-130M)
Re-planner
Instruction: sprinide
Feedback: You sprinited cheddar choese the boundary the platter.
platter Italian seasoning. 2: Add drizzle of extra virgin olive oil. about tablespoon, entiee plamee.
mozzarella cheese on top of the tometo throughout the platter.
Instruction: Now garrish platter with Italian seasoning
Figure Our LIVEMAMBA model architecture. The input video stream is processed by Intern ViT vision head which produces M tokens, and then reduced to K tokens by Q-Former. The language backbone produces feedback and invokes the Re-planner if necessary before the next instruction.
LIVEMAMBA for Step-by-Step Instructions
To deal with the challenge of guiding a user through step-by-step instructions we propose our LIVEMAMBA model. It is vision-language model with vision encoder and language model backbone to produce step-by-step instructions and feedback at the appropriate time-step. Given plan with of step-by-step instructions, the LIVEMAMBA model, shown in Fig. 2, provides the next instruction to the user, waits for the user to complete the instruction or make mistake and provide feedback message. To enable this interactive mode of operation, our LIVEMAMR MBA has the following key novel features: 1. It knows "when to speak", i.e., it is trained to respond at the appropriate time step when the user has successfully completed the provided instruction or has made a mistake. 2. To recognize the diverse set of possible mistakes that could be made by the user, the LIVEMAMRR model trained using a novel data augmentation scheme during the fine-tuning phase. 3. To provide the next relevant instruction, the LIVEMAMBA model re-plans using an external re-planning module, case the user diverges from the provided instruction and performs step out of order. Furthermore, the LIVEMAMBA incorporates recurrent Mamba- 130M [24] language model backbone to enable efficient training and inference over long video streams. Next, we discuss the LIVEMAMBA architecture and training details.
4.1 LIVEMAMBA Architecture
The LIVEMAMB: architecture consists of vision encoder and light-weight but powerful language backbone. A key consideration of the architecture of our LIVEMAMBA model is efficiency. For real-world application of our LIVEMAMBA model, inference using our model should idcally be possible using compute constrained edge devices such mobile phones or smart glasses. Reliance on external servers with larger compute capabilities is both costly and comes with additional latency issues. A second key consideration is that the cooking activities in the Qualcomm Interactive Cooking benchmark requires recognition fine fine-graine grained actions and small objects, e.g., adding the correct amount of salt, coating cup bowl with oil. Next, describe these components more detail.
Vision Encoder. The vision encoder consists of a InternViT-300M 448px V2 5 [10] head along with a Q-Former [32] based adapter. As shown in Fig. 2 the Intern ViT head produces M tokens per input video frame. As we use a light-weight language backbone, we do not want to offload raw visual processing to the language backbone and thus employ powerful Q-Former with multiple transformer layers. For every input video frame, the Q-Former reduces the M tokens to tokens which are then input to the language backbone. To generate these K tokens, the Q-Former uses four

---

cross attention layers to gradually blend in the important visual information from the input video frame. These tokens are then used as imput to the language backbone, interleaved with the text tokens corresponding to the step-by-step plan, instructions and feedback.
"When-to-Say". To respond at the appropriate timestep. following [8, 45] our LIVEMAMBA model uses two special tokens <vision> and <response>. The <vision> token allows the LIVEMAMBA model to interactively ask for the next video frame as input, while <response> allows the LIVEMAMBA to respond with an instruction or feedback at the appropriate time.
Iterative Re-planning, To determine the next instruction, we employ the following strategy. If the user executes the current iestruction-wheee from the original plan is issued. On the other hand, if the user diverges from the prescribed se- quence and performs an out-of-order step, our LIVEMAMBA invokes an external re-planner (sce Fig. 2). The re-planner receives as input the initial step-by-step plan, the set of completed steps, and feedback generated by LIVEMAMBA, and then selects the next optimal step to ensure successful task completion. In yet another scenario, if the intended step is skipped, the re-planner evaluates whether repeating the current instruction is appropriate given the user's progress. Similarly, if a step is completed prematurely, the re-planner determines whether to remove from the remaining plan. This iterative and selective re-planning mechanism provides two principal advantages. First, it enables selective use of large-scale models with enhanced reasoning capabilities only when necessary; in our implementation, Qwen3-32B [16] serves as the re- re-planner. Second, it eliminates the need for LIVEMAMBA to internally store and recall the entire plan, allowing it to focus exclusively on delivering accurate and timely feedback (more details in the appendix).
correctly or with errors- the subsequent step
4.2 LIVEMAMBA Training
We now provide details of our training scheme, which is composed of two stages.
Pre-training. This stage aligns the Q-Former's vision embeddings with the language backhone's 's text embeddings by training only the Q-Former adapter. We use diverse set of image and video datasets to instill two key visual skills: First, for object grounding. LIVEMAMBA is trained on image datasets like LVIS [25] (for diverse household objects) and on video data using VISOR annotations from EPIC-KITCHENS [12]. Tasks include image/video captioning, object recognition, and bounding box prediction. Second, for fine-grained action understanding, LIVEM AMBA learns action recognition from SSv2 [20] and video narration from EPIC-KITCHENS and Ego4D. Narration helps ground the model actions in these large-scale egocentric datasets, particularly cooking activities.
Fine-tuning. During the fine- linettuning phase the LIVEMAMBA model is trained to provide the ap- propriate instructions and feedback at the appropriate time, using the <vision> and <feedback> special tokens as described above. At this stage, both the Q-Former's 's based adapter and the language backbone is trained. The LIVEMAMR, needs to recognize the successful completion of instructions and mistakes. To this end, in addition to training on the step-by-step instructions and feedbacks from Qualcomm Interactive Cooking dataset, apply several augmentations described below.
4.3 LIVEMAMBA Augmentation
We now provide details of our data augmentation scheme during the fine-tuning phase.
Temporal Augmentation. To maintain temporal accuracy of feedbacks while dealing with long videos in the Qualcomm Interactive Cooking benchmark, we introduce temporal jittering during train- ing. Specifically, we jitter the starting timestamp of each instruction by constant =K seconds. This temporal jittering deals that fact that predictions by autoregressive models, e.g., our LIVEMAMBA, accumulate errors. Jittering the starting timestamp of an instruction ensures that the LIVEMAMBA can successfully predict feedbacks irrespective of the previous accumulated error. In practice, we find K 30 to work well.
Instruction Completion Augmentation. To help the LIVEMAMBA recognize the successful comple- tion of instructions, we augment our training set by converting videos from the EPIC-KITCHENS and Ego4D datasets to the step-by-step instruction and feedback format of our Qualcomm Interactive Cooking dataset. In detail, we consider the action descriptions in the EPIC-KITCHENS dataset and

---

Table Zero-shot evaluation on the main set of the Qualcomm Interactive Cooking benchmark.
Instruction
Mistake
Method
IC-Acet
Prec.
Rec.
FIf
BERT
ROUGE-L
LLaVA-NeXT [40]
1.4
0,00
0.00
0.00
0.000
0.000
Video-ChatGPT [43]
1.6
0.00
0.00
0.00
0,000
0,000
VideoChat2 [34]
1.6
0.00
0.00
0.00
0.000
0,000
Video-LLaVA [67, 38]
0.00
0.00
0.00
0.000
0.000
Videol. [62]
1.8
0.00
0.00
0.00
0.000
0.000
Vidcollm-onlinc [8]
0.03
0,02
0.98
0.04
0.332
0.248
Qwen2-VL.-7B [56]
6.3
0.02
0.69
0.05
0.377
0.256
Qwen2.5-VL-7B [19]
18.9
0.18
0.01
0.02
0.299
0.219
2.5ini-2.5-Flash [53]
23.1
0.01
0.22
0.02
0.410
0.342
the Ego4D) Goal-Step datasets and use Qwen2.5-8B model [15] to convert these action descriptions instructions. We provide the instruction at the action start time and feedback message at the action end timestamp. As these datasets do not contain any mistakes, the feedback messages acknowledge the successful completion of the instruction.
Counterfactual Mistake Augmentation. Recognizing mistakes and providing timely feedback is key challenge of our Qualcomm Interactive Cooking benchmark. To this end, we formulate novel data augmentation scheme to generate (counterfactual) mistakes in the EPIC-KITCHENS and Ego4D datasets. First, convert the action description to plausible grounded counterfactual action descriptions. These grounded counterfactual action descriptions are used to generate instructions makes a mistake
and thus construct scenarios where the user tries follow the given instruction (more details in the appendix). To help recognize divergent mistakes, where the user does not follow the provided instruction, we augment the fine-tuning dataset by swapping instructions between recipe steps. The feeback is constructed to explicitly state that the user did not follow the provided instruction and instead performed different action. This format of feedback triggers the re-planning module during the inference stage (more details in the appendix).
5 Experiments
We begin with an introduction to the metrics used to evaluate models for interactive step-by-step task guidance followed by the experimental results.
5.1 Evaluation Metrics.
We use the following metrics measure both the ability of the models detect successfully completed instructions and to provide feedback when mistakes occur.
Instruction Completion Accuracy (IC-Acc). IC-Acc measures the proportion of instructions that the model correctly detects as successfully completed by the user. Specifically, this requires that the model provides the correct instruction, the user completes it, and the model identifies this completion. To mitigate temporal annotation noise, we consider a prediction correct if it falls within small window centered the ground-truth completion time. In practice, a 30-second window is sufficient: typically spans the last ~55 of the current step and the first ~55 of the next step in the Qualcomm Interactive Cooking benchmark, balancing robustness to noise with accuracy.
Mistake Detection Precision (Prec.), Recall (Rec.) and Fl. To calculate mistake detection Precision, Recall, and F1 scores in our interactive streaming setup, we define: Mistake True Positive: A mistake detected by the model within small temporal window centered on the timestamp of ground truth mistake. * Misttkee False Negative: A ground truth mistake that the model fails to detect within this temporal window. Mistake False Positive: mistake detected by the model when no corresponding ground truth mistake occurs within the temporal window. Mistake True Negative: An instruction correctly followed by the user (no ground truth mistake) where the model correctly detects no mistake. We use the same temporal window size as in the IC-Ace metric.
7

---

Table 4: Evaluation of fine-tuned models on the Qualcomm Interactive Cooking benchmark ("indicates models fine-tuned by us).
Instruction
Mistake
Method
IC-Acet
Prec.t
Rec.1
FIf
BERT
ROUGE-LE
Main Set
Videollm-online' [8]
7.6
0.04
0.01
0.01
0.434
0.412
LIVEMAMBA (w/o-ICAug)
7.8
0.05
0,01
0.01
0.605
0.542
LIVEMAMBA (w/o-CFAug)
0.12
0.03
0.05
0.558
0.511
LIVEMAMBA (Ours)
31.5
0.17
0.10
0.13
0.651
0.561
Advanced Planning Set
LIVEMAMBA (woo-reP)
10.9
0.38
0.10
0.16
0.912 0.941
0.901
LIVEMAMBA (Ours)
12.6
0.38
0.13
0.19
0.927
Table 5: Turn-based evaluation of on the main set of the Qualcomm Interactive Cooking benchmark.
Instruction
Mistake
Method
IC-Acet
Prec.t
Rec.1
Flf
BERT
ROUGE-LI
Vide3LL [62]
17.8
0.08
0.61
0.15
0.406
0.346
Qwen2-VL-7B [56]
19.4
0.06
0.46
0.11
0.398
0,293
Qwen2.5-VL-7B [19]
38.9
0.11
0.04
0.06
0.348
0.230
LIVEMAMBA (Ours)
51.0
0.22
0.17
0.19
0.631
0.535
Mistake Feedback Fluency (BERT and ROUGE-L). To measure the fluency of the models in providing appropriate feedback, we use the ROUGE-L and BERT scores, We only consider the fluency of feedback provided in case true positive mistake detections. That is, when the feedback is provided within the temporal window of a ground truth feedback described above. Importantly, these scores are only meaningful when comparing models with similar true positive detection rates, since differences in detection accuracy can confound fluency comparisons. This is because models with lower detection rates produce fewer feedback instances, which can skew the distribution of ROUGE and BERT scores and make fluency appear artificially higher or lower.
5.2 Zero-Shot Evaluation
State of the art multi-modal LLMS are usually limited to turn-based interactions. Applying such models to the streaming setup of the Qualcomm Interactive Cooking benchmark is thus highly challenging. Therefore, we need to employ a special online prompting strategy. This strategy involves prompting the model at regular intervals to detect both successful completions of instructions and mistakes. However prompting after every input frame is not computationally feasible. To balance accuracy and compute requirements, prompt the models at interval of seconds. We provide details of the prompts the appendix.
On the other hand, streaming multi-modal LLMs such as Videollm-online [8] are targeted at providing online narrations of input video streams. Such models are therefore not directly applicable to our Qualcomm Interactive Cooking benchmark. Thus, evaluate such models, we first ask the model to generate online narrations for the whole vidco. We then feed these narrations in an online manner to a "helper" LLM that given the narrations and the action instruction, predicts if the action is completed or not. "yes", we move on to the next instruction. If "not", we ask the "helper" LLM to predict if the narrations suggest mistake has been made and provide feedback for correction. We use Phi-3-mini-4p 4k-strutt [18] as the "helper" LLM.
We report the zero-shot evaluation results in Tab. 3 for the main set of the Qualcomm Interactive Cooking benchmark. Overall, Gemini-2.5-Faas [55] performs best. can recognize 18.9% of instructions being successfully completed by the user. Models such as
[62], Video-LLaVA [38, 67]. VideoChar2 [34], Video-ChatGPT [43], LLaVA-NeXT [40] have trouble following instructions. Even when prompted to detect if the person has completed given instruction

---

these models tend to answer "yes" too early. This highlights gap in understanding scenes as they unfold in a streaming setup. This is likely an artifact of the question-answer of these models where the entire video is always available to the model. Furthermore, Videollm- online [8] predicts narrations which are not fully informative of the action and therefore a mistake is detected very often leading to very high mistake recall and a very low instruction completion detection accuracy.
style training scheme
Overall, when it comes to mistake detection, none of the zero-shot approaches performs well. Qwen2- VL-7B-Instruct [56] overestimates the occurrence of mistakes, leading to higher recall but low precision. Swen2.5-VL-7BB Instructt [19] on the other hand is more precise in detecting mistakes, but with low overall F1 score. This weak performance can be attributed to the fact that detecting mistakes in the Qualcomm Interactive Cooking benchmark is very challenging due to a variety of reasons. First the fine-graie fine-grained nature of many mistakes, i.e., taking teaspoon vs tablespoon or 2 teaspoons of sugar, spilling flour on the kitchen counter etc, requires both fine-graines object recognition and aetion-recognition abilities. This is made additionally challenging by the fact that the model needs to look out of a diverse set of possible mistakes. Secondly, mistakes need to be detected as soon as they occur (similar to the instruction completions). This again exposes the limitations of current models in understanding scenes as they unfold in a streaming setup.
5.3 Evaluation of Fine-tuned Models
Here we evaluate our LIVEMAMBA model line-tuned on the Qualcomm Interactive Cooking dataset. Unlike the zero-shot baselines such as Qwen2.5-VL-7B --nnsruunttt [19] in Tab. 3, our LIVEMAMBA model is fully interactive, i.e., it can decide "when to say" to point out successful completion of instructions and mistakes after every input frame. Thus, it does not require expensive prompting strategies. We also consider two ablations of our LIVEMAMBA model on the main benchmark set: 1. without instruction completion augmentation (w/o-ICAug). 2. without counterfactual augmentation (w/-CFAug). In case of the advanced planning set, we consider an ablation without an external re-planning module (w/o-rP).
Firstly, in Tab. 4 we see that our LIVEMAMBA model significantly improves performance over the zero-shot models in Tab. 3. We show example predictions of our model in Fig. 3. The weaker performance of the fine-tuned Videollm-online model shows that pre- pre-training only on narration data is not sufficient. Furthermore, the use of our efficient Mamba- 130M backbone [24] allows to use a higher number of embedding tokens per frame (32 vs 10) at similar memory costs compared to transformer based models such as Videollm-online In addition to the pre- pre-training data, the key reasons for the improved performance of the LIVEMA AMBA model is our data augmentation scheme during the fine-tuning phase. The addition of instruction completion augmentation (ICAug) leads to significant improvement in IC-Acc from 7.8% to 14 14.3% in the main set along with an improvement in mistake Fl scores. The improvement in mistake Fl score is particularly interesting, as correctly recognizing successful completion of instructions helps the LIVEMAMBA more precisely point out mistakes. Furthermore, the addition of counterfactual mistake augmentation (CFAug) significantly boosts the mistake Fl score from 0.05 to 0.10 in the main set, along with significant jump in feedback fluency. The significant jump in mistake detection accuracy shows that the curation of high quality mistake data is promising direction for future research. Furthermore, we see that the use of the external re- -e-panninng module leads to better performance on the advanced planning set. The external re- re-planning module helps the LIVEMAMBA select the next instruction in case of divergent mistakes over the complex graph structured recipes in the Qualcomm Interactive Cooking benchmark from CaptainCook4DD. Also note that, our counterfactual (divergent) mistake augmentation scheme significantly boosts the mistake detection scores in this set. Overall, providing the correct instruction in the advanced planning set remains highly challenging, shown by the lower IC-Acc scores. Note that, the majority of the mistakes in the advanced planning set are divergent: they point out the intended action given the instruction and the divergent action. Furthermore, the mistake detection scores are calculated only when the provided instruction matches with the groundtruth instruction. Thus, the mistake detection scores are not comparable across the main and advanced planning sets.
In terms of throughput, on a consumer Nvidia H100 GPU, our Live model has a real-time factor of 4 on average: it can process input data four times as fast at 8.1 frames per second as it becomes available at 2 frames per second. In terms of latency, that is time to generate the first token, 1.1 seconds average. Re-planning using the Qwen3-32B[16) model takes 6. 1 seconds on average.

---

Now 1/2
You ground black pepper the cornectly.
teaspoon ground black to the
tablespoon oil in nonstick frying pan.
non-stick frying heat the
New take torneto.
Ground Truth
Now teaspoon ground
LiveMamba
black pepper the
groune black papper the bowl cornectly.
tablespoon oil in nonatick frying
the oil in non- stick frying
Now take tomato.
Figure 3: Our LIVEMAMBA is able to successfully recognize the person has added the black pepper as instructed and points out when the person should heat the oil in non-stick frying pan, in the Qualcomm Interactive Cooking benchmark.
5.4 Turn-based Evaluation
The streaming evaluation in Tabs. and tested models on multi-step user guidance, setup inherently challenging due to enror propagation; for instance, failing to detect instruction completion renders subsequent predictions incorrect. While this reflects real-world scenarios where accurate completion detection is vital for task guidance, such difficulty make it more challenging to measure progress on the Qualcomm Interactive Cooking benchmark. Therefore, we also propose a turn-based evaluation, where models assessed each recipe step independently.
We report the results in Tab. on the main set of the Qualcomm Interactive Cooking benehmark. Overall we observe much higher IC-Acc scores compared to the streaming setup of Tabs. 3 and 4. Most significantly, we observe higher mistake precision, recall, scores of our fine-tuned LIVEMAMBA model compared to the zero-shot models. This again shows that fine-tuning Interactive Cooking dataset along with our data augmentation scheme during fine-tuning allows our LIVEMAMES MBA model to better recognize mistakes compared to zero-shot baselines. Thus, the surn-based evaluation Tab. 5 along with the streaming setup in Tabs. 3 and 4 together provide a more detailed picture of progress the Qualcomm Interactive Cooking benchmark.
Qualcomm
6 Conclusion
We address the challenge of enabling multi-modal LLMs to provide live, interactive step-by-step guidance. To this end, introduce Qualcomm Interactive Cooking dataset and benchmark, featuring densely annotated, timed instructions and feedback, including for mistakes, timestamped to visual occurrences. We also propose LIVEMAMBA MAVEM novel streaming multi-modal LLM designed for this task. LIVEMAMBA utilizes a lightweight Mamba backbone, "when-to-say" mechanism, novel data augmentation for mistake recognition, and iterative re-planning for adaptive delivery. Evaluations show existing multi-modal LLMs struggle with live task guidance, whereas LIVEMAMBA, establishes strong baseline, significantly outperforming others in diverse metrics.
Limitations. Our work is focused on the cooking domain through Qualcomm Interactive Cooking. While LIVEMAMBA establishes strong baseline, detecting subtle mistakes and robustly handling complex scenarios, particularly those involving order errors or missed steps in the advanced planning set, remains challenging, for all state of the art open-source models.
Broader Impacts. Language models can produce harmful and biased content, make incorrect claims and produce wrongful advice. This needs to be taken into account when interacting with, deploying or building on these models, particularly in domains where incorrect advice may lead to physical harm. It also has to be taken into account that any computer vision model processing visual information about human subjects could in principle extract information beyond what is required for the use-case, such biometric information.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
[2]
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooci, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. In NewrIPS, 2022.
[3] Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding. and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal Ilms for video understanding with interleaved visual-textual tokens. arXiv preprini arXiv:2404.03413, 2024.
[4] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization. Text Reading, and Beyond, 2, 2023.
[5] Yuwei Bao, Keunwoo Peter Yu, Yichi Zhang, Shane Storks, Itamar Bar- Yossef,, Alexander De La Iglesia, Megan Su, Xiao-Lin Zheng, and Joyce Chai. Can foundation models watch, talk and guide you step by step to make a cake? In EMNLP Finulings, 2023.
[6] Apratim Bhattacharyya, Sunny Panchal, Mingu Lee, Reza Pourreza, Pulkit Madan, and Roland Memisevic. Look, remember and reason: Visual reasoning with grounded rationales. In ICLR, 2024.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[8] Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In CVPR, 2024.
[9] Joya Chen, Ziyun Zeng, Yiqi Lin, Wei Li, Zejun Ma, and Mike Zheng Shou. Livece: Learning video I1m with streaming speech transcription at scale. arXiv preprint arXhv:2504.16030 2025.
[10] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generie lingultic tasks. CVPR, 2024.
[11] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards seneral-purpose vision-language models with instruction tuning. In NeurlPS, 2023.
[12] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma. Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for spic-kitchens--e In IJCV, 2022.
[13] Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, Tao Zhong, Hao Cheng, Bolin Li, Wanggui He, Fangxun Shu, and Hao Jiang. Streaming video question-answering with in-context video ky-cache retrieval. arXiv preprint arXiv:2503.000 005400 2025.
[14] Xin Ding. Hao Wu, Yifan Yang, Shiqi Jiang, Donglin Bai, Zhibo Chen, and Ting Cao. Stream- mind: Unlocking full frame rate streaming video dialogue through event-gated cognition. arXiv preprint arXiv:2503.06220 2025.
[15] An Yang et. al. Qwen2.5 technical report. CoRR, abs/2412.151115 2024.
[16] An Yang et al. Qwen3 technical report. CoRR, abs/2505.0995 2025.

---

[17] Grauman et. al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.
[18] Marah I Abdin et. al. Phi-3 technical report: A highly capable language model locally on your phone, CoRR, 01122404.142199 2024.
[19] Shuai Bai et. al. Qwen2.5-vl technical report. CoRR, abs/2502.13923. 2025.
[20] Raghay Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Frees Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
[21] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.22 21783, 2024.
[22] Kristen Grauman, Andrew Westbury, Eugene Bymee Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagara- jan, Iijaa Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Erie Zhongcong Xu, Chen Zhao, Siddhant Bansal, et al. Ego4d: Around the world in 3, 000 hours of egocentric video. In CVPR, 2022.
[23] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Tri- antafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first- and third-person perspectives. CoRR, abs/2311 18822, 2023.
[24] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. CoRR, abs/2311..00752, 2023. doi: 10.4555/ARXIV22112.0722. URL https://dol org/ 10. (16550/arXiv..
[25] Agrim Gupta, Piotr Dollár, and Ross B. Girshick. LVIS: A dataset for large vocabulary instance segmentation. In CVPR, 2019.
[26] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
[27] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and langun-a representation learning with noisy text supervision. In International conference on machine learning, pages 4916. PMLR, 2021.
[28] Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et al. Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization. arXiv preprint arXiv:2402.03111,220 2024
[29] Arthur B Kahn. Topological sorting of large networks. Communications of the ACM, 5(11): 558-562, 1962.
[30] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances neural information processing systems, 34:9694-9705, 2021.
[31] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888-12900. PMLR, 2022.
[32] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML., 2023.
[33] KunChang Li, Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305. 06355, 2023.
12

---

[34] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In CVPR, 2024.
[35] Wei Li, Bing Hu, Rui Shao, Leyang Shen, and Liqiang Nie. LION-FS: fast & slow video- language thinker as online video assistant. In CVPR, 2025.
[36] Yanwei Li, Chengyao Wang, and Jiaya Jia, Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323-340. Springer, 2024.
[37] Yifei Li, Junbo Niu, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He. Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, et al. Ovo-bench: How far is your video-llms from real-world online video understanding? arXiv preprint arXiv:2501.00 10, 2025.
[38] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.1012) 2023.
[39] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023.
[40] Haotian Liu, Chumyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https: //1lava-va vl. wiittvauvl.gttuubb..... ht/blog///////////o
[41] Jihao Liu, Zhiding Yu, Shiyi Lan, Shihao Wang, Rongyao Fang, Jan Kautz. Hongsheng Li, and Jose M Alvare. Streamchat: Chatting with streaming video. arXiv preprint MxXiv22422008666 2024.
[42] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL httpps://ppenreviet wwt/ wetffoommmiddieoe hiii:iddd-kkkigc""
[43] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. GPtoo- Towards detailed video understanding via large vision and language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, ACL, August 2024.
[44] Antoine Miech, Dimitri Zhukov, Jean- Caanttaptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning text-video embedding by watching hundred million narrated video clips. In ICCV, 2019.
[45] Sunny Panchal, Apratim Bhattacharyya, Guillaume Berger, Antoine Mercier, Cornelius Böhm, Florian Dietrichkeit, Reza Pourreza, Xuanlin Li, Pulkit Madan, Mingu Lee, Mark Todorovich, Ingo Bax, and Roland Memisevic. What to say and when to say it: Live fitness coaching as a testbed for situated interaction. In NeurlPS, 2024.
[46] Rohith Peddi, Shivvrat Arya, Bharath Challa, Likhitha Pallapothula, Akshay Vyas, Bhavya Gouripeddi, Qifan Zhang, Jikai Wang. Vasundhara Komaragiri, Eric D. Ragan, Nicholas Ruozzi, Yu Xiang, and Vibhav Gogate. Captaincook4d: A dataset for understanding errors in procedural activities. In NeurlPS, 2024.
[47] Reza Pourreza, Rishit Dagli, Apratim Bhattacharyya, Sunny Panchal, Guillaume Berger, and Roland Memisevic. Can rision-language models answer face to face questions in the real-world? arXiv preprint urXiv:2503. 19356, 2025.
[48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference machine learning. PmL.R, 2021.
pages
[49] Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang. and Angela Yao. Assembly101: large-scale multi-view video dataset for understanding procedural activities. In CVPR, 2022.
13

---

[50] Yale Song, Eugene Byrne, Tushar Nagarajan, Huiyu Wang, Miguel Martin, and Lorenzo Torresani. Ego4d goal-step: Toward hierarchical understanding of procedural activities. In NeurlPS, 2023.
[51] Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. COIN: A large-scale dataset for comprehensive instructional video analysis. In CVPR, 2019.
[52] Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In CVPR, 2019.
[53] Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. CoRR, abs/2507.06261, 2025.
[54] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- bean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: A family of highly capable multimodal models. CoRR, abs/23 12.1 122.11880 1805, 2023.
[55] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint wwXiv:2403.055330 2024.
[56] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. arXiv preprint arXiv:2409.121190 2024.
[57] Xin Wang. Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, Neel Joshi, and Mare Pollefeys. Holoassist: an egocentric human interaction dataset for interactive Al assistants in the real world. In ICCV, 2023.
[58] Yuxuan Wang, Yueqian Wang, Bo Chen, Tong Wu, Dongyan Zhao, and Zilong Zheng. Omn- immi: A comprehensive multi-modal interaction benchmark in streaming video contexts. arXiv preprint arXiv:2503.22952 2025.
[59] Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li. Bei Yu, and Jiyy Jia. Visionzip: Longer is better but not necessary in vision language models. In CVPR, 2025.
[60] Zhenyu Yang, Yuhang Hu, Zemin Du, Dizhan Xue, Shengsheng Qian, Jiahong Wu, Fan Yang, Weiming Dong, and Changsheng Xu. Svbench: A benchmark with temporal multi-turn dialogues for streaming video understanding. arXiv preprint arXiv:2502. 108100 2025.
[61] Linli Yao, Yicheng Li, Yuancheng Wei, Lei Li, Shuhuai Ren, Yuanxin Liu, Kun Ouyang, Lean Wang, Shicheng Li, Sida Li, al. Timechat-onling 80% visual tokens are naturally redundant in streaming videos. arXiv preprint arXiv:2504. 17343, 2025.
[62] Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng. Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing. and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding, 2025.
[63] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. In EMNLP System Demonstrations, 2023.
[64] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858 2023. URL https://arxiv.org org/ abs/2306. 02858.
[65] Haoji Zhang, Yigin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-basee real-time understanding for long video streams. arXiv preprint 24X.v22406.08085, 2024.
14

---

[66] Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018.
[67] Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n n-modality by language-base based semantic alignment. arXiv preprint arXiv:2310.01852, 2023.
15

---

Appendix
A Overview
In the following we provide details of the annotation process of our Qualcomm Interactive Cooking dataset and benchmark, details of the re-planner used in the LIVEMAMBA model, training details of our LIVEMA MBA model, additional qualitative results from our LIVEMAMBA model, and details of the prompts used for zero-shot evaluation.
B Qualcomm Interactive Cooking Annotation Details
Our Qualcomm Interactive Cooking dataset and benchmark is built upon the CaptainCook4D dataset [46]. The CaptainCook4D dataset contains 384 videos in total. Each video records person preparing a dish given a recipe, such as breakfast burritos and tomato mozzarella salad, from egocentric view. Each video is associated with graph structrrdd recipe", and annotated with temporal action segments: action descriptions with the corresponding starting time and ending time. If an action contains mistake, then there is description of the mistake. However, there is timestamp annotation for the mistake. The average duration of action segments in CaptainCook4D is about 72.78 seconds, which is about 106 frames if using frame rate 2. This also motivates to use a light-weight Mamba based model to encode more frames given the hardware constraint.
The CaptainCook4DD dataset includes categories of mistakes: preparation error, technique error, measurement error, temperature error, timing error, order error, and missing steps. We annotate the timestamps of the mistakes when they just happened. We remove some noisy mistake annotations in CaptainCook4D, and annotate for all mistake categories except for order error and missing steps. Tab. 6 shows the numbers of mistake instances per mistake category. Note, we show the mistake statistics here for a complete description of the Qualcomm Interactive Cooking dataset and benchmark. This mistake category information is not used in our experiments presented the paper.
Our Qualcomm Interactive Cooking features with step-by-step plan, and timestamped instructions and feedback for each video recording. In what follows, we describe how we obtain those for the main set and advanced planning set respectively.
B.1
Main Set
In the main set (Fig. 4), assume that the user always tries to follow the given instruction. That is, there is no action order error. Given a video, we first sort the actions by their starting time in ascending order (primary key) and ending time in descending order (secondary key). Based this order, we build the step-by-step plan. Usually one action description forms one step in the plan. In cases where one action is temporally contained in another action (actions performed in parallel), we group those action descriptions into one step. Accordingly, we create action groups, where one step the plan corresponds to one action group. The first step in the plan provides the contents for the next instruction to be given.
Given video, treat the first action's starting time the video starting time and the first instruction is given at that time. Once an action finishes, its description is removed from the step-by-ster plan. Once all the actions in the current action group finish, the current step in the plan will be empty. We remove the empty step in the plan and give the instruction for the next step. The instruction sentence containing all the information in the action descriptions in the step. Once an action finishes successfully, we acknowledge the success via summarizing the description of the just finished action, and using the words like successfully, correctly, properly, and etc. This type of feedback is given at the action's 's ending time. If an action contains a mistake, we give the feedback regarding the mistake our annotated timestamp. The feedback is a sentence pointing out the mistake based on the mistake description given in CaptainCook4D. Once all the steps in the initial plan are completed, we output the feedback "You have finished all the steps." to acknowledge the completion. Fig. 4 shows data samples from the main set. Whenever an action finishes, it is removed from the
'Examples be found at https: attps://ccptainncookk..
16

---

Table Mistake category statistics.
Split
Preparation Err.
Technique Err.
Measurement Err.
Temperature Err.
Timing Ert.
Main Set
Training
198
224
159
27
78
Validation
63
79
68
10
37
Testing
110
156
113
17
49
Advanced Planning Set
Training Validation
182
216
152
27
75
54
66
55
10
32
Testing
68
Table Re-plan statistics on the advanced planning set (averaged over videos requiring re- re-planning in each split).
Number of Videos Requiring Re-planning
Average Video Length (minutes)
Average Number of
Average Number of Re-plan Steps
Split
Instructions
Training
94
12.8
13.2
4.3
Validation
29
13.2
13.6
4.8
Testing
36
12.9
13.4
6.1
step-by-step plan. The examples show that recognizing mistakes requires fine-grained understanding of user actions and objects in the scene across long time-horizons.
B.2 Advanced Planning Set
In the advanced planning set (Figs. and 6), we include cases where the user performs an action diverging from the instruction. Given video with its associated graph-structured recipe, we append the actions that are not performed in the video (missing steps) to the ordered actions from the main We then topologically sort the actions using the Kahn's algorithm [29]. We discard some videos which can not be topologically sorted, because there exist actions in the video but absent from the recipe graph. This (newly) sorted action order is used to build the step-by-step plan. Normally one action description forms one step in the plan. We group consecutive action descriptions into the same step only if there exists step in the main set's plan containing the exact same action descriptions. Note that the action groups remain consistent with the main set. An important difference to the main set is that, is possible have divergence between a step the plan and the corresponding action group, in case the user does not follow the given instruction.
As in the main set, once the user finishes an action, we remove its description from the plan and remove any empty step; and once all the actions in the current action group are finished, we provide the instruction for the next step. If the action group matches the corresponding step, that is, the user performs actions following the given instruction, we follow the same convention as in the main set. the user performs an action diverging from the given instruction, we notify the user in the feedback. Specifically, if the action is performed without mistakes but does not follow the instruction, feedback given the action's 's end time, starting with sentence "You did not follow the instruction. and describing the action that is performed. If the action performed contains mistakes, the feedback given when the mistake occurs, starting with the sentence "You are not following the instruction." describing the action that the user is trying to perform based on the initial plan, and pointing out the mistake.
In cases where the current step is not empty after all actions in the current action group finish, that is, the instruction is not followed at all partially completed, we manually decide whether to keep or remove the current step in the plan based on the actions that the user has performed. This corresponds to the cases that require re-planning as the set future steps need to updated. Tab. shows the "re-plan" statistics in our advanced planning set. As can be seen from the table, re-planning is required for about every 2.7 instructions or every minutes on average. Figs. and show data samples from the advanced planning set. Note, in Figs. 5 and 6, "Plan" means that the action follows
17

---

the instruction, and thus once the current step is finished, it is removed from the step-by-step plan. "Re-plan" means that the action diverges from the instruction, and when giving the next instruction, we need to decide whether to (re)instruct the user about the current step or not.
C LIVEMAMBA Re-planner
The re-planner is invoked (during inference) whenever the user does not follow the given instruc- tion. This only occurs in the advanced planning the Qualcomm Interactive Cooking bench- mark. To identify such cases, we train the LIVEMAMBA to add a "You are not following the instruction."
"You did not follow the instruction. prefix to feedback messages in case the user does the follow the given instruction. Note that, during training, the LIVEMAMBA model is trained using the ground truth sequence of instructions (as the LIVEMAMBA model is trained on a single recipe step at time) and the re- re-planner is not used. During inference as we do have have the ground truth sequence of instructions, we use the re-planner to update the step-by-ster instruction plan and provide the correct instructions (and feedbacks). We do this in two steps.
First, the re-planner the instructed recipe step using the following prompt:
the Qwen3-32B [16] model extract the recipe step performed instead of
Retrieve the performed action
You are an expert cooking instructor. You are observing a user cooking a given recipe step by step.
#HINSTRUCTIONS:
Here are the recipe steps: trecipe_steps]. The last instruction that you provided the person is: fast__nsttructio]. The person did not follow your instruction and performed a different recipe step by mistake which did correspond the provided instruction. So you provided this feedback to the person: [last_ feedback). Which recipe step did the person likely perform instead of the step in the last instruction. RETURN THE RECIPE STEP AS A PYTHON STRING. ENSURE THAT YOU OUTPUT A RECIPE STEP AND DO NOT OUTPUT ANYTHING OTHER THAN A RECIPE STEP.
Secondly, the re- re-planner determines if the current instruction needs to be repeated, based on the recipe steps and the past steps completed by the person. We the following prompt for the Qwen3-32B [16] model:
Decide whether repeating the last instruction
You are an expert cooking assistant. You are helping a user to make pacipe_nampe according to the following recipe steps: [recipe_steps].
#IINSTRUCTIONS:
The user has already completed Ipast_ completed_ppeeeeee serpd__tepppeeeeeeeeeeee [eep_coutt]]] steps: [past_ cecesstecmeeeddd__c steps]. Decide whether it is appropriate the effect of all the steps that the user performed. Your answer must begin with 'Yes" or 'No", followed by explanation.
to ask the user fast_instructed, action], considering
If the answer is "No", that is, the last instruction is not to be repeated, find all parent actions of the last action that the user performed from the corresponding recipe graph, and remove those actions from the step-by-step plan if they exist. Otherwise, do not further update the step-by-step plan.
D LIVEMAMBA: Training Details
D.1 Pre-training Data
As discussed the main paper, for object grounding, LIVEMAMBA trained on image datasets, i.e., LVIS [25] (for diverse houschold objects) and video data using VISOR annotations from
18

---

EPIC-KITCHENS [12]. The data follows a question answer format, where the questions are of the following format (following [39]):
LVIS EPIC-KITCHESSS Grounding Questions
Please provide the bounding box coordinates for the A:
Where is the - located in the image?,A: It is located at What are the coordinates of the bounding box encompassing the ?, A:
Can you pinpoint the in the image by giving me its bounding box coordinates?, A: Yes, it is at
Where is the
in the image? Give me the bounding box coordinates,, A:
need the top-left and bottom bottom-right corners of the <1>'s bounding box. What they?, A:
Could you identify the in the image and tell me its location using bounding box coordinates?, A: Yes, it is at
Describe the image concisely., A:
Provide a brief description of the given image., A:
Offer a succinct explanation of the picture presented., A:
Summarize the visual content of the image., A:
Give short and clear explanation of the subsequent image., A:
Share concise interpretation of the image provided., A;
Present compact description of the photo's 's key features., A:
Relay a brief, clear account of the picture shown., A;
Render clear and concise summary of the photo., A:
Write terse but informative summary of the picture., A:
Create compact narrative representing the image presented., A:
For action recognition tasks on SSv2 [20], we use the following format:
SSv2: Action Recognition Questions
Describe the action performed in this video in detail and name the objects that the I interacts with?, A:
Can you provide step-by-step description the action in the video which includes the specific objects that touched manipulated?, A:
Tell me description of the action performed by me in the video that includes the names of any items that interact with., A:
What action happens in the video and what objects involved in the action?, A:
Describe my actions and the objects that come across., A:
For narration tasks EPIC-KITCHENS we convert the action descriptions second person format, e.g. "pick up plate" to "You picked up plate." For Ego4D, we employ the narrations used by [8].
D.2 Fine-tuning Data
As mentioned the main paper, EPIC-KITCHENS and Ego4D datasets included the fine-tuning phase do not include instances of mistakes. Therefore, we formulate novel data augmentation
19

---

scheme to generate (counterfactual) mistakes. First, we describe the process used on Ego4D to generate counterfactual action descriptions detail.
Counterfactual Mistake Augmentation. To this end, we utilize the Ego4D goalstep annotations [50] and the FHO annotations [17]. The FHO annotations largely include fine-grained short duration actions while the goalstep annotations include longer-ranged actions more closely aligned with the recipe steps in our Qualcomm Interactive Cooking benchmark. To create temporally localized counterfactual mistakes, we first find the FHO actions that are included within each goalstep action. Then we ask Qwen-225-33B-IJsstuuc [15] identify the "critical" FHO action for each goalstep action, e.g., for the goalstep action wash green beans in water, the critical FHO identified by Qwen-2.5-32B-Instruee
washes green bean (from the following FHO actions within the goalstep action: collects green bean, puts green beans on the chopping board, puts green beans cooking pan.... ..... ....,ickss green beans, opens tap, washes green bean, to construct two counterfactual actions. Firstly, by changing the noun to an alternative noun in the scene, i.e., washes green bean to washes carrots. The list of nouns in the scene is generated by identifying objects in the scene using DETR [7]. Secondly, by proposing an alternative verb, i.e., washes green bean to mash green bean. Then use this counterfactual action to create an instruction and feedback pair. We use the point-of-no-reture timestamp [17] for the mistake feedback.
...). We then ask the 1reen22.-32B-Imstmue
In case of the EPIC-KITCHENS dataset, most actions are of short duration (<10 seconds long). We 21---..--BB---ss-- to construct two counterfactual actions by using an alternative noun in the scene and by an alternative verb as described above. As the actions are short we use the end of action timestamp generate the (mistake) feedback.
directly use
D.3 Training and Inference Hyperparameters.
We use input video resolution of 448 x 448 at 2 fps. The Intern VTT-300M V2_5 vision head produces N 1025 tokens (including the CLS token) per input frame. We use the mechanism outlined in VisionZip [59] to reduce the number of tokens to 256. Then, our Q-Former reduces this further to K 32 tokens.
In addition to the vision features, the Mamba-130M language backbone of the LIVEMAMBA model prompted with the following prompt:
LIVEMAMBA: Interactive Inference (Language Backbone)
You are expert cooking assistant that is helping person cook the following step by step recipe: |recipe [recipe_steps]. steps].
You just provided the following instruction the person: [last instruction]. Now watch the video and provide the appropriate success or failure messages.
The LIVEMAMB, model is trained using 8 Nvidia H100 GPUs. We use the Adam [42] optimizer. During the pre-training phase, we train only the Q-Former and the LIVEMAMBA model is trained using a learning rate of 10 10-5 for 200k iterations. We again use a learning rate of 1 x 10 10-5, for 120k iterations. During the fine-tuning phase, we train single recipe steps and clip the maximum length to 3 minutes. During inference, we re-initialize the LIVEMAMBA model after every recipe step.
E LIVEMAMBA Results
In Fig. 7, we show additional qualitative examples of predictions by our LIVEMAMBA model from the main set of the Qualcomm Interactive Cooking benchmark. In the top two rows we see that our LIVEM ABA model is able to successfully recognize that the user has completed complex steps from the step-by-step plans, e.g., rinsing tomato and cutting tofu into large cubes. In the third row we show an example where the user incorrectly coats a big cup instead of a 6-oz. cup with cooking spray. Our LIVEMAMBA model is able to provide timely feedback in this case.
20

---

Zero-shot Evaluation
Here we provide the prompts used for zero-shot evaluation. We begin with a more detailed description of the prompting strategy in Section 5.2 of the main paper.
Promoting Strategy. This strategy involves prompting the model regular intervals detect both successful completions of instructions and mistakes. However prompting after every input frame is computationally feasible. To balance accuracy and compute requirements, we prompt the models at an interval of 5 seconds. First, the model is asked if the user has completed the current instruction (obtained from the recipes in the Qualcomm Interactive Cooking benchmark). We use the following prompt for the Gemini-2.5. 5-a-h [55], Qwen2.5-VL-7 V---7BB----BBB... L--B-Innttucct [19], Qwen2-VL-78 In-srrcttt [56], Vide3LLa [62] models:
Gemini-2.5-Flashh complete
Check if instruction
You are expert cooking assistant helping person cook. The person provided with an instruction and your task is to check if the instruction has been completed.
#IINSTRUCTION::
The person has been instructed [instruction]. If the person has completed the instruction answer "yes" else answer "no".. DO NOT OUTPUT ANY OTHER TEXT.
In of VideoChat2 [34] model, the following prompt:
VideoChat2: Check if instruction complete
You are an expert cooking assistant helping person cook. The person provided with an instruction and your task is check if the instruction has been completed. You must check the video content very closely and confirm if the instruction has been completely followed and finished.
#HINSTUUCTOOSS
The person has been instructed to: [instruction]. If the person has completed the instruction answer "yes" else answer "no". Answer "yes" ONLY IF the instructed action is completed in the video, otherwise answer "no".
In of the Video-LLaVA 167, 38), Video-ChatGPT [43], LLaVA-NeXT [40] they do accept system messages we use the following simplified prompt:
Video-ILLaA Video-ChatGPT BaaaaaGGT/LLaV-NeeTTT Check if instruction complete
The person has been instructed (instruction].
the person has completed the instruction answer "yes" else answer "no".
If the model answers "yes", then move to the next instruction in the recipe. If not, then ask the model to check for mistakes.
For VideoL.LM-Online. and mistakes out of the box. Therefore, we use helper L.LM 10l-3-minii44-Instrrre help detect completion and mistakes given narrations. We use the following prompts.
since the model is trained to narrate videos, it can not detect completion
21

---

VideoL.LM-OOline: Check if instruction is complete
VideoLLM-Onnline:
Please narrate the video real-time You
intelligent chatbot that is judging another system which narrates human cooking videos. Given a high level action instruction and a list of narrations generated from the system, your job is decide if the narration is correct and shows completion of the instruction. Answer 'yes' if the instruction is completed otherwise output 'no'.
Helper-ILLM (Phi-3-mini-4k-IIstruccee
Instruction: [instruction]
Narrations:
[videollm-onie onnline narrations]
In the offline (turn-based) evaluation scheme used CaptainCook4D [46], multi-modal LLMs are prompted with sequence of questions per error category detect mistakes. However, such scheme infeasible case of streaming setups due to high computational costs. Therefore, to enable mistake detection in single inference step, design prompt that concisely explains the possible mistakes that are possible while following given instruction. The model can then use this information to recognize possible mistakes. In detail, for given instruction we find similar instructions in the training set and use Qwen-2.5-32B Breen-25-32B-Insstee Instruct [15] to summarize the possible mistakes. The following some examples:
Instruction: Now place 8-inch flour tortilla cutting board.
When guiding the user through this step, the cooking instructor should watch out for these potential mistakes:
1. Users might toast heat the tortilla before placing it the cutting board.
2. Users may place the tortilla plate instead of cutting board.
3. Users could use unclean surface instead of a clean cutting board.
Instruction:
Now add 1/4 tsp salt to bowl.
When guiding the user through this step, the cooking instructor should watch out for these potential mistakes:
1. Spilling salt while measuring adding it.
2. Adding too much salt, specifically 1/2 tsp instead of the required 1/4 tsp.
3. Confusing I/3 tablespoon with 1/3 teaspoon.
4. Accidentally adding the salt to the pan rather than the bowl.
We then these "mistake summaries" to prompt the multi-modal LLM. For Gemini-2.5. Flash [55], VL-77B
[19], Owwen--VL-7B-Innttruc [56], VideoLLaMA3-7B [62],
VideoChat2 [34] the following prompts:
Gemini-2.55Flaan (0wen2x-VL--BBImee V.t.:///---antteec//oooooloaaaaa VideoChat2: Check if instruction complete
Videol
You are expert cooking assistant who is observing person who is provided with step by step instructions for cooking. You should look out for mistakes made by the person.
#HINSTRUCTION::
The person is trying to complete the following instruction: [instruction].
This is how you can check for mistakes: [mistake summary).
Your task is check if the person has already made mistake.
Note that the person may not have completed the provided instruction, that is, the person may have only partially completed the provided instruction.
The answer should be "yes" or "no"". In case of yes, please provide concise feedback to the person describing the mistake (i.e. Yes. <feedbacks.). Directly address the person.
22

---

In case of Video-LLaVA [67, 38], Video-ChatGPT [43]. LLaVA-NeXT [40] models, they do not support complex prompts as they were mainly designed for question answering tasks. They we employ alternative strategy where we ask the model to narrate the video in 30 second chunks. Then we use the (concatenated) past narrations since the last instruction completion to prompt the model to look for mistakes.
Video-LLaVA V Check if instruction complete
The person has been instructed to: [instruction]. ill now the person has done the following: [past narrations].
Your task is to check if the person has made a mistake.
The answer should be "yes" "no". In case of "yes", please provide a feedback the user describing the mistake. Directly address the person.
For VideoLLM--nnlnne we use similar strategy as done in completion detection. We feed VideoLLM-
Online's narration to helper LLM to get mistake detection. We use the following prompts,
Videol.M-Onniine Check if instruction is complete
VideoLLM-Onning Please narrate the video in real-time.
Helper-LLM 35-mini-4k-Instrucche You are an intelligent chatbot that is judging another system which narrates human cooking videos. Given a high level action instruction and a list of narrations generated from the system, your job to decide if mistake has been made. The user has be instructed to do the following: [instruction] Till now the person has done the following: [videollm-online narrations]
Your task is to check if the person has made mistake.
The answer should be yes or no. In case of yes, please provide a feedback to the user describing the mistake. Directly address the person.
23

---

Chpp scallion Drin from Take
into Add the scaltion
skimmed 2:Ade tetspoon saugron Misrowase the mug, concents mug pieces chocolate mug. serve.
Lay large hoeepom srincce rach Season bow! tcaepoon avucado
Instruction: New chop scaltion.
Instrnction: Now fill microwave-safo mug with akimmed milk
170,69s
7.555
228.00 001
65.00
Foodltuck: You chupped sctiliuns scallion.
Fooditakk: mag completely
251.174
72.04
Take ripe avocado. 3: Cut the orncado into thin stross 4:
Reediarkk: You spilled some milk whille filling the mag.
257.54s
77.54s
Instruction: Now drain fhom
leasporo mug Microwave the cuntents mag minute Add 1/5 tougpoon of cinnamon to 4:
Straaming Vidco
271.00
Streaming Vidco
87.684
Instruction: add white mag
Foodlbuck: the cucoa
complesly.
275,99%
122,00s
Take thin slices, 3: chopped scallion the bowi 4
Foedharkk tablospoor, nota teaspoon, of suger.
278.500
131,895
Instuction: Now
mig Add ravgroon cinnamon to
the mag. 3: Mix the contents of the mug. 4:...
309,000
145-41
Instruction: microwave the mug
Feedback: You took the ripe avecado Hansoffill..
335,00s
171.00s
Plan. the chopped scallion the bowl, 31 1.4 cup of msyontine the bowd 4e...
Feedback: You should microwave minute
336 0014
219.154
Instruction Now the evocado into
teaepoon
mug Mix cunteres mug
pieces of chocolate to the 4
347.50a
224-461
Indruction: add 1/5 iczapoon cinmemun the mag.
Figure 4: Data samples from the main set. Left: the user prepares spicy tuna avocado wraps. Right: the user prepares spiced hot chocolate.
24

---

Slice medium Pie the vegetables noodles with popc glattering ncocles about minute miczoware sape. bowl. Mix the flovor packett into the
Plan. Add
Soute ontonns medium
noodles chopped
Geey and trmmuuent. everything 4,Add pan. tomatoes pan. covered op loenatoos sofles. Slowly pour whiikeel cage pas. with syaruls minutes the eggs are almost cookcdd
Instruction: New shoee medium onion into piaces
Instruction: Now add 1/3 teuepoon of pin.
248,824
409.075
263,80 50s
440.00
following instruction. like you are trying clep 144 tommno Burt you chopped 1/2 tomato insted lid tomato
Feedback: You did noe dellow the instuction. You all the vegetables micnoaae--auf boal.
281,00s
471,754
Add the noodles the bewd, Cover the noodles with water, it
Reepinn
piasos
Saute the enions modium they are saft and translianent Add the chilli the
Re-plan, screueon
253,00s
473.56s
Instruction: Now diice medium
Indtrcttion add teaepoton salt pan.
Foddlackk: You did not fullow the insiration You chopped green chill.
Steaming Vidco
264.550
Streaming Vidco
405,00k
Re-plan. scupuoo ane voft and transhnoent Add chilli so the
pai.
madium they
Feedback:: You did not dollow the insruction. You aldal li roslles bowd.
285,30s
$19,08s
Instruction: add 1/3 salt pur.
Re-phan. muslles Cover (or poper towel) present splituzing 3: Micunooe the cemenn for
286,385
$37.505
Instuction: Now noudlas
Foodlookk: 1/2 lcaspoon insted of 1/3 tesspoon.
321.504
$42.504
they are sall and tomsslueeni the Add the garlic the 4:
Feedback: You covered the noodles with water
323,50s
$49.45s
Plan. with paper sowe))
they soct and
przeent glattrinn Mineoave mimnn minute Adi chopped silamio to the bowl.
Fudiback: Yos did not fillow inetration
326 64s
$66 00s
You added the tomatbes the
Instruction
Now paper prevent splatooing
Re-plan.
they are woll innddueenn chilli the pen. 3: Add the garlic the pan. &
342.000
5747744
Indtuction:
the uniame
hest unail they are sort and transhncent.
Figure 5: Data samples from the advanced planning set. Left: the user is making ramen. Right: the user preparing scrambled eggs.
25

---

Entract from Measure fronen teespoon
I: 'Tranefer kettle. Crind coffee umill ooffee while beiling, scunbli filtrr semperatere
Micunsase under cold
Boll
tesspoon of popper powder
Sin houl, som for lime joice howdl Aci souepoon
degrees (ibresheii ur betwoes degrees colsias, water hot, brielly. Place drpper on tup coffiee may.
12:
lime:
Instraction: New extract time juice from 1/3
Instruction: Now transfer the water to kettle.
41.485
293,000
Foodlbckk: You trandferred the water the kettle sacoresfally.
44,50%
301, 50.
Foodltack: You fullowing the insrrcttion,
froeen corn by potting in sieve rurning under cold water, lur you spilled some while
trying
grounal see considency cusees aboun 20 seconds. 2: Beil the water Check the tomperatare of the ...:.
60.975
305 00s
grind grounds consistency
Feedbark: You did noe follow the insmaation You placed the paper filter the draper.
65.005
331.50s
beurs coffiz grounds cursidicccy of zusene sand, about 20 seconds. 2: Hoi the waner F. Check tempeniture
Re-plan. Grind
Re-plan, Extract Mesurr crpe fronen com Add corn to microwove sade bowl. R:...
Straming Vidco
73.75s
Instruction: Now lime
Streaming Mideo
346 346.00
Instruction: the coffee until grounds unmistzzacy
94,,00
352,00
Foodluuck:
You following like you trying the fiber the dripper create cone. However, you did not form the filer into cune
Feedbark:
indruction, looks fike you trying the cam so minnowsvo-edlf hool. Bui you spilled while adding the bow!
97,365
359.985
Replan,
Resplan, Microwave
teaepoon sofirned butter teespoon popper powder the bowl. 4:
zoffie grounds cursieicncy of cusrre sand, about 20 seconds 2: Hol the waner Jt Check the tempeneure the
100.074
360,20x
Instruction: Now
for
Instraction:
grounds the coffee until socodss
129,00s
393.50s
Foedborks shurld grind coffre for about 20 sccunds.
Feedbark: You microwaved the com sucessfully.
139.50s
410.00k
teaepoon
boiling, sescmibl omee Clack temgersture of the water 3: Place the dripper tepof coffee mag.
tenapoon peyper powder to 3: Stir the bowl &...
260,05s
toagpoon of
416.31a
Instruction: bell water, boiling.
Figure 6: Data samples from the advanced planning set. Left: the user is preparing butter corn cup. Right: the user making coffee.
26

---

Ground Truth
Now rinse
You rinsed the tomato corractly.
Now gently dry it wiih paperitea
You dineed the tomato with paperitea towel successfully,
Now gently dry with paper/tea
You driad the tomato with paper/eea towel succassfully.
LiveMamba
Now rinse
You rinsed the correctly
Ground Truth
cut ounces of fresh tofu into large cubes
You into large cubes properly.
Now pat the tofu dry with towel.
You petted the tofu dry properly.
cut ounces of fresh tofu into large cubes
You tafu into large cubes properly.
You patted tofu dry towel successfully,
Now pat dry with towel.
LiveMamba
Now coet tit ramekinn cooking soray.
You are coeting cup ramekin cup.
Now pour 1 into the ramelan cup.
You poured the egg rame kin cup cornectly.
Ground Truth
Now coat 5 oz ramakin cup cooking spray.
You should roar a oz ramekin cup, cup.
Now pour 1 egg into the rameliin cup.
You poured the egg into the ramelin correctly.
LiveMaaba
Figure 7: Predictions from our LIVEMAMBA from the main set of the Qualcomm Interactive Cooking benchmark.
27

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly states that the main contributions of the paper are novel method: LIVEMAMBA and dataset: Qualcomm Interactive Cooking.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the "Limitations" section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
28

---

Justification:
There are no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, experimental details included in the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
29

---

Answer: [No]
Justification:
The Qualcomm Interactive Cooking dataset released at https://
huggingface.com ctt.ecoc/quuueee www.:////aalloomiieeeaeaaaaeeeae
bookiing-dttsse...
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https:
ggtgg/uies////// htte///oouuuisssteeeeeee ssionooiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The details are provided in the main paper and the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: While full statistical analysis of the performance of multi-modal LLMs are computationally prohibitive, our initial analysis suggest limited variance of results on the Qualcomm Interactive Cooking benchmark.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
30

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, the details are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttpa:://///ii...eee/ http:///ww...tttccc.....eeeeeeee
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: include "Broader Impacts" section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our model and data is strictly applicable only to cooking scenarios from an ego-centric viewpoint.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All licenses respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
32

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The Qualcomm Interactive Cooking dataset released at https:// huggingface.com wwtp//aacoon//uual////aaaaaaaa/e www.-aaaaeeeeeoooooooooeeaaeeeee
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The Qualcomm Interactive Cooking dataset and benchmark was not collected by crowd sourcing. The annotations were generated by the authors themselves.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
33

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: Yes, all uses of LLMs are clearly described.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: http:://nnuuuuueeeeee http:ripssee//ooeeeeeneeee/////s for what should or should not be described.
34


Reviews:
[{"review_id": "vBswsY0SCv", "reviewer": "Reviewer_rRyN", "summary": "This paper describes work around building a cooking dataset and a model for providing live instructions and task guidance where the model/system can also identify mistakes (in-step or out of order), re-plan and suggest the next steps. The research addresses gaps in the capabilities of current multi-modal LLMs, which, despite their advancements, largely operate in a turn-based manner and are not suited for providing continuous, real-time guidance and feedback for task operation. The authors introduce the LIVECOOK dataset, which is built upon the CaptainCook4D dataset and is additionally annotated with timed instructions and feedback on user mistakes. This dataset is designed to train and evaluate models on their ability to not only provide instructions but also to detect successful completion and identify errors as they happen.\n\nThe paper proposes LIVEMAMBA, a streaming multi-modal LLM designed for this interactive guidance task. Key features of LIVEMAMBA include its ability to decide \"when to speak,\" a data augmentation scheme for recognizing a diverse set of mistakes, and an iterative re-planning module to adapt to user actions. The evaluations demonstrate that existing state-of-the-art multi-modal LLMs perform poorly in this live guidance scenario, while the fine-tuned LIVEMAMBA model establishes a slightly better baseline.", "questions": "(from comments above:)\n- how good is the timeliness of the feedback (use of 30 second windows for calculating IC-accuracy may be too high)? \n- how good is the re-planner? the re-planner is able to skip steps completely or create new steps, etc - this kind of replanning may work in the cooking domain but not in any mission critical domain for which building these systems may be more pressing (and therefore use of cooking as a proxy as used in many R&D projects) \n- what's the compute and memory requirements for the models? \n- is there a demo for the system?\n- Are there plans to open-source any part of the research?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Real time multimodal task guidance for complex task is a hard problem that hasn't been solved yet since most work in the space aims to address turn by turn task following. This paper establishes an initial baseline for this more ambitious interactive task guidance and error correction problem. Follow up work can build up on this dataset for dynamic planning (when an out of order step is executed), identification of crucial mistakes (that in some other non-creative domain can result in a more serious harm), etc. \n- the notion of timely feedback via the <response> and <vision> tags as well as adapting to the human error are gaps that this paper addresses appropriately. This dataset with ~100 hrs of annotations will help address the task of real time guidance in this domain appropriately\n- the overall system design is interesting with the use of efficient backbone models like Mamba to focus on the core task of vision and language grounding and a more expensive re-planner to address complex reasoning.\n- good evaluations with zero shot studies, fine-tuned model performance as well as ablations (wo data augmentation and re-planner) are very interesting. The choice of creation of a simple and an advanced dataset makes the problem approachable with existing models as well as simple in design. (this adds to the clarity of the research direction)\n\nWeaknesses\n- Authors mention that \"inference using our model should ideally be possible using compute constrained edge devices such as mobile phones or smart glasses.\" However, there is no discussion whatsoever about latency and throughput of the models and system that use a vision encoder, Q-former, Mamba model and a Qwen2.5-32B-Instruct re-planner - what kind of backend infrastructure would this be able to run on, etc (mobile phones, smart glasses?). While the use of Mamba backbone is motivated by efficiency, concrete numbers on processing times are missing especially when this work is differentiated by it's ability to do live task guidance. \n- Many details are missing: how good is the timeliness of the feedback (use of 30 second windows for calculating IC-accuracy may be too high)? how good is the re-planner? the re-planner is able to skip steps completely or create new steps, etc - this kind of replanning may work in the cooking domain but not in any mission critical domain for which building these systems may be more pressing (and therefore use of cooking as a proxy as used in many R&D projects) \n- there is no generalization studies on this model, can the model capture errors in other domains? Is this s solid foundation to build models on top of this for other domains?\n-  this research aligns well with the recent trend of agentic AI models like MM-ReAct and SayCan (also in robotics domain) for task execution. While these models are used for empowering the human better, they can still learn from work in the agentic AI multimodal task execution literature.", "was_revised": true, "final_justification": "Based on the author's responses (about latency details, re-planner details etc), I've increased my score by a point. However, I am still not clear about the re-planner details (besides repeating the steps, what other actions can it do and what are the consequences if the re-planner introduces a new step, how can one verify it, etc) Authors mention they use Nvidia A100 for inference, and still mention that they can run on mobile devices, I am not clear about this.", "rebuttal": "We thank the reviewer for recognizing our contributions: establishing a baseline for the challenging problem of real-time interactive task guidance, the value of our ~100-hour annotated dataset, and the strengths of our evaluations and system design.\n\n**Latency and Throughput of the Models:** Our LiveMamba model consists of a vision encoder, Q-former, Mamba model, and a re-planning module. As explained in L172-176 in the main paper, only when the user does not perform the intended step (on the advanced planning set), the Qwen2.5-32B-Instruct re-planner will be invoked. Otherwise, the re-planning module just returns the next step in the plan. In terms of throughput, on a consumer Nvidia A100 GPU, our LiveMamba model has a real-time factor of  2 on average. This means that it can process input data twice as fast as it becomes available. In detail, we can process 4.1 frames per second on average and the input frame rate is 2 frames per second. In terms of latency, that is time to generate the first token, is 1.1 seconds on average. Re-planning using the Qwen2.5-32B-Instruct re-planner takes 8.2 seconds on average. Note that, as mentioned in L109 in the supplementary material, the Qwen2.5-32B-Instruct re-planner is called once every 2.5 minutes on average on the advanced planning set.\n\nThe target backend infrastructure for our LiveMamba model are edge devices such as mobile phones and smart glasses. A key advantage of the Mamba backbone of our LiveMamba model is its low memory requirement, which scales linearly with sequence length, compared to transformers whose memory requirements scale quadratically with sequence length. The peak GPU memory usage during inference of the LiveMamba model (with the Qwen2.5-32B-Instruct re-planner) is ~32GB to process ~500 frames. This makes our LiveMamba model more suitable for deployment on edge devices with limited memory compared to standard transformer architectures. We will clarify this in the final version. \n\n**Windows for Calculating IC-Accuracy:** Note that the 30 second window is centered at the end of each recipe step and thus includes the last 15 seconds of the current recipe step and the first 15 seconds of the next recipe step. The average length of a recipe step is 62.05 seconds. Thus the window for calculating the IC-accuracy covers on average the last 24% of the duration of a recipe step and the first 24% of the beginning of the next recipe step. In practice, this is sufficient to account for any ambiguities in the data along with any residual inaccuracies in the annotations while not sacrificing on accuracy (L214 in the main paper). We will clarify this in the final version.\n\nFinally, note that even with this 30 second window, the results in Table 3 of the main paper highlight that the LiveCook benchmark is challenging even for current state of the art models like Qwen-2.5-VL-7B.\n\n**The Re-Planning Module:** As described in L172-176 in the main paper, if a recipe step is performed out of sequence (on the advanced planning set), the re-planning module in our LiveMamba model is able to decide if the current recipe step must be repeated and also update the plan, this is when the Qwen2.5-32B-Instruct re-planner is invoked. Otherwise, the re-planning module will just return the next step in the plan. We highlight the advantage of the re-planning module in Table 4 of the main paper on the simple planning set through an ablation of our LiveMamba model without the replanning module (w/o-reP). Furthermore, here we add this LiveMamba (w/o-reP) ablation to the results in Table 4 on the advanced planning set:\n\n|        Model       | IC-Acc↑ | Prec.↑ | Rec.↑ |  F1↑ | BERT↑ | ROUGE-L↑ |\n|:------------------:|:-------:|:------:|:-----:|:----:|:-----:|:--------:|\n| LiveMamba (w/o-reP) | 8.7 | 0.07 | 0.03 | 0.04 | 0.414 | 0.389 |\n| LiveMamba (Ours) | 11.6 | 0.10 | 0.04 | 0.05 | 0.470 | 0.447 |\n\nThese results again highlight the advantage of the re-planning module in our LiveMamba setup. \n\nFinally, note that in this work we are primarily focused on the cooking domain, and we find that this re-planning module is sufficient in such cases. Extension of our re-planning module to other domains is an interesting direction for future research.\n\n**Generalization Studies:** To evaluate the ability of multi-modal LLMs to provide step-by-step guidance, an ideal test domain should be both practical and accessible. We chose cooking because it meets these criteria: its tasks are relevant to daily life, and the specific recipes in our LiveCook benchmark do not require specialized knowledge. These qualities make cooking a robust environment for our study. Thus, we propose LiveCook, the first large scale benchmark and dataset with timed instruction and feedback messages for step by step task guidance. We leave the generalization of our method to other procedural tasks as a direction for future work. We will clarify this in the final version.\n\nBecause the LiveMamba model only tuned to recognize mistakes on the cooking data, currently it may not be able to capture errors in other domains. But our model can be trained with data from other domains based on data availability. \n\n**Learning from Works in Agentic AI:** This is a great point and an interesting direction for future research. We will discuss the MM-ReAct and SayCan papers in the final version.\n\n**Demo:** We will include demo videos of the LiveMamba system with the final version. We also include qualitative examples in Figure 3 in the main paper and Figure 7 in the supplementary material. \n\n**Open-source:** We will release both the LiveCook benchmark and dataset and the LiveMamba model code (pending legal approval)."}, {"review_id": "drZvHdoPlf", "reviewer": "Reviewer_igL6", "summary": "This paper focuses on the challenge of enabling multi-modal large language models (LLMs) to provide real-time, interactive step-by-step task guidance. The authors point out that current multi-modal LLMs excel in conversational abilities but struggle to offer live, interactive step-by-step guidance, which is a critical capability for future AI assistants. Effective guidance requires not only delivering instructions but also detecting their successful execution and promptly identifying and alerting users to mistakes—all in real time. To address this challenge, the authors introduce LIVECOOK, a new benchmark and dataset built upon the CaptainCook4D dataset, which includes user errors during task execution. LIVECOOK features densely annotated, timestamped instructions and feedback messages, particularly mistake alerts precisely aligned with their visual occurrences in the video.", "questions": "see weakness", "limitations": "see weakness", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1.\tNovelty of the benchmark dataset: The LIVECOOK benchmark addresses the limitations of existing datasets by incorporating user errors and providing densely annotated, timestamped instructions and feedback messages. This enables more realistic evaluation of multi-modal LLMs' ability to provide step-by-step task guidance.\n2.\tInnovative model design: The LIVEMAMBA model introduces several novel features, such as the \"when-to-say\" mechanism, data augmentation for mistake recognition, and iterative re-planning. These features enhance the model's ability to provide real-time, interactive guidance.\n\nWeakness\n1.\tLimited generalizability: The research is primarily focused on the cooking domain. The generalizability of the proposed method to other task domains remains unclear.\n2.\tHandling complex tasks: The model struggles with advanced planning scenarios (e.g., order errors or missed steps), as highlighted in the \"Advanced Planning Set\" experiments. This suggests limitations in reasoning about non-linear task structures.\n3.\tComputational efficiency：While LIVEMAMBA is lightweight compared to models like VideoLLaMA, its reliance on external re-planning modules (e.g., Qwen-2.5-32B) may hinder scalability for real-time edge deployment.\n4.\tAnnotation cost: The paper’s annotation process for the LIVECOOK dataset relies heavily on manual labeling of step-by-step instructions and feedback, particularly for mistake detection and timing. While the authors describe leveraging existing annotations from the CaptainCook4D dataset and manually curating timestamps for mistakes, the methodology lacks a scalable, automated pipeline for generating such interactive feedback data. This manual approach is labor-intensive and domain-specific (focused on cooking), making it difficult to generalize to other procedural tasks (e.g., assembly, fitness, or medical procedures) where large-scale, temporally aligned feedback is required.\nMinor mistakes：\n1.\tThe paper's data augmentation for mistakes relies heavily on counterfactual action descriptions (e.g., change nouns/actions), which may not fully capture real-world variability in user errors. Real-world user errors are often nuanced (e.g., partial successes, context-dependent mistakes). Augmenting training data with diverse, real-world examples would improve robustness.\n2.\tThe primary metrics focus on binary correctness but lack nuanced evaluation of feedback quality.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the novelty of the LiveCook benchmark and dataset in enabling more realistic evaluation, and the innovative design of the LiveMamba model, which enhances real-time, interactive task guidance.\n\n**Generalization:** To evaluate the ability of multi-modal LLMs to provide step-by-step guidance, an ideal test domain should be both practical and accessible. We chose cooking because it meets these criteria: its tasks are relevant to daily life, and the specific recipes in our LiveCook benchmark do not require specialized knowledge. These qualities make cooking a robust environment for our study. Moreover, our model can be trained with data from other domains based on data availability, and generalizing to other domains is an interesting direction for future work. We will clarify this in the final version.\n\n**Reasoning About Non-Linear Task Structures:** We agree that the advanced planning set is very challenging for all models, as it requires reasoning about non-linear structures. However, the use of the re-planning module in our LiveMamba model leads to improved performance, as it is able to update the task plan on the fly as described in L168-181 in the main paper. This is highlighted in Table 4 of the main paper where our LiveMamba model performs significantly better compared to the prior work such as Videollm-online [6] on the advanced planning set of our LiveCook benchmark.\n\n**Computational Efficiency:** Re-planning using the Qwen2.5-32B-Instruct re-planner takes 8.2 seconds on average on a Nvidia A100 GPU. Note that, as mentioned in L109 in the supplementary material, the Qwen2.5-32B-Instruct re-planner is called once every 2.5 minutes on average on the advanced planning set. In terms of throughput, on a Nvidia A100 GPU, our LiveMamba model (system with a vision encoder, Q-former, Mamba model and a re-planning module) has a real-time factor of 2 on average. This means that it can process input data twice as fast as it becomes available. In detail, we can process 4.1 frames per second on average and the input frame rate is 2 frames per second.\n\n**Annotation Cost:** We rely on manual annotations for our LiveCook dataset and benchmark to maintain accuracy. It is currently not possible to automate this data generation pipeline as demonstrated by the poor performance of state of the art multi-modal LLMs in detecting mistakes in Table 3 of the main paper. Thus, it is crucial to have human expert annotators provide the ground-truth for the mistakes and their timestamps in a video sequence. This also highlights the importance of our LiveCook benchmark and dataset as the first to provide timed instruction and feedback messages covering a diverse range of mistakes in the cooking domain.\n\n**Counterfactual Action Descriptions:** The counterfactual action descriptions generated using our data augmentation scheme leads to improved performance of our LiveMamba model as highlighted in Table 4 of the main paper (LiveMamba (w/o-DA) vs LiveMamba (ours), where LiveMamba (w/o-DA) is the model trained without the proposed data augmentation scheme). We see a significant increase in performance across all metrics with our data augmentation scheme. We agree that the addition of more nuanced errors could improve performance even further, however, generating such data automatically is challenging with current state of the art multi-modal large language models, and is thus an interesting direction for future research. We will include samples of the augmented data in the final version.\n\n**Nuanced Evaluation of Feedback Quality:** In addition to the “binary” Instruction Completion Accuracy (IC-Acc) and mistake detection precision (Prec.), recall (Rec.) and F1 metrics, we also include the mistake feedback fluency (BERT and ROUGE-L) metrics. This allows for a more nuanced evaluation of the feedback quality in Tables 3, 4 and 5 in the main paper."}, {"review_id": "MlrwGnbENH", "reviewer": "Reviewer_S2qM", "summary": "This paper tackles the challenge of enabling multi-modal large language models to provide live, step-by-step task guidance in interactive settings. The authors introduce LIVECOOK, a new benchmark and dataset built on top of CaptainCook4D, specifically annotated with timestamped instructions and mistake feedback in cooking scenarios. Additionally, the authors propose LIVEMAMBA, a lightweight streaming multi-modal LLM that features a \"when-to-say\" mechanism, mistake-aware data augmentation, and an iterative re-planning module. Through comprehensive evaluations across streaming and turn-based settings, the paper shows that LIVEMAMBA significantly outperforms existing baselines in recognizing completed instructions and detecting fine-grained user mistakes.", "questions": "-  Lack of annotation quality control. The paper introduces LIVECOOK with dense annotations of instructions and feedback, but does not provide sufficient details about the annotation protocol's consistency or quality. A deeper analysis of annotation quality would significantly strengthen the credibility of the dataset.\n\n- The evaluation allows a 30-second window for detecting instruction completion or mistakes, but this choice appears arbitrary and is not empirically justified. It is unclear whether 30 seconds is too lenient or too strict. Maybe an ablation study is needed.\n\n- Benchmark is domain-specific and may limit generalization. Although the authors acknowledge that LIVECOOK is focused on cooking, the benchmark does not include any other experiments or analysis showing how transferable the model is to other task domains. A promising direction would be to evaluate the model on other public datasets, which also involve step-by-step tasks. \n\nIf my concerns can be adequately addressed, I would consider raising my score.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The paper addresses an interesting and meaningful problem: enabling live, step-by-step task guidance in interactive settings.\n\n- The paper is well-structured, clearly written, and easy to follow.\n\n- The authors conduct extensive experiments, and the observed performance improvements are promising, demonstrating the potential of their proposed approach.\n\n**Weakness**\n\n- The paper lacks details on how annotation quality were ensured, making it difficult to assess the reliability of the LIVECOOK dataset.\n\n- The model is only tested on cooking tasks within LIVECOOK, limiting insight into its generalizability across other domains.\n\n- The LIVEMAMBA model depends on a powerful external LLM (Qwen2.5-32B-Instruct) for re-planning, which makes it harder to isolate the model’s own capabilities.", "was_revised": true, "final_justification": "The authors’ rebuttal has addressed my concerns, and I believe the paper’s contributions and novelty are sufficient. I will increase my score by one point.", "rebuttal": "We thank the reviewer for recognizing the importance of the problem we are addressing through the novel LiveCook benchmark and dataset, the clarity and structure of our paper, and the potential of our proposed LiveMamba approach demonstrated through extensive and promising experiments.\n\n\n**Annotation Quality:** All annotations are created manually, and to ensure the quality of the annotations, they are manually verified independently by another annotator. The quality of the dataset is further highlighted in the qualitative example in Figure 1 of the main paper and the additional qualitative examples in Figures 4, 5 and 6 of the supplementary material. We also include the LiveCook benchmark data files in the uploaded supplementary materials. The annotation process is detailed in Section 3 of the main paper and Appendix B of the supplementary material. We will clarify this further in the final version.\n\n\n**Generalization:** To evaluate the ability of multi-modal LLMs to provide step-by-step guidance, an ideal test domain should be both practical and accessible. We chose cooking because it meets these criteria: its tasks are relevant to daily life, and the specific recipes in our LiveCook benchmark do not require specialized knowledge. These qualities make cooking a robust environment for our study. Moreover, our model can be trained with data from other domains based on data availability, and generalizing to other domains is an interesting direction for future work. We will clarify this in the final version.\n\n**The Powerful Re-Planning Module:** We would like to begin by clarifying that (as described in L172-181 in the main paper), if a recipe step is performed out of sequence, the re-planning module invokes the Qwen2.5-32B-Instruct model to decide if the current recipe step must be repeated and also update the task plan. Thus, note that the Qwen2.5-32B-Instruct model is invoked only on the advanced planning set in Table 4 of the main paper. In the simple planning set, the re-planning module only returns the next recipe step (thus, the re-planning module ensures that the LiveMamba model does not need to memorize the recipe steps). \nTo better highlight the capabilities of the LiveMamba model without the re-planning module, we add a LiveMamba (w/o-reP) ablation to the results in Table 4 on the advanced planning set:\n\n\n|        Model       | IC-Acc↑ | Prec.↑ | Rec.↑ |  F1↑ | BERT↑ | ROUGE-L↑ |\n|:------------------:|:-------:|:------:|:-----:|:----:|:-----:|:--------:|\n| Videollm-online | 9.3 | 0.09 | 0.04 |  0.05 |  0.457 |  0.431 |\n| LiveMamba (w/o-reP) | 8.7 | 0.07 | 0.03 | 0.04 | 0.414 | 0.389 |\n| LiveMamba (Ours) | 11.6 | 0.10 | 0.04 | 0.05 | 0.470 | 0.447 |\n\nThese results show that even without the re-planning module, our LiveMamba model is comparable to the Videollm-online baseline.\n\nFurthermore, in the turn-based evaluation in Table 5 of the main paper, we do not use the re-panning module as the models are evaluated on each recipe step independently. We again see that our LiveMamba model outperforms state of the art multi-modal LLMs, even without the powerful re-planning module.\n\n\n**IC-Acc 30-second Window:** Note that the 30 second window is centered at the end of each recipe step and thus includes the last 15 seconds of the current recipe step and the first 15 seconds of the next recipe step. The average length of a recipe step is 62.05 seconds. Thus the window for calculating the IC-accuracy covers on average the last 24% of the duration of a recipe step and the first 24% of the beginning of the next recipe step. In practice, this is sufficient to account for any ambiguities in the data along with any residual inaccuracies in the annotations while not sacrificing on accuracy (L214 in the main paper). We will clarify this in the final version. \n\nFinally, note that even with this 30 second window, the results in Table 3 of the main paper highlight that the LiveCook benchmark is challenging even for current state of the art models like Qwen-2.5-VL-7B.\n\n**Other Public Datasets:** The proposed LiveCook dataset is the only dataset that includes step-by-step task guidance along with a diverse range of timed feedbacks covering mistakes as highlighted in Table 1 of the main paper. Therefore, we restrict our evaluation to our proposed LiveCook dataset."}, {"review_id": "ZuOnt2Wtrt", "reviewer": "Reviewer_P6CK", "summary": "The paper tackles the problem of how to enable multi-modal LLMs to provide step-by-step guidance in an interactive fashion. It introduces the first benchmark for the task, LIVECOOK, that features dense, timestamped instructions + feedback and mistake alerts. Then a model - LIVEMAMBA - is proposed to give instructions and feedbacks (when mistake happens) in real-time, as well as an extra re-planner module for giving next step instructions.\n\nFor evaluation, two modes are considered, real-time mode and turn-based mode. For real-time mode, the paper consider several existing video LLMs as baselines for zero-shot settings, and an existing online video LLM for fine-tuned settings. For turn-based mode, comparison against existing video LLMs are considered. The evaluations are conducted on three aspects: i) instruction completion accuracy ii) mistake prediction performance and iii) language fluency of the mistake feedback.", "questions": "See weaknesses above.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The task setup as well as benchmark is novel. As far as I know, this is the first benchmark that targets at evaluating MLLM's in instruction-guidance capability in an interactive mode.\n\n2. LIVEMAMBA's design seems interesting to me, the “when-to-say” mechanism as well as the extra re-planner effectively enables the model to interact in real-time.\n\nWeaknesses:\n\n1. From application perspective, since the model design is aim for efficiency and is possible for deployment in real-time, I am wondering does the model benchmark the latency aspect for real-time performance? Can such model run successfully on memory-limited devices (memory consumption)? Some discussion around these points will be helpful.\n\n2. How does pretraining affect the final performance of the model? Since the model specifically targets at the cooking settings rather than a general-purpose LLM, perhaps pretraining won't help here.\n\n3. In Table 3, the Mistake prediction accuracy for first 5 baseline models are all 0s performance in prec, recall, F1 and language metrics. Why that happens need more explanations, otherwise, it seems a bit suspicious to me.\n\n4. (More of a suggestion) The benchmark itself is still built on top of offline videos rather than a simulation environment. I think a real interactive environment where a MLLM that can guide simulated user (or agents) is more suitable for building a benchmark since the user/agent can change its actions rather than being fixed every time when a instruction or feedback is given, which is closer to real-world scenario. For example, some prior works in embodied navigation [1] can achieve such functionalities.\n\n[1] CAVEN: an embodied conversational agent for efficient audio-visual navigation in noisy environments. Xiulong Liu, et.al.", "was_revised": true, "final_justification": "After carefully reading through the rebuttal, I think the paper's merit outweighs its limitation. With enough novelty and solid experiments, I lean more towards accept. Please incorporate all the discussion points during the rebuttal into the final version of the manuscript.", "rebuttal": "We thank the reviewer for recognizing the novelty of our benchmark and task setup for interactive guidance, and for finding the design of our LiveMamba model interesting and effective for real-time interaction, particularly the “when-to-say” mechanism and the re-planner which effectively enables the model to interact in real-time.\n\n\n**Latency:** On a Nvidia A100 GPU, our LiveMamba model (with a vision encoder, Q-former, Mamba model, and a re-planning module) has a real-time factor of 2 on average. This means that it can process input data twice as fast as it becomes available. In detail, we can process 4.1 frames per second on average and the input frame rate is 2 frames per second. A key advantage of the Mamba backbone of our LiveMamba model is its low memory requirement, which scales linearly with sequence length, compared to transformers whose memory requirements scale quadratically with sequence length. The peak GPU memory usage during inference of the LiveMamba model (with the Qwen2.5-32B-Instruct re-planner) is ~32GB to process ~500 frames. This makes it more suitable for deployment on edge devices such as phones or smart glasses with limited memory.\n\n\n**Pre-Training:** As described in L184-191 in the main paper, the pre-training phase aligns the Q-Former’s vision embeddings with the language backbone’s text embeddings. We employ a diverse set of image and video datasets for object grounding (LVIS, EPIC-KITCHENS) and fine-grained action understanding (SSv2, EPIC-KITCHENS, Ego4D); these skills are essential for the cooking domain. Without pre-training (LiveMamba (w/o-PT)), we see a significant degradation of performance (on the simple planning set):\n\n|        Model       | IC-Acc↑ | Prec.↑ | Rec.↑ |  F1↑ | BERT↑ | ROUGE-L↑ |\n|:------------------:|:-------:|:------:|:-----:|:----:|:-----:|:--------:|\n| LiveMamba (w/o-PT) | 12.9    | 0.08   | 0.11  | 0.09 | 0.423 | 0.355    |\n| LiveMamba (Ours)   | 19.2    | 0.12   | 0.10  | 0.11 | 0.491 | 0.460    |\n\nWe will clarify this in the final version.\n\n\n**Mistake Prediction Accuracy:** The reason why the performance of the baseline VideoLLaMA3-7B [54], Video-LLaVA [59, 33], VideoChat2 [29], Video-ChatGPT [37], LLaVA-NeXT [35] models on the mistake precision, recall, F1 and language metrics is poor is because they cannot detect mistakes. The reason for this poor performance on mistake detection is the fact that our LiveCook benchmark requires these models to detect mistakes in a streaming setup. However, these models were trained for offline question-answering tasks. More powerful models such as Qwen-2.5-VL-7B [15] can adapt to our streaming setup due to their strong instruction following capabilities. We explain the reasons for the poor performance of the baseline models in L243-253 of the main paper, and we provide the details of the prompts used for zero-shot evaluation in Appendix G of the supplementary material. We will clarify this further in the final version. \n\n\n**Offline Videos:** Thank you for the reference to the highly relevant work of Xiulong Liu, et.al. that highlights the advantages of a real interactive environment. However, there are no extant simulators that can simulate fine grained actions which are common in cooking: e.g., cutting vegetables or adding 1 tablespoon of salt to a dish. Therefore, as described in L88-94 of the main paper, we use real-world offline videos which allows us to create a setup akin to a “non-reactive simulation”, where we task the multi-modal LLM to produce the right instruction or feedback at the appropriate time, but the subject is non-compliant. We will discuss and clarify these further in the final version."}]

Author rebuttal:

Meta-review: The paper introduces LIVECOOK, a densely annotated and timestamped benchmark for real-time, multimodal instruction-following and mistake feedback within the cooking domain. The paper also proposes LIVEMAMBA, a multi-modal LLM designed for this interactive guidance task. 

The paper received scores of 4, 3, 4, 5. The reviewers agree that using multimodal LLMs to support live, interactive, error-aware guidance is timely and important. However, they also pointed out that the dataset is limited to the cooking domain and thus generalizability to other tasks remains unclear. The reviewers also questioned the feasibility of annotation pipeline for LIVECOOK, which is labor-intensive and domain-specific. Its scalability to other tasks is not demonstrated. 
However, given the complexity of the problem space and the absence of comparable prior work, I recommend this paper for acceptance.
Decision: Accept (poster)