Training-free Online Video Step Grounding
Authors: Luca Zanella
Massimiliano Mancini
Yiming Wang
Alessio Tonioni
Elisa Ricci
Venue: NeurIPS 2025 poster ()
Keywords: Video step grounding
training-free
large multimodal models
Primary area: applications

Abstract: Given a task and a set of steps composing it, Video Step Grounding (VSG) aims to detect which steps are performed in a video. Standard approaches for this task require a labeled training set (e.g., with step-level annotations or narrations), which may be costly to collect. Moreover, they process the full video offline, limiting their applications for scenarios requiring online decisions. Thus, in this work, we explore how to perform VSG online and without training. We achieve this by exploiting the zero-shot capabilities of recent Large Multimodal Models (LMMs). In particular, we use LMMs to predict the step associated with a restricted set of frames, without access to the whole video. We show that this online strategy without task-specific tuning outperforms offline and training-based models. Motivated by this finding, we develop Bayesian Grounding with Large Multimodal Models (BaGLM), further injecting knowledge of past frames into the LMM-based predictions. BaGLM exploits Bayesian filtering principles, modeling step transitions via (i) a dependency matrix extracted through large language models and (ii) an estimation of step progress. Experiments on three datasets show superior performance of BaGLM over state-of-the-art training-based offline methods.

Training-free Online Video Step Grounding
Luca Zanella'
Massimiliano Mancini' Yiming Wang²
Alessio Tonioni Elisa Ricci
University of Trento Fondazione Bruno Kessler
Google
https://bucazaneella.ee github. io/baglm/
Abstract
Given task and set of steps composing it, Video Step Grounding (VSG) aims to detect which steps are performed a video. Standard approaches for this task require labeled training set (e.g., with step-level annotations narrations), which may be costly to collect. Moreover, they process the full video offline, limiting their applications for scenarios requiring online decisions. Thus, in this work, we explore how to perform VSG online and without training. We achieve this by exploiting the zero-shot capabilities of recent Large Multimodal Models (LMMs). In particular, we use LMMs to predict the step associated with a restricted set of frames, without access the whole video. We show that this online strategy without task-specific tuning outperforms offline and training-based models. Motivated by this finding, develop Bayesian Grounding with Large Multimodal Models (BAGLM), further injecting knowledge of past frames into the LMM-based predictions. BAGLM exploits Bayesian filtering principles, modeling step transitions via (i) dependency matrix extracted through large language models and (ii) an estimation of step progress. Experiments on three datasets show superior performance of BAGLM over state-of-the saate-of-the-att art training- based offline methods.
Introduction
Grounding procedural steps in videos is crucial for enabling machines to follow along and assist humans in complex tasks like cooking recipe, assembling furniture, or performing maintenance work. This ability is particularly valuable for real-time procedural guidance in AR/XR applications, where recognizing task progress allows users wearing headsets or smart glasses to receive timely, step-specific instructions. Specifically, the task of Video Step Grounding (VSG) takes input a list of procedural steps extracted from an instructional article (e.g., a recipe or how-to guide), and a video performing the same task, with the goal of identifying which of the steps are performed in the video.
Existing VSG approaches align procedural steps descriptions with their corresponding video frames [4] 14 18). However, these strategies face two key limitations. First, they need a training set, entailing the cost of collecting (and potentially annotating) it. Moreover, training set could bias models toward the specific videos and procedural tasks that are depicted, limiting their generalization capability. Second, they assume offline processing, where the entire video available ahead time. This makes them unsuitable for real-world applications processing a live video stream.
To overcome these limitations, we explore how address VSG online and without training, i.e.. operating on a streaming video (Fig. []. This objective is challenging as it requires performing predictions with partial evidence (i.e., having access to only a subset of the video frames) and without the possibility of extracting task-specific representations that would be typically learned from dedicated training set. In this scenario, we explore the zero-shot capabilities of powerful Large Multimodal Models (LMMs) 22 13, 6 for VSG. Specifically, given the set of steps, we prompt LMMs to predict the step corresponding to the current video segment. Surprisingly, models such
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Procedural steps
Video Step Grounding methods
Start a pot of boiling water and cook pasta
Combine the eggs, cheese, salt and pepper in a small bowl and beat with fork
Prior methods
Offline
Training-basee
Add some olive oil to a small saucepan cook pancotta thoroughly
Add the pasta and pancetta to the bowl and begin stirring gently
BaGLM
Finished
(Ours)
Online
Training-fis free
Figure I: We tackle Video Step Grounding with BAGLM, training-free approach which combines Bayesian filtering with Large Multimodal Models to enable online inference over video streams.
as InternVI VL (6] could already surpass state-of-the-ar art training-basse baased methods [8), despite only having access single segment at a time. This highlights strong potential for addressing VSG without training.
Building on the shot capabilities of LMMs, we show how performance can be further improved by modeling temporal dependencies across steps and leveraging past information. To this end, we propose Bayesian Grounding with Large Multimodal Models (BAGLM), training-free approach for VSG that combines Bayesian probabilistic modeling with LMMs. We harness a Large Language Model (L.LM) to estimate a dependency matrix, capturing whether one step is prerequisite of another. From this matrix, we compute transition probabilities to each step in the current video segment (i.e., the predict step of the Bayesian filter). This prior refines the LMM direct prediction (i.e., the update step), injecting past temporal knowledge into the model's output. The transition model is updated over time, following the progress of each step estimated by the LMM. On three publicly available datasets (HT-Step [1]. CrossTask (36), Ego4D Goal-Step (77). BAGLM consistently outperforms existing methods with significant margins, achieving oftte-of-theaar results this challenging task.
We make four key contributions: @ We present the first study to address VSG in an online, training-free setting. eliminating the need for data collection and better aligning with practical application needs; @ We show that the zero-shot LMMs can surpass specialized, training-based methods, revealing their potential for addressing VSG. @ We propose method, BAGLM, which incorporates priors from past video frames into LMMs through Bayesian filtering, modeling the temporal dependencies across steps via LLM queries; c We extensively evaluate BAGLM three datasets, showing that outperforms state-of-the-art offline methods.
2
Related work
Video Step Grounding, Earlier works in VSG adopted weakly-supervised learning approaches. For instance, Zhukov et al. [36] proposed to learn a model from instructional narrations and list of steps derived from temporal constraints, sharing components across tasks with similar actions or objects. Han et al. [9] proposed co-training framework that combines Temporal Alignment Network (TAN) with dual-encoder architecture, predicting step boundaries by aligning videos and narrations, using pseudo-labels derived from cross-modal agreement. VINA [18] considered step descriptions from WikiHow [29] and learned temporally ground them in videos without manual supervision. Recent methods have increasingly leveraged language models and large-scale pretraining. MPTVA introduced a multi-pathway alignment strategy using LLM-filtered narration summaries and multiple sources of alignment, merging them to create robust pseudo-labels for training. NaSVA [14] addressed multi-sentence grounding by leveraging L.LMs to transform noisy ASR transcripts into procedural steps, aligning them with video content using narration-bas based similarity score.

---

Together, these approaches illustrate the progress from weakly supervised models leveraging narra- tions to more sophisticated ones that integrate language models, multimodal alignment, and robust pseudo-labeling. However, most methods still rely on extensive training, domain-specific fine-tuning. and access to the full video, assumptions that limit their real- real-world applicability. To the best of our knowledge, BAGLM is the first triiiing-free online solution that addresses these limitations.
Video-Language Alignment refers the task of measuring the semantic consistency between a video and its corresponding textual description. Early approaches [T] 26] tackled this by relying on the cosine similarity between video frames and captions within the embedding space of CLIP [23] However, these methods are inherently limited by the well-known shortcoming of CLIP, i.e., by its inability to effectively capture temporal dynamics in text descriptions. As result, recent works have shifted toward leveraging LMMs [15] 30,
B1 12 33 and adopting metrics such as VQAScore [[6]. which are obtained from video question answering to better account for the temporal dimension.
Building on these recent studies, we propose to employ an LMM to assess the alignment between instructional steps and temporal video segments, requiring fine-grained video understanding. VSG is particularly challenging because key steps in instructional tasks often involve similar objects or scenes, For instance, "inserting screw" or "aligning parts" may look similar in tasks like "Assemble a chair" and "Fix a table", making them hard to distinguish without nuanced semantic understanding.
3 On using Large Multimodal Models for Video Step Grounding
Large Multimodal Models are powerful pretrained models that have demonstrated impressive zero- shot performance wide variety of tasks without further tuning. As previous solutions for VSG are training-based, we wonder whether off-the-shelf LMMs can address VSG. This section begins by providing formal definition of the VSG task, clearly distinguishing between its offline and online settings (Sec. 3.T). We then present the findings of our preliminary empirical investigation into the use of LMMs for addressing VSG, along with key insights gained from this study (Sec. 3.2)
3.1 Video Step Grounding
Given a video of a task composed of series of actions, video step grounding aims to detect which actions (or steps) appear in the video. Formally, let us denote the set of steps composing a task as
A = fa,) {a=))==1 ==1- where each step a; A is expressed in the natural language space L. and K the number of steps. Moreover, let us denote with V a video in the space V, split into T non-overlapping segments S = [S:)T= VSG aims to identify which steps in A are shown in the video.
The offline setting of VSG, as addressed by previous works [4 18 assumes access to the whole video (i.e., the full set S) when performing the task. In this work, we focus on online VSG, assuming a video stream where segments arrive one after the other, and we perform the task having only access to the current segment and the segments preceding it, i.e., S {S {Si..... S.). Thus, online VSG aims to predict whether segment S, E S depicts a step E A, given only the previous video segments Si.,. While we do not exploit this possibility, a model may also use the current segment to update predictions on past ones, contrary to the single prediction of the offline setting.
3.2 Large Multimodal Models are strong baselines for VSG
the visual encoder fvia, the text encoder fixe, and a text decoder faec- The encoders map their respective inputs into a shared d-dimensional embedding space, i.e., fvia V + Ed and fezt : L + Rd. The decoder maps the visual and textual inputs into probability simplex AlW,, over the L.M vocabulary W ie..
Let us define large multimodal model fIx via three elements:
faec Rd x Rd AIW, The next token is sampled from this probability vector.
Preliminary experiment. We propose to frame the problem of VSG as a multi-chote question answering task where the LMM, prompted with the current video segment and all possible steps. has to predict as answer either one of them or "none". Formally, given the current segment S., we prompt the LMM fusm using information regarding the task and step, obtaining the score for step a, as:
fum Tvsa) = faec (fvia (S:), fext. (Tvsa)) [+]
(I)
For simplicity, we omit the words' tokenization, assuming text prompts and videos are encoded equally.
3

---

Hteet
Cross Task
Ego4D Goal-Step
52,0
52.7
46.5
51.4
53.1
54.3
52.2
47.1
51.1
47.9
46,2
442
Avg. R@1 (%6
36
39.1
R@1 (%)
40
(%6
24.3
20
20
20
10-
IntenVL2.550
Intern VLa-sh
LLaVA-OV-D
Qwenn & VL-
VINA
haave
IatemVV122.900
Intern IntenV1..-66 V7.3-b
LLAVA-OV-7
Q & VL- tb
7b
NasYA
APTVA
InterVL
Intern VL33-b
0
Lia M-V------
Quni & VL. 7h
VSLaer
NasV A
Figure 2: Comparison of VSG performance on HT-Sttp, Cross Task, and Ego4D Goal-Step datasets, prompting LMMs with step options and video segments in an online fashion. For reference, we also show the performance of the top two performing methods from the state of the art (dark bars).
where Tysc is the task prompt (see details in Appendix) and [[] denotes the probability that the next predicted token is (corresponding to step a;), normalizing the scores across the multi-choice options. Note that, to account for no-step occurring, we include additional option "none of the above" (f [K + 1D. By normalizing the LMMs' probabilities over each choice, we map the segment into a probability simplex over the steps and the "none" option, i.e., fL
V x A A K
36 HT-Step [. and Ego4D Goal-Step [27]. HT-Step is a benchmark for procedural step grounding M. where the goal is to align steps from an instructional article with an input how-to video. The dataset provides two types of test sets: one for seen activities and another for unseen activities. The seen validation
Datasets & metrics. We evaluate methods on three public datasets:
CrossTask
and test splits follow [8], each containing 600 videos in total, with videos per activity across 120 activities. We evaluate with the validation set of seen classes, as the evaluation server hosting the test sets for the seen and unseen classes is unavailable.
CrossTask is an established instructional video benchmark for zero-shot step localization [36]. It contains about 4.8k instructional videos, covering 18 primary tasks and 65 related tasks. Only videos in the primary tasks are annotated as steps with temporal segments from a predefined taxonomy. We follow the same evaluation set as indicated in [4] using videos from primary tasks.
Ego4D Goal-Step (27) a subset the Ego4D R. includes 851 videos averaging 26 minutes in length. Unlike CrossTask and HT-Stepp, Ego4D videos collected without predefined tasks. Annotators label them hierarchically, first identifying goals (e.g., Makes the bread), then steps (e.g., Prepares the bread), and finally substeps (e.g., weigh the dough). We evaluate on its validation split.
For both HT-Step and Cross we follow the standard evaluation protocol [4 4, providing for each video the full set of steps for its task as multiple-choice options in the prompt. On average, each task includes about 10 steps in HT-Step and 7.5 in CrossTask. For Ego4D Goal-Step, we use step-level descriptions only (excluding substeps) and apply text normalization with spacs| lowercase, lemmatize (preserving plural nouns and verbal adjectives), and normalize whitespace and punctuation. After preprocessing, the average number of steps per vidco is 17.
Following we report Recall@ (R 1) on HT-Step and Ego4D Goal-Step, measuring whether the top-scoring timestamp for each step falls within the ground-truth interval. For Cross'Task, we report Average Recall@ (Avg.R@ I), computed by averaging per-task R across all primary tasks.
Discussion. Fig. 2 shows the results on the three datasets, using four LMMs with strong performance in video understanding benchmarks 2
[.) Qwen2.5-VL-7B- Instruct (2). Intern VL2.5-8B (6), and Intern VL3-8B [5). For reference, we also include the top two
state-of-tte -ft--e----hrtttt methods on each dataset, considering both in-domain ones, i.e.. trained and evaluated on the same dataset (VINA [18] and NaSVA (14) on HT-Step. VSLNet [34] on Ego4D Goal-Step) and out-of-domain ones (i.e., NaSVA on CrossTask and Fgo4D Goal-Step, MPTVA 4 on CrossTask) Among the LMMss Intern VI.2.588B scores the hest performance on CrossTask and Ego4D Goal-Step, outperforming MPTVA by 6.4% and NaSVA by 16. %, respectively. However, on HT-Step, NaSVA VA
We use the model available at http:://///pppy..oooooo/////////

---

your Agunddais
LLM
Transition Model P(a (c-1)
PREDICT
Dependency matrix
bel
P(a
progress,
S
P(a,
LMM
UPDATE
Figure 3: Overview of BAGLM. Given sequence of steps, LLM is used to estimate dependency matrix among them. This matrix is used to compute step transition probabilities employed during the predict step of Bayesian filter. As the video progresses, the transition model is updated using estimates of each step's progress from an LMM. The update step of the filter merges this with the predictions from the LMM, refining the output.
slightly surpasses VL2.55V (53.1 52.0). Overall, the LMMs outperform prior methods on CrossTask and Ego4D Goal-Step, while showing comparable performance on HT-Step..
Considering the lack of task-specific tuning, these results demonstrate that zero-shot LMMs perform surprisingly well on VSG, accessing only the current segment. natural question whether (i) inject information from past segments into LMM's predictions, refining them, while (ii) maintaining the zero-shot, training-free advantages of LMMs. In the following, we explore how achieve this by drawing inspiration from Bayesian filtering principles.
Remarks.
Bayesian Grounding with Large Multimodal Models
In Sec. 3 we have shown that LMMs are effective VSG without any task-specific tuning. However, they act without memory of past knowledge, performing step prediction by only looking at the current segment. Therefore, uncertain predictions (e.g., due to the segment acquisition) cannot benefit from past evidence (e.g., step performed in the previous segment), leading to potential model mistakes.
Formally, Eq. ①1) provides estimate for P(A oz (s)S, where A is discrete random variable taking values in the set A U none with none denoting any action outside the set of steps. Differently, would like to perform step prediction conditioned all previous segments, estimating P(A as ((p=)))))) without the need of storing the whole history. Inspired by Bayesian filtering, a probabilistic technique addressing sequence modeling from past observations [5]. we propose Bayesian Grounding with Large Multimodal Models (BAGLM). BAGLM (Fig. 3 estimates the transition probabilities across steps through the step dependencies estimated by an LLM, to refine LMMs' predictions.
In the following, we will first present the generic formulation of Bayesian filtering Sec. 4.1 and how revisit the predict (Sec. 4.2) and update (Sec. 4.3) steps for VSG using LMMs and LLMs.
4.1 Bayesian filtering
Let x be state of process, X, be its corresponding random variable at time 1, and be the observation time < l. The goal of Bayesian filtering (5) is to compute the posterior:
bel,(x) P(X, 21, ,2),
via two essential steps: the predict step first computes prior over the possible predictions using past estimations; the update step then estimates the prior with the current observations. In the following, we will denote Z1:t-
{21.-.. .......22 . for simplicity. Using the chain rule, we can write the posterior as:
bel,(x) = P(X, x|zx::) P(zz(Zua-1)
P(z Xx = x, Z1a-1) P(X: = x/21a-1);
(2)
5

---

where the first term is a normalization factor, the second is the likelihood of the current observation given the past ones and the current state, and the last the prior over the states from the observations.
To further simplify Eq. 2 we consider two assumptions: (i) we have a hidden Markov observation model, and thus P(z X x, 11: PX x); (ii) have initial prior over the states
independent from the first observation, i.e., P(Xo = x x/Z1) P(Xo = x).
Adding the first assumption to the second term and using the Chapman-Kolmogen equation on P(X, x|z1:1-1), we obtain:
update product
predict step
bel,(x)= = P(zz|Z1:t-) PiZ
=
P(X: = x x,)
bel;-1(x)
(3)
likelibood
X,EX
transition model
accumulated belief
normalization factor
where X' the set of possible states. The second term corresponds to the predict step, computing an estimate of the current state using the prior belief. The transition model describes the likelihood of state given the previous one, while accumulated belief denotes the likelihood of the previous state as recursively accumulated via Eq. B). The first term refers to the update step, where the predicted state probability is multiplied by the likelihood and normalization factor to obtain the final estimate.
A Bayesian filtering view on VSG. To adapt Eq. (3) to VSG we must define our states and observa- tions. The state what we want to estimate, i.e., the step a the current segment. The observation the input we receive from the environment: the segment S itself. Thus, we obtain the update step as:
(a) P(A, S1.) P(Sa/A, = S1a-1,
a(a = =aa) bel(-1(a;) (a:); (4)
S1.,
a.EA
where Aj random variable over the possible steps for the jth segment. We keep the original model's 's assumptions: an initial prior over steps independent of the first segment, and conditional independence of the current segment given the step (which holds for step-level semantics). In the following, detail the implementation of each component in Eq.
4.2 PREDICT: modeling dependencies among steps via language and progress priors
A peculiarity of VSG is that actions depend on each other: these dependencies provide priors on the actions performed in future segments, allowing us to build transition model, needed in Eq. { We exploit the internal knowledge of an LLM to estimate such dependencies. Specifically, we query the LLM to identify when a step must be completed before another can occur (i.e., is a prerequisite). As the dependency might be ambiguous, we instruct the LLM to estimate probability rather than a binary score, resulting in a matrix D E I x where each entry D., is the probability that
step aj is a prerequisite of step ag. We initialize the transition matrix as T = D allowing for self-transitions (iee., T2; 1), and transitions from all steps to those with prerequisites (i.e.,
Tr = 1f 11
Σ Tı, = 0), normalizing T across rows.
Modeling the action progress. The transition matrix T is static and purely based on the task description, without reflecting how the likelihood of a step evolves during the video. For example, if boil water is prerequisite for cook pasta and boil water has not yet finished, the video cannot transition to cook pasta. Instead, it is more likely to continue showing boil water or switch to parallel steps like prepare the sauce. Conversely, once cook pasta completed, becomes unlikely that the video will return to an earlier step as boil water. We therefore adjust T accounting for both step dependencies and the estimated step progress, introducing two scores: readiness and validity. Intuitively, step is ready when its prerequisites are sufficiently complete, while step is valid candidate for a segment if its successors have not yet been completed (24).
To achieve this, we query the LMM to infer the execution progress of each step within a video segment. Given segment S and the prompt Tprog, we define the estimated progress for step o; as:
progress,
j (Sm Tprog) 1.]

---

where fLMM (t. Tprog)li] denotes the model's s output probabilities over the vocabulary. We treat the model's probability distribution over the tokens {0, ....., 9}. as the distribution over progress levels. From the latter and D, we can compute whether an action is ready to be performed.
Step readiness. For each step ais compute readiness the weighted maximum progress of its prerequisite steps, i.e.,
Dis max, progress,
r:li]
(5)
Σ/=1 Did
where we measure the progress of step its maximum value across all preceding segments (i.e., T < t). performing weighted average across all steps. With Eq. (5), we get high values in case all predecessors of an action have high progress, and low otherwise.
Step validity. Contrary to the readiness, the step validity given by:
K D (1 max,<t progress, (j))
(6)
Σ Djs
This value high when no successors of a; in D have been executed yet, and low otherwise. Finally, adjust the transition matrix ' by accounting for readiness and validity of each step, i.e.
Tūi 5] r [i] velil Σx K
T(i k] rr[k] v.[k]
Predict step. Exploiting the transition model of Eq. (7). the predict step of Eq.
becomes:
(redict((a)) = T i] be,)-1(aj).
(8)
E.E
4.3 UPDATE: using LMM estimates to re-weigh the belief over steps
In the update step, we multiply the step prior for the observation likelihood P(S, A, = a) and normalization factor independent from the steps. However, due the cardinality of V, computing P(S, S,) is intractable. On the other hand, we follow Eq. estimating (S, Ar a) directly from the LMM. Formally, we use the Bayes rule and write:
P(S.) P(S a.) P(A = a)
P(S.) P(A a. P(A a;) fux (St, Tvsa)
where P(S,) is a prior on the segments independent from the steps, P(A, a;) is a prior over the steps independent from the observation. We replace P(A, a/S,) with the LMM prediction. While for P(A, a,) there are various possible choices, in our approach, we consider the prior to be
uniform, ablating this choice in the Appendix.
Final filtering model. Considering the uniform prior over the states and merging Eq. obtain the final belief bel, (a:) for step a, and segment St as:
8 into Eq.
9
1 (a,) = z fum (S., TVsc [i]]
Σ
T.li. bel.-1(a,),
(10)
a,EA
where a normalization factor containing all elements independent of the specific step a..
Experiments
In this section, describe experimental protocol and present the comparison w.r.t the state of the art (Sec. S.I). Finally, we perform a detailed study on BAGLM (Sec. 5.2) We use the same datasets and metrics described in Sec. 3 in our experiments.
Implementation details. Our method is implemented considering Intern V1.2.5-8B 6 as our LMM, based on the results of Sec. We employ LLaMA3-70B-Mne Instrutt [7] as our LLM of choice derive
7

---

Table Comparison between state-of-the-ee offline methods and our online method BAGLM.
(a) HT--ttep and Ego4D Goal-Step
(b) CrossTask
HT-Step R@1
Ego4D Goal-Step
Method
Avg. R@D
Method
1 R@1
Offline
Offline
Zhukov et al.
36
22.4
VSLNET 34
24.3f
HT100M 20
33.6
TAN* 9
30.7
VIDEOCLIP
32
33.9
VINA 18
39.1
MCN R
35.1
NASVA 14
53.1
29.1
DWSA
35.3
MIL-NCE
40.5
Online
VT-TWINS
40.7
NASVA
14
46.1
24.2f
UNIVL 17
42.0
BAGLM
57.4
43.3
VINA 18
44.8
NASVA 14
46.7
MPTVA
47.9
Online
BAGLM
59.8
our transition model. To test our model, we split videos into sequences of non-overlapping 2- seceonnd segments, providing them as input the LMM one after the other. We ran all experiments on single NVIDIA H100 64GB GPU, except for
(7.. which required 4 H100 GPUs.
Baselines. We compare method with several state-of-the-an of-tte--rttttttt approaches for VSG: Zhukov al. [36], HT100M [20), VideoCLIP (32), MCN [3, DWSA [25), MIL-NCE [9). VT-TWINS []],
UniVL [17], VINA (18), TAN* O D. NaSVA [44, and MPTVA 4. We also implement an online variant of NaSVA [4), which introduces causal masking in the transformer encoder's self-attention layers to restrict attention to past segments only. Reported results are taken from the original papers, except for NaSVA on HT-Step and Ego4D Goal-Step, and VSLNet (34 on Ego4D Goal-Step where we use the authors' released code. These are marked with in the tables.
All baselines are training-basee and offline, except for our online variant of NaSVA, and use pre-training. except for VSLNet (trained on Ego4D Goal-
How HowTo 100M 20 training Step), for Zhukov et al. (36] and DSWA (25 (trained on CrossTask) and for MPTVA 4 (trained subset of HowToo T0MM... For CrossTask, VideoCLIP [32 and UniVL [17 perform further fine-tuning step on Cross Task data.
5.1 Comparison with state-of-the-art methods
Tab. la and Tab. Tb report the results of evaluation in the three considered datasets. Overall, BAGLM, built on top of Intern VL2.5-88, outperforms offline methods that rely on noisy supervi- sion from narrations without manual annotations. Notably, it outperforms the current state-of-the-art method, NaSVA, by 4.3% on HT-Step, which consists of videos from HowTo T010M, the same dataset used for NaSVA's self-training. The improvements of BAGLM on Cross Task and Ego4D Goal-Step, with gains of 13.1 % and 14.2%, respectively. The margins in CrossTask are remarkable as there exist methods (i.e.. VideoCLIP, UniVL) that are specifically fine- tuned for this domain. The same applies to Ego4D Goal-Step. This dataset is particularly challenging for models trained HowTo 100M due the distribution shift introduced by its egocentric videos. On this dataset, BAGLM also outperforms VSLNet by significant margin (+19%), despite VSLNet being trained on data from the same domain. Notably, all these results have been achieved in an online setting, with all competitors having access to the whole video, contrary to BAGLM. When evaluated under the same online setting, NaSVA's performance drops to 46.1 R@ on HT-Step and 24.2 on Ego4D Goal-Step (-7% and -4.9% compared to its offline variant), remaining well below BAGLM (-11.3% 1.3% and 19.1%).. These results further highlight the effectiveness of our approach in the challenging online scenario.
NaSVA especially significant

---

Table 2: Ablation study on the transition model.
Table 3: Ablation study on varying the LLM.
Readiness
Validity
HT-Step
CrossTask
Ego4D
Dataset
LLaMA-3.3-70B
GPT-4.1-mini
55.9
58.0
42.1
HT-Step
57.4
57.1
57.0
58.8
42.0
CrossTask
59.8
60.9
56,4
58.8
43.1
Ego4D
43.3
40.6
57.4
59.8
43.3
InteraVi2
IusamVL3-sb InterVL3-8bb (BaGLM)
LLVVA-OV-7b LLAVA-OV-7 (BaGLM)
Qwent S-VL-b
IatemVi22-9bb (BaLLM)
Qwen2 (BaGLM)
HT-Step
Cross
Ego4D Goal-Step
60
RecallA1 (%)
50
4
Segment duration (s)
Segment duration (s)
Segment duration (s)
Figure 4: Ablation study varying the segment duration and the used LMM.
5.2
Ablation studies
In this section, we analyze the key components of BAGLM, evaluating different configurations the transition model, different LLMs for generating the dependency matrix, and experiments with oracle step dependencies and progress. Additional design choices are discussed in the Appendix.
Transition Model. The transition model estimates the conditional probability of moving from one step to the next and is used in the predict step of Bayesian filtering, as per Eq. (8). We first evaluate a static transition matrix. initialized from the dependency matrix. We then study the effect of including the readiness score (i.e., if prerequisites are not completed, Eq. 55), the validity one (i.e., if the step should not be re-executed, Eq. (6). and both. As shown in Tab. 2 readiness improves the static model by 1.1 % and 0.8% on two datasets and performs similarly to the static model on the third (-0.1%). Thus, accounting for the dependency matrix (and if prerequisites met) tends to improve performance by reducing the score of non-executable actions.
Validity improves the static model by +0.5%, +0.8% and +1.0% across the three datasets. By considering the status of actions that occur later time, influences the predictions of their prerequisites (e.g., turn on the stove) and prevents them from being repeated unnecessarily.
The best results come from combining both, improving performance by 1.5%, 1.8%, and 1.2%. This highlights the benefit of including both types of estimated priors when updating the transition model.
Choice of LLMs. Tab. Janalyess how BAGLM is affected by the LLM used to generate dependencies between steps. We compare LLaMA-3. 3.MMMA333-705- 70B Instruct 7 and GPT-4. 1-mini [21]. The two models achieve comparable overall performance: LLaMA-3.3-7088 performs better HT-Step (+0.3%) and Ego4D GoalStep (+2.7%) but worse on CrossTask (1).1%). This trend suggests that -LLMA--33-7000 performs better datasets with more generic step descriptions, whereas proprietary models like 1-i-4.1-mini are more effective at capturing dependencies among more atomic actions.
show how BAGLM's performance varies w.r.t. the segment duration (from 4 seconds), considering different LMMs: InterVL2.55 8B [6), Intern VL3 LLaVA-One Vision 7B [13, and Qwen2.5 7B (2).
LMMs and segment duration. In Fig.
8B 35
From the figure, we can see three trends. First, BAGLM consistently improves the performance of the underlying LMM it is applied to across all segment durations on both HT-Stee and CrossTask with gains becoming more pronounced as segment duration increases. This is related to the second trend, the impact of the segment duration. Segments that are too short may lack sufficient visual cues to evaluate the video-step alignment or the progress, while longer ones may span over multiple actions, making it harder to localize steps precisely. Using 2-second segments offers the best trade-off,

---

Table Results with oracle dependencies and step progress.
Progress Oracle
Dep. Matrix Oracle
HT-Step
CrossTask
Ego4D
57.4
59.8
43.3
44.9
38.6
36.0
54.6
58.3
45.2
62.6
66.9
82.2
improving performance by +1.9%, +1.6%, +0.3%, and +2.3% across the three datasets compared to 1-second segments.
Third, BAGLM does not provide clear advantages on Ego4D Goal-Step (i.e.. -1.9%, -0.8%, -3%, and +0.3%). This can be attributed to challenges specific to this dataset. Ego4D Goal-Step videos are significantly longer on average (28 minutes vs, minutes and less than minutes for HT-Step and Tasks,ask, respectively) and contain coarser step annotations (53 seconds per segment on average vs. 16 seconds and 10.7 seconds for HT-Sttp and CrossTask, respectively). These longer videos and broader annotations result in more generic step descriptions, which in turn make it harder to infer accurate dependencies between steps and to estimate progress, as we will analyze in the following.
Use of annotated step boundaries from datasets. Given the different impact of BAGLM across datasets, analyze the performance of the Bayesian filtering formulation without potential noise from the dependency matrix and/or the estimation of step progresses. With this aim, we conduct an analysis similar Tab. [ this time using step dependencies and action progress derived directly from the original datasets. For each video, we construct a chain of steps based on its ground truth temporal order. Note that while these dependencies reflect the observed execution order, they are not true semantic constraints, as the same steps may occur in different orders in other videos. Progress is computed as the normalized fraction of completion between step's 's start and end timestamps.
Results of this oracle experiment are presented in Tab. 4 showing consistent gains across all datasets (e.g., +5.2% on HT-Step), even the challenging Ego4D Goal-Step (i.e., +38.9% This confirms that the Bayesian filtering formulation is effective, and that jointly improving the elements used to estimate the transition matrix (i.e., progress, dependency) would further boost the results of BAGLM.
Conclusion
In this work, we introduced BAGLM, novel training-free approach for online video step ground- ing. BAGLM uses Bayesian filtering to integrate information from past video frames into LMM predictions. consists two key components: a transition model, initialized from step dependency matrices generated by LLMs and updated over time using action progress and dependency constraints; and an observation model, implemented as an LMM, which refines predictions as the video unfolds. We evaluated BAGLM HT-Step, CrossTask, and Ego4D Goal-Step, showing that it outperforms raining-based methods without requiring additional training or data collection.
Limitations. We identify two main limitations of our approach. First, BAGLM fully relies on pre-trained LLM for estimating the step dependency matrix, so its effectiveness is closely tied the quality of the model predictions. As more advanced L.MSs become available, we expect that BAGLM performance will improve accordingly. Second, prior estimation within our Bayesian framework may be improved with more specific task knowledge. For example, estimate step progress by querying an LMM: future works could refine this approach by incorporating priors on step durations.
Acknowledgements. This work was supported by the Ministero delle Imprese e del Made in Italy (IPCEI Cloud DM 27 giugno 2022 (PCEI-CL-0000007) and by the European Union (Next Generation EU). Additional support was provided by the EU projects ELIAS (No. 101 120237) and ELLIOT (No. 101214398). We acknowledge ISCRA for awarding this project access to the LEONARDO supercomputer, owned by the PProHPC Joint Undertaking, hosted by CINECA (Italy), as well as the EuroHPC Joint Undertaking for awarding us access to NareNost at BSC, Spain. The author also thanks Andrea Pilzer for valuable discussions efficient video decoding.
10

---

References
[1] Triantafyllos Afouras, Effrosyni Mavroudi, Tushar Nagarajan, Huiyu Wang, and Lorenzo Torresani. Ht-step: Aligning instructional articles with how-to videos. NewrIPS, 2023.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song. Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv, 2025.
[3] Brian Chen, Andrew Rouditchenko, Kevin Duarte, Hilde Kuehne, Samuel Thomas, Angie Boggust, Rameswar Panda, Brian Kingsbury, Rogerio Feris, David Harwath, et al. Multimodal clustering networks for self-supervised learning from unlabeled videos. In ICCV, 2021.
[4] Yuxiao Chen, Kai Li, Wentao Bao, Deep Patel, Yu Kong, Martin Renqiang Min, and Dimitris N Metaxas. Learning to localize actions in instructional videos with Ilm- Imm-asedd multi-pathway text-video alignment. In ECCV, 2024.
[5] Zhe Chen. Bayesian filtering: From kalman filters to particle filters, and beyond. Statistics, 2003.
[6] Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open- source multimodal models with model, data, and test-time scaling. arXiv, 2024.
[7] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv, 2024.
[8] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.
[9] Tengda Han, Weidi Xie, and Andrew Zisserman. Temporal alignment networks for long-term video. In CVPR, 2022.
[10] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv, 2021.
[11] Dohwan Ko, Joonmyung Choi, Juyeon Ko, Shinyeong Noh, Kyoung-Woon On, Eun-Sol Kim, and Hyunwoo J Kim. Video-text representation learning via differentiable weak temporal alignment. CVPR, 2022.
[12] Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Xide Xia, Pengchuan Zhang, Graham Neubig, and Deva Ramanan. Evaluating and improving compositional text-to- visual generation. In CVPR, 2024.
[13] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li. Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLLVA--oevissio: Easy visual task transfer. Transactions on Machine Learning Research, 2025.
[14] Zeqian Li, Qirui Chen, Tengda Han, Ya Zhang, Yanfeng Wang, and Weidi Xie. Multi-sentence grounding for long-term instructional video. In ECCV, 2024.
[15] Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, and Deva Ramanan. Revisiting the role of language priors vision-language models. arXiv, 2023.
[16] Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In ECCV. Springer, 2024.
[17] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv, 2020.
[18] Effrosyni Mavroudi, Triantafyllos Afouras, and Lorenzo Torresani. Learning to ground instruc- tional articles in videos through narrations. In ICCV, 2023,
11

---

[19] Antoine Miech, Jean- Can-tappiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, 2020.
[20] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00me Learning text-video embedding by watching hundred million narrated video clips. In ICCV, 2019.
[21] OpenAL. Gpt-4o mini: advancing cost-efficient intelligence. htttps///peeaai com/inde// gpt-4-1/ 2024.
[22] Chiara Plizzari, Alessio Tonioni, Yongqin Xian, Achin Kulshrestha, and Federico Tombari. Omnia de egotempo: Benchmarking temporal understanding of multi-modal Ilms in egocentric videos. CVPR, 2025.
[23] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[24] Yuhan Shen and Ehsan Elhamifar. Progress- aware online action segmentation for egocentric procedural task videos. In CVPR, 2024.
[25] Yuhan Shen, Lu Wang. and Ehsan Elhamifar. Learning to segment actions from visual and language instructions via differentiable weak sequence alignment. In CVPR, 2021.
[26] Yaya Shi, Xu Yang, Haiyang Xu, Chunfeng Yuan, Bing Li, Weiming Hu, and Zheng-Jun Zha. Emscore: Evaluating video captioning via coarse-grained and fine-grained embedding matching. In CVPR, 2022.
[27] Yale Song, Eugene Byrne, Tushar Nagarajan, Huiyu Wang, Miguel Martin, and Lorenzo Torresani. Ego4d goal-step: Toward hierarchical understanding of procedural activities. NeurIPS, 2023.
[28] Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In CVPR, 2019.
[29] wikiHow. wikihow: How-to instructions you can trust. hestps://www.wikiiiii
[30]
Jay Zhangjie Wu, Guian Fang, Haoning Wu, Xintao Wang, Yixiao Ge. Xiaodong Cun, David Jun- hao Zhang, Jia-Wei Liu, Yuchao Gu, Rui Zhao, et al. Towards better metric for text-to-video generation. arXiv, 2024.
[31] Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: unified foundation model integrating visual understanding and generation. arXiv, 2024.
[32] Hu Xu, Gargi Ghosh, Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero- shot video-text understanding. In EMNLP, 2021.
[33] Luca Zanella, Massimiliano Mancini, Willi Menapace, Sergey Tulyakov, Yiming Wang, and Elisa Ricci. Can text-to-video generation help video-language alignment? CVPR, 2025.
[34] Hao Zhang, Aixin Sun, Wei Jing, Liangl Zhen, Joey Tianyi Zhou, and Rick Siow Mong Goh. Natural language video localization: A revisit in span-based question answering framework. IEEE TPAMI, 2021.
[35] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv, 2025.
[36] Dimitri Zhukov, Jean-Baptiste Alayrac Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos. In CVPR. 2019.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: This paper addresses the task of Vidco Step Grounding (VSG), which aims to ground natural-language steps composing a task into a video performing the task. Existing works primarily offline and training-basee limiting their real-world applicability. Moti- vated by the promising performance of prompting Large Multimodal Models (LMMs) in performing VSG, propose novel training- free method BAGLM leveraging LMMs with Bayesian filtering to further reduce uncertain prediction. BAGLM scores new state-of-the- art VSG performance three main benchmark datasets, despite of more restrict setting. Our ablation study also proves our key methodological design.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have created dedicate paragraph in Sec. 6 where we explicitly discussed main limitations in terms of assumption and identified main technical aspects that impact performance.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
13

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We leverage established theoretical results from Bayesian filtering, applying them to construct our method BAGLM.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
We describe our method full details with both equations. The experimental protocol is clearly explained in terms of datasets, metrics, and implementation details. The codebase will be released upon acceptance to ensure experimental result reproducibility.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
14

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code is available at
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
Sublic/guides Codeeuumissionoollie
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experimental details are provided in the Sec. The codebase will also be released upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
15

---

Justification:
We follow the evaluation protocol of prior works (e.g., NaSVA (14) for fair comparison.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provided essential information regarding the compute resources in Sec.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httnhhggttpre//aeerrr..cc///aatt
Answer: [Yes]
Justification: We have carefully inspected the code of ethics, and believe work conforms to it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
16

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We use existing, publicly available dataset that already includes appropriate safeguards. As our work does not involve the release of high-risk models or data, we deem additional safeguards not required.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
17

---

Justification: We have properly referenced all the software, data, and related works that supported the development and evaluation of BAGLM in the Appendix. CrossTask follows BSD 3-Clause License, HT-Step follows Creative Commons Atribution-NoCCommeree 4.0 International Public License and Ego4D Goal-Step follows MIT License Moreover, we publicly release our dataset and code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, papprssitthhaeee thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- se-packaedd, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code and instructions for running our BAGLM will be released alongside the manuscript upon acceptance. Essential implementation details are already included in the manuscript at the submission time.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
18

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Our method exploits the internal knowledge of LLMs to obtain the dependen- cies among steps. The prompts used are listed in the Appendix.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hutt:e///eeurieescc//o/ffrrrrer/ for what should or should not be described.
19

---

In the Appendix, we first provide the prompts used in BAGLM (Sec.Ay We then present additional experimental analyses (B. including the type prompt used to compute Video Step Grounding scores, the prior used in the update step of our method, and step localization results. We extend these analyses with evaluations on the COIN dataset to demonstrate the generalization of BAGLM across domains and tasks, robustness analyses of LLM -egeeerrtteed dependency matrices on HT--Step, and analyses step prediction performance by temporal position and by the number candidate steps. We also describe the computational cost of BAGLM during inference (Sec. C). discuss the broader social impacts of our work (Sec. D) and list the licenses and URLs for all assets used (Sec. Finally, provide qualitative results (Sec.
A
Prompts
use three types of prompts in our approach. For each, the Large Multimodal Model (LMM) prompted to generate answer by selecting from list of options. We then compute the model's score for each choice by applying a softmax to the logits of the first generated token and taking the probability assigned to the first token of each option. Video frames positioned relative to the text according to the input formatting required by the underlying LMM.
The task prompt Tysc is used to estimate the probability that a given video segment corresponds to each step the task, or to the "none" class. The LMM is prompted with the current video segment, the task goal (i.e., the procedural task), and all candidate steps, and must select either one of the steps or the "none" option as the answer.
Tysc
You are watching video segment of someone attempting to {goal}.
What is the main action being performed in this exact moment?
Options: {steps}
Answer with only the letter label of the correct option (e.g., A, B, Z, AA, AB, etc.), with no extra text.
The progress prompt Tprog is used estimate the execution progress of specific step within video segment. The LMM is prompted with the current video segment, the task goal, and given step, and must select progress value from 9, Here, indicates that the action not occurring, I means the action is about to begin, 5 corresponds to the middle of the action, and 9 means the action has just finished.
Trog
You are watching short video clip.
The goal of the person in the video is: {goal}
The specific action of interest is: {step}
Rate how far along this action in terms of execution progress, using a scale from 0 to 9:
0 the action is not present this clip
the action is just beginning about to begin
the action is halfway complete
9 the action is just finishing or about to finish
Respond with single number from 0 to 9. Do not include any other text.
20

---

Table Comparison of the off-the-shelf IntemVL2.55888 model using different prompts for Video Step Grounding, Queries / S indicates the number of LMM queries per video segment S.
Prompt Queries S
HT-Step
CrossTask Avg. R@1
Ego4D Goal-Step
R@1
R@ 1
bin V3G
K
52.5
56.0
41.8
Ry3G
52.0
54.3
45.2
Finally, the prerequisite prompt "prerrg is used estimate step dependencies within task. The LLM is prompted with the task goal, a step, and candidate prerequisite, and must answer either "Yes" or "No".
Tprareq
You are performing the task: {goal}
the following step strictly required before another?
Prerequisite candidate: {prerequisite) Target step: {step}
Answer "Yes" if the target step cannot be completed correctly without first completing the prerequisite step. Otherwise, answer "No".
Answer:
B Additional analyses
In this section, we present additional analyses BAGLM, including ablations on the prompts used to compute Video Step Grounding scores, the choice of step priors, and step localization performance. We also evaluate generalization on the COIN dataset, analyze the robustness of LLM -eg.eeeeedttdd dependency matrices HT-Steep and study step prediction performance temporal position and by the number of candidate steps.
Binary multi-choice VSG prompt. While we compute VSG scores by framing the problem as multi-choice question answering task, an alternative is to treat it as a binary question answering task. In this case, can ask the LMM whether given video segment corresponds to specific step, querying the LMM for each step independently. We compare these approaches by evaluating the off-the-shel model Intern -L2.5-8B using the multi-choice prompt Tysc and the binary prompt myss"
bin.
TTS3
You are watching video segment of someone attempting to fgoal}. the person currently performing the action: "(step}"?
Answer with "Yes" "No" only.
As shown in Table 5 using the multi-choice formulation improves performance by 3.4% on Ego4D Goal-Step. On HT-Step, performance is comparable (a slight drop 0.5%), while on Cross Task, the binary formulation performs better by 1.7%. However, the binary formulation requires querying the LMM times per video segment, once per candidate step, whereas the multi-choice formulation requires only one query per segment.
For example, the HT-Step video -25- 1nb0ki0, which contains 283 segments (each seconds long) and 10 steps, using Tvsc requires 24.79 seconds in total (approximately 0.0876 seconds per
21

---

Table 6: Ablation study on the step prior P(A = az).
HT-Step R@11 t
CrossTask
Ego4D Goal-Step R@1
Avg. R@1
1) fives St, Texxt
50.5
52.6
28.1 1
2) Layed P(A A1 az)) P(A = =a)
55.6
57.3
39.9
3) 847
57.4
59.8
43.3
Table Comparison between NaSVA and our BAGLM on the HT-Step dataset.
HT-Step (1 mAP@ (oU)
Method
@0.3
@0.5
@0.7
@[0.3-0.77
0.7]
NASVA 14]
13.7
6.0
1.4
6.7
BAGLM
15.9
6.7
1.5
7.7
segment), while vys takes 214.04 seconds (about 0.756 seconds per segment) with NVIDIA H100 GPU with 64GB of memory. Given the comparable accuracy and significantly lower inference time, we adopt Tysc as our default prompt for computing Video Step Grounding scores.
Prior over the states P(At = a:). In Eq, (9), describing the update step, we normalize the scores using general prior P(A, = az) over all possible states. In this section, we explore alternatives to the adopted uniform prior in our method.
Table shows the results of our analyses. Row (1) reports results using context-dependent prior, where the LMM is prompted with Tnext to predict the probability distribution over the next most likely action given the current segment, the task goal, and all candidate steps:
Tnext
You are watching video segment of someone attempting to {goal}.
What is the most likely next step the sequence?
Options: {steps}
Answer with only the letter label of the correct option (e.g., A, B, ... Z, AA, AB, ctc.), with no extra text.
Predicting the next action is often similar to predicting the current one, resulting in prior distribution close to the VSG scores. a result, the probability associated with some true positive steps will be erroneously lowered in favor of the "none" class.
Row (2) presents the results of using a prior initialized from uniform distribution and updated each timestamp by applying the state transition model. In this prior, transitions to "none" are often the most likely as they not penalized by step validity readiness. This lowers the likelihood of the "none" class in the posterior, promoting false positive predictions.
Row (3) shows the performance using a uniform prior (our adopted one), which assigns equal probability to all steps. This setting achieves the best results among all the considered prior options.
Step localization. While we report performance using Recall and Avg. Recall@) as our main metrics in the main manuscript, following the evaluation protocol from 14 4.
also evaluate step localization performance (i.e., predicting the start and end timestamps of steps) by leveraging the final belief distribution for each video segment. This results in an alignment matrix of size T x K, where T is the number of video segments and K is the number of task steps. Following [18], we extract temporal segments from this matrix using a 1D blob detection approach. Specifically, we apply Laplacian of Gaussian (LoG) filters at 13 different scales, corresponding to Gaussian standard deviations ranging from to 480. The confidence score for each predicted segment is computed as the average step probability across all video segments within the predicted temporal boundaries. We
22

---

Table Ablation study on varying the used LMM on HT-Step. Bold indicates the best result within each LMM group (i.e., with and without BAGLM).
HT-Stepp (1 mAP@loU)
Method
@0.3
@0.5
@0.7
@10.3-0.7]
LLAVA-OV-78
14.1
5.5
1.3
6.6
+ BAGLM
14.7
5,8
OWEN2.5-VL-77
15.5
1.4
7.3
+ BAGLM
15.8
6.6
1.7
7.7
INTERNVL3-88
35
15.6
5.8
1.4
7.2
+ BAGLM
17.5
6.5
1.6
8.1
INTERNVL2.5 5-88
6
15.0
6.0
1.3
7.0
+ BAGLM
15.9
6.7
1.5
7.7
Table 9: Comparison between NaSVA (best offline competitor) and our online method BAGLM on the COIN dataset.
Method
1 R@1
Offline
NASVA 14
36.8
Online
NASVA 14
32.1
INTERNVL.2.5-88
6
47.9
BAGLM
51.0
compare method against NaSVA 14] the best method (see Tab. in the main manuscript) with publicly available code. NaSVA uses a transformer-basee architecture where steps act as queries that iteratively attend to video features, producing an alignment matrix between steps and video segments. To ensure a fair comparison, we derive the start and end timestamps for each step by applying the same ID blob detection procedure used in our method NaSVA's similarity matrix, computed from its final transformer layer. Before applying blob detection, we convert the similarity scores for each segment into valid probability distribution over steps by applying softmax function.
We conduct this analysis on the HT-Step dataset and report results in Tab. 7 We evaluate local- ization performance using the HT-Step protocol, which measures article- soreee--roudiing mean Average Precision (mAP) at various IoU thresholds. This involves treating each step as a class-agnostic text query, computing AP per activity, and averaging the results to obtain overall mAP. BAGLM performs better than NaSVA at all loU thresholds (i.e., +2.2%, +0.7%, and +0.1%).
We complement the results of Fig. 4in the main manuscript by evaluating how BAGLM performs when applied different LMMs, i.e., InternVL2.5 8B (6) LLaVA-One Vision 7B [3] Qwen2.5 7B (2), and Intern 8B (35. using mAP as the evaluation metric. From Tab, g we can see that adding BAGLM to any LMM consistently improves performance. The improvements are largest at lower IoU thresholds (e.g., +1.9% at 0.3 w.r.t. the off-the-shed Intern 8B) and gradually diminish higher thresholds (e.g., +0.2% 0.7).
COIN. To further demonstrate the generalizability of BAGLM, we evaluate it on the COIN 28 dataset, which contains instructional YouTube videos spanning various domains. We the test split, consisting of 2,797 videos, of which 2,354 were successfully downloaded (the remainder being unavailable or set to private). These videos cover 12 domains (i.e., Dish, Drink and Snack, Electrical Appliance, Furniture and Decoration, Gadgets, Housework, Leisure and Performance, Nursing and Care, Pets and Fruit, Science and Craft. Sport, and Vehicle) and are categorized into 180 tasks.
We compare BAGLM against three baselines: (1) Interm 6 (off-the-shelf), (2) NaSVA 14 (best offline competitor), (3) an online version of NaSVA. As shown in Tab. 9 BaGLM outperforms NaSVA by 14.2 (offline) and 18.9 (online), and achieves a gain 3.1 over Intern VL2.5-8B. These results suggest that our method generalizes well across diverse domains and tasks.
Dependency matrix robustness. We evaluate the robustness L.LM-generated dependency matrices on the HT-Step dataset, which includes 120 cooking-relatee related tasks, each with 5 videos (600 videos
23

---

Table 10: Dependency violations and fraction of tasks with violations for different threshold values.
Metric
Model
0.30
0.70
0.80
1.00
Violated dependencies (%)
LLaMA-3.3
26.1
8.6
8,4
8.4
8,4
8,4
8.3
8.3
GPI-4,1-mmin
26.1
15.4
12.5
11.6
10.8
10.2
9.6
8.9
8.2
7.6
0.0
LLaMA-3.3
100.0
72.0
71.3
71.3
69.3
69.3
69.3
68.7
68.7
68.7
68.7
Tasks w/ violation (%)
GPT-4.I-mii
100.0
74.0
65.3
64.0
59.3
54.0
51.3
48.0
44.7
42.0
0.0
Table 11:
R@)
performance across step positions on HT-Step.
Model
Bin 0
Bin
Bin 2
Bin 3
Bin 4
InternVL2.5-8b
53.6
52.7
$4.6
49.7
49.3
BAGLM
56.1
57.2
60.2
56.8
56.6
in total). For each task, we convert the soft dependency matrix generated by either GPT-4.1 1-mini Instruutt into binary (hard) dependencies at various thresholds. To evaluate robustness, we measure the ratio of violated dependencies. A dependency Daj- indicating that step aj is a prerequisite for step 0; is considered violated if a, occurs before the first occurrence of aj in any of the videos for that task (i.e., the prerequisite should always occur before the dependent step whenever both are present).
Tab. reports the percentage of violated dependencies and the percentage tasks with at least one violation for both LLaMA-3.3-700 -.0.....-70-BIItrrrue and GPT--.1-mmni. Both models perform well across different tasks, with roughly 10% of dependencies being violated depending on the threshold. In particular, -LaMA-3.3-708 IIntructt shows flat violation ratio (~-8%) across wide range of thresholds, suggesting that its outputs are mostly binary (i.e., close to 0 or 1). In contrast, GPT-4.1- mini shows smoother decline in violation rate the threshold increases, indicating confidence distribution. Even at low thresholds, portion of tasks (28% for LLaMA-3.3 and 26% for GPT-44.-minn at threshold 0.1) have violated dependencies, suggesting that the predicted matrices can approximate the oracle matrix.
graded
Step prediction by temporal position. We analyze how step prediction performance varies across the temporal positions of steps in HT-Step. For each step, we normalize its occurrence within the video to the range [0, 1] and divide these positions into five bins, then compute R @@! for each bin. Tab. reports R@I across these bins, from early (bin 0) to late (bin 4) steps. The off-the-shelf model (Intern VL),.5-8B), which does not leverage past context, maintains relatively stable performance. performs better on early and mid-range steps (bin O: 53.6%, bin 2: 54.6%) than on the final steps (bin 4: 49.3%) suggesting that initial to mid-range steps steps are harder to recognize.
semantically distinctive while later
In contrast, BAGLM shows upward trend from early to mid-range bins, likely due to its use of accumulated temporal context: R@ I increases by +1.1% from bin to bin I and +3.0% from bin to bin 2. Its performance then decreases toward the final bin, likely due to limitations inherited from the base model (i.e., InternVL2., 2-B performs worst in the final part of the video) as well as error propagation.
Step prediction by number of candidate steps. We analyze how performance changes with the number of candidate steps in the HTT-Step dataset by grouping videos based on the number of steps in their tasks. For each group, we compute the average R @@ using the off off-the-shelf Intern VI.2.5-8B model (Fig. 5). The results show non-monotonic trend: except for tasks with very few steps (fewer than 4), there is no clear decline performance as the number of steps increases. This suggests that Intern VL2.5-8B can handle prompts with many candidate steps. Note that some bins contain few videos (e.g., 14 and 29 for tasks with 17-18 steps), which can cause large fluctuations in the measurements (e.g., from 57.1% to 34.5%).
C Inference speed analysis
We measure inference speed on single NVIDIA H100 GPU with 64GB of memory. Figure 6 shows the average processing time per video segment for three main operations: video loading and preprocessing, computing VSG scores, and estimating action progress. These estimates were
24

---

90
Recall (%)
80
70
60
332
50-
40
70
30
0
Number of Steps (K)
Figure 5: R@ on the HT-Step dataset, grouped by the number of candidate steps in each task.
Load preproc
2.0
Firox
Fina
Time per Segment
0.5
Number of Steps
Figure 6: Average processing time per video segment for computing VSG scores, estimating action progress, and video loading/preppe peating/prproccssingg for HT-Step videos grouped by their number of steps.
computed HT-Step videos with the same number of steps. Our results show that the total processing time per segment is largely dominated by action progress estimation, which grows linearly with the number steps in the task. Importantly, the majority of the segments are processed in under seconds, implying that the 2-second segments can be processed in real time.
As an example, consider the video -5 1nb0ki0 from the HT-Step dataset. This video is 567 seconds long and corresponds to the task Make Nigerian Style Jollof Rice, which includes 10 steps. We gener- ate the step dependency matrix for this task using GPT-4.1-min with prompt Tpzerequ requiring 87. 8.84 seconds to compute. This matrix generated only once per task and reused for all related videos. We process each 2-second video segment as follows: we compute VSG scores using Intern V1.2.-81 with prompt Tysa. taking 0.088 seconds per segment. We estimate action progress using prompt Tprog. requiring 0.814 seconds per segment. Additional time is spent model initialization (per- formed once) and video decoding and preprocessing. Using the torchcoded library, decoding and preprocessing take approximately 1.201 seconds per segment on CPU, with possible speedups using GPU acceleration. In comparison, the Bayesian filtering step is computationally negligible. Overall, excluding step dependency generation and model loading, each 2-second segment of this video can be processed in approximately 2.1 seconds.
D Broader impact
Our approach enables online understanding of instructional video steps without requiring task-specific training. This makes suitable for deployment real environments, providing feedback users performing tasks. Such capabilities can benefit a wide range of applications, including education, skill development, and assistive technologies, especially contexts where annotated data limited or users require immediate feedback (e.g., remote learning, home cooking, or physical rehabilitation).
25

---

Table 12: List of URLs and licenses for all assets used.
Name
URI.
License
LLAVA-OV-7B
https:
www.iioonngnen--eeeeeeee 7b-ov
Apache 2.0 Apache 2.0
OWEN2.5-VLL2
Instrruct
INTERNVL2.5-8B
http:///ggaaaaaoooo////////aaaaa 5-88
MIT MIT BSD-3-Clause
INTERNVL.3-8R
CROSSTASK
HT-STEP
CC-BY-NC 4.0
EG04D GOAL-STEP
MIT CC BY-NC 4.0
COIN
However, as our method depends large pretrained multimodal models. may also inherit their limitations, such as biased predictions or inconsistent performance across different cultural or domain- specific tasks. Future work should explore strategies for mitigating such biases, ensuring equal performance across diverse user populations.
E
Assets
Tab.
12
lists URLs and licenses for all the assets used in the paper.
Qualitative results
Fig. 7 showcases qualitative results produced by BAGLM on two test videos: one from the HT-Step dataset (task: Make Milanesa) and one from the CrossTask dataset (task: Make a Latte). For each video, plot the ground truth step boundaries and the step probabilities over video segments, as predicted by our method and by the off-the-shelf version the LMM. We also show keyframes at selected time points, indicated by arrows.
In the Make Milanesa video, the off-the-shelf LMM assigns high probability the step Dip the steak at the very beginning of the video, as the person is seen picking up the steak from plate, as shown by the corresponding keyframe. For same step, BAGLM initially assigns a low probability but increases it over time, consistent with the true presence of the step in the video. Around timestamp 00: 20, the off-the-shell LMM mistakenly assigns high probability to step Prep your egg mixture, likely due to the presence of the egg mixture in the video segment. In contrast, BAGLM consistently predicts Dip the steak at this point, leveraging prior context to avoid a false positive. At timestamp 00:30, although the ground truth still indicates Dip the steak is in progress, the step is already completed and BAGLM correctly detects this.
In the Make a Latte video, BAGLM effectively suppresses probabilities for steps that fall outside their actual temporal boundaries. Leveraging LMMs to compute VSG scores proves particularly beneficial in distinguishing visually similar actions such pour milk and pour espresso. Importantly, steps not present in the video are not falsely identified by BAGLM, showcasing its precision.
26

---

Melanesa
Gusker ingeddects
Poop your egg wisten
Gret tostug pea.
HoCLM AibraVLL..88
Gucker impediensa
Popp you upe mosure
Gralis frying
BES
Sarve up.
Pop indoran
Soea abon.
Video
Cround Taath
wakes
ponr repress
cadi
shalt
odd odher
22.--812
reprens
Nox of the sheve
sald calle
gaves coher
pour
H&CLM
soee undi
poer
muse
Buui
Video
Figure Qualitative results of BAGLM on test videos from HT-Step (Make Milanesa) and CrossTask (Make a Lutte). Ground truth step boundaries and predicted step probabilities per segment are shown for both BAGLM and the off-the-shelf LMM. Arrows point to the timestamps selected keyframes.
27


Reviews:
[{"review_id": "CXrtf1FxcR", "reviewer": "Reviewer_oKLx", "summary": "This paper introduces the first training-free online video step grounding method, powered by large multimodal models (LMMs). The authors start by demonstrating the capability of vanilla LMMs on the task of video step grounding, where they show fairly competitive results. To further improve the performance, Bayesian filtering is applied with two stages, i.e., prediction and update, which delicately model the inter-dependency among steps. Through the extensive experiments on three benchmarks, the effectiveness of the proposed method is validated.", "questions": "Please refer to weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n+ This paper showcases the successful exploitation of large multimodal models within the field of temporal step grounding.\n+ The idea of adopting Bayesian filtering is interesting, and it is seamlessly integrated within the framework. \n+ The proposed method largely surpasses the existing state of the arts, despite the more challenging online setting.\n\n## Weaknesses\n- Despite the effectiveness of the Bayesian filtering method, its practicality is quite limited, considering the performance degradation on Ego4D. Although the authors demonstrate that the performance could be further improved under the Oracle situations (Table 4), it does not seem to be the case for a real-world scenario. Obtaining more accurate transition matrices of the target dataset would lessen the advantage of the training-free setting.\n- The baselines under the online setting are missing. I would encourage the authors to build simple baselines or extend existing approaches to be capable of handling online scenarios. This would make the superiority of the proposed method clear.", "was_revised": true, "final_justification": "This paper introduces a method for incorporating LLMs into a streamlined pipeline for online video step grounding.\n\nDespite its strengths, I find some major concerns that need to be handled, including the practicality and the selection of online baselines.\n\nThrough the rebuttal, the authors addressed most of my concerns.\n\nAfter all, I would vote for a borderline accept.", "rebuttal": "We thank reviewer `oKLx` for acknowledging the **successful exploitation of large multimodal models** for temporal step grounding, the **interesting and seamless use of Bayesian filtering** within the framework, and that our method **largely surpasses state-of-the-art** methods despite the **more challenging online setting**.\n\n**1.1 Applicability in real-world scenarios:**\n\nOur method performs consistently well on CrossTask and HT-Step, which are as representative of real-world scenarios as Ego4D Goal-Step. This suggests that the method remains practical despite the lower performance on Ego4D. Each dataset used in our evaluation has its own challenges. CrossTask includes over 2,300 videos spanning 18 diverse everyday tasks beyond cooking, such as household and craft activities. These videos are shorter, averaging under 5 minutes, with concise action segments (\\~10.7 seconds). HT-Step focuses on cooking-related tasks with medium-length videos (~6 minutes) and an average segment duration of 16 seconds. Ego4D Goal-Step videos are significantly longer, with an average duration of about 28 minutes, step annotations of about 53 seconds per segment, and a focus on the cooking domain. These longer segments result in more generic step descriptions, which in turn make it more difficult to infer accurate dependencies between steps.\n\n**1.2 Availability of step transitions in practice:**\n\nThe Oracle analysis (Table 4) shows the potential performance gain when step dependencies are derived from the actual execution order. In some real-world scenarios, such transition matrices may already be available. For example, in industrial manufacturing, processes often follow standardized, predefined protocols. In these settings, step transition data can be reliably obtained (e.g., from existing documentation), making the Oracle setup not just a theoretical upper bound, but a realistic scenario. Our method is flexible and works in both cases, whether the dependency matrix is inferred or provided as ground truth.\n\n**2. Online baselines:**\n\nWe thank the reviewer for the constructive feedback. Following the reviewer's suggestions, we developed an online version of NaSVA [14], our best-performing offline competitor (see Table 1), which is based on the Transformer architecture. In this model, visual features extracted from each 1-second clip are encoded and treated as key-value pairs, while the textual features of steps serve as queries. The queries then attend to the visual features through cross-attention, and alignment scores are computed based on their similarity.\nThe original NaSVA is offline because its transformer encoder allows each video segment to attend to both past and future segments. To enable online processing, we introduce causal masking within the transformer encoder’s self-attention layers, restricting attention to only past segments and preventing access to future information.\nThe online version of NaSVA achieves an R@1 of 46.1 on HT-Step and 24.2 on Ego4D Goal-Step, representing a drop of 7% and 4.9% compared to its offline variant. These scores are also substantially lower than BaGLM (i.e., -10.2% on HT-Step and -18.6% on Ego4D Goal-Step). These results further highlight the effectiveness of our approach in the challenging online setting."}, {"review_id": "9YsJ7vIktj", "reviewer": "Reviewer_JSj1", "summary": "The paper addresses the task of Video Step Grounding (VSG), which aims to detect specific procedural steps described in text within corresponding video frames. Traditionally, this task relies heavily on offline models trained on annotated datasets, limiting real-time applicability and generalizability. To overcome these limitations, the authors propose Bayesian Grounding with Large Multimodal Models (BAGLM), a novel online and training-free approach. BAGLM leverages the zero-shot capabilities of recent large multimodal models (LMMs), particularly InternVL2.5-8B, and integrates temporal dependency modeling through Bayesian filtering. The dependencies between steps are inferred from a large language model (LLM), refining the predictions of LMMs by incorporating knowledge of past video frames. Experiments conducted on three benchmarks (HT-Step, CrossTask, and Ego4D Goal-Step) demonstrate that BAGLM significantly outperforms state-of-the-art training-based offline methods.", "questions": "Please see my questions in the weakness points.", "limitations": "The authors didn't discuss the limitations or potential negative societal impact of their work", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n- The paper presents the first study to address Video Step Grounding in an online, training-free setting, which reduces data annotation and training overhead and better aligns with real-world, real-time application scenarios.\n\n- Authors presented BAGLM, the first training-free, online approach that incorporates priors from past video frames into LMMs through Bayesian filtering, modeling the temporal dependencies across steps via LLM queries. \n\n- Extensive experiments show compelling results where the proposed BAGLM surpasses existing state-of-the-art methods, including those specifically trained or fine-tuned on respective datasets. Authors also provided detailed analyses to validate design choices, such as the influence of different LMMs/LLMs and the transition model, demonstrating their impacts on the model performance\n\nWeakness:\n\n- Authors show that the Zero-shot LMMs show better performance than previous offline training-based methods. I wonder if it’s because the LMMs, e.g., the InternVL2.5-8B, have seen such instructional videos of those evaluation benchmarks during pre-training? Authors should provide a more detailed analysis of why such Zero-Shot LMMs provide even better performance than training-based methods.\n\n- BAGLM’s performance strongly relies on the quality of the dependency matrix generated by the LLM. Although the authors tested two LLMs (GPT-4.1-mini and LLaMA-3.3-70B-Instruct), the robustness of this dependency inference across other domains and tasks remains unclear.", "was_revised": true, "final_justification": "My questions have been well addressed by the authors. I has been reflected in my current rating.", "rebuttal": "We thank reviewer `JSj1` for recognizing our work as the **first training-free, online approach** for Video Step Grounding, which **reduces annotation and training costs** and better aligns with real-world, real-time applications. We also appreciate the acknowledgment of our **extensive experiments showing that BaGLM outperforms state-of-the-art methods**, and the **analysis validating our key design choices**.\n\n**1. Zero-shot LMMs vs. training-based methods:**\n\nThe strong performance of zero-shot LMMs is not due to data contamination. As shown in Table 4 of the InternVL 2.5 report [6], HT-Step, CrossTask, and Ego4D Goal-Step are not included in its pre-training data mixture.\n\nTraining-based methods like NaSVA predict alignment score matrices, used to localize each step, by computing similarities between visual features of video segments and textual features of steps. These alignments are typically obtained using discriminatively pre-trained vision-language models such as CLIP. However, CLIP struggles with fine-grained alignment [a2,b2,c2,d2], particularly in instructional videos where different steps often involve similar objects or scenes. For example, the steps *\"tighten the screw\"* and *\"loosen the screw\"* involve the same objects in nearly identical visual settings. What distinguishes them is the direction of motion and the intent, subtle cues that CLIP-like models often miss due to their limited temporal modeling and contextual reasoning (L77-88).\n\nIn contrast, LMMs have stronger alignment capabilities thanks to the reasoning power of their underlying LLMs, which are trained on massive text corpora with complex question-answer tasks. Recent work [16] shows that computing the generative likelihood of answers to simple questions leads to more reliable alignment scores than traditional similarity metrics like CLIPScore, even outperforming models fine-tuned with human feedback.\nThis explains why zero-shot LMMs can outperform training-based methods, even without having seen the evaluation datasets during pre-training.\n\n**2. Dependency matrix robustness:**\n\nFollowing reviewer `JSj1`'s suggestion, we evaluate the robustness of the LLM-generated dependency matrices on the HT-Step dataset, which includes several (120) cooking-related tasks, each with 5 videos (600 videos total). For each task, we have a soft dependency matrix generated using either GPT-4.1-mini or LlaMA-3.3-70B-Instruct. To evaluate robustness, we measure the ratio of violated dependencies. We first threshold the soft matrices to obtain binary (hard) dependencies. A dependency $\\mathbf{D}_{i,j}$, indicating that step $a_j$ is a prerequisite for step $a_i$, is considered violated if $a_i$ appears before the first occurrence of $a_j$ in any of the videos for that task (i.e., the prerequisite should occur before the dependent step whenever both are present).\n\nThe tables below report the percentage of violated dependencies and the percentage of tasks with at least one violation for both LLaMA-3.3-70B-Instruct and GPT-4.1-mini. Both models perform reasonably well across different tasks in this domain, with roughly 10% of dependencies being violated, depending on the threshold value. In particular, LLaMA-3.3-70B-Instruct shows a flat violation ratio (~8%) across a wide range of thresholds, suggesting that its outputs are mostly binary (i.e., close to 0 or 1). In contrast, GPT-4.1-mini shows a smoother decline in violation rate as the threshold increases, indicating a more graded confidence distribution. Even at low thresholds, a portion of tasks (28% for LLaMA-3.3 and 26% for GPT-4.1-mini at threshold 0.1) show no violated dependencies, suggesting that the predicted matrices can approximate the oracle matrix.\n\n**LLaMA-3.3**\n\n| Threshold               | 0.00 | 0.10 | 0.20 | 0.30 | 0.40 | 0.50 | 0.60 | 0.70 | 0.80 | 0.90 | 1.00 |\n|-------------------------|------|------|------|------|------|------|------|------|------|------|------|\n| Dependency violated (%) | 26.1 | 8.6  | 8.4  | 8.4  | 8.4  | 8.4  | 8.4  | 8.3  | 8.3  | 8.1  | 8.1  |\n| Tasks w/ violation (%)  | 100.0 | 72.0 | 71.3 | 71.3 | 69.3 | 69.3 | 69.3 | 68.7 | 68.7 | 68.7 | 68.7 |\n\n**GPT-4.1-mini**\n\n| Threshold               | 0.00 | 0.10 | 0.20 | 0.30 | 0.40 | 0.50 | 0.60 | 0.70 | 0.80 | 0.90 | 1.00 |\n|-------------------------|------|------|------|------|------|------|------|------|------|------|------|\n| Dependency violated (%) | 26.1 | 15.4 | 12.5 | 11.6 | 10.8 | 10.2 | 9.6  | 8.9  | 8.2  | 7.6  | 0.0  |\n| Tasks w/ violation (%)  | 100.0 | 74.0 | 65.3 | 64.0 | 59.3 | 54.0 | 51.3 | 48.0 | 44.7 | 42.0 | 0.0  |\n\n\n**References**\n\n[a2] Bagad, Piyush, Makarand Tapaswi, and Cees GM Snoek. \"Test of time: Instilling video-language models with a sense of time.\" CVPR 2023.\n\n[b2] Momeni, Liliane, et al. \"Verbs in action: Improving verb understanding in video-language models.\" ICCV 2023.\n\n[c2] Wang, Zhenhailong, et al. \"Paxion: Patching action knowledge in video-language foundation models.\" NeurIPS 2023.\n\n[d2] Yuksekgonul, Mert, et al. \"When and why vision-language models behave like bags-of-words, and what to do about it?.\" ICLR 2023."}, {"review_id": "SvrR7F0zb7", "reviewer": "Reviewer_WbTE", "summary": "The paper proposes an online, training-free approach to step grounding in procedural videos by leveraging the inherent procedural and task knowledge of LLMs, combined with the visual understanding capabilities of LMMs. The authors first demonstrate that LMMs like InternVid-2.5-8B can outperform state-of-the-art VSG methods out of the box. This motivates further improvement through Bayesian filtering and the incorporation of language and progress priors, such as step transition probabilities, step readiness, and step validity. The resulting method, BAGLM, is fully online and training-free, and outperforms existing state-of-the-art offline VSG methods across three datasets.", "questions": "I expect a response to the following questions in the rebuttal:\n\n- **Q1.** Response to W2.\n\n- **Q2.** While using LMMs for both offline and online VSG, all the steps are provided as multiple-choice options in the prompt. For datasets like HT-Step and Ego4D Goal-Step, which contain over 500 steps in the form of sentence descriptions, doesn’t the performance degrade due to long-context limitations of the LMM? I could not find the actual LMM prompts with the full list of steps in the main paper or supplementary. Is there any filtering process used to reduce the number of tokens required to represent each step? Otherwise, it would be helpful to include an analysis showing how performance varies as the number of candidate steps increases.\n\n- **Q3.** A small clarification: is the ground-truth start and end information used at all in the original BAGLM setup in Table 1, or is it only used in the Oracle setting reported in Table 4?\n\n- **Q4.** This is low priority, but I would appreciate it if the authors could consider W1 and benchmark against any existing streaming VideoLLMs.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\n**S1.** The paper is very well-written, and the motivations are clearly articulated.  \n**S2.** The method is simple and elegant, demonstrating that VSG can be effectively addressed using LMMs. This opens up a promising, training-free, and resource-efficient alternative to existing SOTA approaches.  \n**S3.** The experimental results are strong, and the ablations are thorough. The accompanying discussion is particularly well-explained and contributes meaningfully to understanding the factors behind performance gains.\n\n**Weaknesses:** The paper does not have any major flaws, but there are a few limitations related to the online setup that prevent me from rating it higher.\n\n**W1.** In L32, the authors mention “operating on a streaming video.” While the method does function in an online manner, there may be practical latency concerns, as video is processed in fixed 2-second chunks. Streaming VideoLLMs have recently gained attention, and I would like to point the authors to models such as Dispider [a] and LiveCC [b], which operate more efficiently in streaming, real-time settings. Replacing the offline LMMs with a truly streaming VideoLLM and demonstrating performance in a low-latency, real-time setup would significantly strengthen the paper and likely raise my rating to a strong accept.\n\n**W2.** Another aspect missing from the paper is the analysis of step prediction performance across different positions in the sequence. Specifically, in a video with steps [$s_1, s_2, ..., s_n$], one would expect early steps (e.g., $s_1$) to be more difficult to predict due to limited context, while later steps (e.g., $s_n$) may benefit from accumulated temporal evidence. It would be helpful to understand whether this trend holds in practice. Additionally, if early predictions are incorrect, how does the error propagate or accumulate over the rest of the video?\n\n[a] R. Qian et al., Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction, CVPR 2025\n\n[b] J. Chen et al., LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale, CVPR 2025", "was_revised": true, "final_justification": "I thank the author for their detailed responses to be queries. I am satisfied with the rebuttal and am maintaining my rating of Accept. \n\nI would recommend the author to include the ablations concerning Q1 and Q2 to the revised supplementary.", "rebuttal": "We appreciate reviewer `WbTE`'s positive feedback on the **clarity and motivation** of our paper. We are glad that the **simplicity and elegance** of our method, leveraging **LMMs for Video Step Grounding**, is recognized as a **promising, training-free, and resource-efficient alternative** to existing training-based approaches. We also thank the reviewer for highlighting the **strength of our experimental results**, **thorough ablations**, and the **well-explained discussion**.\n\n**1.1 Step prediction performance by temporal position:**\n\nWe thank the reviewer for this insightful suggestion. To analyze performance across step positions, we compute R@1 on HT-Step by grouping steps based on their normalized position within each video. For every step, we normalize its occurrence to the \\[0, 1\\] range. We then divide these positions into five bins and compute R@1 for each.\n\nThe table below shows R@1 performance across these bins, from early (bin 0) to late (bin 4) steps. The off-the-shelf model (InternVL2.5-8b), which does not use past information, maintains relatively stable performance across bins. It performs best on early steps (bin 0) and worst on the final ones (bin 4), suggesting that initial steps are more semantically distinctive, while later ones are harder to recognize.  In contrast, BaGLM shows an upward trend from early to mid-range bins, likely due to its use of accumulated temporal context. For example, its R@1 increases by +1.0% from bin 0 to bin 1, +1.7% from bin 1 to bin 2, and +0.7% from bin 2 to bin 3. The only exception is the final bin, likely due to limitations inherited from the base model (i.e., InternVL2.5-8b performs worst in the final part of the video) as well as error propagation.\n\n| Model            | Bin 0 | Bin 1 | Bin 2 | Bin 3 | Bin 4 |\n|------------------|--------|--------|--------|--------|--------|\n| InternVL2.5-8b   | 55.1%  | 50.7%  | 53.8%  | 50.7%  | 49.0%  |\n| BaGLM            | 55.3%  | 56.3%  | 58.0%  | 58.7%  | 54.5%  |\n\n**1.2 Error propagation:**\n\nWe analyze error propagation by measuring R@1 performance on HT-Step across five temporal bins for BaGLM. To gain insights into the source of mistakes, we show the results of BaGLM variants augmented with oracle signals. BaGLM with Oracle Progress (but predicted dependencies) outperforms BaGLM across all bins, highlighting the importance of an accurate progress estimate for stable performance. Instead, BaGLM with Oracle Dependency Matrix (but predicted progress) does not show the same upward trend as BaGLM. This suggests that when the progress contains errors, the oracle dependency matrix amplifies those, ruling out valid steps based on flawed readiness/validity. \nWhen both oracle signals are provided, performance improves across the mid-range bins (bins 1, 2, and 3), showing that accurate progress and dependencies together help the model select the correct step. The two exceptions are the initial and final steps, where performance drops likely due to noise in the estimation of the belief in the initial states for the former, and the limits of the base model for the latter.\n\n| Model                          | Bin 0 | Bin 1 | Bin 2 | Bin 3 | Bin 4 |\n|--------------------------------|--------|--------|--------|--------|--------|\n| BaGLM                          | 55.3%  | 56.3%  | 58.0%  | 58.7%  | 54.5%  |\n| BaGLM w/ Oracle Progress       | 58.6%  | 60.0%  | 59.3%  | 58.7%  | 57.8%  |\n| BaGLM w/ Oracle Dep. Matrix    | 56.1%  | 54.3%  | 59.0%  | 54.4%  | 49.0%  |\n| BaGLM w/ Oracle                | 56.3%  | 65.2%  | 66.0%  | 71.6%  | 53.7%  |\n\n**2.1 Steps preprocessing:**\n\nFor HT-Step and CrossTask, we follow the standard evaluation protocol [4,14] by providing, for each video, the full set of steps associated with its corresponding task as multiple-choice options in the prompt. On average, each task includes about 10 steps in HT-Step and 7.5 in CrossTask. As reviewer `WbTE` correctly noted, Ego4D Goal-Step includes a larger number of annotated segments, which are divided into steps and substeps. To reduce the number of candidate steps in our approach, we only use the step-level descriptions (i.e., no substeps). Additionally, many of these step descriptions differ only in minor grammatical ways (e.g., capitalization). To address this, we apply a simple preprocessing step using the spaCy Python library: we lowercase the text, lemmatize it (while preserving plural nouns and verbal adjectives), and normalize whitespace and punctuation. After this preprocessing, the average number of steps per video is reduced to 17.\n\n**2.2  Analysis on the number of candidate steps:**\n\nWe analyze how performance changes with the number of candidate steps in the HT-Step dataset by grouping videos based on the number of steps in their respective tasks. For each group, we compute the average R@1 using the off-the-shelf InternVL2.5-8B model. We summarize the results in the table below. The values in the table depict a non-monotonic trend. Except for tasks with very few steps (e.g., fewer than 5), there is no clear decline in performance as the number of steps increases. This suggests that InternVL2.5-8B is capable of handling prompts with a large number of candidate steps. Note that for some bins, the number of matching videos is small, which can lead to large fluctuations in the reported measurements (e.g., from 24.1% to 48.1% for 19 to 20 steps).\n\n| Candidate Steps | 3    | 4    | 5    | 6    | 7    | 8    | 9    | 10   | 11   | 12   | 13   | 14   | 15   | 16   | 17   | 18   | 19   | 20   | 22   | 23   | 25   | 26   | 28   |\n|---------------------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|\n| Mean R@1 (%)        | 100  | 72.2 | 42.1 | 52.9 | 54.7 | 44.4 | 49.0 | 49.4 | 62.4 | 50.9 | 59.1 | 45.0 | 53.9 | 65.7 | 50.5 | 50.0 | 24.1 | 48.1 | 44.4 | 70.6 | 43.8 | 53.8 | 42.9 |\n\n**3. Ground-truth start and end information:**\n\nThe ground-truth start and end information are only used in the oracle analysis reported in Table 4. All other experiments are performed without access to any ground-truth timestamps.\n\n**4. Streaming VideoLLMs:**\n\nWe thank the reviewer for the suggestion! We benchmarked Dispider [a3] and LiveCC [b3] using the authors' official implementations on the HT-Step dataset, following the temporal video grounding prompt from ETBench [c3] (also used in Dispider's evaluation). The prompt is:\n\n> You are given a video about daily activities. Watch the video carefully and find a visual event described by the sentence: '{step}'. The format of your response should be: 'The event happens in \\<start time\\> - \\<end time\\>'. You must represent start and end times in seconds. For example: 'The event happens in 10.2 - 12.8 seconds'.\n\nUnfortunately, both models either failed to produce outputs or simply repeated the example from the prompt. We believe it is worthwhile to further investigate how these models can be adapted to online video step grounding. However, this was not feasible within the limited time of the rebuttal period.\n    \nIt is worth noting that, as we report in Appendix Section C (*Inference Speed Analysis*), our method achieves low latency even with a traditional LMM. On HT-Step, the total processing time per 2-second segment, including video decoding, VSG scoring, and progress estimation, remains under 2 seconds for tasks with up to 24 steps (more than the average step count of any dataset we evaluate on), using a single H100 GPU with 64GB of memory. This suggests that BaGLM can operate in real time. We can further reduce latency by parallelizing step progress estimation, which requires as many independent forward passes as the number of steps in a task, either by batching multiple inferences on a single GPU (if there is sufficient memory) or by distributing them across multiple GPUs. In the optimal scenario, latency can be reduced approximately linearly with the number of parallel executions.\n\n**References:**\n\n[a3] Qian, Rui, et al. \"Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reaction.\" CVPR 2025.\n\n[b3] Chen, Joya, et al. \"Livecc: Learning video llm with streaming speech transcription at scale.\" CVPR 2025.\n\n[c3] Liu, Ye, et al. \"Et bench: Towards open-ended event-level video-language understanding.\" NeurIPS 2024."}, {"review_id": "H42hp1NElY", "reviewer": "Reviewer_pBnT", "summary": "This paper tackles the problem of Video Step Grounding (VSG)—identifying which instructional steps are being performed in a video. \nThe key novelties of the paper are: 1) using a LMM for step recognition, which can operate online, 2) using a bayesian framework to incorporate past predictions into the decision making. The paper shows that a LMM alone achieves SoTA results on this problem, and the bayesian framework further improves the performance. The experiments show performance gains on three public benchmarks: HT-Step, CrossTask, and Ego4D Goal-Step.", "questions": "* It looks like step validity doesn’t add much value on top of step readiness. Would it make sense to only use one to have a simpler method?\n*  I wonder if the authors thought of estimating transition probabilities differently. Instead of asking an LLM for the probability of co-occurence of two steps, an interesting approach would be to sample task step sequences from an LLM and calculate the probability empirically.", "limitations": "Nothing the authors do not mention in the paper", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* A very timely approach - showing that one can use the current SoTA LMMs to solve classical computer vision problems zero-shot, without task-specific training, is an important realization for the community\n* I like the bayesian approach to incorporating previous predictions into the current decision making, as the problem is indeed sequential.\n* Using an LLM to elicit the transition probabilities is a really good idea, especially in the cooking domain as it has seen so many text recipes. \n\nWeaknesses:\n* While the method is advertised as online, it is important to show that it is indeed practical to run it in the online fashion (i.e., near real-time), but the runtime analysis is missing in the paper. If the method is not able to run in the real-time, the online assumption is not very useful. If that so, one could instead estimate the probabilities of steps for the whole video sequence, and than run classical offline algorithms (that could also be bayesian). As it stands, the pitch about the method being online isn't helping the paper.\n* Is this really reasonable to ask an LLM for a probability score? It’s unlikely the LLMs \"know\" the actual probability, thus making this a threat of validity.", "was_revised": true, "final_justification": "I thank the authors for providing their response. All of that answers my questions. I recommend the paper to be accepted.", "rebuttal": "We thank reviewer `pBnT` for recognizing our **timely use of zero-shot LMMs** to solve classical computer vision tasks. We are glad the **Bayesian integration of past predictions into current decisions** is well received. We also appreciate the positive feedback on **using an LLM to estimate transition probabilities**.\n\n**1. Runtime analysis:**\n    \nSection C *\"Inference Speed Analysis\"* in our Appendix reports the average processing time per video segment for computing VSG scores, estimating step progress, and video decoding/preprocessing on the HT-Step dataset. Videos are grouped by the number of steps in their tasks, and the times were measured on a single H100 GPU with 64GB of memory. For tasks with up to 24 steps, which exceeds the average number of steps in any of the datasets we used, the total inference time remains under 2 seconds per segment. This suggests that BaGLM can process 2-second video segments in real-time. Importantly, the time needed to estimate step progress can be reduced linearly by either batching multiple inferences on a single GPU (if there is sufficient memory) or by distributing them across multiple GPUs.\n\n**2. Validity of LLM-estimated probabilities:**\n\nWe would like to clarify a possible misunderstanding: we do not ask the LLM/LMM for a direct probability score. Instead, we prompt it to generate an answer by choosing from a multiple-choice list. We then take the logits output by the LLM for the first token generated after the prompt, convert these logits into probabilities over the vocabulary using a softmax, and select the probability corresponding to the first token of each choice. For Video-Step Grounding, the choices correspond to the full set of steps of the task associated with the video (see prompt $\\pi_{\\texttt{VSG}}$  in the Appendix). For step progress estimation, the choices are a list of discrete progress values (see prompt $\\pi_{\\texttt{prog}}$ in the Appendix). For step dependencies, the choices are *Yes* or *No* (see prompt $\\pi_{\\texttt{prereq}}$ in the Appendix).\n\n**3. Step validity:**\n\nWhile step validity alone brings only a small improvement (i.e., +0.7% on Ego4D Goal-Step, as shown in Table 2 of the main manuscript), the best performance is achieved by combining it with step readiness. This combination leads to gains of 1.9%, 2.5%, and 1.2% across datasets, representing additional improvements of +0.3%, +0.6%, and +0.6% over using step readiness alone. Importantly, step validity adds negligible computational overhead compared to the variant that does not use it, and proves especially useful in scenarios where certain steps should not be repeated once completed. For example, without step validity, a model might repeatedly predict *\"turn on the stove\"* after it has already been activated. By enforcing validity, the model can avoid such errors, producing more reliable predictions.\n\n**4. Alternative transition probability:**\n\nSampling step sequences from an LLM and estimating transition probabilities empirically can reveal how often steps appear together, but our goal is different: we aim to model causal dependencies, not just co-occurrence. In procedural tasks, it is important to know what must happen before something else. Rather than asking the LLM which steps co-occur, we ask whether one step is a prerequisite of another. This allows us to build a dependency matrix that captures step ordering constraints, which we then use to define the transition model (L197-205). \n\nTo see why this matters, consider the steps: (1) *attach the tabletop*, (2) *attach the table legs*, and (3) *tighten the screws*. A sampling-based approach might generate plausible sequences, for example:\n- *attach the tabletop &rarr; tighten the screws &rarr; attach the table legs &rarr; tighten the screws*\n- *attach the table legs &rarr; tighten the screws &rarr; attach the tabletop &rarr; tighten the screws*\n    \nFrom these samples, a co-occurrence-based method might incorrectly infer that *attach the tabletop* and *attach the table legs* depend on *tighten the screws*, simply because they appear after it in the sequence. In reality, both are prerequisites for *tighten the screws*, which should only happen after either the tabletop or the legs are attached. Only a method that explicitly models dependencies, not just co-occurrence, can capture this causal structure."}]

Author rebuttal:

Meta-review: This paper presents a training-free method for step-wise grounding of the individual steps in a given procedural video.   It incorporates Bayesian filtering together with LLMs to model the step-wise transitions in the procedure.  Performance is excellent, especially when compared to supervised offline methods.

All four reviewers support this paper and think it would be a nice contribution to the NeurIPS program, and the AC agrees.  The extensive discussion between the authors and the four reviewers highlights various areas of confusion and missing considerations in the analysis.  The authors are requested to incorporate these points when revising their manuscript for the final version.
Decision: Accept (poster)