Title: Training-free Online Video Step Grounding
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video step grounding

training-free

large multimodal models
Abstract: Given a task and a set of steps composing it, Video Step Grounding (VSG) aims to detect which steps are performed in a video. Standard approaches for this task require a labeled training set (e.g., with step-level annotations or narrations), which may be costly to collect. Moreover, they process the full video offline, limiting their applications for scenarios requiring online decisions. Thus, in this work, we explore how to perform VSG online and without training. We achieve this by exploiting the zero-shot capabilities of recent Large Multimodal Models (LMMs). In particular, we use LMMs to predict the step associated with a restricted set of frames, without access to the whole video. We show that this online strategy without task-specific tuning outperforms offline and training-based models. Motivated by this finding, we develop Bayesian Grounding with Large Multimodal Models (BaGLM), further injecting knowledge of past frames into the LMM-based predictions. BaGLM exploits Bayesian filtering principles, modeling step transitions via (i) a dependency matrix extracted through large language models and (ii) an estimation of step progress. Experiments on three datasets show superior performance of BaGLM over state-of-the-art training-based offline methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of Video Step Grounding (VSG)—identifying which instructional steps are being performed in a video. \nThe key novelties of the paper are: 1) using a LMM for step recognition, which can operate online, 2) using a bayesian framework to incorporate past predictions into the decision making. The paper shows that a LMM alone achieves SoTA results on this problem, and the bayesian framework further improves the performance. The experiments show performance gains on three public benchmarks: HT-Step, CrossTask, and Ego4D Goal-Step.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* A very timely approach - showing that one can use the current SoTA LMMs to solve classical computer vision problems zero-shot, without task-specific training, is an important realization for the community\n* I like the bayesian approach to incorporating previous predictions into the current decision making, as the problem is indeed sequential.\n* Using an LLM to elicit the transition probabilities is a really good idea, especially in the cooking domain as it has seen so many text recipes. \n\nWeaknesses:\n* While the method is advertised as online, it is important to show that it is indeed practical to run it in the online fashion (i.e., near real-time), but the runtime analysis is missing in the paper. If the method is not able to run in the real-time, the online assumption is not very useful. If that so, one could instead estimate the probabilities of steps for the whole video sequence, and than run classical offline algorithms (that could also be bayesian). As it stands, the pitch about the method being online isn't helping the paper.\n* Is this really reasonable to ask an LLM for a probability score? It’s unlikely the LLMs \"know\" the actual probability, thus making this a threat of validity.\n\nQUESTIONS:\n* It looks like step validity doesn’t add much value on top of step readiness. Would it make sense to only use one to have a simpler method?\n*  I wonder if the authors thought of estimating transition probabilities differently. Instead of asking an LLM for the probability of co-occurence of two steps, an interesting approach would be to sample task step sequences from an LLM and calculate the probability empirically.\n\nLIMITATIONS:\nNothing the authors do not mention in the paper", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an online, training-free approach to step grounding in procedural videos by leveraging the inherent procedural and task knowledge of LLMs, combined with the visual understanding capabilities of LMMs. The authors first demonstrate that LMMs like InternVid-2.5-8B can outperform state-of-the-art VSG methods out of the box. This motivates further improvement through Bayesian filtering and the incorporation of language and progress priors, such as step transition probabilities, step readiness, and step validity. The resulting method, BAGLM, is fully online and training-free, and outperforms existing state-of-the-art offline VSG methods across three datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n**S1.** The paper is very well-written, and the motivations are clearly articulated.  \n**S2.** The method is simple and elegant, demonstrating that VSG can be effectively addressed using LMMs. This opens up a promising, training-free, and resource-efficient alternative to existing SOTA approaches.  \n**S3.** The experimental results are strong, and the ablations are thorough. The accompanying discussion is particularly well-explained and contributes meaningfully to understanding the factors behind performance gains.\n\n**Weaknesses:** The paper does not have any major flaws, but there are a few limitations related to the online setup that prevent me from rating it higher.\n\n**W1.** In L32, the authors mention “operating on a streaming video.” While the method does function in an online manner, there may be practical latency concerns, as video is processed in fixed 2-second chunks. Streaming VideoLLMs have recently gained attention, and I would like to point the authors to models such as Dispider [a] and LiveCC [b], which operate more efficiently in streaming, real-time settings. Replacing the offline LMMs with a truly streaming VideoLLM and demonstrating performance in a low-latency, real-time setup would significantly strengthen the paper and likely raise my rating to a strong accept.\n\n**W2.** Another aspect missing from the paper is the analysis of step prediction performance across different positions in the sequence. Specifically, in a video with steps [$s_1, s_2, ..., s_n$], one would expect early steps (e.g., $s_1$) to be more difficult to predict due to limited context, while later steps (e.g., $s_n$) may benefit from accumulated temporal evidence. It would be helpful to understand whether this trend holds in practice. Additionally, if early predictions are incorrect, how does the error propagate or accumulate over the rest of the video?\n\n[a] R. Qian et al., Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction, CVPR 2025\n\n[b] J. Chen et al., LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale, CVPR 2025\n\nQUESTIONS:\nI expect a response to the following questions in the rebuttal:\n\n- **Q1.** Response to W2.\n\n- **Q2.** While using LMMs for both offline and online VSG, all the steps are provided as multiple-choice options in the prompt. For datasets like HT-Step and Ego4D Goal-Step, which contain over 500 steps in the form of sentence descriptions, doesn’t the performance degrade due to long-context limitations of the LMM? I could not find the actual LMM prompts with the full list of steps in the main paper or supplementary. Is there any filtering process used to reduce the number of tokens required to represent each step? Otherwise, it would be helpful to include an analysis showing how performance varies as the number of candidate steps increases.\n\n- **Q3.** A small clarification: is the ground-truth start and end information used at all in the original BAGLM setup in Table 1, or is it only used in the Oracle setting reported in Table 4?\n\n- **Q4.** This is low priority, but I would appreciate it if the authors could consider W1 and benchmark against any existing streaming VideoLLMs.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the task of Video Step Grounding (VSG), which aims to detect specific procedural steps described in text within corresponding video frames. Traditionally, this task relies heavily on offline models trained on annotated datasets, limiting real-time applicability and generalizability. To overcome these limitations, the authors propose Bayesian Grounding with Large Multimodal Models (BAGLM), a novel online and training-free approach. BAGLM leverages the zero-shot capabilities of recent large multimodal models (LMMs), particularly InternVL2.5-8B, and integrates temporal dependency modeling through Bayesian filtering. The dependencies between steps are inferred from a large language model (LLM), refining the predictions of LMMs by incorporating knowledge of past video frames. Experiments conducted on three benchmarks (HT-Step, CrossTask, and Ego4D Goal-Step) demonstrate that BAGLM significantly outperforms state-of-the-art training-based offline methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n- The paper presents the first study to address Video Step Grounding in an online, training-free setting, which reduces data annotation and training overhead and better aligns with real-world, real-time application scenarios.\n\n- Authors presented BAGLM, the first training-free, online approach that incorporates priors from past video frames into LMMs through Bayesian filtering, modeling the temporal dependencies across steps via LLM queries. \n\n- Extensive experiments show compelling results where the proposed BAGLM surpasses existing state-of-the-art methods, including those specifically trained or fine-tuned on respective datasets. Authors also provided detailed analyses to validate design choices, such as the influence of different LMMs/LLMs and the transition model, demonstrating their impacts on the model performance\n\nWeakness:\n\n- Authors show that the Zero-shot LMMs show better performance than previous offline training-based methods. I wonder if it’s because the LMMs, e.g., the InternVL2.5-8B, have seen such instructional videos of those evaluation benchmarks during pre-training? Authors should provide a more detailed analysis of why such Zero-Shot LMMs provide even better performance than training-based methods.\n\n- BAGLM’s performance strongly relies on the quality of the dependency matrix generated by the LLM. Although the authors tested two LLMs (GPT-4.1-mini and LLaMA-3.3-70B-Instruct), the robustness of this dependency inference across other domains and tasks remains unclear.\n\nQUESTIONS:\nPlease see my questions in the weakness points.\n\nLIMITATIONS:\nThe authors didn't discuss the limitations or potential negative societal impact of their work", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

comments: SUMMARY:
This paper introduces the first training-free online video step grounding method, powered by large multimodal models (LMMs). The authors start by demonstrating the capability of vanilla LMMs on the task of video step grounding, where they show fairly competitive results. To further improve the performance, Bayesian filtering is applied with two stages, i.e., prediction and update, which delicately model the inter-dependency among steps. Through the extensive experiments on three benchmarks, the effectiveness of the proposed method is validated.

STRENGTHS AND WEAKNESSES:
## Strengths
+ This paper showcases the successful exploitation of large multimodal models within the field of temporal step grounding.
+ The idea of adopting Bayesian filtering is interesting, and it is seamlessly integrated within the framework. 
+ The proposed method largely surpasses the existing state of the arts, despite the more challenging online setting.

## Weaknesses
- Despite the effectiveness of the Bayesian filtering method, its practicality is quite limited, considering the performance degradation on Ego4D. Although the authors demonstrate that the performance could be further improved under the Oracle situations (Table 4), it does not seem to be the case for a real-world scenario. Obtaining more accurate transition matrices of the target dataset would lessen the advantage of the training-free setting.
- The baselines under the online setting are missing. I would encourage the authors to build simple baselines or extend existing approaches to be capable of handling online scenarios. This would make the superiority of the proposed method clear.

QUESTIONS:
Please refer to weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-08