Video Action Differencing
Authors: James Burgess
Xiaohan Wang
Yuhui Zhang
Anita Rau
Alejandro Lozano
Lisa Dunlap
Trevor Darrell
Serena Yeung-Levy
Venue: ICLR 2025 Poster (2025)
Keywords: Video
Actions
Differencing
Zero-shot
benchmark
multimodal
lmm
llm
Primary area: datasets and benchmarks
TL;DR: A new task and benchmark for comparing how an action is performed between two videos, with a zero-shot method

Abstract: How do two individuals differ when performing the same action? In this work, we introduce Video Action Differencing (VidDiff), the novel task of identifying subtle differences between videos of the same action, which has numerous applications, such as coaching and skill learning. To enable development on this new task, we first create VidDiffBench, a benchmark dataset containing 549 video pairs, with human annotations of 4,469 fine-grained action differences and 2,075 timestamps indicating where these differences occur. Our experiments demonstrate that VidDiffBench poses a significant challenge for state-of-the-art large multimodal models (LMMs), such as GPT-4o and Qwen2-VL. By analyzing the failure cases of LMMs on VidDiffBench, we highlight two key challenges for this task: localizing relevant sub-actions over two videos and fine-grained frame comparison. To overcome these, we propose the VidDiff method, an agentic workflow that breaks the task into three stages: action difference proposal, keyframe localization, and frame differencing, each stage utilizing specialized foundation models. To encourage future research in this new task, we release the benchmark and code.

Published as a conference paper at ICLR 2025
VIDEO ACTION DIFFERENCING
James Burgess', Xiaohan Wang', Yuhui Zhang', Anita Rau', Alejandro Lozano', Lisa Dunlap', Trevor Darrell', Serena Yeung-Levy' Stanford, UC Berkeley
ABSTRACT
How do two individuals differ when performing the same action? In this work, we introduce Video Action Differencing (VidDiff), the novel task of identifying subtle differences between videos of the same action, which has many applications, such as coaching and skill learning. To enable development on this new task, we first create VidDiffBench, benchmark dataset containing 549 video pairs, with human annotations of 4,469 fine-gra fine-grained action differences and 2,075 localization times- tamps indicating where these differences occur. Our experiments demonstrate that VidDiffBench poses a significant challenge for state-of-the-ar large multimodal models (LMMss, such as GPT-4o and Qwen2-VI. VI. By analyzing failure cases of LMMs VidDiffBench, we highlight two key challenges for this task: localiz- ing relevant sub-actions over two videos and fine-grained frame comparison. To overcome these, propose the VidDiff method, agentic workflow that breaks the task into three stages: action difference proposal, keyframe localization, and frame differencing, each stage utilizing specialized foundation models. To en- courage future research in this new task, we release the benchmark and code_
INTRODUCTION
The ability to compare two videos of the same action and discern their detailed differences plays critical role in wide variety of applications. For instance, in fitness coaching, novice learning to perform a barbell squat typically watches instructional videos and then compares their actions in recorded video to identify discrepancies between their movements and those of an expert. In medical training, junior surgeons compare videos of themselves performing surgical procedures with reference videos from experts to identify errors and improve surgical skills.
There are two critical obstacles. First precise localization of sub-actions: finding differences requires finding the sub-action frames where the differences might occur, and aligning those frames between the two videos. Second is Fine-graina understanding: the ability to perceive subtle visual differences in motions.
Current research on video difference understanding largely emphasizes feature visualization (Bal- akrishnan et al.] (2035) coarse coarse-grained comparisons between different actions or interacting ob- jects (Nagarajan & Torresani] 2024). However, many real-world applications demand fine-grained comparisons between videos of the same action, a challenge that has received little attention.
We introduce a new task, Video Action Differencing (VidDiff). Given two videos of the same action, (UA, UB), along with a description of the action, the task is to generate two sets of statements: one that is more true for VA and another for up. For example, in video pair featuring an expert and novice performing barbell squat, key differences might include "knees caving in more in video A" and "the squat is deeper in video B" (Figure D]. Since generating the initial difference candidates relies heavily on language capabilities, we also introduce a simpler 'closed" setting that focuses on video analysis. In this setting, the target difference strings are provided, and the task is to predict whether each applies more to video A or B.
To facilitate research in this new direction, we present VidDifiBench, comprehensive benchmark designed for video action differencing. VidDiflBench contains 549 video pairs drawn from domains
"Benchmark; https::/hugginggface. http://iww.......taaaaa////////a
Project page: http://jmhhh.giihuh.oom

---

Published as a conference paper at ICLR 2025
"Deadlift"
SSurgical suturing skills"
A
B
B
"Fet stance wider B" speed B"
"Instrument applies tissue
"Grasper positioned quickly A'
"Soccer penalty kick"
"Competitive diving from 10m"
2,8m
0.0c
2.75
A
"More hip rotation 8" "Harder kick B"
"B is more perpendicular to the water water entry"
The Video Action Differencing task and benchmark (VidDiffBench). Given pair of videos and an action, the task is to generate a list of differences as natural language descriptions. Our VidDiffBench consists of annotated differences across diverse domains, where the differences are relevant to human skill learning. The first row emphasizes the first key challenge: localization of sub-actions between segments of the video for comparison. The second highlights the second key challenge: fine grained image understanding of actions in order to perform comparison.
Figure 1:
that require expert feedback, such as fitness, sports, music, and surgery. The videos are annotated with 4,469 fine-grained differences (~88 per video pair), along with 2,075 timestamp annotations that identify where these differences occur. To ensure the annotated differences relevant to skill learning, we create a taxonomy of action differences that leverages domain expertise. This makes VidDiffBench the first large-scale dataset dedicated to video action differencing.
In addition to introducing a new task and benchmark, we propose the VidDiff Method, an agentic workflow (Anthropic. 2025) for solving VidDiff in zero-shot. The method incorporates large lan- guage models (LLMs) to propose differences, localizes relevant frames using contrastive language- image models (CLIP), and compares frames for differences using vision-language models (VLMs). The key idea is that localizing specific video segments where differences occur enables more effec- tive visual comparison with VLMs. We further benchmark both open-source (Qwen2-VL, LLaVA- Video) and proprietary (GPT-4o, Gemini-1.5 pro, Claude 3.5 Sonnet) large multimodal models (LMMs) on VidDiffBench. Our results demonstrate that VidDiff is a very challenging task for zero shot models, while the structured approach in the VidDiff Method enhances video comparison.
2 RELATED WORK
Skilled Action Understanding in Videos Video comparison has many potential applications, and benchmark focuses on the specific goal of natural language feedback in skill learning. Most of the video action comparison papers from this section's 's first paragraph are systems for skill feedback, showing that skill feedback is well-motivated. Many works give feedback by classifying coarse mo- tion errors, or by visualizing motions, with applications in yoga (Zhao et al. 2022] Thoutam et al.)
2022 Chen af. 2018 Dittakavi et al.) 2022; Chen & Yang 2020; Xie et al. 2019), physical ther- apy (Velloso et al. 2013)) weightlifting (Prrmr et al. 2022 Ogata et al. [2119) and general fitness (Fieraru et al.J Ashwin et al. 2023). The feedback tends to be coarse- coarse-grained. In contrast, our task focuses on open natural language feedback, and identifying fine-grained feedback. Recently, the Ego-Exo4D dataset (Grauman et al. 2023) provides videos with expert commentary on skilled actions, which is promising for developing instructional feedback systems. This, along with exist-
Ashwin
ing works that give language feedback (Li et al.) 2024b) Fieraru et al.) [002]] Parmar et al. 2022] Velloso al. 2013), support our claim that language is good medium for providing skill feed-
2

---

Published as a conference paper at ICLR 2025
back to humans. Zooming out from skills feedback, skilled action understanding which includes foundational capabilities for feedback systems - has attracted enormous interest. For example. in sports, music, dance, and surgery, prior works have tackled action recognition (Verma et al.] 2020
Shahroudy et al. 2016 Soomro et al. 2012 ZZang et al. 2013; Wang & Zemel. 2016] et al. 2021]; spatial and temporal action localization segmentation (Shao et 20.0 Liu et al. 2022] Li
et al.
2021b Zhang et al.) 2023b Ibrahim et al. 2016; Garrow et al. [202]] Li et al. [202] 1b Aklilu
al.
2024);
human pose and motion estimation reconstruction (Cai al. [022 Tang et al. 2023b
Wang
et al.
2023 Andriluka et al.
2014; Li et al. 202 Fieraru et al. | et al. [0022] Bera
al.
2023
Liu et al. 2024] Grauman et al., 2023); and hand and tool pose estimation (Doosti)
2019
Johnson et al.; 2020 2016 Gao et al. 2014] Grauman et al., (023). Skilled action domains
also tackle higher level reasoning tasks like question answering (Li et 2024a), and action quality
assessment (Pirsiavash et al. 2014 Parmar & Tran Morris 2017).
Visual Difference Understanding Only a few prior works have considered video comparison in actions. They mostly emphasize skill learning in similar categories to our benchmark, but their methods tend to tackle single domains. One approach visualizes the user's motion against target expert motion in video or in augmented reality (AR) (Trout 2013 Motokawa & Saito 2006 Han itt al. 2016,; Kyan ct al. 20155 Kurillo et al. 2008). Since interpreting discrepancies between mo-
tions is chal engingg especially for novices, other works generate visualizations of differences (L.iu et al. 2023 Liao et al. 2023 Balakrishnan et al.] 2015). In contrast, we summarize action differ- ences in natural language, which enables direct and interpretable feedback. Also, our benchmark covers many skill categories, encouraging the development of generalizable methods that do not require domain-specific training data and methods. The most related work by NNaaaaaan & Torre- sani (2024) focuses coarse- -grinerrnnddd step differences in instructional videos using question-answee pairs. In contrast, our approach targets line-grained action differences, such as a "deeper squat", which offers more detailed insights for skill learning. Additionally, VidDiff method is zero-shot for a benchmark spanning multiple skilled domains, while their method requires instruction tun- ing data and is specialized to cooking. Beyond inference-time comparison, a number of important works in skill assessment leverage video pairs in training here the supervision signal is typically a binary variable indicating which video demonstrates greater skill Doughtt et al. (2018) 2019);
Pan et al. (2021): Zhang et al. (2023a). In appendix E. we discuss all related datasets having video pairs, finding that none have labels for fine-grained comparison while being large scale, unlike our VidDiffBench
Describing differences between images in language is an established task called 'difference caption- ing' or 'change captioning' (Jhamtani & Berg- Seng-Kirkkpatree Kirkpatrick] 2018 Park et al.) 2019 Kim et al.) 2021] Yao et al., 2022; Hu et al., 2023). LMM evaluation and eestrruttttunng papers address image differ-
encing for pairs small sets of images (Alayrac et al. 2022 Li et al. 2023 Achiam et al.] 2023
Jiang et al. 2024). The task of image set differencing with large sets was introduced in (Dunlap
et al. (2023) Our VidDiff method uses image differencing with LMMs subroutine, however the
task of video action differencing with natural language has not previously been explored.
3 VIDEO ACTION DIFFERENCING
Video Action Differencing (VidDiff) is a novel and challenging task, offering significant potential for applications in coaching, skill acquisition, and automated performance feedback. To facilitate the development of models capable of handling such a task, we define two complementary task settings: closed setting, evaluated via multiple-choice format, and more complex open setting, requiring generation of action differences. Both are essential for advancing video understanding, especially in contexts where precise feedback on actions is critical.
3.1
TASK DEFINITION
The goal of video action differencing is to identify based differences between two videos of the same action, in a zero-shot setting. We first introduce the simpler closed-set version, followed by the more difficult open-set variation.
Closed-Set Video Action Differencing: In the closed-set task, the input is an action deserip- tion string 8, video pair UB), and list of i candidate difference statements D
3

---

Published as a conference paper at ICLR 2025
{do, di..... d such as "the jump is higher". For each k, the model makes a predictions = {Po.Pl Pi..... Ps-1). where each prediction is either 'A' if the statement applies more to V,a, or 'B' if it applies more to Un. This setup simulates real-world scenarios like coaching, where spe- cific differences of interest are already known. For benchmark purposes, the dataset only includes instances where there is clear ground-truth label ('A' 'B') for each difference, which makes evaluation both reliable and automatic.
Open-Set Video Action Differencing: In the open-set task, the input is the action description string 8, a video pair vn), and an integer Nait-- The model must generate at most Naw difference statements D and their associated predictions P, which label the differences as 'A' for video V, or ''' for video ug. This setting is more challenging, as the model must not only identify differences, but also generate those differences without any pre-defined options, closely mimicking real-world conditions.
3.2 EVALUATION METRIC
Our choice of benchmark evaluation metries is driven by two major challenges for designing annota- tions: ambiguity and calibration. First, there is ambiguity around what differences important for performing an action skillfully. Second, annotators are calibrated differently - they have different thresholds for whether a difference like "wider feet stance" is different enough be annotated.
Closed-Set Evaluation: In the closed-set task, the evaluation is straightforward: prediction ac- curacy is measured the percentage of correct predictions, where 50% corresponds to random guessing and 100% represents perfect performance (assuming a balanced evaluation set). There is no ambiguity because we provide the possible differences. There no calibration issue because the answer must be 'A' or 'B' (and not 'C' for "not different"). Overall, it's an automatic metric that focuses on video understanding.
Open-Set Evaluation: In the open-set task, we use an LLM query (GPT-40) to match the ground truth difference strings predicted difference strings in a 'partial matching". Then we only consider "positive differences" where the ground-truth label is 'A' or 'B' and not 'C', Then the recall @ Naur calculated as the number of correctly matched and predicted positive differences, divided by the total number of positive differences. To handle the ambiguity of what differences are relevant, we set Nair to be 1.5 times the number of labeled differences, so models can predict more differences without penalty. This is reasonable number because the annotation taxonomy is designed to cover skill-relevant differences. Morcover, we handle the calibration challenge of whether a difference is 'above threshold' by only considering the positive differences where ground truth is 'A' 'B'.
4 BENCHMARK DATASET AND ANNOTATIONS
The Video Action Differencing task presents a novel challenge in video understanding, requiring precise comparison of subtle action differences. As no comprehensive benchmark to evaluate this task exists, we introduce VidDiffBench, comprehensive benchmark specifically designed to test and advance the ability of models to detect fine- -ee-gradded differences in complex actions. Our bench- mark consists of publicly available videos and our human-created annotations are freely available on HuggingFace Hutf] VidDiffBench covers a wide range of actions relevant to skill learning and performance feedback, and is constructed to challenge models across varying levels of difficulty, en- suring its relevance for long-term model development. Table 4 summarizes the key dataset statistics.
4.1
VIDEO DATASETS
The video collection for VidDiffBench was designed capture diverse range of actions where performance feedhack is essential, ranging from simple exercises to complex professional tasks. This diversity ensures that models challenged not only temporal localization but also on the subtlety and complexity of visual differences. Actions in VidDifiBench span multiple levels of difficulty-from the basic "hip rotations" in fitness exercises to the intricate "surgical knot tying." This wide coverage tests models across varying degrees of granularity and action complexity. The are five categories:
4

---

Published as a conference paper at ICLR 2025
Category
Source Dataset
Activity
Video Pair
Difterence
Timestamp
Fitness
HuMMan Cai al.
2022
8
193
1,466
310
Ballsports
Ego-Exo4d
GGrauman
20233
996
595
Surgery
JIGSAWS
Gao el al.
2014
166
1,386
672
Music
Ego-Exo4d
Grauman
al.
2023
2
29
180
320
Diving
FineDiving
Xu et al.
2022
63
441
140
Total
18
549
4,469
2,075
Table 1: Summary of VidDiffBench statistics across categories and datasets: number of unique activities, video pairs, annotations for differences, and timestamps.
Fitness videos simple, single-human exercises sourced from HuMMan (Cai et al. )022), characterized by clean consistent backgrounds, consistent camera viewing angles, and consistent movement patterns.
Ballsports includes basketball and soccer actions from Ego-Exo4D (Grauman et al. 2023 recorded across various environments with some diversity in background and camera angle, as well as action detail.
Diving features high-level Olympic performances from the FineDiving dataset (Xu et al. 2022 capturing subtle and complex movements in professional diving. The backgrounds may be different, but the camera angles consistent.
Music contains guitar and piano exercises from Ego-Exo4D (Grauman al. 2023). focus- ing detailed finger and hand movements. Background and camera angles can vary.
Surgery includes long, intricate procedures such "knot tying" and "needle passing" from the JIGSAWS dataset (Gao et al. (014). The background and camera angles are consistent.
Within each action, video pairs are randomly sampled to ensure wide range of comparison diffi- culty. The range of tasks is broad in terms of action complexity and background variation.
4.2 VIDEO ACTION DIFFERENCE ANNOTATIONS
A critical innovation of VidDiffBench is its detailed human annotated dataset, designed to address two major challenges in annotating the video differencing task: ambiguity identifying relevant differences and calibration consistency among annotators. To tackle ambiguity, we introduce structured difference taxonomy for each action, ensuring clarity on what aspects are being com- pared. Then assign annotators to label video pairs with differences to handle the calibration challenge we ensure labeling consistency by maintaining a consistent annotator identity within each action. Additionally, we provide frame-level localization annotations of differences, which can able analysis for future model development. In the following section, we describe these components in greater detail.
4.2.1 ANNOTATION TAXONOMY
For each action, we define a structured difference taxonomy a list of key visual differences relevant the task. For instance, in the basketball jump shot, skill-relevant difference might be "the ball is more in front of the body"; on the other hand, we do not include differences not directly relevant skill performance like "the athlete taller". Annotators assign labels video pairs follows: 'A' if the difference is more pronounced in video A, 'B' if it's more pronounced in video B, and 'C' if the difference negligible. By fixing this taxonomy, we address the ambiguity challenge that different annotators may not focus on the same differences. This allows for more objective and consistent comparisons.
We consulted domain experts create the taxonomies for each action category. For Fitness and Surgery, we worked with a personal trainer and an attending surgeon, respectively, to identify vi- sually salient differences between novice and expert performers. For Ballsports and Music, extracted relevant differences from expert commentary in the Ego-Exo4D dataset using a large lan- guage model (LLM). For Diving, leveraged the FINA diving manual, processed by L.M., to
5

---

Published as a conference paper at ICLR 2025
identify key differences. We filtered differences that were difficult visually assess, such as "more wrist snap" in basketball jump shot (because video resolution was not high enough).
This method resulted in 148 distinct difference descriptions, which detailed in Appendix G.2 This fixed taxonomy allows for precise evaluation model performance across video pairs and helps identify failure cases where models struggle with particular types of differences.
4.2.2 ANNOTATING ACTION DIFFERENCES
For each action aj and its corresponding differences, annotators reviewed video pairs (UA, UB) side- by-side, with the ability step through frames. Each difference was labeled as 'A' if it applied more to video VA, 'B' if it applied more to uB, or 'C' if the difference was insignificant. Consis- tent annotation was achieved by assigning single annotator to each action, ensuring that models are evaluated uniformly across all samples. This avoids the calibration challenge, that different annotators may have different thresholds for significance.
To verify annotation quality, second annotator reviewed 25% of the samples. We assessed dis- agreements where one annotator marked 'A' and the other marked 'B', which occurred in only 2% of cases, indicating low error rates. Annotators were provided with clear visual guidelines to en- sure accurate and impartial labeling. On average, annotators spent three minutes per video pair to evaluate about eight differences.
4.2.3 ANNOTATING DIFFERENCE LOCALIZATIONS
In addition to action differences, VidDiffBench provides localization annotations, pinpointing the exact frames in each video where key differences occur. Since identifying localizing frames and aligning them across videos is a key step in performing video action differencing, these annotations enable analysis of model weaknesses, for example through ablation tests in our results section.
We define specific key points for each action, representing critical frames where important move- ments occur. For example, in squat, key points might include "knees start to bend" and "reaches lowest position." Differences are then linked to these key points: for example the difference "faster squat descent" is defined as the frame spanning "knees start to bend" and "reaches lowest position". Further details are provided in Appendix
4.3 DATASET SPLITS AND STATISTICS
Dataset Splits To account for varying levels of difficulty VidDiffBench, we categorize actions into easy, medium, and hard splits. GPT-4o was used to assign actions these splits based de- scriptions, difference lists, and video lengths. The easy split includes simple movements like Fitness exercises, while medium and hard splits contain sic, and Surgery. This ensures that models are challenged across a range of difficulties, from basic movements subtle, fine-grained comparisons.
complex actions like Ballsports, Diving, Mu-
Dataset Statistics VidDifiBench includes 549 video pairs, 4,469 annotated differences, and 2,075 key point annotations across Fitness, Weightlifting, Ballsports, Surgery, Music, and Diving domains. Video lengths range from few seconds to several minutes, providing comprehensive coverage of different action complexities. This diversity ensures that VidDifiBench is robust benchmark for testing and advancing models in fine-grained action comparison. Under the closed setting, the A/B ratio 0.493/0.507, and in the open setting, the A/B/C ratio 0.259/0.26440.47.
VIDDIFF METHOD
We propose a three-stage framework, the VidDiff Method, that addresses the Video Action Dif- ferencing task. The method follows an agentic workflow (Anthropic) (025) consisting of three components: Difference Proposer, Frame Localizer, and Action Differencer Figure 2 The stages decompose the differencing task into logical steps, and leverage strong zero-shot models for each step. The method described is for the open setting. The method for the closed setting is the same, except the LLM query for candidate differences in stage is replaced with the ground truth differ- ences.

---

Published as a conference paper at ICLR 2025
"Weighted Squad"
Properee OLLNO
Propssed Action deftemeces: "Wizer foot stance stant "Focur decent" "Longer Sqaar: Deepar Separt "Faster secent"
Frame Localieer
Retroval Strings EAA person five squat burbbul thei back", person
"Pster ascend!
the ltting from squa ausilion with
ILMI
Number of taane neaded
Which frones containca fyster AnRARY 2, nrnthwrr
Prame Localzer (2.P1
Video
Action beffeenccar (VQA)
"Viceo has fastor ascent than video A
Frame Localizat
Video
Figure 2: VidDiff Method. One input is an action description (c.g. "weighted squat"). The Dif- ference Proposer generates potential differences using large language model (LLM). The Frame Localizer assigns frames where these differences are observable. Finally, the Action Differencer checks each difference using a vision-language model, determining whether it applies more to video A or video B, or neither.
The Difference Proposer module generates candidate differences for given action description 8. It leverages the extensive knowledge embedded in large language models (LLMs) predict likely differences between the videos. For example, given the description "A practice basketball jump shot", the module might generate difference candidates such as "the athlete jumps higher". These difference statements, which are visually assessable, form the basis for further analysis. The goal of this stage is to create diverse set of meaningful and relevant comparisons.
Difference Proposer:
The Frame Localizer module focuses on identifying the most relevant frames in the video where the proposed differences can be observed. By retrieving the most salient segments from both frames, we solve the key challenge of temporal localization of sub-actions, which makes the next stage more effective. Our approach is to do temporal sub-action segmentation. The LLM takes uses the action description string to produce a list of sub-actions, along with retrieval strings to guide localization. A pretrained CLIP model (Radford et al. 2021 is used to compute frame similarity based on these retrieval strings, and then we assign each frame to one of the sub-actions. Here,
Frame Localizer:
Viterbi-based algorithm (Kukleva al., 2019). which assigns each frame sub- action based on its similarity score, while enforcing that the frames follow the fixed sequence of sub-actions. Finally, the LLM predicts a mapping between the sub-actions and their corresponding differences, yielding set of precisely localized frames for each difference.
3. Action Differencer: In the final stage, the Action Differencer module validates the proposed differences using lasion-language models (VLMs). Given the localized frames from both videos, this module poses multiple-choiee questions (derived from the generated difference candidates) to VLM, which determines whether each difference is more pronounced in UAA vB. or if it is indistin- guishable. This stage transforms the problem into a structured multiple-choic task. Moreover, by providing the maalized-framed relevant to each difference.
Overall the VidDiff method is structured to localize the key parts of the video where differences are possible, which should make visual comparison with the VLM easier.
RESULTS
In this section, we present the results of evaluating large multimodal models (LMMs) and our Vid- Diff Method and on the challenging task of video action differencing on our VidDiffBench bench- mark. Our experiments show the complexity of this task, particularly capturing subtle, fine- grained action differences across diverse video categories. We demonstrate that existing state-of- the-art LMMs, such as GPT-4o and Gemini, struggle with these challenges, while our proposed VidDiff Method outperforms the baselines, especially in the close-set evaluation. Through detailed error analysis and ablation studies, we uncover key factors that influence model performance, shed- ding light future directions for improving video-based model capabilities.
7

---

Published as a conference paper at ICLR 2025
6.1 MAIN RESULTS
As described in Section 3.2 we evaluate our approach on both the closed-set and open-ser tasks. In the closed-set task, models are provided with predefined difference descriptions and must pre- dict whether the difference applies to video A or B. In the open-set task, models are tasked with both generating the difference description and making a prediction. These tasks are fundamental to assessing models' capabilities in fine-grained action comparison.
For our experiments, we benchmark large multimodal models (LMMs) that have demonstrated strong performance in video tasks. Specifically, we use top models from the Video-MME bench- mark (Fu et al. 22024]: GPT-4o (Achiam et al. 2023), Gemini-1.5-Pie (Reid et al. 2024), Claude 3.5 Sonnet Anthropic (2024), and the leading open-source models, Qwen2-VL-7B Wang et al. 2024 Bai et 2023) and LLa (Zhang et al. 2024). Following model guidelines, we provide Gemini, Qwen, and VideoLLaVA with video inputs, while for GPT-4o and Claude give frames, with text prompts explaining which frames belong to which video. For categories with shorter, fine- grained actions (e.g., Fitness, Ballsports, and Diving), we sample frames 4-6 fps, while for longer actions (e.g., Music and Surgery), we sample at 2 fps. Our method, VidDiff, is evaluated alongside these baselines, were the proposer LLM is gpt 4 2024-02222202 -8-06, the localizer embedding model is CLIP results shown in Table 2 and Table
14, and frame differencer VLM is gpt - 2o---00-0000000 2022024 - 08 -06. The results are
Closed-Set Benchmark Performance The closed-set results are in Table revealing that video action differencing is highly challenging task. While some models surpass the random-guessing baseline of 50% where gray shading indicates better-than-rain their improvements are modest, especially in the harder splits where no model performs signifi- cantly better than chance. Gemini, which has emphasized its results in video understanding, has the strongest overall performance. Our VidDiff Method, which uses GPT-4o as visual perception backbone, outperforms GPT-44 on the raw video frames and is second overall, demonstrating the value of our scaffolding for this task. LLava-Vides competitive with GPT and Claude, while DL performs poorly, possibly related instruction-followigg challenges appendix G.4
with statistical significance
Table 2: Results for closed setting (accuracy). Best scores in bold, second best underlined. Scores are better than random, with statistical significance highlighted in gray. Significance is p-value< 0.05 on a binomial test.
Easy
Med
Hard
Avg
GPT-4o
58.3
53.2
48.9
53.5
Gemini-1.5-Pro
67.8
53.6
51.7
57.7
Claude-3.5-Somnet
57.1
50.5
52.5
53.4
LLa -ideeo
56.6
52.0
48.3
52.3
Qwen2-VL-7B
49.0
52.6
49.6
50.4
VidDiff (ours)
62.7
56.2
50.0
56.3
Open-Set Benchmark Performance In the open-set task (Table our method outperforms all other models across most splits, except on the medium difficulty. Among the LMMs, GPT-4o per- forms much better than Gemini. We analyze this gap by breaking down errors into two categories: difference recall error, where the model fails to generate the ground-truth difference, and flipped pre- diction error, where the generated difference is correct but the prediction ('A' or 'B') is incorrect. The closed-set results show that Gemini has lower flipped prediction error, suggesting that Gemini's main weakness is difference recall. Specifically, on the easy split, Gemini's 's recall error 66% compared to GPT-4o's 30%. Despite generating similar number of differences as GPT-4o, Gemini struggles to identify the most important ones in our taxonomy, which hampers its performance. Suc- cess in the open setting requires strong language capabilities, and this limitation is the bottleneck for handling subtle differences. This explains why, when using the same language proposer, our model performs similarly GPT-40.
8

---

Published as a conference paper at ICLR 2025
Table 3: Results for open setting (recall @ Nai)). Best scores in bold, second best underlined.
Easy
Med
Hard
Average
GPT-40
45.7
41.5
38.0
41.7
Gemini-1.5-Pro
30.3
30.5
24.1
28.3
Claude-3.--Sonnet
37.8
34.6
34.3
35.6
LLa V.-VViieiee
7.8
9.0
8.5
8.4
Qwen2-VL VL-7B
11.2
8.8
1.6
7.2
VidDiff (ours)
49.9
37.9
38.5
42.1
6.2 ABLATION STUDIES
conducted ablation studies to better understand the individual contributions of different compo- nents within VidDiff. These studies focus on the Closed setting, isolating the effects of the frame differencing and frame localization stages.
Frame Differencer Image Comparison In the final stage of VidDiff, the model performs visual question answering (VQA) frames retrieved from the videos. To evaluate the effectiveness of this process, we conducted a test using the ground-truth timestamp annotations from VidDiffBench. The results (Table 4 show that even with perfect frame alignment, zero-shot VLMs struggle to consistently detect subtle differences in images. Performance decreases significantly on the medium and hard splits, which suggests room for improvement in zero-shot VLMs* image understanding capabilities.
Frame Localization Design We also analyzed the per- formance of the Frame Localizer in the closed-set case for the easy split, using ground truth difference proposals to measure VQA accuracy. Table []] shows that random frame retrieval leads significant performance drops, while the addition Viterbi-based decoding (which en- forces fixed action transcript) substantially improves ac- curacy. The improvement suggests that temporal align- ment plays a critical role in achieving robust video differ- encing.
Split
Easy
Medium
Hard
Ace
78.6
61.2
51.0
Table 4: Ablation study results for frame differencing VQA with ground truth frames. Questions are 3-way multiple-choice.
In summary, these ablation studies confirm that both ac- curate frame localization and careful VQA processing are essential achieving strong performance in video action differencing.
Method
Accuracy
Oracle (GT timestamps)
78.6
Random
50.1
Ours w/o Viterbi Decoding Ours
57.4
6.3 DIFFERENCE-LEVEL ERROR ANALYSIS
62.7
VidDifiBench's predefined taxonomy allows us to ana- lyze model performance on 148 specific types of action differences, highlighting where models succeed and fail. The results for each difference are detailed in Appendix Table 14, and we perform a statistical significance test to compare models against the aadom-guessing baseline.
Table 5: Ablation on frame localization using different retrieval techniques easy.
We find that model performance is highly dependent on the visual complexity of the action and the difficulty of localization. Successful examples (Figure left column) show high accuracy for simple, easily localized actions, such as "wider foot stance" in hip rotations (83% accuracy) "guiding the ball" in basketball layup (90% accuracy). These cases feature coarse differences that are apparent in most frames, or require only approximate localization.
Conversely, failure cases (Figure right column) often involve precise localization differences. For instance, identifying the angle of a diver's entry into the water in a 10m dive' re- quires frame-perfect alignment, and recognizing subtle changes speed in "piano scales" is difficult
ine-grained

---

Published as a conference paper at ICLR 2025
when reasoning
These challenges highlight the limitations of current models in handling fine-gres mra-grained video analysis.
Success cases
Failure cases
"Wider foot stance", Accuracy: 80%
"Body closer 90 water entry". Anpuracy: random
Coarse
Moderately fine difference Difflcult localization
"Non-shooting hand guides
Accuracy:
"Plays
Coarse visual difference
Easy localization
Fine-grained difference over multiple frames Easy localization
"The step out is faster". Accuracy: 93%
*Instrument applies force to the tissue". Accuracy: random
MMlti-frame reasoning Easy localization
Fine-grained differencing over multiple frames Complex motion
Examples of 'success cases' (left) differences where GPT-4o has high accuracy and failure cases (right). Success cases typically involve coarse differences, easy localization, simple actions, while failure cases often involve fine differences, precise localization or complex actions.
Figure 3:
7 CONCLUSION
In this paper, we introduce the novel task of Video Action Differencing (VidDiff), aimed at com- paring actions in videos. We define this task, compile meticulously annotated benchmark, and propose a zero-shot agent-based framework. Our findings demonstrate that this task is feasible with current foundation models, although more challenging splits the benchmark reveal significant op- portunities for further methodological improvements. We believe that Video Action Differencing represents a promising research direction with broad applications in fields such as skill acquisition, sports analytics, and scientific research.
8 FUTURE WORK AND LIMITATIONS
While our work demonstrates the potential of Video Action Differencing, there are areas for future improvement. Enhancing frame retrieval techniques could improve performance. While many large models have emergent capabilities not trained for (Tang et al. 2023a: Burgess et al., 2024), explicit training for comparing fine-grat fine-grained features of Vision-Language Models (VLMs) is likely underuti- lized. Further, developing methods tailored to specialized domains such as healthcare education could unlock more targeted applications. Limitations in our current approach include reliance on general foundation models, which may struggle with domain-specifee tasks fine ame-grained compar- isons. We hope this work encourages further exploration into broader video comparison methods and inspires advancements in these areas.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXivv2303.0877 2023.
Josiah Aklilu, Xiaohan Wang, and Serena Yeung-Levy. 7ero-shot action localization via the confi- dence of large vision-lange lsiounlanguagg models. arXiv preprint arXiv:2410.14340, 2024.
Cean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estima- tion: New benchmark and state of the art analysis. In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition, pp. 3686-3693, 2014.
Anthropic. The claude 3 model family: Opus, sonnet, haiku. Model card, Anthropic, 2024. URL httpps://i wwt.:wawthhopicc.cooooooo.
Anthropic. Building effective agents. Blog post, 2025. URL https: wwww anthropic..
com/ http:///wrrccbuuulaagggaaeeeefee agents Retrieved from https::/ :ttp:://wwwr /www. aaturppic..comrreeeeerr//bdddddp
agents on February 27, 2025.
TS Ashwin, Vijay Prakash, and Ramkumar Rajendran. systematic review of intelligent tutoring systems based gross body movement detected using computer vision. Computers and Educa- tion: Artificial Intelligence, 4: 4:100125, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308. /2966, 2023.
Guha Balakrishnan, Frédo Durand, and John Guttag. Video diff: Highlighting differences between similar actions videos. ACM Transactions on Graphics (TOG), 34(6):1-10. 2015.
Asish Bera, Mita Nasipuri, Ondrej Krejear, and Debotosh Bhattacharjee. Fine- line-grained sports, yoga, and dance postures recognition: A benchmark analysis. IEEE Transactions on Instrumentation and Measurement, 2023.
James Burgess, Kuan-Chich Wang, and Serena Yeung- Viewpoint textual inversion: Discov- ering scene representations and 3d view control in 2d diffusion models. In European Conference on Computer Vision, pp. 416-435. Springer, 2024.
Zhongang Cai, Daxuan Ren, Ailing Zeng, Zhengyu Lin, Tao Yu, Wenjia Wang, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, al. Humman: modti 4d human dataset for versatile sensing and modeling. In European Conference on Computer Vision, pp. 557-577. Springer, 2022.
Hua-Tsung Chen, Yu-Zhen He, and Chun-Chieh Hsu. Computer-assisted yoga training system. Multimedia Tools and Applications, 77:23969-23991, 2018.
Steven Chen and Richard Yang. Pose trainer: correcting exercise posture using pose estimation. arXiv preprint 2rXiv:2006.1177 17118 2020.
Jihoon Chung, Cheng-hsin Wuu, Hsuan-ru Yang, Yu-Wing Tai, and Chi-Keung Tang. Haa500: Human-centrie atomic action dataset with curated videos. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pp. 13465-13474, 2021.
Bhat Dittakavi, Divyagna Bavikadi, Sai Vikas Desai, Soumi Chakraborty, Nishant Reddy, Vineeth N Balasubramanian, Bharathi Callepalli, and Ayon Sharma. Pose tutor: explainable system for pose correction the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3540-3549, 2022.
Bardia Doosti. Hand pose estimation: A survey. arXiv preprint arXiv: 1903.0101, 2019.
11

---

Published as a conference paper at ICLR 2025
Hazel Doughty, Dima Damen, and Walterio Mayol-Cuevas. Who's better? who's best? pairwise deep ranking for skill determination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6057-6066, 2018.
Hazel Doughty, Walterio Mayol-Cuevas, and Dima Damen. The pros and cons: Rank-aware tempo- ral attention for skill determination in long videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7862-7871, 2019.
Lisa Dunlap, Yuhui Zhang. Xiaohan Wang, Ruiqi Zhong. Trevor Darrell, Jacob Steinhardt, Joseph E Gonzalez, and Serena Yeung-Levy. Describing differences image sets with natural language. arXiv preprint wwXh:2312..0294.. 2023.
Mihai Fieraru, Mihai Zanfir, Silviu Cristian Pirlea, Vlad Olaru, and Cristian Sminchisescu. Aifit: feedback models for fitness training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9919-9928, 2021.
Automatic 3d numan-interpreiie
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.21005, 2024.
Yixin Gao, S Swaroop Vedula, Carol E Reiley, Narges Ahmidi, Balakrishnan Varadarajan, Henry C Lin, Lingling Tao, Luca Zappella, Benjamin Béjar, David D Yuh, et al. Jhu-isi gesture and skill assessment working set (jigsaws): A surgical activity dataset for human motion modeling. In MICCAI workshop: M2cai, volume 3. pp. 3, 2014.
Carly Garrow, Karl-Friedrich Kowalewski. Linhong Li, Martin Wagner, Mona W Schmidt, Sandy Engelhardt, Daniel A Hashimoto, Hannes G Kenngott, Sebastian Bodenstedt, Stefanie Speidel, et Machine learning for surgical phase recognition: a systematic review. Annals of surgery, 273(4))684-693 2021.
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. arXiv preprint vrXiv:2311.18259 2023.
Ping-Hsuan Han, Kuan-Wad Chen, Chen-Hsii Hsieh, Yu-Jie Huang, and Yi-Ping Hung. Ar-arm: Augmented visualization for guiding arm movement in the first- pestpperonn perspective. In Proceed- ings of the 7th Augmented Human International Conference 2016, pp. 1-4, 2016.
Xinyue Hu, Lin Gu, Qi A. An, Mengliang Zhang, Liangchen Liu, Kazuma Kobayashi, Tatsuya Harada, Ronald M. Summers, and Yingying Zhu. Expert knowledge-awee aware image difference graph representation learning for difference-awart aware medical visual question answering. Proceed- ings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023. URL httlhhtts:://ee /api....... semant wcccoolscclllc.. ar www..or....oooeeeee.eeeeeeeeeeee art..rrg/coorrpsst:0205555311
Mostafa S Ibrahim, Srikanth Muralidharan, Zhiwei Deng, Arash Vahdat, and Greg Mori. A hierar- chical deep temporal model for group activity recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1971-,980, 2016.
Harsh Jhamtani and Taylor KinggKirkaatrick. Learning to describe differences between pairs of similar images. arXiv preprint arXiv:1808, 100111888........ 2018.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning, April 2024. URL htttp:://tiger ainiraiilabba lab.
github. 10/ B Blw.///o//oooooo htt.//log/maanissss mantis
David Johnson, Isabelle Dufour, Daniela Damian, and George Tzanetakis. Detecting pianist hand posture mistakes for virtual piano tutoring. In Proceedings of the international computer music conference, pp. 168-171, 2016.
David Johnson, Daniela Damian, and George Tzanetakis. Detecting hand posture in piano playing using depth data. Computer Music Journal, 43(1):59-78, 2020.
12

---

Published as a conference paper at ICLR 2025
Omar Khattab, Arnv Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self- improving pipelines. 2024.
Hoeseong Kim, Jongseok Kim, Hyungseok Lee, Hyun a Park, and Gunhee Kim. Viewpoin--agnostic change captioning with cycle consistency. 2021 IEEE/CVF International Conference Com- puter Vision (ICCV), pp. 2075-2084, 2021.
Anna Kukleva, Hilde Kuehne, Fadime Sener, and Jurgen Gall. Unsupervised learning of action classes with continuous temporal embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12066-12074, 2019.
Gregorij Kurillo, Ruzena Bajesy, Klara Nahrsted, and Oliver Kreylos. Immersive 3d environment for remote collaboration and training physical activities. 2008 IEEE Virtual Reality Conference, pp. 269-270. IEEE, 2008
Matthew Kyan, Guoyu Sun, Haiyan Li, Ling Zhong, Paisarn Muneesawang, Nan Dong, Bruce Elder, and Ling Guan. approach ballet dance training through ms kinect and visualization in cave virtual reality environment. ACM Transactions on Intelligent Systems and Technology (TIST), (2):1-37, 2015.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Fanyi Pu, Jingkang Yang. Chunyuan Li. and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.005425 2023.
Haopeng Li, Andong Deng, Qiuhong Ke, Jun Liu, Hossein Rahmani, Yulan Guo, Bernt Schiele, and Chen Chen. Sports-qa: A large-scale video question answering benchmark for complex and professional sports. arXiv preprint arXiv:240.001505, 2024a.
Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music condi- tioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 13401-13412, 2021a.
Yixuan Li, Lei Chen, Runyu He, Zhenzhi Wang, Gangshan Wu, and Limin Wang. Multisports: A multi-person video dataset of spatio-temporally localized sports actions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13536-13545, 2021b.
Yuan-Ming Li, Wei-Jin Huang, An-Lan Wang, Ling-An Zeng, Jing-Ke Meng, and Wei-Shi Zheng- Fgoexo-fitness: Towards egocentric and exocentric full-body action understanding. In European Conference on Computer Vision, pp. 363-382. Springer, 2024b.
Chen-Chieh Liao, Dong-Hyun Hwang, Erwin Wu, and Hideki Koike. Ai coach: motor skill training system using motion discrepancy detection. In Proceedings of the Augmented Humans International Conference 2023, pp. 179-189, 2023.
Ruofan Liu, Erwin Wu, Chen-Chieh Liao, Hayato Nishioka, Shinichi Furuya, and Hideki Koike. Pianosyncar: Enhancing piano learning through visualizing synchronized hand pose discrepancies in augmented reality. In 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. $59-868. IEFF, 2023.
Yanchao Liu, Xina Cheng, and Takeshi Ikenaga. Motion-awan and data independent model based multi-view 3d pose refinement for volleyball spike analysis. Multimedia Tools and Applications, 83(8):22955230115 2024.
Yi Liu, Limin Wang, Yali Wang, Xiao Ma, and Yu Qiao. Fineaction: A fine-grained video dataset for temporal action localization. JEEE transactions on image processing. 31:6937-6950 2022.
Yoichi Motokawa and Hideo Saito. Support system for guitar playing using augmented reality display. In 2006 IEEE/ACM International Symposium Mixed and Augmented Reality, pp- 244-244. IEEE, 2006.
Tushar Nagarajan and Lorenzo Torresani. Step differences in instructional video. arXiv preprint arXiv:2404. 16222, 2024.
13

---

Published as a conference paper at ICLR 2025
Ryoji Ogata, Edgar Simo-Serra, Satoshi lizuka, and Hiroshi Ishikawa. Temporal distance matrices for squat classification. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition workshops. pp. 0-0, 2019.
Jia-Hui Pan, Jibin Gao, and Wei-Shi Zheng. Adaptive action assessment. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 44(12):8779-8799 2021.
Dong Huk Park, Trevor Darrlll, and Anna Rohrbach. Robust change captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4624-4633, 2019.
Paritosh Parmar and Brendan Tran Morris. Learning to score olympic events. In Proceedings the IEEE conference on computer vision and pattern recognition workshops, pp. 20-28, 2017.
Paritosh Parmar, Amol Gharat, and Helge Rhodin. Domain knowledge-informed self-supervised representations for workout form assessment. In European Conference on Computer Vision, pp- 105-123. Springer, 2022.
Hamed Pirsiavash, Carl Vondrick, and Antonio Torralba. Assessing the quality of actions. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Part VI 13, pp. 1.6-571. 2014.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055500 2024.
Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. Ntu rgb+ A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1010-1019, 2016.
Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A hierarchical video dataset for fine- grained action understanding. In Proceedings of the IEEE/CVE conference on computer vision and patlern recognition, pp. 2616-2625, 2020.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 12 22.204042 12.0402, 2012.
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36: 1363-1389, 2023a.
Yansong Tang, Jinpeng Liu, Aoyang Liu, Bin Yang, Wenxun Dai, Yongming Rao, Jiwen Lu, Jie Zhou, and Xiu Li. Flag3d: 3d fitness activity dataset with language instruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22106-22117, 2023b.
Vivek Anand Thoutam, Anugrah Srivastava, Tapas Badal, Vipul Kumar Mishra, GR Sinha, Aditi Sakalle, Harshit Bhardwaj, and Manish Raj. Yoga pose estimation and feedback generation using deep learning. Computational Intelligence and Neuroscience, 2022, 2022.
Josh Trout. Digital movement analysis in physical education. Journal of Physical Education, Recre- ation Dance, 84(7):47-50, 2013.
Eduardo Velloso, Andreas Bulling, and Hans Gellersen. Motionma: by demonstration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1309-1318, 2013.
Motion modelling and analysis
14

---

Published as a conference paper at ICLR 2025
Manisha Verma, Sudhakar Kumawat, Yuta Nakashima, and Shanmuganathan Raman. Yoga-82. 82: new dataset for one-grained classification human poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. pp. 1038 (038-1039, 2020.
Kuan-Chieh Wang and Richard Zemel. Classifying nba offensive plays using neural networks. In Proceedings of MIT Sloan sports analytics conference, volume 4, 2016.
Kuan-Chieh Wang, Zhenzhen Weng, Maria Xenochristou, João Pedro Araújo, Jeffrey Gu, Karen Liu, and Serena Yeung. Nemo: Learning 3d neural motion fields from multiple video instances of the same action. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22129-22138 2023.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.1219) 2024.
Haoran Xie, Atsushi Watatani, and Kazunori Miyata. Visual feedback for core training with 3d human shape and pose. In 20/9 Nicograph International (Nic (Nicolnt), pp. 49-56. IEEE, 2019.
Jinglin Xu, Yongming Rao, Xumin Yu, Guangyi Chen, Jie Zhou, and Jiwen Lu. Finediving: A fine- grained dataset for procedure-awas action quality assessment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3949-2958 2022.
Linli Yao, Weiying Wang, and Qin Jin, Image difference captioning with pre-training and contrastive learning. In AAAI Conference on Artificial Intelligence, 2022.
Shao-Jie Zhang. Jia-Hui Hui Pan, Jibin Gao, and Wei-Shi Zheng. Adaptive stage-aware assessment skill transfer for skill determination. IEEE Transactions on Multimedia, 2023a.
Shiyi Zhang, Wenxun Dai, Sujia Wang, Xiangwei Shen, Jiwen Lu, Jie Zhou, and Yansong Tang. Logo: A fong-form video dataset for group action quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2405 2415-4414, 2023b.
Weiyu Zhang, Menglong Zhu, and Konstantinos G Derpanis. From actemes to action: A strongly- supervised representation for detailed action understanding. In Proceedings of the IEEE interna- tional conference on computer vision, pp.
2013.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024.
Ziyi Zhao, Sena Kiciroglu, Hugues Vinzant, Yuan Cheng, Isinsu Katircioglu, Mathieu Salzmann, and Pascal Fua. 3d pose based feedback for physical exercises. In Proceedings of the Asian Conference on Computer Vision, pp. 1316-1332, 2022.
Kevin Zhu, Alexander Wong, and John McPhee. Fencenet: Fine-grained footwork recognition in feneing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 2589-3598, 2022.
15

---

Published as a conference paper at ICLR 2025
A BENCHMARK: DOWNLOAD INSTRUCTIONS
Our benchmark is released at http://gggnnnaa..oooo/o////////a instructions on how access annotations, how download external datasets, and all licenses for our annotations and the source video datasets.
It has complete
B BENCHMARK: DIFFERENCE ANNOTATION TAXONOMY GENERATION
Each dataset underwent a thorough taxonomy generation process. Details for each dataset are pre- sented in this section. Most datasets were first processed with the Difference Proposer.
B. GENERATING JIGSAWS DIFFERENCE ANNOTATION TAXONOMY SURGERY VIDEOS
To produce the JIGSAWS taxonomy for surgery videos, we first used the Difference Proposer to gen- erate difference candidates. As we found many proposed variations be irrelevant, we consulted surgeon from janonymous hospital,, We first showed them variety of videos from the JIGSAWs dataset and brainstormed visually discernible differences. Next, we shared the GPT-generated varia- tions with the surgeon and discussed which ones should be added to the brainstormed listed. Lastly, we dropped all differences that could not be annotated consistently. For instance, we removed "Sur- geon ex exploits robotic instrument's range of movement efficiently in Video A than in Video B", as the difference subject to interpretation.
B.2 GENERATING EGO-EXO4D DIFFERENCE ANNICCEANOONNN ANNOTATION TAXONOMY
For datasets with expert annotation, such as Basketball, Soccer, and Music, we processed the ex- pert commentary with GPT instead of asking for difference proposals directly. We asked GPT to summarize the expert commentary using the following prompt:
Bolow are sequence of text strings.
The strings are written by exports who are watching vidcos of a task with this description: "(action "(action_doscrttt doscriptions)". As the experts watch each video, they pause the video and record verbal commentary about how that person is perforing the task.
Return a 1ist of text strings that sumariizes what are the key visual cues that the
expert is Looking when they provice feedback.
Each iten in the 1ist should be specific and testable, that anyoody watching single
video can assess whether that visual cue applies to particular video.
Your response should be a json with this structure: "sunmary_tee" ttrtsst ["text0", "text1", ...] least 15 items.
The list should
Here are the texts to sumarizz: (texts)
As GPT has no knowledge of the specifics of the dataset, we then manually parsed the proposed list and kept items that could be distinguished by non-experts based on visual information only. Some differences were not visible in our data. For instance, the "wrist snap" in basketball was excluded, as it could not be discerned from the videos.
As we found that GPT lost lot of information from the expert commentary, we also manually parsed the expert comments and added key visual cues that were mentioned by the experts.
B.3 GENERATING FINEDIVING DIFFERENCE ANNOTATION TAXONOMY
For the diving dataset, we used the Difference Proposer. As all videos are of experts, there little variation between the videos. We thus used the diving score annotations given in the dataset, find pairs of images with more variability and used those proposed differences that were discernible in these pairs.
16

---

Published as a conference paper at ICLR 2025
C BENCHMARK: ANNOTATION DETAILS
Two annotators were provided with a list of variations and folder containing all videos as well as concatenated videos of two actions next to each other. They were instructed to annotate differences only as such, if they were obvious. The instructions for the annotators were as follows:
You' 11 get pairs of videos and be asked questions about how they' re different. They' 11 be speci f querstions ons:
K.g. "which one has wider foot stance: (a) video (b) vi dio H, (c) they' re the sane.
if it's hard tell whether there really a difference, then say "c", Rule of thunb: once you've ve found the important point in the video, if it takes you more thar. 10 seconds to make decision, then say "c"
C.I BENCHMARK: THE DIFFERENCE TAXONOMY
The difference taxonomy is available as part of the benchmark release at this link The full list of differences can also be previewed in the analysis table 14
C.2 BENCHMARK: RETRIEVAL. ANNOTATION GENERATION
For the closed-evaluation scenario, we need to temporally align the videos we wish to compare. For this alignment, annotate retrievals, which important and identifiable moments within an action. To generate retrievals, we used the Frame localizer, where we prompt GPT4-V to propose stages for a given action (see Section C.3 for details). We found that the stages to coarse. We thus either manually identified key moments that helped retrieve sections of the videos important for comparing differences, or, for JIGSAWS consulted our expert. The retrieval annotations are available as part of the VidDifßBench benchmark release, and are in the folder called 'retrieval'.
sometimes
C.3 PROMPTS FOR ACTION ASSIGNMENT TO THE EASY/MEDIUM/HARD SPLITS
I'm designing benchmark for comparing pairs videos of the same action.
We have many actions and each action has list of differences we look for. The benchmark's task is to examine di fifrrnnces and say whether the statement applies more "video A" "video 3"
Below we show dictionary where each element a single action. Each action has an "action description" describing the action.
It also has daveragg_scconde_eg scccscdc_ooooeeeeeeeeeeeeeeeeeee per___eooo__""""" for the modian length of videos in seconds. Each action has dictionary of "differences", where each di difference has these keys: namc" for the difference
'description' describing the fferencee
Finally, for each action, there are two unfinished options:
'split' which currently says 'easy Imddim//aad
reasor' which currently says '...'
Your task is to fill in these values:
Decide whether the 'split' value is 'easy', 'mediun or "hard' This evaluation judges the fifcullyy of performing actionn difference comparison for all diffrrnnces within an action. Having high number of actions should not considered as criteria for difficulty.
Justify your choice in spplitrresson
Return the dictionary as json, with the values of 'split' and 'split_reascrt reasor populated.
Here are the actions. (actions)
The ict ions field is replaced with a json with the structure that is described in the prompt.
17

---

Published as a conference paper at ICLR 2025
C.4 DIFFICULTY SPLITS
The result of the difficulty splits is in table
Table 6: The difficulty splits with action code and names
Split easy
Action
Action Name lipp circle anticlockwise
Action description
fitness exercise called standing hip circle with hands on hips, one rota- tion anriclockwise
easy
Giness. 3
lucky cut
fitnews exercise called two arm standing lucky cat starting with arms up,
repetition
casy easy
fitness.
Squat Knee Raise side view Hip circle clockwise
squat without weights, then knee raise on left side
fitnes_6
fitness exereise called standing hip circle with hands on hips, reta-
tion clockwise
medium
ballsports_0 O
baskethall jump shot
person doing haskethall mid range jump shot, stariing hall in their hand, no defense, practice only
medium
ballsports..
baskethall mikan layup
person does the Baskethull drill called the Mikan layup where they start under the basket, do layup with the right hand and catch do left hand layup and catch it, no defense. practice only
medium
ballsports_22
basketball reverse layup
person playing basketball does a reverse layup starting from the side of the basket and lays it with their right hand on the right hand side, no defense, practice only
medium
ballsports_3
soccer penalty kick
person does a soccer drill where they do a single penalty kick, practioe only, no defense, no goalie
medium medium
diving.0 O fitness_]
diving
competirive diving from 10m fitness exercise called opening and closing step on left side and then opening closing step on right side
Opening and closing step left side first
medium
fitnes__ 2
car deadlift
single free weight deadlift without any weight
medium medium
fitness_5 fitnes_.7
Squat Knee Raise diagonal view
squat then knee raise left side
Opening and closing step right side first
fitness exercise called opening and closing on right side and then opening and closing slep on lelt side
hard
music.0
piano
person is playing scales the piano person is playing scales on the guitar The subject picks up one
hard hard
music_1 surgery.J LD
guitur
Knot Tying
suture tied to flexible tube attached at its ends to the surface of the bench top model, and ties a single loop knot.
hard
surgery
Suturing
The subject picks up acedle, proceeds the incision (designsted vertical line on the bench- top modell. and passes the needle through the fake tissue, entering the dot marked one side of the incision and exiting at the correponningg dol marked on the other side of the incision. After first of the tissue, passes it to the right hand and repeats the needle pass three more times.
pass, subject extracts
hard
surgery
Needle Passing
The subject picks up the needle (in some not captured in the video) and passes it through four smmll metal hoops from right to left. The hoops are attached small beight above the surface of the bench-top model.
C.5 VALIDATING SPLIT GENERATION HUMAN STUDY
Choosing the difficulty splits requires a holistic view of all the actions, so we decided didn't make sense for experts to suggest them, since they are only familiar with few actions each. On the other hand, we didn't want to rank the splits based on performance of current models since this felt like biasing towards current models; and besides, the performance for many actions in 'medium' and 'hard' is already random, so it would be hard to differentiate these actions. LLMs are good candidate because they have a good understanding of the actions and are relatively free of the biases of this paper's s authors Furthermore, human annotators could not do the ranking, because no human annotated all the actions.
To further support the choice of an LLM, we asked 3 humans to rank the action comparisons from easiest to hardest, and compared against the LLM ranking. We then computed the Spearman's rank correlation between all ranking sets, and the results are in table 7 The mean of the pairwise correlations between the humans was 0.602, while the mean of pairwise correlations between the LLM and humans was higher at 0.673. This shows (i) that there is non-negligible variability in human rankings, and (ii) that the LLM ranking is reasonable, and actually better correlated with most humans compared to several of the human annotations.
18

---

Published as a conference paper at ICLR 2025
Table 7: Results human evaluation study choosing the splits. This is the Spearman's is rank corrlation between the ranks of action dificulty, comparing our LLM approach and 3 humans.
LLM
Human
Human 2
Human
LLM
53.1
68.0
80.6
Human
53.1
45.9
64.5
Human
68.0
45.9
70.3
Human 3
80.6
64.5
70.3
Average 67.3
54.5
61.4
71.8
C.6 FURTHER DATASET CONSTRUCTION CONSIDERATIONS
Camera angles The change of camera angle perspective does make the task harder. For samples in the 'Fitness' category, the camera angle is the same because the source dataset has fixed camera rig, and chose camera angle is approximately the same. On the other hand, samples from 'ballsports' and 'music' categories change. A related attribute (not mentioned here) is differences in background sim- ilarly the 'ballsports' and 'music' categories often had different backgrounds as well. Importantly, these attributes were considered when assigning the difficulty splits. This may partly explain why the fitness exercises are all in the easy and medium split.
the same camera angle. For samples in 'diving' and 'surgery' categories, the
Each video pair has the same fps. In case others want to leverage our code with new videos, our code does handles the case where FPS is different. Specifically, the input configuration has value for the target FPS for running inference, and we subsample the video to have this FPS. (If the videos cannot be subsampled to have the exact target fps, then warning is printed).
FPS
Impact of different actor heights For annotator instructions, we clarified that all differences on things like distance should be relative the actor's height. We gave the example of 'wider foot stance', saying that a 5ft actor and 6ft actor both had their legs 3ft apart, then the shorter actor has "wider foot stance' relative their height. This reflects what commonly understood by descriptions like these in skills coaching.
D BENCHMARK STATISTICS
Beyond the main statistics the main, table shows further statistics broken down by difficulty splits.
Table 8: Detailed data statistics by split
Split # video pairs
medium
hard
Overall
95
265
197
557
Ave video length (sees)
3.9
18.7
Total video length (mins) # differences lagged
6.5
34.7
122.5
163.7
1224
4788
3542
9554
StdDDy within retrieval type
8.4%
5.2%
4.1%
5.9%
SidDey across retrieval lypes
17.3%
25.7%
20.2%
21.0%
annotations
4719
Difference annotations A/B/C distribution
167/190/221
622/605/1
143
435/452/884
1224//1447/2248
Average video length is longer the difficulty gets higher: 1//3.9/18.7 seconds, for easy/medi- um/hard. Compared video QA datasets, the lengths are relatively shorter because we focus on fine-grained action understanding in how actions are performed. The total length of videos is 163 minutes.
Retrieval tags, temporal bias For the 'retrieval tags', we first show the number of retrieval tags 9554 total. To give insight into their distribution within each video, each instance is normalized to the video length, and compute its 'video location'. E.g. in squat, the starting position might be position 0.1, the bottom of the descent 0.45, and the squat finish at 0.87. Within each retrieval type, compute 'StdDev within retrieval type', which intuitively measures how well-aligned are
19

---

Published as a conference paper at ICLR 2025
the key points in the video. For example, if the average squat video records 'bottom of descent' location 0.45, and 'within StdDev' is 0.06, then the mean distance from the average is 0.06 (so at 0.39 or 0.51). The 'within StdDev' is on average 0.059, indicating there is some variation in retrieval position, but there is temporal bias. This is expected since each video is trimmed and contains an atomic action. Future benchmarks could use untrimmed videos to make retrieval annotations less aligned, but the present benchmark is already difficult for SOTA models, so this is unnecessary now.
Retrieval tags, coverage We also measure 'StdDev across retrieval types'. meaning the standard deviation of different retrieval classes within one video. Intuitively this measures how much of the video is 'covered' by retrieval keypoints. This is 0.21 on average. So if the mean of retrieval keypoints were 0.5, then the average retrieval annotations is around 0.29 or 0.71 the video.
A/B/C distribution Additionally, we have shown the count of difference annotations and the A/B/C distribution; the 'no difference' annotation of 'C' is the most prevalent.
E BENCHMARK: RELATED VIDEO PAIR DATASETS
A small number of prior works have datasets of paired videos with some label of the difference.
However none have labels for fine-grained comparison while also having a large scale.
Nagarajan & Torresani (2024) has a large-scale dataset of video differences in instructional video large enough to be used for instruction tuning. However their differences are very coarse-graie coarse-grained, for example 'cooking video forgot to add salt'. Here, large-scale is pos- sible because differences can be derived automatically from annotated instructional video datasets.
(Balakrishnan et al. 2015) considers fine-grained action differences. However their dataset is very small (less than 50), and they have labels.
(Doughty al. 2018) has dataset of paired actions called EPIC-Skills2018. Here, the scale is large, but the difference label is more coarse: binary for which video shows more skill.
F EVALUATION
The evaluation code is available this GitHuu repo https:/ggithbb.omm//mmb/odddddd
F.1 CLOSED EVALUATION DESIGN CHOICE
Our closed evaluation setting has options for 'A' and 'B', but not 'c'. Our initial approach to for- mulating this did include option 'C' for insignificant differences. However, the challenge of calibration made fair evaluation difficult. For example, when comparing two videos of a basketball shot to evaluate stance width, the question arises: how different is "different enough" to be both relevant for skill learning and perceptible? Different annotators may apply varying thresholds for what constitutes a significant difference, leading inconsistencies. Introducing option 'C' further complicates evaluation because it requires calibrating not only the human annotators but also the VLMs, which may have different internal thresholds for perceiving significance. To address these challenges, we adopted the following approach:
Annotators were instructed to choose either 'A' or 'B' only when the difference was clearly perceptible.
We limited the evaluation of VLMs to cases where there was a very clear ground truth answer of either 'A' 'B'.
This method ensures faimess by focusing scenarios with unambiguous ground truth, avoiding complications introduced by subjective calibration thresholds. While we briefly discuss this in the section on annotation creation, recognize that this is nuanced point.
20

---

Published as a conference paper at ICLR 2025
F.2 EVALUATION: MATCHING WITH LLMS
As described in section 3.2 we use an LLM
We "gpt-4o-2024-08-00 with the following prompt for matching ground truth to predicted de- scriptions. We do one prompt per video pair sample.
You are analyzing videos of people performing a specific action described as "( action action_descriptinn description)."
In this task, "di fference" refers to how two people might perform the same action distinct ways.
Each difference consists of:
A name (key) and
A description that explains how the performances di differ visually.
You are provided with two dictionaries:
Dictionary 0: (differences0)
Dictionary 1: (differencesl)
Your task is to natch the fffrenneces from Dictionary to Dictionary 1. Here's s how:
1. For each entry in Dictionary 0, find the best match in Dicti Dicticnary 1.
2. Each item in Dictionary 1 can only be used once.
3. Only match entries if their description strings visually similar, even if the word choices differ. no suitable match exists, return "None." "
Your output should be a JSON object where:
The keys from Dictionary 0: (dict0
The values key from Dicticnary "None" if ratch is found.
The available keys from Dictionary are: (dictikkess)
Example output forma::
"O"; "3",
"I": "None",
"2": "I",
"3": "5", "(final)" "0"
Important: the keys in this dictionary should be" (dict0 keys)
We replace:
/action.description with a string describing the action
[differences0] with dictionary where keys are gt difference keys ("O", "1 and values which are strings describing differences.
[differences]] is the same as (differencesO), except for the predicting differ- ences.
(dict.keys) are the keys in [differences(]
[kect] are the keys in (differences [final} the highest key in (di fdireneee)))
An issue with matching is that the prediction actually has the opposite value the ground truth e.g. "the arms are more straight" vs "the arms are more bent". If this the case, then the prediction 'a' should be reversed to 'b' and vise versa. To identify those cases, use this prompt, and evaluate them in batches of 6 difference sequences at once.
Task:
You will be given pairs of statements. Your task is to deternine the Logical relationship between each pair.
21

---

Published as a conference paper at ICLR 2025
Instructions:
Analyze Each Pair: For each pair of statements, carefully analyze their meaning and relationship.
2. Catogorization:
Return "0" if the statements are equivalent, very similar, differ only in minor details.
Example: "X bigger than and is larger than should both return "o".
Return "1" if the statements are direct opposites in meaning.
Example: "X is bigger than y" and "X is snaller than Y" should return "1".
3. Edge Cases:
Avoid returning "1" for statements that are not true opposites, even if they have some differences in detail or degree.
Example: "x is much bigger than y" and "x is slightly bigger than y" should still return "g".
Output Format:
Your response should be a USON object with a single koy "results" and an array of string values "0" or "1" as its value. The array should exactly match the number of statement pairs given in the input.
Input: Fornat:
The 1ist of statement pairs will be provided in the fol fowlewing format:
(statements)
Important Regentemetts:
Ensure that each value in the output array is either "C" or "1"
The length of the "results" array must exactly match the number of input pairs.
We replace the (statements) with list, where each element is two element list of strings, which are matched difference descriptions.
F.3 PROMPTS FOR LMM BASELINES
The LMM baselines GPT, Gemini, and Qwen all receive the same prompts. Prompts for action assignment to the easy/medium/hard splits:
Here are two videos of an action with the tollowing description: "lactico dessripteosc (video representation description)
Below is a set of identified differences that describe how the action be performed differrntly..
Each diffrrence is associated with unique key:
[llfferrnces__amottted)) annotated) (di.fferences
Your task is to predict, for each difference, whether it is more true for video 'ar or video 'b".
(target_out)
The (act on description) replaced with a string describing the action which is the same as in table 13 The (di feifffeeceeeannee _anooettsnnttteeeeeaee is dictionary mapping the ground truth difference key to a difference description, and are the same stringes used in table [4 The tells the model how the video data is passed in. If 2 videos for Gemini and Qwen then it's:
We have passed 'video and 'video b' as videos to the prompe in that crder.
If passing in the videos as frames for GPT then it's:
We have passed a sequence of images into the prompt. The first
are video The last (vidm_nframes) video b. frano rate the same and is (fps).
22

---

Published as a conference paper at ICLR 2025
The open prompt is:
Here are two videos of action with the following description: /ecctioo_descriptiion""
(video__ree represerraticn iupressratiicc.sscrriiiiii description)
Return a list of 'differences' in how the action is being performed. Each differenes should have a 'descriptior' that is specific statements that is more true in onc video compared the other vidco.
Then there is 'predictiorf which by if it applics more to video b.
if the statement applics more to video and
The di fference descriptions should be visual and about how the action is performed. For example 'the jump is higher', or "the
more straight'.
The difference descriptions should not refer to a specific video. For example you would not say the jump in video B is higher'. Instead, the 'description' would be tte jump is higher', and the 'prediction is lb.
Suggest no more than (n_diiffeescee) differences.
Return json like this, replacing with actual content:
"O"
"description" .....",
"prediction" *a/b"
"1"
"description" .....", "prediction" "alb"
F.4 VALIDATION OF MATCHING PROCESS
We leverage LLMs in open evaluation to identify matching between the ground truth difference description strings and the predicted differences. Here we validate that this is a reasonable approach.
Robustness to multiple LLM runs The LLM evaluation is robust to random seed. We repeated the evaluation five times with different random seeds and observed standard deviation of only 0.7 in the final evaluation score. This indicates that the results are consistent across runs. Although the prompt specifically engineered for the fixing the model for evaluations, treating all comparisons under identical conditions.
model, ensured consistency by
Comparison with Human Evaluation measure alignment with humans, we recruited 3 human annotators to perform open evaluation matching, each with 44 video pairs and 347 individual dif- ferences. For each video pair, they were provided with a list of ground truth differences, and asked to match each to predicted difference from list, or suggest match. We calculated inter- rater agreement across annotators and the automated LLM system. The results are in table We can semantic matching proved to be challenging for humans the mean of pairwise rater agreement from each human to the other humans was 75.7%. Meanwhile, the mean agreement between our automated system and human annotators was 73.9%. Therefore, our LLM-based approach is on par with human annotators, while being completely automatic.
Details of Prompt for LLM Evaluation The LLM prompt was carefully developed using a prompt engineering workflow. We selected set of four evaluation samples, covering two actions and two models, and iteratively refined the prompt based on performance individual runs. For example, added the instruction: "Only match entries if their description strings are visually sim- ilar, even if the word choices differ." This adjustment was necessary because the LLM struggled match equivalent descriptions phrased differently (e.g., "the feet stance wider" "the legs
23

---

Published as a conference paper at ICLR 2025
Table 9: Agreement rate of LLM and human predictions for the evaluation matching.
LLM
Human
Human 2
Human
LLM
72.4
74.0
70.1
Human 72.4
75.0
78.2
Human 74.0
75.0
73.9
Human 70.1
78.2
73.9
Average 72.2
75.2
74.3
74.0
are spread wider apart"). While this approach achieved satisfactory results, we acknowledge that the prompt could be further optimized using more systematic methods, such as DSPy Khattab et al. 2024). Exploring such techniques is promising direction for future work.
G RESULTS: MORE ANALYSES
G. RESULTS: ACTION-LEVEL. MODEL COMPARISON
We have performed a more thorough comparison of the different ofate-of-the-ars LMMs on VidDIff- Bench, added a small subsection to the results, and discussion in appendix. Specifically we look each action, and compare the different L.MMs.
First, we show the correlations in the per-action scores between models in table 10
Table 10: Correlations between models where the data is the action-level accuracy.
GPT
Gemini
Claude
-Lava-Vider
Qwen2-V
GPT-40
0.152
0.375
0.243
0.273
Gemini-1..-Pro
0.152
0,215
0.111
0.223
Claude-3.5-Sonnet
0.375
0.215
0.261
0.220
LLa VL-VA--ida 0.243
0.111
0.261
0.376
Qwen2-VL-7b 0.273
0.223
0.220
0.376
The correlations are generally low, but there are clusters models. LLaVA- and Qwen-2- VI. in cluster; they both open-source, and have the same L.M backbone. Then GPT-4o and Claude-Sonnet cluster together. and Gemini is not similar to any other model. We can speculate that for video data, Claude and GPT have similar training strategies, while Gemini's 's different.
Next we compare model performance within one action, and this is two large tables. table [1 is the action-level performance of each model. Then table 12 is the 'relative performance': the difference between the model score on that action compared to the mean score across all models for the action. The most sigmificant results in the benchmark are the easy split. Here, the improvement score is uniform for all models. The models generally close perform similarly each other. The relative performance usually less than points when is higher, the sample size is very small.
By comparing models at the level of actions, we are considering smaller sample sizes than in the main results, which compare models the level of easy/medium/hhrd splits. There therefore lower statistical power to identify significant result differences, the results are less certain. We elected not to compare model performance at the level action differences, because here the sample sizes very small, any correlations would not meet significance thresholds.
G.2 DETAILED DIFFERENCE ANALYSIS
In Section 6.3 we discuss an analysis of the accuracy at the difference level. The vary long table 14 gives the deffdifference accuracies and p-values compared for the accuracy against random guessing baselines. Each difference is associated with an action key, whose description is in table
24

---

Published as a conference paper at ICLR 2025
Table 11: Action-level scores for each model, and their differences compared to the average model score for that action. The model names are abbreviated and the full model names are GPT-4o, Gemini-1..-Pro, Claude-3.5-Sonnee LLaVA-Video-7B, Qwen2-VL--B
Split
Action
Action Name
Count
Scores GPT
Gemini
Claude
LLaVA-Vid
Qwen2
Avg
Hip circle
fimess
anticlockwise
129
56.6
58.1
56.6
$1.9
45.0
53.6
easy
Oiness 3
lucky car
62
53.2
58.1
43.5
58.1
45.2
51.6
Squat
casy
Gimess.
side view
43
65.1
69.8
37.2
69.8
55.8
59.5
easy
Hip circle basketball
76.4
medium
ballsports.
jump shot basketball
medium
ballsports.11
148
56.8
49.3
46.6
51.4
52.0
mikan layup basketball
medium
ballsports,2
125
46.4
55.2
49.6
44.0
50.4
49.1
revense layup soccer
medium
ballsports_3
70
51.4
60.0
57.1
70.0
54.3
58.6
penalty kick
medium
diving.
diving
240
53.8
52.1
53.3
50.8
54.2
52.8
Opening and closing step left side first
medium
liness_.1
186
57.5
54.8
52.7
51.1
51.6
53.5
deadlift
55.5
Squat Knee Raise diagonal
medium
fitness_
70
35.7
32.9
38.6
62.9
42.9
Opening and closing
medium
step right side first
52.9
52.9
49.7
hard
music_0
51.1
51.1
58.5
$1.1
52.8
hard
music_1
guitar
20
55.0
40.0
45.0
50.0
65.0
$1.0
surgery
Tying
47.7
hard
surgery.
Suturing
309
48.9
51.5
48.2
47.6
50.2
49.3
hard
surgery 2
Noodle Passing
211
51.7
46.9
49.8
48.8
49.5
Table 12:
Action-level difference scores for each model relative to the mean model score on that action. This is the difference with respect to the table 11 The model names abbreviated and the full model names are GPT-40, Gemini- Prrorr, Claude Claude-3.5-Sone 3.5---,.5,,,n,,,,, LLaVA-Video-72 Qwen2-VL- 7B
Split
Action
Action Name
Count
Differences
GPT
Gemini
Claude
LLaVA-Vid
Hip circle
easy
anticlockwisee
4.5
easy
Ginnsss
lucky cal
62
1.6
6.5
-8.1
6.5
6.5
Squat Knee Raise
titness
side
43
5.6
10.2
-2..3
10.2
-3.7
fimess
Hip circle clockwise basketball
123
4.2
13.7
7.2
6.7
9.9
medium
ballsports.0
shot
96
-15
0.6
4.6
0.6
4,8
basketball
medium
ballsports.1
148
4.7
-2.7
-5.4
-07
4.1
mikan layup basketball
mediunn
bullsports_2
125
2.7
6.1
0.5
5.1
1.3
reverse layup
medium
ballsports_3
soccer
penalty
-7.1
1.4
-1.4
114
433
medium
diving
diving
240
0.9
-08
0.5
2.0
1.3
Opening and clusing step left side first
medium
1
186
4,0
1.3
-.99
225
-1.9
medium
Giness
car deadlift
137
499
6.4
5.7
8.9
2.3
Squat
medium
fitness_5
diagonal view
70
-7.1
-10.0
4,3
20.0
1.4
Opening and closing
medium
fitness
step right side first
155
-1.4
-1.4
89
-466
-1.4
hard
music 0
piano
94
-1.7
-1.7
5.7
.1.7
.0.6
hard
music_l
guitar
20
4.0
-11.0
-1.0
140
hard
sargery
Knoi Tying
237
2.6
-1.6
-1.6
1.4
4.8
surgery
Suturing
hard surgery.2 2
Needle Passing
211
2.2
-2.6
0.3
-0.7
0.8
25

---

Published as a conference paper at ICLR 2025
Table 13: Actions keys and their descriptions
Action
Action description
ballsports_0
a person is doing basketball mid-range jump shot, starting with the ball in their hand, no defense, practice only
ballsports_1
a person does the Basketball drill called the Mikan layup where they start under the basket, do layup with the right hand and catch it, do left hand layup and catch it, no defense, practice only
ballsports 2
a person playing basketball does reverse layup starting from left side of the basket and lays it up with their right hand on the right hand side, no defense. practice only
ballsports 3
person does soccer drill where they do single penalty kick, practice only, no defense, no goalic
diving Q
competitive diving from 10m
fitness_0
fitness exercise called standing hip circle with hands hips, one rotation anti- clockwise
fitness. 1
fitness exercise called opening and closing step on left side and then opening and closing step on right side
fitness 2
single free weight deadlift without any weight
fitness_3
fitness exercise called two arm standing lucky cat starting with arms up, one repetition
fitness 4
squat without weights, then knee raise left side a squat then knee raise on left side
fitness
fitness_6 6
fitness exercise called standing hip circle with hands on hips, one rotation clock- wise
fitness.7 7
fitness exercise called opening and closing step on right side and then opening and closing step on left side
music_0
person is playing scales on the piano
music_1
a person is playing scales on the guitar
surgery. O
The subject picks one end of a suture tied to flexible tube attached at its ends to the surface of the bench-top model, and ties single loop knot.
surgery_1
The subject picks up needle, proceeds the incision (designated as vertical line on bench-top model), and passes the needle through the fake tissue, entering at the dot marked on one side of the incision and exiting at the cor- responding dot marked on the other side of the incision. After the first needle pass, the subject extracts the needle out of the tissue, passes to the right hand and repeats the needle pass three more times.
surgery 2
The subject picks up the needle (in some cases not captured in the video) and passes through four small metal hoops from right to left. The hoops are at- tached small height above the surface of the bench-top model.
26

---

Published as a conference paper at ICLR 2025
Table Differenne--een accuracy scores VidDidf, 'action" values can looked up table columas indicate a pivalue 0:05 for the two tailed binomial significance test
The grayed
Split
Action
Difference Description
Mean
Num Sam- ples
p-value
finess. 6
the head remoins vertical during the recation
15
deaullift they gither the ball with both hands
guze feeward
medium
ballsports 2
0.5
hand
The player
plectrum.
0.125
medium
Ninee.. 7
the modion frater
0.93
14
0.011
Seedium
ballsperis
uses the noo shooting hand (the guide band) for stabilizing the ball duning the abot in the 1st sbot botiom deaditt
0.9
10
0.02
medhum
lines_2
knees
0.89
casy
liness 6
Bee locs are moore pointed out
0.88
16
0.0034
medium
ballspurrs..
higher the lirst shos
0.83
12
0.107
casy
6
Gie stance wider
0.83
medium
hiness_2
the lest stance is Wider
0.8
15
0.028
essy
easy
hinessJ)
the fect stunce is wider
0.8
25
0,003
meedium
balisports, 1
uses the mon-shooring hand (the guide hand) for stahilizing the ball during the aboo as the 2nd shot
088
10
0.088
msedium
the shooter's arm more exsended towards the basket
0.79
0.122
mediam
liness_7
the arms are elevated an uneves way on the first step
0.75
20
0.05
liard
supary.i
graser supports the right graspes. by pressing down lissue.
0.75
hallsports D
as shooter begins estending elhow shoot, the non-shooting hand (th guide hand) is on sidee
0.75
12
0.107
the ball and does not inffuence the balls trnjectccy
medium
ballsparts
the bull harder
0.75
12
0.107
medium
hallsports, 2
the non-jumping log has more elevated
0.73
15
0.183
uediium
hinesL2
the gaze Torward at the start the mation
0.75
11
0.322
hand
surpey_2.
(ocoloded
0.73
medium
hiness.7
Die arms reachs lighes on the nist step
0.71
17
0.189
hand
sungoy_2
The second gropper used to stabaline the torget.
0.69
0.093 26
medium
Roness
the hands are lower botton of deadlift
0.68
19
0.192
easy
Rines_] 1
the upper amss are porallel to the ground the start of the motion
0.68
19
0.192
medium
the
further closer foot during
step
0.67
easy
Rineesh 5
the nage mution the hips is larger
0.67
0.156
essy
limess 6
The bodly rocks more in forwand-backwwed
0.67
0.492
liness)
body more locked out
top
0.67
medium
liness I
the arms meads higher on the first step
0.67
18
0.245
medium
ballsporis
the ball is closer the
of the square the backhoard
0.87
0,092
easy
Rinces Q
The upper hody rocks more in forwoni-buckwwn way
0.67
18
0.343
medium
3
they pulase their hips more during the strike
0.67
18
0.243
mediumm
ballsporis_
facing ball leuding
0.67
hard
music
Rhythmic consistency is better mainained in Video A than in Video B.
0.87
12
0.387
eayy
linees
the
puinted
0.65
0,297 17
divangJ)
size volume the splisis created upon entry langer video
0.03
essy
finesij 6
fiee speed of hip cotezion is fister
0.64
25
0.122
medium
ballsperts.]
jump with
0.64
0,451
medium
diving_J JI
Diver enters the water at an ingle closer to degroes in video vidlon
0.63
30
0.161
mediuu
Riness.1
the arms are elevated
uneven on the second step
0.43
19
0,192
essy
mage mation
hips larger
0.63
hard
sargory D
The tiee in Video A moves more thas in Video B.
0.63
27
0.194
fimess4
the
derper, messured by angle of the thigh to the ground
0.63
0.344 16
essy
the toes are moore pointed
0.63
0.458
medium
lines__
the loes are more pointed outwards on the lins step out
0.63
0,458 13
hand
control the sevond step out wider
movement
Video A than Video B.
0.62
0.419
medium
linee_]
0.62
13
0,419
medium
fimess
the arms reach higher on the sacond stop
0.61
18
0.334
easy
linescD
remoins more vertical during the rotation
0.61
medium
ballsports
the won-jumping log has more elevated lonoe in the 2nd shot
0.6
0.625
medium
ballsporis
the shooker's
verlical than forward
0,6225
medium
ballsporis. 2
the guve mor up and forward indiradl of down on the 2nd last step
0.6
10
0.41
medium
balisperts
the arm is more fully estended lowards the basket in the tollow through in the 2nd shot
0.5
15
0,305
methum
ballsporis..
shooter's feet stance wider staring
modion
10
0.41
medium
ballsperts..
the body moves more forward dunig the 2nd shot, rather than up or back
0.6
10
0.41
medium
ballsporis
the not kicking loot planted closer to ball
10
0.41
J
tension
sututing materaal the uisse is beteer controtled Video than in B.
0.59
32
easy
6
the hand position higher on the body
0.58
12
0.451
mediuum
ballsports_
nou-kicking foot plamee
ball and less behind ball
0.58
12
0.451
mediuum
hallsports,]
they eatch in higher position en shot
0.58
12
easy
RinesLD)
the hand position higlter on the body
0.58
12
0.451
hand
misie_)
speed playing higher Video
Video
0.58
hard
sargar.1
The instummnt tigs are never out of view (oceladed by instruments, or out frane)
0.58
19
0.352
hanl
sumger.
The suturing spead is higher in Video A than in Vidoo B
0.58
52
0,157
hard
surgary
movement of noddle through the hoop more radiol Video A than in Video R.
0.58
26
0.288
bunl
singery.
The tession on the suturing ihread is lower in Video A thhm in Video B.
0.57
0.279 28
mediom
the elevated uneven way
stop
0.56
16
medium
linees_
the arms are elevoted uneven way on the lirst step
0.56
16
0.349
hand
sungory
The number of movemente to amango the neodle before threading is lower Video than in Video B.
0.56
0.223 41
sungety..
movement more Bud Video
B.
0.36
hand
singery
The grasper in Video is more quickly positioned on the neddle thin in Video B.
0.56
43
0.216
medium
hiness.
the
outwards the second step out
0.36
0.092
medium
dicing 0
Diver roloies forward rel relative to themselves.
0.56
0.299
mediuu
balisperts
they zei deeper line bend betore jumpung
0.50
0.892
easyy
(ines)))
speed hip rilaaion
0.55
80
0.32
medium
ballsports
thy jump higher on the second shot
0.53
0.451
liness_
the
pointed
0.3
13
0.419
isedium
ballsports.0
the shooter's feet position more staggored, mecning at ditferent distance from the basket
0.54
27

---

Published as a conference paper at ICLR 2025
isedium
diving B
Duration from jump off the hoard to water eatry longer in video than video
0.54
39
0.251
D
the Shooter's anees maoee best before laking
0.54
13
0.419
mediim
2
mey use the non-hhoiing hand (the guide hand) for stahilizung the ball during the shor
0.53
17
0.371
mediuum
hiness_3
the squit deeper, measured by of thish ground
0.55
17
0.371
hard
2
noodle is graspad closee to tip Video
0.52
0,336
hard
surgery
The force on the turget is lower Video A than in Video i
0.51
57
0.257
medium
diving_)
body
strigght video than vidko B.
0.51
medium
ballsporis.
they huve beiter bulance when landing on the shot
medium
diving 0
Speed at which divers rotate deriing the dive in lorger in video A than video
0.5
38
0.257
medium
liness.
arms reach higher the secund step
mediom
bullsports
they have better balance when landing on the 2nd shot
0.5
hand
music_
Smooth transitions between
mimmal disruption rhythm tempo. Transiioons Video
are smoother than in
hand
misic_]
Gniariist uses finger vibrato.
0.5
medium
ballsports_3
body totso)
facing
'square' the
12
medium
Rines_7
the step out wider
0.47
17
0.297
liard
sargary.o
The movements in Video are more procise than Viden B.
0.47
0.216
easy
spred the
mediom
hallsports
mey land on two feet
0.47
17
0.297
medium
ballsporis.2 2
follow tbrough
and lowards the busket
0,47
0.305 15
medium
Biness
the loes ane more pointol out
0.47
15
0.805
medium
1
Usere a pouse al the botiom of the deadlift
0.47
0.305 15
medium
step wider
0.46
15
hand
suppty
The necills inserted in the lahric more perpendicular to the incision.
0.46
13
0.314
medium
linex.
the
puinted outwards the finst
0.45
15
0.314
music
smocthoess
crossing mure
0,40
13
0.314
easy
finess 3
Bie upper amis more stable through the enture motion
0,40
0.314 13
dium
ballsperis..
jump off the
night hand shot (the 2nd shot) shoos, hall moree the right
0.43
0.322
medium
hallsports,
before raising the
hip
0,45
band
surpery.i
The torce applaed in more radial way in Video A than in Video B.
0.45
31
0.192
medium
Riness_7
the
further
foot during
step
0.44
18
medium
hines
the arms are in fromt of the body at the bottom of the deadlitt
0.44
0.328
hand
sargory.2
The presage of the neodle berween two honds more fluid in Viden A than Video B.
0.44
25
0,266
asedium
the stance
0,44
16
hand
misie
The wrist should be stragght and nor dipped or mised, facilitarngg fluid motion and avoiding stmin. The uppropriate Video than in Video B.
0.43
14
0.244
position
the entre motion taster
0.43
0.194
band
surgety.J J0
The movements in Videos A are faster than in Video
0.43
92
0.116
liard
surarry
More
Video than Video
0.42
hard
suppsy
The thread more efficlensly managed in Viceo A than in Video B.
0.82
0,156
meddium
liness_1
the locs are more pointed outwards on the second step
0.41
17
0.189
surpory.i
instrument upolies more to tissue and
Video than in
0.41
laad
sargesy D
The movements in Video are more efficient than in Video
0.4
0.076
med medium
ballsperts..
forward during the shonter's foot are oeiented more square to booket when starting the shooting motion, morsing
back
0,625
medium
hallsports
0.38
13
0.175
faet point moce forward
mediuum
bullsperts.j
shooier begins extending the elbew shoot, the ball
front the body, than behind
0.38
16
0,244
hund
simgery.
The dot is more accurately hit in Video A than in Video
0.36
14
0.122
band
The body closer the
Video than Video B.
0.35
0,094
medium
linee.. 3
the speed of the whole motion fister
0.35
0.094
meedium
balisports
they release the ball at higher position
0.35
0.148 20
mechum
hallsporss..
the non- non-jumping
more elevated the shot
hard
surpesy J
Both graspers are used efficieatiy.
0.33
0,141
hand
Video A stops often plan steps than the surgeon Video
0.33
0,25
hard
music
are more wrong note coneetions Video than Video
0.33
0.188
medium
finess_. I
the mocion faster
0.33
18
0.065
medium
hallsports.]
flaid mosion asoving between
0.33
12
0,147
hand
music_l
left fingers in Video A more curvel collapped than video
0.33
0.25
mediam
Riness.7 7
the tasso moves out further closer to the foot during the second out
0.35
0.141
medium
diving_)
water jump
0.32
19
mediom
finees. I
the firs step out wider
0.31
16
0.153
medium
3inees_
during the desentt the knees inwands, instead of tracking over the foel
0.3
20
0,074
medium
limess
the taso moves out forther closer foot during first step out
0.29
0.03
bund
sungary.l
The grasper grusps the needle opproximately 2/3 from the noodle lip. The needle grasped more precesely
0.29
0.006
Video than Video
essy
Riness,4
the stance
0.29
0.024
hand
masic_1
The unused left Binger tips Video stay closer to the board than in video
0.25
0.5
hand
surpory. D
thread tangles.
0.34
0.01
mediam hallsports.)
tie arm is more fully extended towards the basket in the tollow through the ist shot
0.18
11
0.011
hand
music_1
Fingers should press. strings the conter of the frees, aooddng the metal fiet hars for clear wound production.
0
0.5
Video shows
finger placement
fiethoard Video
hard
musicl
Only one hinger of the lett hands rests on string at time.
0.25
28

---

Published as a conference paper at ICLR 2025
Table 15: Evaluating "easy" split with variable video fps for three models. Our evaluation protocol chooses 4fps.
fps
2 fps
fps
fps
average
GPT-40
58.0
59.4
58.8
59.10
58.8
Gemini-1.5-Pro
59.7
66.9
65.8
66.9
64.8
Claude-3.5-Sonnet
58.1
58.5
56.6
52.9
56.5
G.3 RESULTS: DEPENDENCE ON FPS
The frame sampling rate, fps, is an important consideration for evaluating fine-grained actions. While typical video benchmarks like Video-MME Fu et al. (2024) sample videos lfps, we have sampled at a higher rate depending category. The categories with shorter videos were sampled at higher rate: 4fps for 'fitness', 5fps for 'ballsports' and 6fps for 'diving' (they are slightly dif- ferent they can be compatible with fps in the source dataset). We chose this relatively higher rate because we are interested in more fine- grained differences, while prior benchmarks are more coarse- grained; however we did not sample even higher due practical too many frames. The longer videos 'surgery' and 'music' were sampled at Ifps: these are longer videos where differences are discernible at lower sampling rates, and where the longer videos make high-fps sampling impractical.
constraints of processing
To show that our fps is reasonable, we tested the three closed-source models on range of fps levels on the "easy" subset of closed evaluation. We chose this set because this is where statistically significant differences were clear. The results are table 15
Across all models. the sampling rate that we use, 4 fps. has reasonable scores, either at or above the average over the other fps values. For all models, the variability is low: GPT's scores are within 0.8 points of the average; all other models have scores within 2.1 points of the average (except for the low sampling rate of 1fps Gemini, where degrades by 5.2 points). Moreover, the optimal fps is different for different models.
To help explain the results, we refer to the qualitative examples in the main results sections. The only 'success cases' for all our models were those having easy localization, and coarse differences. We hypothesize that fps is not important for these cases. Where fps is likely important - fine- ann-ggrined multiframe reasoning the current LMMs cannot perform better than random. So although 2fps currently has good performance, we believe that as LMMs improve, they will perform better on subtle motions and using higher fps will be important.
G.4 RESULTS: QWEN2-VLL OPEN EVALUATION
Qwen2-VL VL performs especially poorly in open evaluation, which we investigate here. The key issuc that Qwen2-VL-7h was failing to follow the evaluation prompt, while the other compared models did follow it. We sampled 3 video pairs for each action and manually inspected Qwen's responses, identifying multiple key issues. Below, we list each issue, and provide quantitative estimate for the prevalence of each issue.
(45% of differences) Proposing differences not relevant to *how to perform actions. but instead are visual things like "The person in video a is wearing a blue jacket, while the person in video b is wearing plaid shirt." We estimated prevalence by using a gpt-4o query that we manually prompt engineered.
(26% of differences) Proposing difference that actually not a difference, e.g. "The person in video is performing the exercise with their arms out to the sides, while the person video is performing the exercise with their arms out to the sides." We estimated prevalence by using gpt-4o query that manually prompt engineered.
(56% of differences) are repeated, meaning when trying to propose multiple differences, it proposes the same difference multiple times. We could directly measure this prevalence exactly.
29

---

Published as a conference paper at ICLR 2025
(23% of actions) Proposing only small number of differences less then half many as what is prompted for. We could directly measure this prevalence exactly.
(55% of differences) Proposing vague differences that are harder to interpret visually like "The player in video has more versatile and adaptable skill set than the player in video b". We estimated prevalence by using a gpt-4o query that we manually prompt engineered.
Overall, only 31.9% of proposed differences by Qwen did not suffer from any of these errors. (Note that some differences suffered from multiple errors at the same time)
G.S No MULTIPLE CHOICE BIAS IN CLOSED EVALUATION
In multiple choice benchmarking, models may be biased towards one particular option, which can impact evaluation robustness. We find no evidence of this. Firstly, in closed evaluation, the A/B ratio is 0,493/0.507. Second, we test the impact of video order on GPT-4o for the 'fitness' category, which has samples the easy and medium subsets(sample size 193). We test flipping the order of videos which flips the A/B answer. The performance is 54.8% in the original evaluation, and reversing the order of videos gives performance of 55.5%, showing a 0.7% difference. This result suggests that the performance VidDiftBench is not significantly sensitive video order.
G.6 EXPERIMENT ON DUPLICATING VIDEO
One idea to validate the reasonable-nee of the benchmark is to check what happens when passing an identical video as and B to the system we should expect that in closed evaluation, the predictions should be A/B 50% of the time. We did this experiment the closed setting, for the "easy" subset for GPT-4o. Over two random seeds, the results were 49.3 and 50.2. This is an interesting validation check that the benehmark passes.
G.7 No MULTIPLE CHOICE BIAS IN CLOSED EVALUATION
In multiple choice benchmarking, models may be biased towards particular option, which can impact evaluation robustness. We find no evidence of this. Firstly, in closed evaluation, the A/B ratio is 0.493/0.507. Second, we test the impact of video order on GPT-4o for the 'fitness' category, which has samples in the easy and medium subsets(sample size 193). We test flipping the order of videos which flips the A/B answer. The performance is 54.8% in the original evaluation, and reversing the order of videos gives performance of 55.5%, showing a 0.7% difference. This result suggests that the performance on VidDifiBench not significantly sensitive to video order.
G.8 SAMPLE RETRIEVALS
In our methods, we leverage frame localizer to find either a single frame or small sequence of frames. These localized frames are then passed to the next stage. In fig. 4 we show a few examples of predictions vs ground truth for some frames
H VIDDIFF METHOD
The code for VidDiff method is available at this GitHub repo httpss//ghhhbb.om//////hiiiitttt
Here show the prompts used in the different components.
Proposer stage Part chooses candidate differences (Open setting only):
I have two videos of action with the following description: "(action)".
Propose a set of 'differences' in how this action may be performed between the two videos.
For each difference, give a 'name' ddscrrppiior, 'query_string', and 'rumeframes".
The 1 "name" is very short name that can be used as key.
30

---

Published as a conference paper at ICLR 2025
"hhe arms reach higher the first stop"
"there a longer pause the bottom"
Frnd
Predicet
the higher with their right
"dlver laces the water
Figure 4: Sample frame localizations: prediction ground truth.
The 'description' is slightly longer description of the di fference.
The "query_string is the same as descriptior
The descriptions should be visual and about how action is performed. For example the jump higher', or 'the am is more straight'.
The difference descriptions should not refer to a specific video. For example you would not say "the jump in video B is higher'.
Instead, the 'description' would be 'the jumo is higher', and the statemet could be more true of one video or the other.
Now suppose you had to judge whether the di fffernnnce was stronger in one video, but you could only look at individus frames.
What's the smallest number frames you need, assuming the frame is well 1-chosen?
the answer in 'num frames' The answer should '1' or 'gt_11 (meaning 'greater than 1r).
Once you have the franes to coparre, create 'query 'query_string that is sirple statement about this difference that is more true in one video vs the other video (based on the frames only). For examlle "the left leg is higher" or "movement faster".
List (n infiiffereeees) di.fferences Return json like this, where the di fffrences keys are stringified ints starting from 0.
'0' !
"name" "...", "description" " "..." "query_string"
"num_frames": "1 [gt_1", 1,
Proposer part 2 estimates sub-action stages:
I have two videos of an action with the following description: "(action))"
list. These sub-actions that make up that action. Give steps or fewer in the action transcript.
Provide "stage transcript'
For each stage, give 'naee" for the stage, and a descrrpttion of that stage. each stage, gi ve 1ist retrieea" _sttiiiissttsst_._____
31

---

Published as a conference paper at ICLR 2025
These strings that describe what is visible in the frame.
Only describe the visual features. Only describe what is visible in single frame. Focus on appearance. Pocus on pose, Do not use the name of the action. Start each string with something similar to "A photo of a ...... Give at least (n in_retrieval_Ke0 _eys) retrieval strings stage.
Return a jaon like this:
"stages"
"name ""
"description" "...",
"retriewal_string"" ["A photo of a a..., ...l,
}}
And proposer part does linking between those stages
I have two videos of an action with the following description: "(action)".
Here 1ist of stages subactions that make up that action: (stages)
We also have differences in how this action may be performed between the two videos. The di fferences are speci fic statements that are more true in one video vs another. Here they are: (di fferences)
Now need to match each differences to stage. Return 1ist of the stages using their names. If difference is relevant a particular stage, put its name the 'difference 1ist
It's okay for fiffrrnncee to be visible in multiple stages. It's okay for some stages to have ne difference. Refer stages and fferences by their 'name! attribute.
Return a json like this:
"<stage name@> " I"differrece nameO>", Herence namel>", ...l,
"<stage_namee [],
"<stage_name?> iffrenofrree_nmmel Mddi fference_name2:: ...l,
Please be careful. Every fifrenncce must appear least
Frame Differencer
The prompt also takes the image frames.
have two videos of people performing an action with description: "factica)" The first (num frames) frames are from video A and the last (num franes) frames are from video B.
For each video, the frames very close together in the video: they are (time_diff) seconds apart.
Which one shows more of the variation with this description: "/query_stringg""
(a) video 1, (b) video 2, (c) similar or can't tell. Answer in ison: "'answerddetailed "..." 'answer' :a "b""""""
I COMPUTATIONAL COSTS FOR VIDDIFF METHOD
Our method's 's runtime less than one minute per video pair using an A6000 GPU for running CLIP inference Radford al. (002). Additionally, utilize the GPT API average cost of $0.2
32

---

Published as a conference paper at ICLR 2025
per sample AAcham al. (2023). Notably, over 95% of the GPT cost arises from verbose VLM responses. Attempts to prompt for shorter responses resulted in degraded performance.
Methodologically, rely pre-trained zero-shot models, which limits their applicability in spe- cialized domains, as discussed our results. For evaluation, the Open setting formulation neces- sitates an LLM in the evaluation pipeline. One challenge is the subjectivity in annotations: deter- mining which differences are relevant and what magnitude of difference is significant, though we thoroughly discuss this problem and mitigations.
33


Reviews:
[{"review_id": "OPuTIrlWlT", "reviewer": "Reviewer_nLbM", "summary": "This paper introduces Video Action Differencing, a novel task of identifying subtle differences between videos of the same action. It also introduces a new benchmark sourced from mutliple video datasets with new annatations. A new method is proposed for this new task with state-of-the-art performance.", "questions": "see above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed task has not been explored, which has key applications for some scenarios in real life.\n2. The construction process for the dataset is technically sound with different splits.\n3. The visulizations are clear and interesting.", "weaknesses": "Weakness and questions:\n1. Do the authors consider factors like fps for videos, which may impact the restults of answering questions like \"the speed of the arms is faster\" for distinguishing videos A and B.\n2. For the open-set benchmark, have the authors analyzed the reasons for why QWen2-VL performs so worse?\n3. Have the authors visualized the selected frames by the frame localizer compared to the ground-truth frames? What's about the effects of frame localizer compared to the ground-truth frames?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xAW3DtuWXi", "reviewer": "Reviewer_ThzX", "summary": "The authors introduce a method and dataset designed to compare subtle action differences across video pairs. Their method, VidDiff, uses a three-stage process to generate, localize, and verify these differences using multimodal models.\n\nMain Contributions:\n\n- A Dataset includes 557 video pairs across domains like sports, fitness, and surgery, annotated with over 4,700 fine-grained action differences. The dataset is designed to help models learn and evaluate nuanced action comparisons.\n- A Framework uses a three-step pipeline to identify differences: (1) generating potential differences with a language model, (2) aligning frames between videos using CLIP and Viterbi algorithms, and (3) verifying differences with a vision-language model.\n- A method is compared with leading multimodal models (e.g., GPT-4o and Gemini-1.5 Pro), showing improved performance in closed-set settings. This comparison also highlights challenges in current models for frame alignment and action detail recognition.", "questions": "Please refer to weakness for more details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The method is shown to outperform baseline large multimodal models by systematically isolating action differences through a three-stage approach, excelling in both closed and open settings.\n- The introduction of benchmark, with extensive annotations across varied domains (e.g., fitness, surgery, sports), provides a unique and structured dataset for fine-grained video action comparison.\n- Evaluations and ablation studies demonstrate the robustness and effectiveness of the method, especially in tasks that require precise frame alignment and action differentiation.\n- The proposed task and methods address real-world challenges in skill-based learning environments.", "weaknesses": "- The performance of the leading multimodal models on the dataset is not clearly demonstrated, and examples comparing success and failure cases across models would enhance understanding of their effectiveness.\n- The Difference Proposer stage’s reliance on large language models (LLMs) may introduce biases or inaccuracies, especially when generating action differences for complex or nuanced tasks. Providing more details on the generated proposer queries and their corresponding ground truth labels would enhance clarity.\n- Although the multi-stage approach is well-structured, it presents a risk of error propagation. Inaccuracies in early stages could impact the final outputs, potentially reducing overall reliability, particularly in the open-set task, where the Difference Proposer’s effectiveness is not fully evaluated.\n- While the paper introduces a detailed taxonomy for annotations, the reasonableness of some annotations remains unclear. For example, the “surgery_0” annotation includes the description \"the movements in video A are more efficient than in video B,\" which lacks a concrete definition and could be interpreted inconsistently even by human evaluators. Scaling this annotation approach to larger datasets or adapting it to new domains could also present significant challenges.\n- Minor issue: The table mentioned as Table 7 in Section 5.3 is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HqXyDMc7C2", "reviewer": "Reviewer_dA39", "summary": "In this paper, a new task called video action differencing is proposed which aims for models to be able to understand fine-grained differences between multiple videos of people performing the same action. A new benchmark dataset is collected, named VidDiffBench, which includes 5 categories from 4 different existing datasets. Annotations are collected from pairs of videos with statements given per pair of video based on the action (for example video A includes someone jumping higher than Video B for a lay-up shot). There are two main evaluation protocols for this task, a closed set setting, in which the model must predict A or B for each possible description, and a closed set setting in which the method must generate the description. A new method which combines stages named VidDiff is proposed which outperforms standard LMMs on the dataset.", "questions": "1. Does the sampling of pairs of videos mean that these might not contain the same action (see above)? Or the actions are sampled first to ensure a wide range of comparison difficulty over actions before video pairs are sampled within action?\n2. Were the annotators skilled/experts/knowledgeable in the actions that they were annotating? Or was this found to not be that important given the annotation task? Additionally, how many total annotators were used and were they renumerated for their time?\n3. Has the potential bias of the video pairs in the closed task been checked to ensure that naive performance should be 50% instead of video A (or B) occurring as the answer more than 50% of the time. Additionally, I would be interested to know if candidates which could be categorised as C (for insignificant differences) can be understood by the model as this would also increase the naive difficulty to 33% before taking into account a non-uniform distribution.\n4. The evaluation protocol for the open-set task seems like it could include errors/inconsistencies depending on the LLM output. Has there been any investigation into this and how much it differs per run and how much it aligns with a human? Currently, the prompts are also given in the appendix with little to no discussion as to why these prompts were chosen, if they were developed over multiple iterations to find the best performing prompt, etc.\n5. Did the easy/medium/hard classifications align with experts' opinions for each of the actions? It would be good to know the types of actions that are classed as easy/medium/hard as these are not present within the paper as far as I could tell. It's not clear why an LLM was chosen to do this task.\n6. Could more qualitative results and statistics be provided about the dataset? For example, there is very little in the paper regarding the retrieval task of localising the differences: How much of the video does the method need to localise? Are there any temporal biases regarding the timestamps from the videos? Additionally, under the closed set task, more statistics over the number of As, Bs, and Cs that have been annotated and included for each action would be interesting to see. Other statistics that feel like they are missing are the average length of each video (potentially broken down per category) as well as the total number of hours within the dataset.\n7. As a thought, has an experiment where the same video is duplicated and provided into the methods, would the output predictions (esp. for the closed set task) give a 50% response rate? Ideally, this is where a method could predict the difference is negligible also.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 4, "strengths": "* The new task of Video Action Differencing is an interesting new task for video understanding, forcing models to recognise and understand fine-grained differences between two very similar videos.\n* The collected dataset combines four datasets with 5 different categories of video, providing a varied test bed for this new task.\n* The proposed method performs well on the dataset, outperforming off the shelf LMMs on the task yet still showcase that there is a lot still to work on in this area for future work.", "weaknesses": "# Weaknesses\n\n* There are some missing references for skill determination within the related work [a, b, c, d] as another example of fine-grained differences between videos containing the same action.\n* Line 196: It is mentioned here in the text that *\"Video pairs are randomly sampled within each dataset to ensure a wide range of comparison difficulty, from simple actions to more advanced tasks requiring fine-grained understanding\"* This implies that videos of differing actions are compared against one another. \n* Section 3.3.2: There are some missing information about the annotators, regarding skill level, total number, renumeration etc.\n* For the closed set, a binary classification setup was used as all candidate difference statements which is mentioned to be unbiased on Line 298. However, has this been checked? If videos are not randomly swapped at inference/training time there could have been a bias towards one video or another.\n* The open set evaluation seems like it could be prone to some errors/inconsistencies depending on the LLM chosen and how much it could hallucinate/not understand the task and doesn't represent a potentially sound evaluation protocol.\n* It is not clear within the paper as to why an LLM was used to choose the easy/medium/hard splits for each of the actions.\n* This paper did not feel like an easy read, whilst the grammar/sentence clarity was good. There was a lot of information that is split across the main paper and the appendix which necessitates jumping between them. The structure of the paper could also be improved, the method details occur within the experiments yet are given as a main contribution within the introduction with only a small amount of space given to explain the model. Another major factor for this is that details of the dataset are given before the task is formally defined, which given this is a new task, makes it harder to read than it should be.\n\n# Additional Comments\nLine 158 is referring to the wrong table, this should be Table 1\nLine 1040 (in supp.) vary -> very\nSection D.1 in the appendix is empty, maybe D.2 is meant to be a subheading of D.1?\nFor results tables, it would be good to include a random performance row.\n\n# References\n[a] Doughty, Hazel, Dima Damen, and Walterio Mayol-Cuevas. \"Who's better? who's best? pairwise deep ranking for skill determination.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n\n[b] Doughty, Hazel, Walterio Mayol-Cuevas, and Dima Damen. \"The pros and cons: Rank-aware temporal attention for skill determination in long videos.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n\n[c] Pan, Jia-Hui, Jibin Gao, and Wei-Shi Zheng. \"Adaptive action assessment.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 44.12 (2021): 8779-8795.\n\n[d] Zhang, Shao-Jie, et al. \"Adaptive stage-aware assessment skill transfer for skill determination.\" IEEE Transactions on Multimedia (2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qqzm3PvjdP", "reviewer": "Reviewer_8a8a", "summary": "This paper introduces the first large-scale video action differencing dataset, presenting a novel task of identifying differences between videos depicting the same action. The authors compile over 500 video pairs from existing datasets across five categories: Fitness, Ball Sports, Diving, Music, and Surgery. These videos are then assigned to annotators along with 147 distinct descriptions. Annotators must indicate which video (A or B) most closely aligns with each description. For example, given two videos of different actors performing a squat, a description might read \"deeper squat,\" and the annotator would select A or B based on which video demonstrates the deeper squat. To ensure dataset quality, 25% of the initial annotations undergo re-annotation, revealing a very low discrepancy rate. The dataset also includes action localization (pinpointing where the action occurs in the video) and specific key points for each action (e.g., when knees start to bend).\n\nThe authors also develop an agentic model called VidDiff to address the action differencing challenge. VidDiff employs several Large Language Models (LLMs) and Vision Language Models (VLMs) as agents to solve specific aspects of the problem: proposing potential differences based on the action description, localizing frames where such actions might occur, and finally specifying which video (A or B) corresponds to the observed difference. VidDiff outperforms other zero-shot VLMs in this task.\n\nLastly, the authors provide ablation experiments that highlight the challenges presented by their new benchmark.", "questions": "See weaknesses, plus the following:\n\n- Which LLMs/VLMs are used for the *Difference Proposer* and *Action Differencer*?\n- How does the benchmark handle cases of inverse correlation? For example, would *lower squat in video A* be equivalent to *higher squat in video B*?\n- Since the videos are not curated, factors such as different camera angles, varying FPS, or differences in the actor's height could introduce biases in the annotations and results. How do the authors address these potential biases?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "### Originality\n- **A novel task**: This paper introduces the new task of video action differencing with natural language. While related tasks, such as difference captioning, have been explored to provide a coarse comparison between videos, no prior work has tackled video action differencing in the same way—focusing on fine-grained differences described in natural language.\n- **A challenging benchmark**: The proposed benchmark, VidDiffBench, is comprehensive, covering five categories of instructional videos. It has proven to be highly challenging, even for top-performing closed-source vision-language models (VLMs).\n- **An agent-based system**: The paper presents an agent-based system that decomposes the task, achieving better performance than existing VLMs.\n### Clarity\nThe flow of ideas is straightforward, making the paper easy to follow and understand.\n\n### Significance\nThe paper convincingly demonstrates the importance of video action differencing, and the introduction of the new benchmark is likely to inspire further research in this area.", "weaknesses": "### Unproven claims\n- In the introduction, the authors claim they will address the challenges of *precise temporal alignment and the need for fine-grained understanding of action dynamics*. However, it remains unclear how they specifically solve the issue of temporal alignment. Could you elaborate on how you solve this issue or point us to the location where it is addressed?\n\n### Benchmark and results\n- Similar datasets are presented in the related work section; however, since this work is primarily a benchmark paper, more comparisons with existing benchmarks would be make the differences clearer (e.g., similar to Table 1 but with other datasets in the first column). Consider adding what is unique about each dataset and how the current dataset differs.\n- As a benchmark paper, we would expect more results from other open-source VLMs (especially those addressing video data such as LLaVA-video) to better understand their limitations and make it easier for other researchers to work with this benchmark. \n\n### Clarity\n- 557 or 656 video pairs? In the abstract, the authors state that the dataset contains *557 video pairs...  4,719 fine-grained action differences* (line 013-014), but on line 260, they mention *656 video pairs, 5,580 annotated differences*. Clarification needed on which is correct. \n- Figure 1: The distinction between the first and second row is unclear, yet the caption claims these represent two different challenges. These two challenges are not discussed elsewhere in the paper and don't seem to be related to the dataset splits. Please clarify this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)