Video Action Differencing

Authors: James Burgess, Xiaohan Wang, Yuhui Zhang, Anita Rau, Alejandro Lozano, Lisa Dunlap, Trevor Darrell, Serena Yeung-Levy
Primary area: datasets and benchmarks
Keywords: Video, Actions, Differencing, Zero-shot, benchmark, multimodal, lmm, llm

Abstract:
How do two individuals differ when performing the same action? In this work, we introduce Video Action Differencing (VidDiff), the novel task of identifying subtle differences between videos of the same action, which has numerous applications, such as coaching and skill learning. To enable development on this new task, we first create VidDiffBench, a benchmark dataset containing 549 video pairs, with human annotations of 4,469 fine-grained action differences and 2,075 timestamps indicating where these differences occur. Our experiments demonstrate that VidDiffBench poses a significant challenge for state-of-the-art large multimodal models (LMMs), such as GPT-4o and Qwen2-VL. By analyzing the failure cases of LMMs on VidDiffBench, we highlight two key challenges for this task: localizing relevant sub-actions over two videos and fine-grained frame comparison. To overcome these, we propose the VidDiff method, an agentic workflow that breaks the task into three stages: action difference proposal, keyframe localization, and frame differencing, each stage utilizing specialized foundation models. To encourage future research in this new task, we release the benchmark and code.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729222996269

thread_type:
official_review

<thread_object>
creator: reviewer_8a8a

<content>
summary:
This paper introduces the first large-scale video action differencing dataset, presenting a novel task of identifying differences between videos depicting the same action. The authors compile over 500 video pairs from existing datasets across five categories: Fitness, Ball Sports, Diving, Music, and Surgery. These videos are then assigned to annotators along with 147 distinct descriptions. Annotators must indicate which video (A or B) most closely aligns with each description. For example, given two videos of different actors performing a squat, a description might read "deeper squat," and the annotator would select A or B based on which video demonstrates the deeper squat. To ensure dataset quality, 25% of the initial annotations undergo re-annotation, revealing a very low discrepancy rate. The dataset also includes action localization (pinpointing where the action occurs in the video) and specific key points for each action (e.g., when knees start to bend).

The authors also develop an agentic model called VidDiff to address the action differencing challenge. VidDiff employs several Large Language Models (LLMs) and Vision Language Models (VLMs) as agents to solve specific aspects of the problem: proposing potential differences based on the action description, localizing frames where such actions might occur, and finally specifying which video (A or B) corresponds to the observed difference. VidDiff outperforms other zero-shot VLMs in this task.

Lastly, the authors provide ablation experiments that highlight the challenges presented by their new benchmark.

strengths:
### Originality
- **A novel task**: This paper introduces the new task of video action differencing with natural language. While related tasks, such as difference captioning, have been explored to provide a coarse comparison between videos, no prior work has tackled video action differencing in the same way—focusing on fine-grained differences described in natural language.
- **A challenging benchmark**: The proposed benchmark, VidDiffBench, is comprehensive, covering five categories of instructional videos. It has proven to be highly challenging, even for top-performing closed-source vision-language models (VLMs).
- **An agent-based system**: The paper presents an agent-based system that decomposes the task, achieving better performance than existing VLMs.
### Clarity
The flow of ideas is straightforward, making the paper easy to follow and understand.

### Significance
The paper convincingly demonstrates the importance of video action differencing, and the introduction of the new benchmark is likely to inspire further research in this area.

weaknesses:
### Unproven claims
- In the introduction, the authors claim they will address the challenges of *precise temporal alignment and the need for fine-grained understanding of action dynamics*. However, it remains unclear how they specifically solve the issue of temporal alignment. Could you elaborate on how you solve this issue or point us to the location where it is addressed?

### Benchmark and results
- Similar datasets are presented in the related work section; however, since this work is primarily a benchmark paper, more comparisons with existing benchmarks would be make the differences clearer (e.g., similar to Table 1 but with other datasets in the first column). Consider adding what is unique about each dataset and how the current dataset differs.
- As a benchmark paper, we would expect more results from other open-source VLMs (especially those addressing video data such as LLaVA-video) to better understand their limitations and make it easier for other researchers to work with this benchmark. 

### Clarity
- 557 or 656 video pairs? In the abstract, the authors state that the dataset contains *557 video pairs...  4,719 fine-grained action differences* (line 013-014), but on line 260, they mention *656 video pairs, 5,580 annotated differences*. Clarification needed on which is correct. 
- Figure 1: The distinction between the first and second row is unclear, yet the caption claims these represent two different challenges. These two challenges are not discussed elsewhere in the paper and don't seem to be related to the dataset splits. Please clarify this.

questions:
See weaknesses, plus the following:

- Which LLMs/VLMs are used for the *Difference Proposer* and *Action Differencer*?
- How does the benchmark handle cases of inverse correlation? For example, would *lower squat in video A* be equivalent to *higher squat in video B*?
- Since the videos are not curated, factors such as different camera angles, varying FPS, or differences in the actor's height could introduce biases in the annotations and results. How do the authors address these potential biases?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730450431936

thread_type:
official_review

<thread_object>
creator: reviewer_da39

<content>
summary:
In this paper, a new task called video action differencing is proposed which aims for models to be able to understand fine-grained differences between multiple videos of people performing the same action. A new benchmark dataset is collected, named VidDiffBench, which includes 5 categories from 4 different existing datasets. Annotations are collected from pairs of videos with statements given per pair of video based on the action (for example video A includes someone jumping higher than Video B for a lay-up shot). There are two main evaluation protocols for this task, a closed set setting, in which the model must predict A or B for each possible description, and a closed set setting in which the method must generate the description. A new method which combines stages named VidDiff is proposed which outperforms standard LMMs on the dataset.

strengths:
* The new task of Video Action Differencing is an interesting new task for video understanding, forcing models to recognise and understand fine-grained differences between two very similar videos.
* The collected dataset combines four datasets with 5 different categories of video, providing a varied test bed for this new task.
* The proposed method performs well on the dataset, outperforming off the shelf LMMs on the task yet still showcase that there is a lot still to work on in this area for future work.

weaknesses:
# Weaknesses

* There are some missing references for skill determination within the related work [a, b, c, d] as another example of fine-grained differences between videos containing the same action.
* Line 196: It is mentioned here in the text that *"Video pairs are randomly sampled within each dataset to ensure a wide range of comparison difficulty, from simple actions to more advanced tasks requiring fine-grained understanding"* This implies that videos of differing actions are compared against one another. 
* Section 3.3.2: There are some missing information about the annotators, regarding skill level, total number, renumeration etc.
* For the closed set, a binary classification setup was used as all candidate difference statements which is mentioned to be unbiased on Line 298. However, has this been checked? If videos are not randomly swapped at inference/training time there could have been a bias towards one video or another.
* The open set evaluation seems like it could be prone to some errors/inconsistencies depending on the LLM chosen and how much it could hallucinate/not understand the task and doesn't represent a potentially sound evaluation protocol.
* It is not clear within the paper as to why an LLM was used to choose the easy/medium/hard splits for each of the actions.
* This paper did not feel like an easy read, whilst the grammar/sentence clarity was good. There was a lot of information that is split across the main paper and the appendix which necessitates jumping between them. The structure of the paper could also be improved, the method details occur within the experiments yet are given as a main contribution within the introduction with only a small amount of space given to explain the model. Another major factor for this is that details of the dataset are given before the task is formally defined, which given this is a new task, makes it harder to read than it should be.

# Additional Comments
Line 158 is referring to the wrong table, this should be Table 1
Line 1040 (in supp.) vary -> very
Section D.1 in the appendix is empty, maybe D.2 is meant to be a subheading of D.1?
For results tables, it would be good to include a random performance row.

# References
[a] Doughty, Hazel, Dima Damen, and Walterio Mayol-Cuevas. "Who's better? who's best? pairwise deep ranking for skill determination." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[b] Doughty, Hazel, Walterio Mayol-Cuevas, and Dima Damen. "The pros and cons: Rank-aware temporal attention for skill determination in long videos." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[c] Pan, Jia-Hui, Jibin Gao, and Wei-Shi Zheng. "Adaptive action assessment." IEEE Transactions on Pattern Analysis and Machine Intelligence 44.12 (2021): 8779-8795.

[d] Zhang, Shao-Jie, et al. "Adaptive stage-aware assessment skill transfer for skill determination." IEEE Transactions on Multimedia (2023)

questions:
1. Does the sampling of pairs of videos mean that these might not contain the same action (see above)? Or the actions are sampled first to ensure a wide range of comparison difficulty over actions before video pairs are sampled within action?
2. Were the annotators skilled/experts/knowledgeable in the actions that they were annotating? Or was this found to not be that important given the annotation task? Additionally, how many total annotators were used and were they renumerated for their time?
3. Has the potential bias of the video pairs in the closed task been checked to ensure that naive performance should be 50% instead of video A (or B) occurring as the answer more than 50% of the time. Additionally, I would be interested to know if candidates which could be categorised as C (for insignificant differences) can be understood by the model as this would also increase the naive difficulty to 33% before taking into account a non-uniform distribution.
4. The evaluation protocol for the open-set task seems like it could include errors/inconsistencies depending on the LLM output. Has there been any investigation into this and how much it differs per run and how much it aligns with a human? Currently, the prompts are also given in the appendix with little to no discussion as to why these prompts were chosen, if they were developed over multiple iterations to find the best performing prompt, etc.
5. Did the easy/medium/hard classifications align with experts' opinions for each of the actions? It would be good to know the types of actions that are classed as easy/medium/hard as these are not present within the paper as far as I could tell. It's not clear why an LLM was chosen to do this task.
6. Could more qualitative results and statistics be provided about the dataset? For example, there is very little in the paper regarding the retrieval task of localising the differences: How much of the video does the method need to localise? Are there any temporal biases regarding the timestamps from the videos? Additionally, under the closed set task, more statistics over the number of As, Bs, and Cs that have been annotated and included for each action would be interesting to see. Other statistics that feel like they are missing are the average length of each video (potentially broken down per category) as well as the total number of hours within the dataset.
7. As a thought, has an experiment where the same video is duplicated and provided into the methods, would the output predictions (esp. for the closed set task) give a 50% response rate? Ideally, this is where a method could predict the difference is negligible also.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730471545126

thread_type:
official_review

<thread_object>
creator: reviewer_thzx

<content>
summary:
The authors introduce a method and dataset designed to compare subtle action differences across video pairs. Their method, VidDiff, uses a three-stage process to generate, localize, and verify these differences using multimodal models.

Main Contributions:

- A Dataset includes 557 video pairs across domains like sports, fitness, and surgery, annotated with over 4,700 fine-grained action differences. The dataset is designed to help models learn and evaluate nuanced action comparisons.
- A Framework uses a three-step pipeline to identify differences: (1) generating potential differences with a language model, (2) aligning frames between videos using CLIP and Viterbi algorithms, and (3) verifying differences with a vision-language model.
- A method is compared with leading multimodal models (e.g., GPT-4o and Gemini-1.5 Pro), showing improved performance in closed-set settings. This comparison also highlights challenges in current models for frame alignment and action detail recognition.

strengths:
- The method is shown to outperform baseline large multimodal models by systematically isolating action differences through a three-stage approach, excelling in both closed and open settings.
- The introduction of benchmark, with extensive annotations across varied domains (e.g., fitness, surgery, sports), provides a unique and structured dataset for fine-grained video action comparison.
- Evaluations and ablation studies demonstrate the robustness and effectiveness of the method, especially in tasks that require precise frame alignment and action differentiation.
- The proposed task and methods address real-world challenges in skill-based learning environments.

weaknesses:
- The performance of the leading multimodal models on the dataset is not clearly demonstrated, and examples comparing success and failure cases across models would enhance understanding of their effectiveness.
- The Difference Proposer stage’s reliance on large language models (LLMs) may introduce biases or inaccuracies, especially when generating action differences for complex or nuanced tasks. Providing more details on the generated proposer queries and their corresponding ground truth labels would enhance clarity.
- Although the multi-stage approach is well-structured, it presents a risk of error propagation. Inaccuracies in early stages could impact the final outputs, potentially reducing overall reliability, particularly in the open-set task, where the Difference Proposer’s effectiveness is not fully evaluated.
- While the paper introduces a detailed taxonomy for annotations, the reasonableness of some annotations remains unclear. For example, the “surgery_0” annotation includes the description "the movements in video A are more efficient than in video B," which lacks a concrete definition and could be interpreted inconsistently even by human evaluators. Scaling this annotation approach to larger datasets or adapting it to new domains could also present significant challenges.
- Minor issue: The table mentioned as Table 7 in Section 5.3 is missing.

questions:
Please refer to weakness for more details.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730536827043

thread_type:
official_review

<thread_object>
creator: reviewer_nlbm

<content>
summary:
This paper introduces Video Action Differencing, a novel task of identifying subtle differences between videos of the same action. It also introduces a new benchmark sourced from mutliple video datasets with new annatations. A new method is proposed for this new task with state-of-the-art performance.

strengths:
1. The proposed task has not been explored, which has key applications for some scenarios in real life.
2. The construction process for the dataset is technically sound with different splits.
3. The visulizations are clear and interesting.

weaknesses:
Weakness and questions:
1. Do the authors consider factors like fps for videos, which may impact the restults of answering questions like "the speed of the arms is faster" for distinguishing videos A and B.
2. For the open-set benchmark, have the authors analyzed the reasons for why QWen2-VL performs so worse?
3. Have the authors visualized the selected frames by the frame localizer compared to the ground-truth frames? What's about the effects of frame localizer compared to the ground-truth frames?

questions:
see above
</content>
</thread_object>

</thread>
<thread>
creation date:
1732432426968

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of all reviewer responses (1/2)

comment:
We thank the reviewers for their valuable feedback and for recognizing the novelty of our work as well as the contributions of our proposed task and benchmark. The reviewers provided many constructive suggestions, which we have carefully addressed. In response, we conducted several new experiments and revised both the main paper and supplementary materials. All text changes are highlighted in blue for clarity. We believe we have thoroughly addressed each point raised in the reviews and significantly improved the manuscript as a result.

## Summary of most important new results
We now briefly summarize the most important new results

### More benchmarking
We added more models to our evaluation baselines: Claude-3.5-Sonnet, and the recently released LLaVA-Video-7B. Here are the updated results for closed evaluation, showing that LLaVA-Video exceeds Qwen2VL-7B, while Claude outperforms open-source models, but not GPT and Gemini.

|                       | Easy  | Med   | Hard  | Avg   |
|-----------------------|-------|-------|-------|-------|
| **GPT-4o**            | 58.8% | 53.0% | 50.1% | 54.0% |
| **Gemini-1.5-Pro**    | 65.8% | 51.9% | 49.8% | 55.8% |
| **Claude-3.5-Sonnet** | 56.6% | 53.5% | 48.3% | 52.8% |
| **LLaVA-Video**       | 56.6% | 52.0% | 48.3% | 52.3% |
| **Qwen2VL-7B**        | 49.0% | 52.6% | 49.6% | 50.4% |
| **VidDiff (ours)**    | 65.3% | 55.4% | 50.4% | 57.0% |

### Verification of open eval
Our open evaluation setting requires matching ground-truth difference strings to predicted difference strings, and we propose to use an LLM for that purpose. We added two experiments to verify the trustworthiness of this approach. 

First, it is robust to random seed: over 5 runs, the standard-deviation was 0.7 points.

Second, we recruited 3 human annotators to perform matching, and then we computed inter-annotator agreement scores:

|             | LLM  | human 1 | human 2 | human 3 |
|-------------|------|---------|---------|---------|
| **LLM**     |      |    72.4 |    74.0 |    70.1 |
| **human 1** | 72.4 |         |    75.0 |    78.2 |
| **human 2** | 74.0 |    75.0 |         |    73.9 |
| **human 3** | 70.1 |    78.2 |    73.9 |         |
| **avg**     | 72.2 |    75.2 |    74.3 |    74.0 |

This shows that (i) the matching task is challenging, with mean agreement amongst humans at 75.7%, and (ii) LLMs have comparable agreement to the humans at 72.2%. 

This supports that LLM-based matching is reasonable, allowing us to enjoy the benefits of automatic evaluation that is consistent and reproducible.

### LLMs in assigning splits
We chose LLMs for determining easy/medium/hard difficulty splits because they have action understanding, and because this avoids biasing towards either current models or towards the opinions of authors. We recruited 3 humans to rank the actions by difficulty, and computed the Spearman rank correlation between the LLM and all humans:

|             | LLM   | human 1 | human 2 | human 3 |
|-------------|-------|---------|---------|---------|
| **LLM**     |       |   0.531 |   0.680 |   0.806 |
| **human 1** | 0.531 |         |   0.459 |   0.645 |
| **human 2** | 0.680 |   0.459 |         |   0.703 |
| **human 3** | 0.806 |   0.645 |   0.703 |         |
| **avg**     | 0.673 |   0.545 |   0.614 |   0.718 |


The mean of the pairwise correlations between the humans was 0.602, while the mean of pairwise correlations between the LLM and humans was higher at 0.673. This shows (i) that there is non-negligible variability in human rankings, and (ii) that the LLM ranking is reasonable, and actually better correlated with most humans compared to several of the human annotations.

### Baseline model comparison
The main results table compares the different models at different splits, but for a more fine-grained comparison, we computed the accuracy on a per-action basis. Given these scores, we then computed the correlations between the models:


|                       | GPT   | Gemini | Claude | LLava-Video | Qwen2-VL |
|-----------------------|-------|--------|--------|-------------|----------|
| **GPT-4o**            |       |  0.152 |  0.375 |       0.243 |    0.273 |
| **Gemini-1.5-Pro**    | 0.152 |        |  0.215 |       0.111 |    0.223 |
| **Claude-3.5-Sonnet** | 0.375 |  0.215 |        |       0.261 |    0.220 |
| **LLaVA-Video**       | 0.243 |  0.111 |  0.261 |             |    0.376 |
| **Qwen2-VL-7b**       | 0.273 |  0.223 |  0.220 |       0.376 |          |

The correlations are generally low, but there are 3 clusters of models. LLaVA-Video and Qwen2-VL are in a cluster; they are both open-source, and have the same LLM backbone. Then GPT-4o and Claude-Sonnet cluster together, and Gemini is not similar to any other model. We can speculate that for video data, Claude and GPT have similar training strategies, while Gemini’s is different.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733196493024

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author summary of the discussion period.

comment:
Thanks again to all the reviewers for engaging thoughtfully during the discussion period. The comments raised many interesting points, and have greatly improved the paper. The main paper and supplementary pdfs have been updated, with changes marked in blue.

Our paper introduces Video Action Differencing with three key contributions: a novel task, a benchmark (VidDiffBench), and a multi-stage method (VidDiff).

During the rebuttal period, we added new experiments and improved the writing to address the reviewer’s concerns. The most significant were:
- Improved breadth of benchmarking by adding new large multimodal models (LMMs), specifically Claude-3.5-Sonnet and LLaVA-Video.
- Validated the use of LLMs in the evaluation protocol for the ‘ open-set' setting, using human studies and robustness tests.
- On dataset quality, showed that there is negligible label bias, removed a small set of potentially ambiguous action differences, and added text to the manuscript describing dataset statistics and comparisons to prior datasets.
- Better justified the motivation for the task, especially arguing that language is a natural way to receive feedback in skill learning.
- Supported the benchmark split into easy/medium/hard using a human study.
- Added ablation studies over frame sampling rates (fps) and showed the robustness of our design choices across multiple LMMs.
- Deeper analysis into the failure cases of QwenVL-2 in open evaluation, finding issues with instruction-following.
- Restructured the paper's organization and expanded related work sections to better position our contributions in the context of video-pair datasets

As a result of these discussions, all reviewers concluded the open discussion period with scores above the acceptance threshold.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a new task and a benchmark for action differencing, to tell the differences between different actors performing the same action. Various baselines (included latest LLM-based baselines) are compared with the proposed approach, VidDiff. 

All reviewers appreciate the novelty of the task and recommend to accept the paper. Most of the discussions are about further clarifications and asking additional ablations and analysis. The author(s) did a great job in providing additional experiment as well as further elaborate on motivation and insights which help to convince all reviewers to support accepting the paper. The area chair reads all reviews and discussions and agrees with the reviewers, thus recommends an acceptance.