ICLR 2025

Published as a conference paper at ICLR 2025
VIDEO ACTION DIFFERENCING
James Burgess1, Xiaohan Wang1, Yuhui Zhang1, Anita Rau1, Alejandro Lozano1,
Lisa Dunlap2, Trevor Darrell2, Serena Yeung-Levy1
1Stanford, 2 UC Berkeley
ABSTRACT
How do two individuals differ when performing the same action? In this work, we
introduce Video Action Differencing (VidDiff), the novel task of identifying subtle
differences between videos of the same action, which has many applications, such
as coaching and skill learning. To enable development on this new task, we first
create VidDiffBench, a benchmark dataset containing 549 video pairs, with human
annotations of 4,469 fine-grained action differences and 2,075 localization times-
tamps indicating where these differences occur. Our experiments demonstrate that
VidDiffBench poses a significant challenge for state-of-the-art large multimodal
models (LMMs), such as GPT-4o and Qwen2-VL. By analyzing failure cases of
LMMs on VidDiffBench, we highlight two key challenges for this task: localiz-
ing relevant sub-actions over two videos and fine-grained frame comparison. To
overcome these, we propose the VidDiff method, an agentic workflow that breaks
the task into three stages: action difference proposal, keyframe localization, and
frame differencing, each stage utilizing specialized foundation models. To en-
courage future research in this new task, we release the benchmark1 and code2.
1
INTRODUCTION
The ability to compare two videos of the same action and discern their detailed differences plays a
critical role in a wide variety of applications. For instance, in fitness coaching, a novice learning
to perform a barbell squat typically watches instructional videos and then compares their actions
in a recorded video to identify discrepancies between their movements and those of an expert. In
medical training, junior surgeons compare videos of themselves performing surgical procedures with
reference videos from experts to identify errors and imp...