Sharingan: Extract User Action Sequence from Desktop Recordings
Authors: name: Yanting Chen; name: Yi Ren; name: Xiaoting Qin; name: Jue Zhang; name: Kehong Yuan; name: Lu Han; name: Qingwei Lin; name: Dongmei Zhang; name: Saravan Rajmohan; name: Qi Zhang
Published: 2024-11-13 16:53:29

Abstract: Video recordings of user activities, particularly desktop recordings, offer a
rich source of data for understanding user behaviors and automating processes.
However, despite advancements in Vision-Language Models (VLMs) and their
increasing use in video analysis, extracting user actions from desktop
recordings remains an underexplored area. This paper addresses this gap by
proposing two novel VLM-based methods for user action extraction: the Direct
Frame-Based Approach (DF), which inputs sampled frames directly into VLMs, and
the Differential Frame-Based Approach (DiffF), which incorporates explicit
frame differences detected via computer vision techniques. We evaluate these
methods using a basic self-curated dataset and an advanced benchmark adapted
from prior work. Our results show that the DF approach achieves an accuracy of
70% to 80% in identifying user actions, with the extracted action sequences
being re-playable though Robotic Process Automation. We find that while VLMs
show potential, incorporating explicit UI changes can degrade performance,
making the DF approach more reliable. This work represents the first
application of VLMs for extracting user action sequences from desktop
recordings, contributing new methods, benchmarks, and insights for future
research.