MotionSight: Boosting Fine-Grained Motion Understanding in Multimodal
  LLMs
Authors: name: Yipeng Du, name: Tiehan Fan, name: Kepan Nan, name: Rui Xie, name: Penghao Zhou, name: Xiang Li, name: Jian Yang, name: Zhenheng Yang, name: Ying Tai

Despite advancements in Multimodal Large Language Models (MLLMs), their
proficiency in fine-grained video motion understanding remains critically
limited. They often lack inter-frame differencing and tend to average or ignore
subtle visual cues. Furthermore, while visual prompting has shown potential in
static images, its application to video's temporal complexities, particularly
for fine-grained motion understanding, remains largely unexplored. We
investigate whether inherent capability can be unlocked and boost MLLMs' motion
perception and enable distinct visual signatures tailored to decouple object
and camera motion cues. In this study, we introduce MotionSight, a novel
zero-shot method pioneering object-centric visual spotlight and motion blur as
visual prompts to effectively improve fine-grained motion understanding without
training. To convert this into valuable data assets, we curated MotionVid-QA,
the first large-scale dataset for fine-grained video motion understanding, with
hierarchical annotations including SFT and preference data, {\Theta}(40K) video
clips and {\Theta}(87K) QAs. Experiments show MotionSight achieves
state-of-the-art open-source performance and competitiveness with commercial
models. In particular, for fine-grained motion understanding we present a novel
zero-shot technique and a large-scale, high-quality dataset. All the code and
annotations will be publicly available.