IF-VidCap: Can Video Caption Models Follow Instructions?
Authors: name: Shihao Li, name: Yuanxing Zhang, name: Jiangtao Wu, name: Zhide Lei, name: Yiwen He, name: Runzhe Wen, name: Chenxi Liao, name: Chengkang Jiang, name: An Ping, name: Shuo Gao, name: Suhan Wang, name: Zhaozhou Bian, name: Zijun Zhou, name: Jingyi Xie, name: Jiayi Zhou, name: Jing Wang, name: Yifan Yao, name: Weihao Xie, name: Yingshui Tan, name: Yanghai Wang, name: Qianqian Xie, name: Zhaoxiang Zhang, name: Jiaheng Liu
Organization: NJU-LINK Lab

Although Multimodal Large Language Models (MLLMs) have demonstrated
proficiency in video captioning, practical applications require captions that
follow specific user instructions rather than generating exhaustive,
unconstrained descriptions. Current benchmarks, however, primarily assess
descriptive comprehensiveness while largely overlooking instruction-following
capabilities. To address this gap, we introduce IF-VidCap, a new benchmark for
evaluating controllable video captioning, which contains 1,400 high-quality
samples. Distinct from existing video captioning or general
instruction-following benchmarks, IF-VidCap incorporates a systematic framework
that assesses captions on two dimensions: format correctness and content
correctness. Our comprehensive evaluation of over 20 prominent models reveals a
nuanced landscape: despite the continued dominance of proprietary models, the
performance gap is closing, with top-tier open-source solutions now achieving
near-parity. Furthermore, we find that models specialized for dense captioning
underperform general-purpose MLLMs on complex instructions, indicating that
future work should simultaneously advance both descriptive richness and
instruction-following fidelity.