AuroraCap: Efficient, Performant Video Detailed Captioning and a New
  Benchmark
Authors: name: Wenhao Chai; name: Enxin Song; name: Yilun Du; name: Chenlin Meng; name: Vashisht Madhavan; name: Omer Bar-Tal; name: Jeng-Neng Hwang; name: Saining Xie; name: Christopher D. Manning
Published: 2024-10-04 00:13:54

Abstract: Video detailed captioning is a key task which aims to generate comprehensive
and coherent textual descriptions of video content, benefiting both video
understanding and generation. In this paper, we propose AuroraCap, a video
captioner based on a large multimodal model. We follow the simplest
architecture design without additional parameters for temporal modeling. To
address the overhead caused by lengthy video sequences, we implement the token
merging strategy, reducing the number of input visual tokens. Surprisingly, we
found that this strategy results in little performance loss. AuroraCap shows
superior performance on various video and image captioning benchmarks, for
example, obtaining a CIDEr of 88.9 on Flickr30k, beating GPT-4V (55.3) and
Gemini-1.5 Pro (82.2). However, existing video caption benchmarks only include
simple descriptions, consisting of a few dozen words, which limits research in
this field. Therefore, we develop VDC, a video detailed captioning benchmark
with over one thousand carefully annotated structured captions. In addition, we
propose a new LLM-assisted metric VDCscore for bettering evaluation, which
adopts a divide-and-conquer strategy to transform long caption evaluation into
multiple short question-answer pairs. With the help of human Elo ranking, our
experiments show that this benchmark better correlates with human judgments of
video detailed captioning quality.