AnyCap Project: A Unified Framework, Dataset, and Benchmark for
  Controllable Omni-modal Captioning
Authors: name: Yiming Ren; name: Zhiqiang Lin; name: Yu Li; name: Gao Meng; name: Weiyun Wang; name: Junjie Wang; name: Zicheng Lin; name: Jifeng Dai; name: Yujiu Yang; name: Wenhai Wang; name: Ruihang Chu
Published: 2025-07-17 07:04:05

Abstract: Controllable captioning is essential for precise multimodal alignment and
instruction following, yet existing models often lack fine-grained control and
reliable evaluation protocols. To address this gap, we present the AnyCap
Project, an integrated solution spanning model, dataset, and evaluation. We
introduce AnyCapModel (ACM), a lightweight plug-and-play framework that
enhances the controllability of existing foundation models for omni-modal
captioning without retraining the base model. ACM reuses the original captions
from base models while incorporating user instructions and modality features to
generate improved captions. To remedy the data scarcity in controllable
multimodal captioning, we build AnyCapDataset (ACD), covering three modalities,
28 user-instruction types, and 300\,k high-quality data entries. We further
propose AnyCapEval, a new benchmark that provides more reliable evaluation
metrics for controllable captioning by decoupling content accuracy and
stylistic fidelity. ACM markedly improves caption quality across a diverse set
of base models on AnyCapEval. Notably, ACM-8B raises GPT-4o\'s content scores
by 45\% and style scores by 12\%, and it also achieves substantial gains on
widely used benchmarks such as MIA-Bench and VidCapBench.