Painting with Words: Elevating Detailed Image Captioning with Benchmark
  and Alignment Learning
Authors: name: Qinghao Ye; name: Xianhan Zeng; name: Fu Li; name: Chunyuan Li; name: Haoqi Fan
Published: 2025-03-10 22:53:56

Abstract: Image captioning has long been a pivotal task in visual understanding, with
recent advancements in vision-language models (VLMs) significantly enhancing
the ability to generate detailed image captions. However, the evaluation of
detailed image captioning remains underexplored due to outdated evaluation
metrics and coarse annotations. In this paper, we introduce DeCapBench along
with a novel metric, DCScore, specifically designed for detailed captioning
tasks. DCScore evaluates hallucinations and fine-grained comprehensiveness by
deconstructing responses into the smallest self-sufficient units, termed
primitive information units, and assessing them individually. Our evaluation
shows that DCScore aligns more closely with human judgment than other
rule-based or model-based metrics. Concurrently, DeCapBench exhibits a high
correlation with VLM arena results on descriptive tasks, surpassing existing
benchmarks for vision-language models. Additionally, we present an automatic
fine-grained feedback collection method, FeedQuill, for preference optimization
based on our advanced metric, showing robust generalization capabilities across
auto-generated preference data. Extensive experiments on multiple VLMs
demonstrate that our method not only significantly reduces hallucinations but
also enhances performance across various benchmarks, achieving superior detail
captioning performance while surpassing GPT-4o.