Describe Anything: Detailed Localized Image and Video Captioning
Authors: name: Long Lian; name: Yifan Ding; name: Yunhao Ge; name: Sifei Liu; name: Hanzi Mao; name: Boyi Li; name: Marco Pavone; name: Ming-Yu Liu; name: Trevor Darrell; name: Adam Yala; name: Yin Cui
Published: 2025-04-22 17:51:41

Abstract: Generating detailed and accurate descriptions for specific regions in images
and videos remains a fundamental challenge for vision-language models. We
introduce the Describe Anything Model (DAM), a model designed for detailed
localized captioning (DLC). DAM preserves both local details and global context
through two key innovations: a focal prompt, which ensures high-resolution
encoding of targeted regions, and a localized vision backbone, which integrates
precise localization with its broader context. To tackle the scarcity of
high-quality DLC data, we propose a Semi-supervised learning (SSL)-based Data
Pipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets and
expands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmark
designed to evaluate DLC without relying on reference captions. DAM sets new
state-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, and
detailed multi-sentence localized image and video captioning.