Describe Anything: Detailed Localized Image and Video Captioning
Authors: name: Long Lian, name: Yifan Ding, name: Yunhao Ge, name: Sifei Liu, name: Hanzi Mao, name: Boyi Li, name: Marco Pavone, name: Ming-Yu Liu, name: Trevor Darrell, name: Adam Yala, name: Yin Cui

Generating detailed and accurate descriptions for specific regions in images
and videos remains a fundamental challenge for vision-language models. We
introduce the Describe Anything Model (DAM), a model designed for detailed
localized captioning (DLC). DAM preserves both local details and global context
through two key innovations: a focal prompt, which ensures high-resolution
encoding of targeted regions, and a localized vision backbone, which integrates
precise localization with its broader context. To tackle the scarcity of
high-quality DLC data, we propose a Semi-supervised learning (SSL)-based Data
Pipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets and
expands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmark
designed to evaluate DLC without relying on reference captions. DAM sets new
state-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, and
detailed multi-sentence localized image and video captioning.