DOCCI: Descriptions of Connected and Contrasting Images
Authors: name: Yasumasa Onoe; name: Sunayana Rane; name: Zachary Berger; name: Yonatan Bitton; name: Jaemin Cho; name: Roopal Garg; name: Alexander Ku; name: Zarana Parekh; name: Jordi Pont-Tuset; name: Garrett Tanzer; name: Su Wang; name: Jason Baldridge
Published: 2024-04-30 17:56:24

Abstract: Vision-language datasets are vital for both text-to-image (T2I) and
image-to-text (I2T) research. However, current datasets lack descriptions with
fine-grained detail that would allow for richer associations to be learned by
models. To fill the gap, we introduce Descriptions of Connected and Contrasting
Images (DOCCI), a dataset with long, human-annotated English descriptions for
15k images that were taken, curated and donated by a single researcher intent
on capturing key challenges such as spatial relations, counting, text
rendering, world knowledge, and more. We instruct human annotators to create
comprehensive descriptions for each image; these average 136 words in length
and are crafted to clearly distinguish each image from those that are related
or similar. Each description is highly compositional and typically encompasses
multiple challenges. Through both quantitative and qualitative analyses, we
demonstrate that DOCCI serves as an effective training resource for
image-to-text generation -- a PaLI 5B model finetuned on DOCCI shows equal or
superior results compared to highly-performant larger models like LLaVA-1.5 7B
and InstructBLIP 7B. Furthermore, we show that DOCCI is a useful testbed for
text-to-image generation, highlighting the limitations of current text-to-image
models in capturing long descriptions and fine details.