Building and better understanding vision-language models: insights and
  future directions
Authors: name: Hugo Laurençon; name: Andrés Marafioti; name: Victor Sanh; name: Léo Tronchon
Published: 2024-08-22 17:47:24

Abstract: The field of vision-language models (VLMs), which take images and texts as
inputs and output texts, is rapidly evolving and has yet to reach consensus on
several key aspects of the development pipeline, including data, architecture,
and training methods. This paper can be seen as a tutorial for building a VLM.
We begin by providing a comprehensive overview of the current state-of-the-art
approaches, highlighting the strengths and weaknesses of each, addressing the
major challenges in the field, and suggesting promising research directions for
underexplored areas. We then walk through the practical steps to build
Idefics3-8B, a powerful VLM that significantly outperforms its predecessor
Idefics2-8B, while being trained efficiently, exclusively on open datasets, and
using a straightforward pipeline. These steps include the creation of Docmatix,
a dataset for improving document understanding capabilities, which is 240 times
larger than previously available datasets. We release the model along with the
datasets created for its training.