Multimodal Autoregressive Pre-training of Large Vision Encoders
Authors: name: Enrico Fini; name: Mustafa Shukor; name: Xiujun Li; name: Philipp Dufter; name: Michal Klein; name: David Haldimann; name: Sai Aitharaju; name: Victor Guilherme Turrisi da Costa; name: Louis Béthune; name: Zhe Gan; name: Alexander T Toshev; name: Marcin Eichner; name: Moin Nabi; name: Yinfei Yang; name: Joshua M. Susskind; name: Alaaeldin El-Nouby
Published: 2024-11-21 18:31:25

Abstract: We introduce a novel method for pre-training of large-scale vision encoders.
Building on recent advancements in autoregressive pre-training of vision
models, we extend this framework to a multimodal setting, i.e., images and
text. In this paper, we present AIMV2, a family of generalist vision encoders
characterized by a straightforward pre-training process, scalability, and
remarkable performance across a range of downstream tasks. This is achieved by
pairing the vision encoder with a multimodal decoder that autoregressively
generates raw image patches and text tokens. Our encoders excel not only in
multimodal evaluations but also in vision benchmarks such as localization,
grounding, and classification. Notably, our AIMV2-3B encoder achieves 89.5%
accuracy on ImageNet-1k with a frozen trunk. Furthermore, AIMV2 consistently
outperforms state-of-the-art contrastive models (e.g., CLIP, SigLIP) in
multimodal image understanding across diverse settings.