PaLI-3 Vision Language Models: Smaller, Faster, Stronger
Authors: name: Xi Chen, name: Xiao Wang, name: Lucas Beyer, name: Alexander Kolesnikov, name: Jialin Wu, name: Paul Voigtlaender, name: Basil Mustafa, name: Sebastian Goodman, name: Ibrahim Alabdulmohsin, name: Piotr Padlewski, name: Daniel Salz, name: Xi Xiong, name: Daniel Vlasic, name: Filip Pavetic, name: Keran Rong, name: Tianli Yu, name: Daniel Keysers, name: Xiaohua Zhai, name: Radu Soricut

This paper presents PaLI-3, a smaller, faster, and stronger vision language
model (VLM) that compares favorably to similar models that are 10x larger. As
part of arriving at this strong performance, we compare Vision Transformer
(ViT) models pretrained using classification objectives to contrastively
(SigLIP) pretrained ones. We find that, while slightly underperforming on
standard image classification benchmarks, SigLIP-based PaLI shows superior
performance across various multimodal benchmarks, especially on localization
and visually-situated text understanding. We scale the SigLIP image encoder up
to 2 billion parameters, and achieves a new state-of-the-art on multilingual
cross-modal retrieval. We hope that PaLI-3, at only 5B parameters, rekindles
research on fundamental pieces of complex VLMs, and could fuel a new generation
of scaled-up models.