PaLI-X: On Scaling up a Multilingual Vision and Language Model
Authors: name: Xi Chen, name: Josip Djolonga, name: Piotr Padlewski, name: Basil Mustafa, name: Soravit Changpinyo, name: Jialin Wu, name: Carlos Riquelme Ruiz, name: Sebastian Goodman, name: Xiao Wang, name: Yi Tay, name: Siamak Shakeri, name: Mostafa Dehghani, name: Daniel Salz, name: Mario Lucic, name: Michael Tschannen, name: Arsha Nagrani, name: Hexiang Hu, name: Mandar Joshi, name: Bo Pang, name: Ceslee Montgomery, name: Paulina Pietrzyk, name: Marvin Ritter, name: AJ Piergiovanni, name: Matthias Minderer, name: Filip Pavetic, name: Austin Waters, name: Gang Li, name: Ibrahim Alabdulmohsin, name: Lucas Beyer, name: Julien Amelot, name: Kenton Lee, name: Andreas Peter Steiner, name: Yang Li, name: Daniel Keysers, name: Anurag Arnab, name: Yuanzhong Xu, name: Keran Rong, name: Alexander Kolesnikov, name: Mojtaba Seyedhosseini, name: Anelia Angelova, name: Xiaohua Zhai, name: Neil Houlsby, name: Radu Soricut

We present the training recipe and results of scaling up PaLI-X, a
multilingual vision and language model, both in terms of size of the components
and the breadth of its training task mixture. Our model achieves new levels of
performance on a wide-range of varied and complex tasks, including multiple
image-based captioning and question-answering tasks, image-based document
understanding and few-shot (in-context) learning, as well as object detection,
video question answering, and video captioning. PaLI-X advances the
state-of-the-art on most vision-and-language benchmarks considered (25+ of
them). Finally, we observe emerging capabilities, such as complex counting and
multilingual object detection, tasks that are not explicitly in the training
mix.