PaLI-X: On Scaling up a Multilingual Vision and Language Model
Authors: name: Xi Chen; name: Josip Djolonga; name: Piotr Padlewski; name: Basil Mustafa; name: Soravit Changpinyo; name: Jialin Wu; name: Carlos Riquelme Ruiz; name: Sebastian Goodman; name: Xiao Wang; name: Yi Tay; name: Siamak Shakeri; name: Mostafa Dehghani; name: Daniel Salz; name: Mario Lucic; name: Michael Tschannen; name: Arsha Nagrani; name: Hexiang Hu; name: Mandar Joshi; name: Bo Pang; name: Ceslee Montgomery; name: Paulina Pietrzyk; name: Marvin Ritter; name: AJ Piergiovanni; name: Matthias Minderer; name: Filip Pavetic; name: Austin Waters; name: Gang Li; name: Ibrahim Alabdulmohsin; name: Lucas Beyer; name: Julien Amelot; name: Kenton Lee; name: Andreas Peter Steiner; name: Yang Li; name: Daniel Keysers; name: Anurag Arnab; name: Yuanzhong Xu; name: Keran Rong; name: Alexander Kolesnikov; name: Mojtaba Seyedhosseini; name: Anelia Angelova; name: Xiaohua Zhai; name: Neil Houlsby; name: Radu Soricut
Published: 2023-05-29 18:58:38

Abstract: We present the training recipe and results of scaling up PaLI-X, a
multilingual vision and language model, both in terms of size of the components
and the breadth of its training task mixture. Our model achieves new levels of
performance on a wide-range of varied and complex tasks, including multiple
image-based captioning and question-answering tasks, image-based document
understanding and few-shot (in-context) learning, as well as object detection,
video question answering, and video captioning. PaLI-X advances the
state-of-the-art on most vision-and-language benchmarks considered (25+ of
them). Finally, we observe emerging capabilities, such as complex counting and
multilingual object detection, tasks that are not explicitly in the training
mix.