Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction
  Tuning
Authors: name: Lili Yu; name: Bowen Shi; name: Ramakanth Pasunuru; name: Benjamin Muller; name: Olga Golovneva; name: Tianlu Wang; name: Arun Babu; name: Binh Tang; name: Brian Karrer; name: Shelly Sheynin; name: Candace Ross; name: Adam Polyak; name: Russell Howes; name: Vasu Sharma; name: Puxin Xu; name: Hovhannes Tamoyan; name: Oron Ashual; name: Uriel Singer; name: Shang-Wen Li; name: Susan Zhang; name: Richard James; name: Gargi Ghosh; name: Yaniv Taigman; name: Maryam Fazel-Zarandi; name: Asli Celikyilmaz; name: Luke Zettlemoyer; name: Armen Aghajanyan
Published: 2023-09-05 21:27:27

Abstract: We present CM3Leon (pronounced "Chameleon"), a retrieval-augmented,
token-based, decoder-only multi-modal language model capable of generating and
infilling both text and images. CM3Leon uses the CM3 multi-modal architecture
but additionally shows the extreme benefits of scaling up and tuning on more
diverse instruction-style data. It is the first multi-modal model trained with
a recipe adapted from text-only language models, including a large-scale
retrieval-augmented pre-training stage and a second multi-task supervised
fine-tuning (SFT) stage. It is also a general-purpose model that can do both
text-to-image and image-to-text generation, allowing us to introduce
self-contained contrastive decoding methods that produce high-quality outputs.
Extensive experiments demonstrate that this recipe is highly effective for
multi-modal models. CM3Leon achieves state-of-the-art performance in
text-to-image generation with 5x less training compute than comparable methods
(zero-shot MS-COCO FID of 4.88). After SFT, CM3Leon can also demonstrate
unprecedented levels of controllability in tasks ranging from language-guided
image editing to image-controlled generation and segmentation.