Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction
  Tuning
Authors: name: Lili Yu, name: Bowen Shi, name: Ramakanth Pasunuru, name: Benjamin Muller, name: Olga Golovneva, name: Tianlu Wang, name: Arun Babu, name: Binh Tang, name: Brian Karrer, name: Shelly Sheynin, name: Candace Ross, name: Adam Polyak, name: Russell Howes, name: Vasu Sharma, name: Puxin Xu, name: Hovhannes Tamoyan, name: Oron Ashual, name: Uriel Singer, name: Shang-Wen Li, name: Susan Zhang, name: Richard James, name: Gargi Ghosh, name: Yaniv Taigman, name: Maryam Fazel-Zarandi, name: Asli Celikyilmaz, name: Luke Zettlemoyer, name: Armen Aghajanyan

We present CM3Leon (pronounced "Chameleon"), a retrieval-augmented,
token-based, decoder-only multi-modal language model capable of generating and
infilling both text and images. CM3Leon uses the CM3 multi-modal architecture
but additionally shows the extreme benefits of scaling up and tuning on more
diverse instruction-style data. It is the first multi-modal model trained with
a recipe adapted from text-only language models, including a large-scale
retrieval-augmented pre-training stage and a second multi-task supervised
fine-tuning (SFT) stage. It is also a general-purpose model that can do both
text-to-image and image-to-text generation, allowing us to introduce
self-contained contrastive decoding methods that produce high-quality outputs.
Extensive experiments demonstrate that this recipe is highly effective for
multi-modal models. CM3Leon achieves state-of-the-art performance in
text-to-image generation with 5x less training compute than comparable methods
(zero-shot MS-COCO FID of 4.88). After SFT, CM3Leon can also demonstrate
unprecedented levels of controllability in tasks ranging from language-guided
image editing to image-controlled generation and segmentation.