MUMU: Bootstrapping Multimodal Image Generation from Text-to-Image Data
Authors: name: William Berman; name: Alexander Peysakhovich
Published: 2024-06-26 23:21:42

Abstract: We train a model to generate images from multimodal prompts of interleaved
text and images such as "a <picture of a man> man and his <picture of a dog>
dog in an <picture of a cartoon> animated style." We bootstrap a multimodal
dataset by extracting semantically meaningful image crops corresponding to
words in the image captions of synthetically generated and publicly available
text-image data. Our model, MUMU, is composed of a vision-language model
encoder with a diffusion decoder and is trained on a single 8xH100 GPU node.
Despite being only trained on crops from the same image, MUMU learns to compose
inputs from different images into a coherent output. For example, an input of a
realistic person and a cartoon will output the same person in the cartoon
style, and an input of a standing subject and a scooter will output the subject
riding the scooter. As a result, our model generalizes to tasks such as style
transfer and character consistency. Our results show the promise of using
multimodal models as general purpose controllers for image generation.