PUMA: Empowering Unified MLLM with Multi-granular Visual Generation
Authors: name: Rongyao Fang; name: Chengqi Duan; name: Kun Wang; name: Hao Li; name: Hao Tian; name: Xingyu Zeng; name: Rui Zhao; name: Jifeng Dai; name: Hongsheng Li; name: Xihui Liu
Published: 2024-10-17 17:59:57

Abstract: Recent advancements in multimodal foundation models have yielded significant
progress in vision-language understanding. Initial attempts have also explored
the potential of multimodal large language models (MLLMs) for visual content
generation. However, existing works have insufficiently addressed the varying
granularity demands of different image generation tasks within a unified MLLM
paradigm - from the diversity required in text-to-image generation to the
precise controllability needed in image manipulation. In this work, we propose
PUMA, emPowering Unified MLLM with Multi-grAnular visual generation. PUMA
unifies multi-granular visual features as both inputs and outputs of MLLMs,
elegantly addressing the different granularity requirements of various image
generation tasks within a unified MLLM framework. Following multimodal
pretraining and task-specific instruction tuning, PUMA demonstrates proficiency
in a wide range of multimodal tasks. This work represents a significant step
towards a truly unified MLLM capable of adapting to the granularity demands of
various visual tasks. The code and model will be released in
https://github.com/rongyaofang/PUMA.