MonoFormer: One Transformer for Both Diffusion and Autoregression
Authors: name: Chuyang Zhao; name: Yuxing Song; name: Wenhao Wang; name: Haocheng Feng; name: Errui Ding; name: Yifan Sun; name: Xinyan Xiao; name: Jingdong Wang
Published: 2024-09-24 17:51:04

Abstract: Most existing multimodality methods use separate backbones for
autoregression-based discrete text generation and diffusion-based continuous
visual generation, or the same backbone by discretizing the visual data to use
autoregression for both text and visual generation. In this paper, we propose
to study a simple idea: share one transformer for both autoregression and
diffusion. The feasibility comes from two main aspects: (i) Transformer is
successfully applied to diffusion for visual generation, and (ii) transformer
training for autoregression and diffusion is very similar, and the difference
merely lies in that diffusion uses bidirectional attention mask and
autoregression uses causal attention mask. Experimental results show that our
approach achieves comparable image generation performance to current
state-of-the-art methods as well as maintains the text generation capability.
The project is publicly available at https://monoformer.github.io/.