ControlAR: Controllable Image Generation with Autoregressive Models
Authors: name: Zongming Li; name: Tianheng Cheng; name: Shoufa Chen; name: Peize Sun; name: Haocheng Shen; name: Longjin Ran; name: Xiaoxin Chen; name: Wenyu Liu; name: Xinggang Wang
Published: 2024-10-03 17:28:07

Abstract: Autoregressive (AR) models have reformulated image generation as next-token
prediction, demonstrating remarkable potential and emerging as strong
competitors to diffusion models. However, control-to-image generation, akin to
ControlNet, remains largely unexplored within AR models. Although a natural
approach, inspired by advancements in Large Language Models, is to tokenize
control images into tokens and prefill them into the autoregressive model
before decoding image tokens, it still falls short in generation quality
compared to ControlNet and suffers from inefficiency. To this end, we introduce
ControlAR, an efficient and effective framework for integrating spatial
controls into autoregressive image generation models. Firstly, we explore
control encoding for AR models and propose a lightweight control encoder to
transform spatial inputs (e.g., canny edges or depth maps) into control tokens.
Then ControlAR exploits the conditional decoding method to generate the next
image token conditioned on the per-token fusion between control and image
tokens, similar to positional encodings. Compared to prefilling tokens, using
conditional decoding significantly strengthens the control capability of AR
models but also maintains the model's efficiency. Furthermore, the proposed
ControlAR surprisingly empowers AR models with arbitrary-resolution image
generation via conditional decoding and specific controls. Extensive
experiments can demonstrate the controllability of the proposed ControlAR for
the autoregressive control-to-image generation across diverse inputs, including
edges, depths, and segmentation masks. Furthermore, both quantitative and
qualitative results indicate that ControlAR surpasses previous state-of-the-art
controllable diffusion models, e.g., ControlNet++. Code, models, and demo will
soon be available at https://github.com/hustvl/ControlAR.