Randomized Autoregressive Visual Generation
Authors: name: Qihang Yu; name: Ju He; name: Xueqing Deng; name: Xiaohui Shen; name: Liang-Chieh Chen
Published: 2024-11-01 17:59:58

Abstract: This paper presents Randomized AutoRegressive modeling (RAR) for visual
generation, which sets a new state-of-the-art performance on the image
generation task while maintaining full compatibility with language modeling
frameworks. The proposed RAR is simple: during a standard autoregressive
training process with a next-token prediction objective, the input
sequence-typically ordered in raster form-is randomly permuted into different
factorization orders with a probability r, where r starts at 1 and linearly
decays to 0 over the course of training. This annealing training strategy
enables the model to learn to maximize the expected likelihood over all
factorization orders and thus effectively improve the model's capability of
modeling bidirectional contexts. Importantly, RAR preserves the integrity of
the autoregressive modeling framework, ensuring full compatibility with
language modeling while significantly improving performance in image
generation. On the ImageNet-256 benchmark, RAR achieves an FID score of 1.48,
not only surpassing prior state-of-the-art autoregressive image generators but
also outperforming leading diffusion-based and masked transformer-based
methods. Code and models will be made available at
https://github.com/bytedance/1d-tokenizer