Title: LINA: Exploring Linear Autoregressive Image Generative Models with Continuous Tokens
Primary area: generative models
Keywords: Linear attention; autoregressive image generation

Abstract: In this paper, we systematically explore how efficient \underline{lin}ear attention mechanisms, as alternatives to full attention, should be designed in the context of \underline{a}utoregressive image generative models with continuous tokens.
We offer \textbf{\model}, a simple and empirically validated text-to-image generation baseline built on pure linear attention, capable of rapidly generating high fidelity 1024$\times$1024 images from user instructions. 
Our key contributions are: 
(1) An empirical study on scaling behavior \wrt parameter counts.
We examine two crucial design choices: (i) normalization paradigms—division-based \vs addition-based, and (ii) the use of a depthwise convolution module on image features for locality capacity enhancement. We then compare which variants of linear attention scale most effectively in autoregressive image generation. 
(2) KV gate mechanism: 
We empirically find that applying data-independent learnable parameters to the key and value states, thereby enabling flexible memory management and benefiting sample quality. 
We run thorough ablations on the KV gate design choices and visually show off the patterns it learns. 
Building on these designs, \model achieves promising results on both class-conditional (C2I) and text-to-image (T2I) generation. Compared to diffusion models of similar scale and autoregressive models with full attention, \model delivers competitive performance: FID of 2.18 on ImageNet ($\sim$1.4B) and an overall score of 0.74 on GenEval ($\sim$1.5B). Code and models will be open-sourced.

TL;DR: A successful practice of efficient linear autoregressive image generation model with continuous tokens.