Continuous Visual Autoregressive Generation via Score Maximization
Authors: name: Chenze Shao; name: Fandong Meng; name: Jie Zhou
Published: 2025-05-12 17:58:14

Abstract: Conventional wisdom suggests that autoregressive models are used to process
discrete data. When applied to continuous modalities such as visual data,
Visual AutoRegressive modeling (VAR) typically resorts to quantization-based
approaches to cast the data into a discrete space, which can introduce
significant information loss. To tackle this issue, we introduce a Continuous
VAR framework that enables direct visual autoregressive generation without
vector quantization. The underlying theoretical foundation is strictly proper
scoring rules, which provide powerful statistical tools capable of evaluating
how well a generative model approximates the true distribution. Within this
framework, all we need is to select a strictly proper score and set it as the
training objective to optimize. We primarily explore a class of training
objectives based on the energy score, which is likelihood-free and thus
overcomes the difficulty of making probabilistic predictions in the continuous
space. Previous efforts on continuous autoregressive generation, such as GIVT
and diffusion loss, can also be derived from our framework using other strictly
proper scores. Source code: https://github.com/shaochenze/EAR.