Continuous Visual Autoregressive Generation via Score Maximization
Authors: Chenze Shao
Fandong Meng
Jie Zhou
Venue: ICML 2025 poster (2025)
Keywords: visual autoregressive generation
strictly proper scoring rules
continuous
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: This paper introduces a framework of continuous visual autoregressive generation based on the theory of strictly proper scoring rules.

Abstract: Conventional wisdom suggests that autoregressive models are used to process discrete data. When applied to continuous modalities such as visual data, Visual AutoRegressive modeling (VAR) typically resorts to quantization-based approaches to cast the data into a discrete space, which can introduce significant information loss. To tackle this issue, we introduce a Continuous VAR framework that enables direct visual autoregressive generation without vector quantization. The underlying theoretical foundation is strictly proper scoring rules, which provide powerful statistical tools capable of evaluating how well a generative model approximates the true distribution. Within this framework, all we need is to select a strictly proper score and set it as the training objective to optimize. We primarily explore a class of training objectives based on the energy score, which is likelihood-free and thus overcomes the difficulty of making probabilistic predictions in the continuous space. Previous efforts on continuous autoregressive generation, such as GIVT and diffusion loss, can also be derived from our framework using other strictly proper scores. Source code: \url{https://github.com/shaochenze/EAR}.

Continuous Visual Autoregressive Generation via Score Maximization
Chenze Shao
Fandong Meng' Jie Zhou'
Abstract
from finite vocabulary. Therefore, for continuous modal- ities such visual data, Visual AutoRegressive modeling approaches (van den Oord et al., 2017; Razavi et al., 2019; Esser et al., 2021; Yu et al., 2024a) to cast the data into a discrete space.
Conventional wisdom suggests that autoregres- sive models used process discrete data. When applied to continuous modalities such vi- sual data, Visual AutoRegressive modeling (VAR) typically resorts to quantization- based approaches to cast the data into a discrete space, which can introduce significant information loss. To tackle this issue, we introduce a Continuous VAR frame- work that enables direct visual autoregressive generation without vector quantization. The un- derlying theoretical foundation is strictly proper scoring rules, which provide powerful statistical tools capable of evaluating how well a gener- ative model approximates the true distribution. Within this framework, we need to select strictly proper score and set it as the train- ing objective to optimize. We primarily explore class of training objectives based on the en- ergy score, which is likelihood-free and thus overcomes the difficulty of making probabilis- tic predictions in the continuous space. Pre- vious efforts on continuous autoregressive gen- eration, such as GIVT and diffusion loss, can also be derived from our framework using other strictly proper scores. Source code: https: //github. com/ httmm//hachhneeeeeecmmma EEAR.
(VAR') typically resort to quantization-
Discrete visual representation based on vector quantization provides support for autoregressive generation, yet the pri- mary concern lies in the information loss due to quantization errors. During visual generation, quantization errors de- grades the reconstruction quality of discrete image tokeniz- ers, which upper-bbunds the generation quality (Rombach et al., 2022). Moreover, discrete representations compro- mise the model's perception of low-level details, restricting its ability to capture continuous variations and subtle differ- ences. Consequently, in terms of visual understanding, the performance of discrete tokenizers often lags behind that of continuous tokenizers (Wu et al., 2024; Xie et al., 2024).
Given the limitations associated with vector quantization, there is a growing interest in continuous visual autoregres- sive generation. However, without a finite vocabulary, generally intractable to explicitly predict the likelihood over continuous spaces and train with likelihood maximization. Prior to this work, autoregressive generation in continuous spaces has been explored through GIVT (Tschannen et al., 2023) and diffusion loss (Li et al., 2024). Nevertheless, the expressive capability of GIVT is confined to the pre-defined family of Gaussian mixtures (Tschannen et al., 2023), and the per-token diffusion procedure necessitates multiple de- noising iterations to recover the token distribution, which significantly increases the inference latency (Li et al., 2024).
1. Introduction
Autoregressive large language models (Achiam et al., 2023;
In this work, we introduce Continuous VAR framework that enables direct visual autoregressive generation without vector quantization. The underlying theoretical foundation is strictly proper scoring rules (Brier, 1950; Good, 1952; Gneiting & Raftery, 2007), which provide powerful sta- tistical tools capable of evaluating how well generative model approximates the true distribution. Specifically, scor- ing rules are functions to assess the quality of a probability distribution based on the observed sample. A scoring rule is considered strictly proper encourages the model to make honest predictions. In other words, the expected score
Touvron et al., 2023; Team et al., 2023; Bai et al., 2023) have demonstrated remarkable scalability and generalizability in understanding and generating discrete text, which has in- spired the exploration of autoregressive generation on other data modalities. However, autoregressive models equipped with cross-entropy loss are limited to handle discrete tokens
Pattern Recognition Center. WeChat AI, Tencent Inc. Corre- spondence to: Chenze Shao <chenzeshao@ tencent.com>. Fan- dong Meng <fandongmeng @ @temeeet.com>> Jie Zhou <with- tomzhou@tonn @tencent.com>>
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Herre, VAR refers to the autoregressive modeling of visual content, not limited to next scale prediction (Tian et al., 2024).

---

Continuous Visual Autoregressive Generation via Score Maximization
is maximized only when model predictions follow the true distribution, and any deviation from the truth will result in decrease in expected score.
(Chang et al., 2022; Li et al., 2023; Chang et al., 2023). However, the information loss incurred during the quantiza- tion process becomes bottleneck for the generation quality, Consequently, recent focus has shifted towards finding a better image tokenizer (Yu et al., 2022; Mentzer et al., 2024; Yu et al., 2024a;b; Weber et al., 2024). In parallel, there is a growing interest in employing continuous tokenizers for autoregressive image generation, with Gaussian mix- ture models (Tschannen et al., 2023) and diffusion models (Li et al., 2024) being used to represent the token distribu- tion. Our approach advances this direction by establishing a universal framework for predicting continuous tokens.
The intrinsic property of strietly proper scoring rules makes them well-suited training objectives for generative mod- els. A prominent example is the cross-entropy loss used in discrete autoregressive models, which corresponds to the maximization of logarithmic score (Good, 1952). Within the Continuous VAR framework, all we need is to select strictly proper score for continuous variables and set it as the training objective to optimize. Previous efforts on continu- ous autoregressive generation, such as GIVT and diffusion loss, can also be derived from this framework, where they are respectively aligned with the logarithmic score (Good, 1952) and the Hyvärinen score (Hyvärinen, 2005).
Strictly Proper Scoring Rules. Strictly proper scoring rules, initially introduced in Brier (1950) for the verification of weather forecasts, have since evolved into a compre- hensive theoretical framework for evaluating probabilistic forecasts. In the realm of deep learning, the most exten- sively applied scoring rule is the logarithmic score (Good, 1952), which is closely linked with maximum likelihood estimation, cross-entropy loss, and perplexity evaluation. The Brier score is also widely used for training classifica- tion networks (Shoemaker, 1991; Hung et al., 1996; Kline & Berardi, 2005; Hui & Belkin, 2021) and evaluating their cali- bration (Lakshminarayanan et al., 2017; Ovadia et al., 2019; Gruber & Buettner, 2022). Recently, Shao et al. (2024) proposed using scoring rules as the training objective for autoregressive language modeling. In the continuous space, the Hyvärinen score (Hyvärinen, 2005) plays important role in score matching, which gives rise to score-based dif- fusion models (Song & Ermon, 2019; Song et al., 2021). The energy score has also been employed in generative mod- eling, with applications spanning image generation (Belle- mare et al., 2018), speech synthesis (Gritsenko et al., 2020), time series prediction (Pacchiardi et al., 2024; Pacchiardi & Dutta, 2022), and self- sulf-supe learning (Vahidi et al., 2024). In this work, we focus on leveraging scoring rules to enable the autoregressive modeling of continuous data.
Under the Continuous VAR framework, we primarily plore a class of training objectives based on the energy score (Székely, 2003), which is likelihood-free and thus overcomes the difficulty of making probabilistic predictions in the continuous space. The associated energy loss incen- tivizes the model to generate samples close to the target label, while maintaining the diversity between independent samples. The model architecture remains largely analogous to a discrete Transformer ( (Vaswani et al., 2017), with the key difference being the substitution of the softmax layer with a small MLP generator, both of which transform the hid- den representation into a distribution. Similar to Generative Adversarial Networks (Goodfellow et al., 2014), the MLP generator is an implicit generative model that takes random noises additional inputs, and the predictive distribution is implicitly represented by its sampling process.
Experiments on the ImageNet 256x 256 benchmark (Deng et al., 2009) show that our approach achieves stronger visual generation quality than the traditional autoregressive Trans- former that uses a discrete tokenizer. Compared to diffusion- based methods, our approach exhibits substantially higher inference efficiency, as it does not require multiple denoising iterations to recover the target distribution.
3. Continuous Visual Autoregressive Generation
2. Related Work
In this section, we begin by introducing the essential back- ground of strictly proper scoring rules. Following that, we present the Continuous VAR framework, which enables continuous visual autoregressive generation via score max- imization. For simplicity of notation, we assume a setting of unconditional generation, and the conclusion can be ex- tended to conditional generation scenarios.
Visual Autoregressive Generation. Early efforts ap- proached visual autoregressive generation by treating the image sequence of pixels (Gregor et al., 2014; Parmar et al., 2018; van den Oord et al., 2016 To mitigate the expensive cost of autoregressive modeling at the pixel level, van den Oord et al. (2017) introduced the vector quantiza- tion technique to represent an image as a set of discrete tokens, paving the way for more effective autoregressive image generation with both causal (Razavi et al., 2019; Esser al., 2021; Ramesh et al., 2021; Yu al., 2022; Sun et al., 2024a; Tian et al., 2024) and masked Transformers
3.1. Strietly Proper Scoring Rules
In statistical decision theory, scoring rules serve quantita- tive measures to assess the quality of probabilistic predic-
2

---

Continuous Visual Autoregressive Generation via Score Maximization
tions, by assigning numerical score based on the predicted distribution p and the observed sample x. Let X' represents the sample space and P be the set of probability measures on X. A scoring rule S takes values in the extended real line R = |- oo]. indicating the reward or utility of predicting p when sample æ is observed:
with multiple potential minimizers, making it challenging for the model to converge the correct distribution. A triv- ial but telling example is the constant-valued score. While being technically proper, it does not offer meaningful guid- ance for model training.
When dealing with intricate samples like long texts, videos, or high-resolution images, direct generation poses signifi- cant challenges, which necessitates breaking down the pro- cess into several steps for autoregressive modeling. In this case, the direct calculation of Equation 4 is not always fea- sible, but we can evaluate scoring rules at each time step to calculate the following sequence loss (Shao et al., 2024):
S x) P
The role of scoring rules is to assess whether the prediction p honestly represents the underlying sample distribution q- This is reflected in the expected score with respect to x ~ q, denoted as S(p, 9):
S(p g) = 1
(2)
Ls(D,0) - - Σ ((p()1re)))))))))
(5)
A proper scoring rule should encourage the model to make honest predictions. Formally, a scoring rule is proper the expected score is maximized when the model reports true probabilities:
The expected loss is minimized only when every expected score ((tl)r=r)).g((ll=))))) is maximized, which conse- quently encourages honest sequential predictions p = q. However, for continuous-valued generative models, the ex- plicit likelihood estimation is sometimes intractable due to the lack of a finite vocabulary, which makes the score calculation also infeasible. Under these circumstances, we can adopt an unbiased estimator of Equation 5 as the loss function. Since the expectation of loss remains unchanged, the model will still be trained towards approximating the true distribution.
S(p.9) S S(g.9),
V.
(3)
It is strictly proper when the equality holds if and only if The strict propricty means that the score maximizer is unique, where any deviation from the truth will result in a decrease in expected score.
The study of scoring rules, which dates back to the Brier score (Brier, 1950) for the verification of weather forecasts, has evolved into comprehensive framework offering a wealth of useful scores, such as the logarithmic score (Good, 1952), Brier score (Brier, 1950), and spherical score (Roby, 1965). For continuous variables, the choices expand to strictly proper scores such as the energy score (Székely, 2003), CRPS (Matheson & Winkler, 1976), and Hyvärinen score (Hyvärinen, 2005), well as proper scores like the keree score (Eaton, 1981) and Variogram score (Scheuerer & Hamill, 2015). Gneiting& & Raftery (2007) provides comprehensive literature review on scoring rules.
3.3. Examples
Here, we revisit the previous methodologies of continuous visual autoregressive generation, namely GIVT and diffu- sion loss. We will show that these methods can be derived from the perspective of score maximization, falling within our Continuous VAR framework.
Example 1 (GIVT, Tschannen et al., 2023). Generative unfnite-Vocabule Transformers (GIVT) is perhaps the first visual Transformer that directly generates vector sequences with real-valued entries. The loss function for GIVT is still the cross-entropy, which corresponds to the maximization of the logarithmic score S(p, x) log p(x). To estimate the likelihood, GIVT employs an invertible flow model (Dinh et al., 2015) to simplify the latent distribution, and then approximates it with a Gaussian Mixture Model (GMM), Recently, this methodology has been adapted for speech synthesis (Lin & HE, 2025). The primary limitation of GIVT lies in its expressive capability, which is constrained by the capacity of GMM and its assumption of channel-wise independence.
3.2. Continuous VAR via Score Maximization
The property of strictly proper scoring rules naturally align with the objective of generative models. With a loss function that promotes the maximization of a strictly proper score, the model will be trained to approximate the data distribution. direct approach is to take the negative of a strictly proper score as the loss:
Cs(p 2).
(4)
For example, maximizing the logarithmic score S(p, x) = log p(z) recovers the cross-entropy loss. The emphasis on the strict propriety of the scoring rule is crucial. Unlike strictly proper scores which guarantee& a unique optimizer, proper but not strictly proper scores results in loss function
Example 2 (Diffusion Loss, Li et al., 2024). Recently, diffu- sion loss is proposed to model the per-token distribution by a diffusion procedure, which has soon gained widespread
3

---

Continuous Visual Autoregressive Generation via Score Maximization
applications such as video generation (Deng al., 2024; Liu et al., 2024), text-to-image generation (Fan et al., 2024; Yu et al., 2025), speech synthesis (Turetzky et al., 2024), and multi-modal generation (Sun et al., 2024b). Diffusion mod- els (Sohl-Dickstein et al., 2015; Ho et al., 2020), also known as score-based generative models (Song & Ermon, 2019; Song et al., 2021), require multiple denoising iterations to recover the target distribution, which results in a significant inference lateney for per-token diffusion procedures.
where 41, T2, a E IR are independent samples with distri- bution p. The expected energy score is associated with the generalized energy distance (p, q) by constant:
(p, q) = 2 y y/ El[ - x2| [1 Bly - y/2 [1]
Sp q) + C(g).
(9)
For a E (0, (", g) 2 0 with equality to zero if and only if p = (Székely, 2003; Szěkely & Rizzo, 2013), which implies that the energy score is strictly proper. Note that the distance (' q) = |B - Elyll is minimized as long as their expectations match, so the energy score 2 is proper but not strictly proper.
From the perspective of score matching". one needs to esti- mate the gradients of the data distribution to reverse a diffu- sion process with Langevin dynamics, which is facilitated by maximizing the Hyvärinen score (Hyvärinen, 2005):
The energy score can be unbiasedly estimated using two independent samples T1, 422 drawn from the distribution p. In this way. the energy loss is defined as:
Sp,, =
logp(z)) + logg())))? (6)
where tr(.) denotes the trace of matrix. The expected score is equivalent to the score matching objective up constant, which shows that the score is strictly proper:
C(p, v) = |x] y/" + | y/" |21 221", (10)
which incentivizes the model to generate samples close to the target label, while maintaining the diversity between independent samples. Notably, the energy loss does not require explicit likelihood estimations but merely the capa- bility to sample from the model distribution, which gives much flexibility in the following architecture design of en- ergy Transformer.
Sp, q) = Egg | v p(x) - Vz log ((x) 21 + C(a)-
(7)
The diffusion training objective an estimation of S(p. q) through denoising score matching over multiple noise scales (Vincent, 2011; Song & Ermon, 2019). It implies that the per-token diffusion loss also falls within our Continuous VAR framework with respect to the Hyvärinen score.
4.2. Energy Transformer
Figure 1 illustrates the architecture of energy Transformer, where the energy loss is employed to supervise each au- toregressive generation step. The continuous- energy Transformer remains largely analogous to a discrete Trans- former (Vaswani et al., 2017), with the key difference being the substitution of the softmax layer with small MLP gen- erator, both of which transform the hidden representation into distribution. Similar to Generative Adversarial Net- works (Goodfellow et al., 2014), the MLP generator is an implicit generative model, whose predictive distribution is implicitly represented by its sampling process.
4. Energy-based Autoregressive Generation
Under the Continuous VAR framework, we develop a Energy-based AutoRegression (EAR) approach via max- imizing the energy score (Székely, 2003). The estimation of energy score does not require explicit likelihood estimations but merely the capability to sample from the model distri- bution. This reduction of constraints facilitates the design of a more expressive energy Transformer. Moreover, the energy Transformer is highly efficient in inference, capable of predicting the next continuous token in a single forward pass. Further details are elaborated below.
The energy Transformer is designed accept continuous tokens inputs. In the embedding layer, the lookup table is replaced with linear projection, which maps each token of size deokn to dmedel- The representations extracted by Transformer are then mapped to dmip. serving as inputs for the MLP generator. The MLP generator also takes a random noise e as an additional input to perturb the representation for the sampling purpose. The random noise of size dnoise is drawn from a uniform distribution ranging from [-0.5, 0.5). which is then embedded to size dmip.
4.1. Energy Loss
The energy score is a family of strictly proper scoring rules for continuous variables in Rdd To avoid symbol confusion, we denote the samples drawn from the model distribution p as x, and the samples from the data distribution q as Let ar E (0, 2), the energy score is defined as:
S(p. v) = E||r| - 22 - 2BI - yl"
(8)
The MLP generator consists of few residual blocks that gradually inject noises into the prediction. Each residual block contains a two-layer FFN network with SiLU activa- tion (Elfwing et al., 2018), and the noise is incorporated via
'Pleass note that the concept of "score" differs between scoring rules and score matching. In scoring rules, "score" is measure used to assess the quality of prohabilistic predictions. In score matching, "score" refers to the gradient Vz p(z).
4

---

Continuous Visual Autoregressive Generation via Score Maximization
softmax(logit
(L,
MLP E)
Softmax
MLP Generator
Gate
Transformer
Transformer
XN
FFN
Embedding
Linear
Scale, Shift
12
0.9
0.3
2.1
Layer Norm
Linear
0.8
-1.5
0.7
discrete tokens
0.4
0.1
-0.3
continuous tokens
h
Figure 1. Comparison between the discrete-token standard Transformer and continuous-toker energy Transformer. At the input side, the embedding lookup table is replaced with a linear projection. At the output side, the softmax classification layer is replaced with a small MIP generator, which takes random noise input perturb the hidden state.
adaptive layer normalization (Peebles & Xie, 2023), which perturbs the prediction with shift, scale, and gate layers. Specifically, assuming that the input to the i-th residual block ht, its output is given by:
Tirain 1, as it would cause the loss function unbounded and hackable. During inference, directly modifying the scale of the noise would corrupt the generated images. Instead, we propose to only scale the shi by a temperature Tinfer- while keeping scale(e) and gate(e) unchanged.
hh = (1 scale(E)) LN(K") + shift(e), h' = h² gate(e) (F)(
(11)
Classifier- Guidance. We employ Classifier-Free Guidance (CFG, Ho & Salimans, 2022) to improve the qual- ity conditional generation. At training time, we replace the condition with a dummy token for 10% of the samples. At inference time, the Transformer model is run with both the given condition and the dummy token, providing two outputs he and hy- The combination of the two outputs h = cfg he + (1 -fg) h,, is fed to the MLP generator, where cfg is the guidance scale. Following Chang et al. (2023), we linearly increase the guidance scale during the autoregressive generation. We sweep the optimal guidance scale for each model.
where shiff(()), scale(.), and gate(-) are linear transforma- tions that interpret the input noise perturbation signals, L-(() represents layer normalization, and (-) denotes two-layer feed-forward neural network with a intermediate dimension of dmlp- Finally, the MLP generator concludes by predicting the next continuous token via a linear layer.
4.3. Other Techniques
In this section, we present several techniques that have proven effective in improving the visual generation quality of EAR.
Masked Autoregressive Generation. Masked autoregres- sive models can be regarded as a type of autoregressive model that predicts a set unknown tokens based on ex- isting tokens (Chang et al., 2022; Li et al., 2023: Chang et al., 2023; Li et al., 2024). They supports bidirectional attention, which facilitates a more effective representation learning compared causal attention. Consistent with Li et al. (2024), we find that masked autoregressive generation performs better than causal generation. During training, we randomly sample a masking ratio the range of [0.7, 1.0). During inference, we generate tokens in random order, progressively reducing the masking ratio from 1.0 to fol-
Temperature. The temperature hyperparameter 7 is widely used the sampling process of generative models, which trades diversity for accuracy. For EAR, we can incorpo- rate temperature hyperparameters during both training and inference, denoted as Tiriin and Tinfer, respectively. During training, the energy loss is composed of two components: |a y/ and |zj 22 where the latter measures the di- versity of the generated outputs. Therefore, we can assign a weight Trrin < to |z] _21" la in the energy loss, which can enhance the generation quality with a short period of fine-tum tme-ttning. However, this approach not applicable for
5

---

Continuous Visual Autoregressive Generation via Score Maximization
Table Model comparisons on ImageNet 256x256 conditional generation. Metrics include Fréchet Inception Distance (FID), Inception Score (IS), Precision (Pre) and Recall (Rec). "4" or "f" indicate lower or higher values are better.
w/o guidance
w/ guidance
Type
Model
#Params
FID4
1St
Pref
Reef
FID4
1St
Pref
Reet
BigGAN (Brock et al., 2019)
112M
6.95
224.5
0.89
0.38
GAN
GigaGAN (Kang al., 2023)
569M
3,45
225.5
0.84
0.61
StyleGan-XL (Sauer et al., 2022)
166M
2.30
265.1
0.78
0.53
ADM (Dhariwal Nichol, 2021)
554M
10.94
101.0
0.69
0.63
4.59
186.7
0.82
0.52
LDM-4f (Rombach al., 2022)
400M
10.56
103.5
0.71
0.62
3.60
247.7
0.87
0.48
Diff
DiT-XL/2 (Peebles & Xie, 2023)
675M
9.62
121.5
0.67
0.67
2.27
278.2
0.83
0.57
L-DiT-7B (Alpha-VLLM, 2024)
7B
5.06
153.3
0.70
0.68
2.28
316.2
0.83
0.58
VDM++ (Kingma & Gao, 2023)
2B
2.40
225.3
2.12
267.7
VQGAN (Esser et al., 2021)
1.4B
15.78
78.3
RQ- To-Transformer (Lee et al., 2022)
3.8B
7.55
134.0
LlamaGen-3B (Sun al., 2024a)
3.1B
2.18
263.3
0.84
0.54
MaskGIT (Chang et al., 2022)
227M
6.18
182.1
0.80
0.51
AR
MAGE (Li et al., 2023)
230M
6.93
195.8
MAGVIT-v2 (Yu et al., 2024a)
307M
3.65
200.5
1.78
319.4
VAR-d30 (Tian et al., 2024)
2.0B
1.92
323.1
0.82
0.59
GIVT (Tschannen et al., 2023)
304M
5.67
0.75
0.59
3.35
0.84
0.53
MAR (Li et al., 2024)
943M
2.35
227.8
0.79
0.62
1.55
303.7
0.81
0.62
EAR-B
205M
5,46
155.9
0.76
0.57
2.83
253.3
0.82
0.54
EAR
EAR-L
474M
3.69
183.4
0.77
0.59
2.37
273.8
0.81
0.57
EAR-H
937M
3.16
204.2
0.76
0.61
1.97
289.6
0.81
0.59
lowing cosine schedule. By default, we 64 generation steps in this schedule.
We the decoder-only Transformer architecture following the implementation in ViT (Dosovitskiy et al., 2021) for masked autoregressive generation. The class condition is represented as 64 class tokens at the start of the decoder sequence. We use the discrete VQ-16 tokenizer (Rombach et al., 2022) for the standrard Transformer and the con- tinuous KL-16 tokenizer (Li et al., 2024) for our energy Transformer. The stride of both tokenizers is 16.
Learning Rate for MLP Generator. our experiments using regular learning rates, we found that the model failed converge. Given that the Transformer backbone remains unchanged, we hypothesize the MLP generator may need smaller learning rate to ensure its training stability (Singh et al., 2015; Howard & Ruder, 2018; Xu et al., 2025). To address it, we adjust the MLP generator's 's learning rate by applying a constant multiplier
Following Li et al. (2024), we explore the scaling behavior of Energy-based AutoRegression (EAR) with three sizes of energy Transformer, referred to as EAR-B, EAR-L, and EAR-H, respectively. They respectively have 24, 32, 40 Transformer blocks and width of 768, 1024, and 1280. The MLP generators account for approximately 15% the parameter size of energy Transformer, which respectively have 6, 8, 12 blocks and width of 1024, 1280, and 1536.
throughout the training. Empirically, setting 0.25 strikes balance between training efficiency and stability.
5. Experiments
5.1. Settings
The random noise for the MLP generator has size of dnoise = 64, independently drawn from uniform distribu- tion 0.5, 0.5) at each time step. We by default set a = 1 to calculate the energy loss. We train our model for a total of 800 epochs, where the first 750 epochs use the standard energy loss and the last 50 epochs reduces the temperature Ttrain to 0.99. The inference temperature Tinfer fer is set to
We evaluate visual generation capability on the class- conditional ImageNet 256x256 benchmark (Deng et al., 2009). We use Fréchet Inception Distance (FID, Heusel al., 2017) as the main metric, and also provide Incep- tion Score (IS, Salimans et al., 2016) and Precision/Recall (Kynkäänniemi et al., 2019) as secondary metrics. We fol- low the evaluation suite of Dhariwal & Nichol (2021).
6

---

Continuous Visual Autoregressive Generation via Score Maximization
EAR
Continuous w/o dfg
MAR-B
Continuous w/ clg
10a
MAR-L
Discrete cfę
Quality (FID)
Discrete w/ cfg
MAR-
10
101
10"
10a
80
160
320
400
Inference Latency (s)
Epoch
Figure 2. The speed/quality trade-off for EAR and MAR. The num- ber of autoregressive steps fixed at 64. For MAR, we vary the number of diffusion steps (10, 20, 25, 30, 40, 50) to generate outputs under different inference latencies. For EAR, the curve is obtained by using different model sizes (EAR-B, BB, EAR-L, EAR-H). The inference time is measured on single A 100 GPU.
Figure 3. FID curves of the continuous- valuee energy Transformer (205M) and the discrete-valuce standard Transformer (196M). The guidance scale 3.0.
denoising iterations recover the target distribution. Con- versely, the energy-style supervision enables EAR make predictions within single forward computation.
0.7. Our models are optimized by the AdamW optimizer (Loshchilov & Hutter, 2019) with 0.9, A82 0.95. The batch size is 2048. The learning rate is 8c-4 and the constant learning rate schedule is applied with linear warmup of 100 epochs. We use a weight decay of 0.02, gradient clipping of 3.0, and dropout of 0.1 during training. Following Peebles & Xie (2023); Li et al. (2024), we maintain the exponential moving average of the model parameters with a momentum of 0.9999.
The continuous tokenizer we use exhibits strong recon- struction quality of 1.22 FID. In contrast, a VQ tokenizer with the same model architecture only achieves a recon- struction FID 5.87 (Rombach et al., 2022), which could become bottleneck in generation quality. In Figure 3, we compare our continuous-valued energy Transformer with a discrete-vallee Transformer that uses the VQ tokenizer. The results show that continuous tokenization with the energy loss consistently outperforms discrete tokenization with the cross-entropy loss, highlighting the great potential of Con- tinuous VAR.
5.2. Main Results
In Table we compare EAR with popular image genera- tion models, including GANs, diffusion models, and VQ- based autoregressive models. Notably, EAR-B obtains strong FID of 2.83 with only 205M parameters, and EAR-H achieves competitive FID of 1.97, while maintaining relatively modest model size among the leading systems. The scalability EAR suggests that the generation quality could be further boosted through scaling.
For causal autoregressive modeling, our findings align with Li et al. (2024) that both continuous- and riscrete-valued Transformers can only achieve FID scores around 20. We hypothesize that causal models may suffer from overfitting due to the absence of random masking mechanism during training-a key feature of masked autoregressive modeling that enhances generalization.
Our approach is most closely aligned with MAR (Li al., 2024), they both model the distribution of continuous tokens with an MLP module on top of masked autore- gressive Transformer. They can both be viewed as instances under the Continuous VAR framework, with EAR and MAR maximizing the energy score and the Hyvärinen score, re- spectively. Figure illustrates the inference latency (average time to generate an image) and generation quality (measured by FID) of the two methods. EAR is significantly more effi- cient in inference, capable of producing a high-quality im- age in roughly 1s, while MAR takes nearly 10 times longer to produce images of comparable quality. The efficiency ad- vantage stems from the difference in probabilistic modeling. Trained with the diffusion loss, MAR necessitates multiple
5.3. Importance of Being Strictly Proper
In the energy loss presented in Equation 10, the exponen- tial coefficient a was empirically set to in previous ex- periments. While any choice of (0, 2) ensures the strict propriety, energy losses with duce rapid training collapse due to gradient instability, For example, the gradient of | x2 a can be expressed as #|21-zal" = Σ 11- 2- 88 ing begins, independent samples æn and 22 are typically nearly indistinguishable. which causes the denominator | 2 to approach zero exponentially faster than the numerator when a < 1, resulting in unbounded gradient magnitudes that destabilize optimization.
invariably in-
a(zi-z))
- When train- 00

---

Continuous Visual Autoregressive Generation via Score Maximization
Additionally, while the energy score remains proper at 2, its non-strict propriety (only expectation alignment [p|r]] = lE, [u] is enforced) proves insufficient for effective training. As evidenced in Table 2, training with = 2 fails to generate meaningful contents (FID> > 100), whereas lever- aging strictly proper energy scores with a E [1,2) achieves decent generation quality. It validates our claim on the im- portance of being strictly proper for scoring rules, which guarantees unique global minimum.
40
200
Incepton Score
330
100
10
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Figure 4. Generation quality of the Gaussian Transformer under different standard deviations during inference. The model size 184M. efy is disabled since it does not work well here.
Table 2. The performance of EAR-B when being trained with dif- ferent exponential term a in the energy loss. The number of training epochs is 400. The guidance scale 3.0.
1r=8e-4,
10a
1==2o-4,2-1
1.0
1.25
1.75
2.0
FID
FID
3.55
3.73
4.10
4.32
188.1
230.3
223.1
212.1
204.2
6.4
10a
40
80
120
160
200
Epoch
5.4. Importance of Being Expressive
Figure 5. The results of varying learning rates for EAR-B.
While the logarithmic score also serves strictly proper scoring rule, it necessitates explicit knowledge of the pre- dictive probability density, which poses challenge for continuous-valued models. To obtain an explicit likelihood cstimation, it generally requires constraining the predictive distribution of the model, as exemplified in Tschannen al. (2023). For instance, consider model that parameterizes Gaussian distribution with mean vector μ and covariance matrix Σ. The corresponding negative log-likelihood objec- tive becomes:
5.5. Ablation Study
Effect of Learning Rate. We observed that the model failed to converge when using standard learning rates, and reduc- ing the learning rate specifically for the MLP generator was found effective to enhance training stability. As illustrated in Figure 5, when the entire model was trained with a global learning rate = Se - 4, the training process eventually collapsed after several epochs. By selectively lowering the learning rate for the MLP generator ing process is stabilized. While reducing the learning rate of the entire model also enables successful training, this approach results in relatively lower training efficiency.
0.25 x, the train-
C = log f(// )) = -((x (x- ((t=-)²--- -- -w)
(12)
1
+ log(2x) + log E
Effect of Noise. In our experiments, we observed that both the type and dimension of random noise can affect model performance. We experimented with Uniform noise and Gaussian noise, setting the noise dimension, dnoise to 32, 64, and 128. As shown in Table 3, uniform noise consistently outperforms Gaussian noise, and davise = 64 performs better than other settings. Therefore, we adopt the 64-dimmesiionll uniform noise in EAR.
This objective reduces to the Mean Squared Error (MSE) loss under the common assumption of channel independence with a fixed standard deviation a. We employ this MSE loss train Gaussian Transformer and experiment with different during inference. The results are depicted in Figure 4.
As seen, an appropriate variance selection yields non-trivial generation quality, but the performance gap compared to EAR remains substantial, suggesting that the token distri- bution is complex and challenging to be explicitly repre- sented using predefined distributions. Our proposed energy Transformer addresses this challenge through its inherently expressive architecture: the model implicitly defines the predictive distribution through its sampling process, which enables the automatic learning of complex data distributions without restrictive prior assumptions.
Table 3. The results of varying random noises for EAR-B. The number of training epochs is 400. The guidance scale 3.0.
Uniform 64
Gaussian
dnoise
32
128
32
64
128
w/o cfg
9.87
7.95
7.04
9.45
7.79
7.17
w/ cfg
3.89
3.55
4.34
4.01
3.60
4.51
8

---

Continuous Visual Autoregressive Generation via Score Maximization
clg-1
cfg=2
cfg=3
cfg 4
Figure 6. Samples of EAR under different gudiance scales. We fix the random seed and apply the constant cfg schedule during sampling.
3.00
E6R-B
3.
6.4
FAR-1
(AB-1
2.95
24
3.0
EI
5.8
2:85
20
40
50
2:80
0.5
0.7
0.8 Temparature
2.9
1.0
Epoch
Figure 7. The results of varying classifier-free guidance scales for EAR models.
Figure 8. The results of varying temperatures (left) and Tinker (right) for EAR-B.
Effect of CFG. Classifier-free guidance (Ho Salimans, 2022) plays a crucial role in the inference stage of EAR. Figure 7 illustrates the variations of FID and Inception Score across different efg scales. The image quality, measured by the Inception Score, consistently improves with increasing cfg. However, the FID metric reaches its optimal value around cfg=3.0, as excessive guidance scale can compromise the generation diversity. Figure illustrates the sampling outputs EAR-H under different guidance scales, where a larger cfg scale generally produces more fine-grained images.
vector quantization. The Continuous VAR framework is grounded in strictly proper scoring rules, and we primarily explore a class of maaey training objectives, which is likelihood-free and induces an expressive enery Trans- former architecture. Our experimental results demonstrate competitive performance in both generation quality and in- ference efficiency, while leaving substantial room for future improvements. Promising research directions include: 1) architectural optimization of the energy Transformer, 2) in- corporation of alternative strictly proper scoring rules as training objectives, 3) extension to more continuous modal- ities such as video and audio, and 4) continuous language modeling through conversion of discrete text into latent vector representations.
Effect of Temperature. We employ temperature hyperpa- rameters Timin and Tinfer to trade diversity for accuracy. Fig- 8 shows the impact varying temperatures during the fine-tuning and inference of EAR-B. These results induce temperature combination of Tuain 0.99 and Tinter 0.7.
Impact Statement
6. Conclusion
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences our work, none which we feel must be specifically highlighted here.
This paper introduces a Continuous VAR framework that enables direct visual autoregressive generation without

---

Continuous Visual Autoregressive Generation via Score Maximization
References
Dhariwal, P. and Nichol, A. Diffusion models beat gans image synthesis. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 8780-8794, Curran Associates, Inc. 2021. URL
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
cc cc/pppee__file/ppppe///////// neurips... 99ad23ddeccFabbaee77222266...... pdf.
Alpha-VLLM.
Large-dit-mmagened 2024.
URL
https: btttp://githube com/ powmAApaa-ViiMM (LaMAA2-Accesso/ttree.
Large-DiT-ImageNes
Dinh, L., Krueger, D., and Bengio, Y. NICE: non-linear independent components estimation. In Bengio. and LeCun, Y. (eds.), 3rd International Conference on Learn- ing Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015. URL http: arxiv org. abs 1410 0. 8516.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y. Ge. W., Han, Y., Huang, F, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Bellemare, M. G., Danihelka, I., Dabney, W., Mohamed, S., Lakshminarayanan, B., Hoyer, S., and Munos, R. The cramer distance as solution to biased wasserstein gradients, 2018. URL https: httpss//opppeeveeeeee net/
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021. URL https:// openreview net worum Fww.umiideiiooooo feieeiiddiiiii wit.uuiiiiiiicb..yy.....
Brier, G. W. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Brock, A., Donahue, J., and Simonyan, K. Large scale
GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL https: http://pppweeeeeeeeeeeee net/ forum? id-Bixsqj09Em
Eaton, M. A method for evaluating improper prior distribu- tions. Technical report, University of Minnesota, 1981.
Elfwing. S., Uchibe, E., and Doya, K. Sigmoid- weighted linear units for neural network function approximation in reinforcement learning. Neu- rul Networks, 107:3-11, 2018. ISSN 0893-6080. doi: http:://iiong//ooooooooccmoooo.p URL https://www..cceeceediiie science/arte
Chang, H., Zhang, H., Jiang, I.,, Liu, C., and Freeman, Maskgit: Masked generative image transformer. Pro- ceedings of the IEEE/CVF Conference Computer Vi- sion and Puttern Recognition (CVPR), pp. 10.11111, 11315-11325, June 2022.
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, I.., Yang, M.-H., Murphy, K. P., Freeman, W. T., Rubinstein, M., Li, Y., and Krishnan, D. Muse: Text-to-image generation via masked generative trans- formers. In Krause, A., Brunskill, E., Cho, K., En- gelhardt, B., Sabato, S., and Scarlett, J. (eds.), Pro- ceedings of the 40th International Conference chine Learning, volume 202 of Proceedings of Machine Learning Research, 2023. URL https:::: pttp///ecceeng...e/ee/pppp mlr. ptess// v202 h012/2cann22bbb.m html.
Special issue on deep reinforcement learning.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12873-12883, June 2021.
Ma-
Fan, L., Li, T., Qin, S., Li, Y., Sun, C., Rubinstein, M., Sun, D., Hc, K., and Tian, Y. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410. 13863, 2024.
4055-4075, PMLR, 23-29 Jul
Deng, H., Pan, T., Diao, H., Luo, Z., Cui, Y., Lu, H., Shan, S., Qi, Y., and Wang, X. Autoregressive video generation without vector quantization. arXiv preprint arXiv:2412.14169, 2024.
Gneiting, T. and Raftery, A. E. Strictly proper scoring rules, prediction,, and estimation. Journal the American statistical Association, 102(477):55-337)
2007. 2007.
Good, I. Rational decisions. Journal of the Royal Statisti- cal Society: Series B (Methodological), 14(1):107-114, 1952.
Deng, J., Dong, W., Socher, R., L...,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vi- sion and Pattern Recognition, pp. 248-255, 2009. doi: 10.1 109/CVPR22009 50060848..888...
Goodfellow, L., Pouget-Abadic, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio,
10

---

Continuous Visual Autoregressive Generation via Score Maximization
Y. Generative adversarial nets. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Wein- berger, K. (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, 2014. URL http://///p.eeeeenng...eeee.ree cc /pppper_fiileeeppprrr///////eeee
volume 33, pp. 6840-6851. Curran Associates, Inc.,
2020. URL httpss:/pprcceeednggg nng cc/ httpppper_filee/pseeeeeee cc/sapee_iiileeppprr/////////
4 4cfioffe655aa0000771aa00009aa339 pdf.
Howard, J. and Ruder, S. Universal language model fine-tuning for text classification. In Gurevych, I. and Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- une 1: Long Papers), pp-
Paper.
pdf.
Gregor, K., Danihelka, I., Mnih, A., Blundell, C., and Wierstra, D. Deep autoregressive networks. In Xing, E. P. and Jebara, T. (eds.), Proceedings of the 31st International Conference on Machine Learning, vol- ume 32 of Proceedings of Machine Learning Research, pp. 1242-1250, Bejing, China, 22-24 Jun 2014. PMLR. URL htt.tttts//// http///ooeddiiie ngs s. mll mwr.m..rreee ptt....rs/s//// gregori4.htm.
Melbourne, Aus- tralia, July 2018. Association for Computational Lin- guistics. doi: 11188663/11/11-100111 URL https: / /acclanthooogy...... hwtpttthhoooooo.ooooooooeo wttp:/y..rr///10/00011000 10311..
Hui, L. and Belkin, M. Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks. International Conference on Learning Represen- tations, 2021. URL https: / openreview net/ forum?
Gritsenko, A., Salimans, T., van den Berg, R., Snoek, J. and Kalchbrenner, N. A spectral energy distance for parallel speech synthesis. In Larochelle, H., Ran- zato, M.. Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 13062-13072. Curran Associates, Inc., 2020. URL cc /iaape__fillesppppr////////// (093eaaa133363300116eeee3aa33333 Paper. pdf.
Hung, M., Hu, M., Shanker, M., and Patuwo, B. Estimat- ing posterior probabilities in classification problems with neural networks. International Journal of Computational Intelligence and Organizations, 1(1):49, 1996.
Hyvärinen, A. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(24))695-709, 2005. URL http::/mmr/ jmlr. org org/par hts//papers////hyyvrrre
html.
Gruber, S. and Buettner, F. Better uncertainty calibration via
Kang, M., Zhu, J.-Y., Zhang, R., Park, J., Shechtman, E., Paris, S., and Park, T. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10124- 10134, 2023.
proper scores for classification and beyond. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 8618-8632, Curran Associates, Inc., 2022.
Kingma, D. P. and Gao, R. Understanding diffusion objec- tives as the ELBO with simple data augmentation. In Teirty-sevntth Conference on Neural Information Pro- cessing Systems, 2023. URL https: opwnrevie..eeeeeeeeeee net/ forumm wiwammiiididieaaadaaaaa
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Guyon, I,, Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Process- ing Systems, volume 30. Curran Associates, Inc., 2017. URL https::/prrcee https://proceeiingg.. neurips. cc/paper cc/paper_filestee http////eseeepeer/0////111ee B41664700ee0eee66511187772222228 pdf.
Kline, D. and Berardi, V. Revisiting squared-error and cross-entropy functions for training neural network clas- sifiers. Neural Computing and Applications, 14:310-318, 12 2005. doi: 10. 00.100770005210055-46667
Kynkäänniemi, T., Karras, T. Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assessing generative models. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Bue, F., Fox, E., and Gameett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.,
Ho, J. and Salimans, T. Classifier-free diffusion guid- ance, 2022. URL https:// org/ abs/ 2207. 12598.
Ho. J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems,
2019. URL httpss//proceeedingg.oo neurip.... cc he//appe__fleeeppper////////// 22345510Dcc500222227011133333333 pdf.

---

Continuous Visual Autoregressive Generation via Score Maximization
Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
0ew.ee4400616761662212126 058 pdf.
Pacchiardi, L. and Dutta, R. Likelihoodd inference with generative neural networks via scoring rule minimiza- tion, 2022. URL httptttt///aatee 15784.
Lee, D., Kim, C., Kim, S., Cho, M., and Han, W.-S Autore- gressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11523-11532 2022.
org/ harg/abe abs/2205.
Pacchiardi, L., Adewoyin, R. A., Dueben, P., and Dutta, Probabilistic forecasting with generative networks via scoring rule minimization. Journal of Machine Learning Research, 25(45):1-64, 2024. URL http:// /mmr. org/ htts/aapera/v25///2 00388 .hmml.
Li, T., Chang, H., Mishra, S., Zhang. H., Katabi, D., and Krishnan, Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2112-215222 June 2023.
Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer, N., Ku, A., and Tran, D. Image trans- former. In Dy, J. and Krause, A. (eds.), Proceed- ings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learn- ing Research, pp. 4055-4064. PMLR, 10-15 Jul 2018. URL https: : / proceedingg www. pwe.s/peeesvv000 pa parmara Ba html.
Li, T., Tian, Li, H., Deng, M., and He, K. Autoregres- sive image generation without vector quantization. arXiv preprint arXiv:2406.1 118888 1138,, 2024.
Lin, W. and HE, C. Continuous autoregressive modeling with stochastic monotonic alignment for speech synthesis. In The Thirteenth International Conference on Learning Representations, 2025. URL https: / openreview. net/ Eorum? :u ZE8J1vb.
Peebles, W. and Xie, S. Scalable diffusion models with trans- formers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4195-4205, October 2023.
Liu, H., Liu, S., Zhou, Z., Xu, M., Xie, Y., Han, X., Pérez, J. C., Liu, D., Kahatapitiya, K., Jia, M., et al. Mardini: Masked autoregressive diffusion for video generation at scale. arXiv preprint 4rXiv:2410.20280, 2024.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- Meila, M. and Zhang, T, (eds.), Proceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8821-8831. PMLR, 18-24 Jul 2021. WRRLhttss///poceeee v139 htt99//mmee22a.... html.
image generation.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019, https: hpppeeveen.e..eee net font/forumiid--a0 Bikoumiiiddddkkkkk.....
mlr press/
Matheson, E. and Winkler, R. L. Scoring rules for contin- uous probability distributions. Management science, 22 (10):1087-1066, 1976.
Razavi, A., van den Oord, A., and Vinyals, O. Generating di- verse high-fidelity images with vq-vae-2. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buu F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019. URL https::/pproceedinggeee/////// .eerrr.. occ/sppee_ffles//ppprr///////// 5 Affe22fal77.ddlcccafflo00//////. pdf.
Mentzer, F., Minnen, D., Agustsson, E., and Tschannen, M. Finite scalar quantization: VQ-VAE made simple. In The Twelfth International Conference on Learning Representationss 2024. https:::: htttt:::///peerriiee. net/ hutt/oormm?id -is.d-sshhhhANN...
Ovadia, Y., Fertig, E., Ren, J., Nado, 2., Sculley, D., Nowozin, S., Dillon, J., Lakshninarayanan, B., and Snock, Can you trust your model's un- certainty? evaluating predictive uncertainty under dataset shift. In Wallach, H., Larochelle, H., Beygelz- imer, A., d'Alché-Buc, F., Fox, F.., and Garnett, R. (eds.). Advances in Neural Information Process- ing Systems, volume 32. Curran Associates, Inc., 2019. URL bttps://prcceeding/ hewprrocceeiinnn...er.......... neurips. ca/
Roby, T. B. Belief states: A preliminary empirical study. Behavioral Sci, 10(3):55--70,, 1965.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
12

---

Continuous Visual Autoregressive Generation via Score Maximization
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X., and Chen, X. Im- proved techniques for training gans. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Process- ing Systems, volume 29. Curran Associates, Inc., 2016. URL https::/prcceeddnng... neuriii co/paper_ [pappp__iiee//ppprr//////////
Song. Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative mod- eling through stochastic differential equations. In In- ternational Conference on Learning Representations, 2021. URL htttts:://ppeneeiiiiceo http///ppeeeeeeeeeeeeeeeeeeeeeee net/ www.re//owww.eeeooom forum? id-PxTIG12RRES.
Sun, P.. Jiang, Y. Chen, S., Zhang, S., Peng, B., Luo, P., and Yuan, Z. Autoregressive model beats diffusion: Llama for scalable image generation, 2024a. URL https: / //arxiv. org/ abs /2406.
pdf.
Sauer, A., Schwarz, K., and Geiger, A. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022.
Sun, Y., Bao, H., Wang, W., Peng, Z., Dong, L., Huang, S., Wang, J., and Wei, F. Multimodal latent language modeling with next-token diffusion. arXiv preprint 1:Xi::2412008633 2024b.
Scheuerer, M. and Hamill, T. M. Variogram- baased proper scoring rules for probabilistic forecasts of multivariate quantities. Monthly Weather Review, 143(4):1311-1334, 2015.
Szěkely, G. E-statistics: The energy of statistical sam- ples. Bowling Green State University, Department of Mathematics and Statistics Technical Report, 3(05):1-18. 2003.
Shao, C., Meng, F., Liu, Y., and Zhou, J. Language gen- eration with strictly proper scoring rules. In Forty-first International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL hwtph:tp///// htttpp:://pennreveerrt forumm id-LALS286Xpx.
Székely, J. and Rizzo, M. L. Energy statistics: A class of statistics based on distances. Journal of Statistical Planning and Inference, 143(8):1249-1272, 2013. ISSN 0378-3758. doi:
018. URL http:ttpt:/ww..ccieeeeeeeeeeeeee sci /tteencc//rrticcee///////7777773
com/
Shoemaker, P. A note on least-squares learning procedures and classification by neural network models. IEEE Trans- actions on Neural Networks, 2(1):158-16) 160, 1991. doi: 10.1109/72.80304.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: family of highly capable multimodal models. arXiv preprint arXivv2312.. 1805, 2023.
Tian, K., Jiang, Y., Yuan, Z., PENG, B., and Wang, L. Vi- sual autoregressive modeling: Scalable image generation via next-scale prediction. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024. URL https: https:// /twpn//eeeeeeeeeeee/eeeeee...eee net/ forum? id=go JL6CCES8.
Singh, B., De, S., Zhang, Y., Goldstein, T., and Tay- lor, G. Layer-specific adaptive learning rates for deep networks, 2015. URL /ectps://arxte org/ abs/ 1510.04609.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N.. and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Ma- chine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256 2256-2265, Lille, France, 07-
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 13971, 2023.
09 Jul 2015. PMLR. URL httpt:tt//// httssttt:///prreedii /ptp://d//occ.c....//oooo mlr p.r.spssss//. / v37 7 schl wwwss/hiccssseilll.s1acc hhtmmmm.sttei.mmmmm........
Tschannen, M., Eastwood, C., and Mentzer, F. Givt: Gener- ative infinite-vocabulary transformers. urXiv:2312.02111 16, 2023.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. In Wallach, H., Larochelle, H., Beygelzimer, A., d' d'Alcéé-Bue, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL ce ce/paper_fileeeeeepeeeeeeeeeeeee pppppee_ffleee/pppprr////////111 200lef23740ddaa3711s6c6677777771 Pape.. pdf.
Turetzky, A., Shabtay, N., Shechtman, S., Aronowitz, H., Haws, D., Hoory, R., and Dekel, A. Continuous speech synthesis using per-token latent diffusion. arXiv preprint arXiv:2410.16048 2024.
Vahidi, A., Schosser, S., Wimmer, L., Li, Y., Bischl, B., Hüllermeier, E., and Rezaei, M. Probabilistic self- supervised representation learning via scoring rules min- imization. In The Twelfth International Conference
13

---

Continuous Visual Autoregressive Generation via Score Maximization
Learning Representations, 2024. URL https:// openreview. net/ foo.:mmm..ssssccccdo
Xie, J., Mao, W., Bai, Z., Zhang, D. Wang, W., Lin, K. Q.. Gu, Y., Chen, Z.,, Yang. Z., and Shou, M. Z. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. /2528, 2024.
van den Oord, A., Kalchbrenner, N., Espeholt, L.., kavukcuoglu, k., Vinyals, O., and Graves, A. Condi- tional image generation with pixelenn decoders. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016a. URL https://proceedingg..eo neurips........................p ca ca/paper_fiiccccc cct/ppppe__fleessppperr/////// b Paper. pdf.
Xu, S., Bu, Z., Zhang, Y., and Barnett, A hessian- informed hyperparameter optimization for differential learning rate, 2025. URL https://arxiv... abs 2501 06954.
Yu, H., Luo, H., Yuan, H., Rong, Y., and Zhao, F. Frequency autoregressive image generation with continuous tokens. arXiv preprint arXiv:2503.05306 2025.
van den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. Pixel recurrent neural networks. In Balcan, M. F. and Weinherger, K. Q. (eds.), Proceedings of The 33rd International Conference ume 48 of Proceedings of Machine Learning Research, pp. 1747-1756, New York, New York, USA, 20-22 Jun 2016b. PMLR. URL https: http:s:////cccedngg.... mlrr press/v48 8 oordl 6 html.
Yu, J., Li, X., Koh, J. Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., and Wu, Y. Vector- quantized image modeling with improved VQGAN. In International Conference Learning Representations, 2022. URL https::/p///......... /openreview formm? id=pfNyExj7z2.
Machine Learning, vol-
Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y., Gupta, A., Gu, X., Haupt- mann, A. G., Gong, B., Yang, M.-H., Essa, I., Ross, D. A., and Jiang, L. Language model beats diffusion tokenizer is key to visual generation. The Twelfth International Conference on Learning Representations, 2024a. URL ht..s:/:: /owprevveeeereeeeeeereeeeerreeee net/ forum? i id=gzgrANCF4g.
van den Oord, A., Vinyals, O., and kavukcuoglu, k. Neural discrete representation learning. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceddingg www.:eiiiiiiinnnein..ieo. oc papper_fileespppprr//1////1 /
Yu, Q., Weber, M., Deng, X., Shen, X., Cremers, D., and Chen, L.-C. An image is worth 32 tokens for recon- struction and generation. In The Thiny--ightt Annual Conference Neural Information Processing Systems, 2024b. https::: http:::////ppeeeieew net/ forum? id=tOXogPRaPL.
7a98af1 Teteaeeeaaa0eeeeee000000000oooo0 Paper. pdf.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,
L., Gomez, A. N.. Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.). Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:/ nee oc cc/paper_ffie ht//ppper_flleee/ppprr///////// 33be223337cce3111b03aaaaaa333333 Paper. pdf.
Vincent, P. A connection between score matching and de- noising autoencoders. Neural Computation, 1667):1661- 1674, 2011. doi: 162/NECO a 00142.
Weber, M. Yu, L., Yu, Q. Deng, X., Shen, X., Cre- mers, D., and Chen, L.-C. Maskbit: Embedding-fre image generation via bit tokens, 2024. URL https: // //arxiv. org/ abs/24099.6211
Wu, Y., Zhang, Z., Chen, J., Tang, H., Li, D., Fang, Y., Zhu, L.,, Xie, E., Yin, H., Yi, L., et al. Vila-u: a unified foundation model integrating visual understanding and generation, arXiv preprint ND.:2409.04422 2024.
14

---

Continuous Visual Autoregressive Generation via Score Maximization
A. Additional Results
Table 4. Model comparisons on ImageNet 512x512 conditional generation. The clş scale is set to 4.0.
w/o guidance
w/ guidance
Type
Model
#Params
FID
ISt
FID
ISt
ADM (Dhariwal & Nichol, 2021)
554M
23.24
58.1
7.72
172.7
Diff
DiT-X1/2 (Peebles & Xie, 2023)
675M
12.03
105.3
3.04
240.8
VDM++ (Kingma & Gao, 2023)
2B
2.99
232.2
2.65
278.1
MaskGIT (Chang et al., 2022)
227M
7.32
156.0
MAGVIT-v2 (Yu et al., 2024a)
307M
3.07
213.1
1.91
324.3
AR
GIVT (Tschannen et al., 2023)
304M
8.35
MAR (Li et al., 2024)
481M
2.74
205.2
1.73
279.9
EAR
EAR-B
205M
7.75
141.5
3.38
227.0
Table The effect of attention masking on EAR-B. The number of training epochs is 400.
w/o guidance
w/ guidance FID
Type
FID
ISt
IS
Causal
17.83
78.6
8.10
144.5
Bidirection
7.95
130.5
3.55
230.3
15


Reviews:
[{"review_id": "iYo6I07fgm", "reviewer": "Reviewer_gkkh", "summary": "This paper introduce energy-based autoregressive to train a continuous autoregressive models. The continuous model bypasses the traditional approach of using discrete representation to train an autoregressive model, therefore reduce the information loss during discrete quantization. The experiment shows the effectiveness of continuous autoregressive framework on imagenet compared to other models.\n\n## update after rebuttal\nThe novelty has partially resolved my concern. Therefore, I increase my score to weak accept. Please include the ablation for causal and full attention in your revised paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the Reviewer's time and efforts in reviewing our work. We provide discussions about the concerns as follows.\n\n> The ablation for masked autoregressive should be provided with causal and full attention like setting in MAR.\n\nWe thank the Reviewer for raising this concern. We will enrich the ablation studies and provide comparisons between causal and full attention.\n\n> The experiment in table 1 lacks of comparison with continuous AR like [1], [2]. The paper states that in [2] GMM could limit expressivity line 155 2nd column. However, the performance of proposed method still lag behind GIVT. The author should include the performance of [1] and [2] in the main table and make a comprehensive comparison.\n\nWe will follow the suggestion and include comparisons with continuous AR models (MAR and GIVT) in the main table. We would also like to respectfully clarify that our method (EAR) does not lag behind GIVT in performance. Specifically:\n\n+ Base-scale models: GIVT (304M params) achieves an FID of ​3.35, while our EAR-B (205M params) achieves ​2.83.\n \n+ Large-scale models: GIVT-L (1.67B params) achieves an FID of ​2.59, while our EAR-H (937M params) achieves ​1.97.\n\n> Why the random noise in MLP is uniform but not normal. Why uniform is better choice ?. The author should ablate this choice.\n\nWe empirically observe that uniform noise yields slightly better performance compared to Gaussian noise. Specifically, for EAR-B: w/o CFG, FID increases from ​5.46 (uniform) to ​5.74 (Gaussian); w/ CFG, FID increases from ​2.83 (uniform) to ​2.89 (Gaussian). We will report this comparison in the revised manuscript.\n\n4. The inference process is significantly affected by variance σ in section 5.4. This could be similar to Truncated trick in Big GAN. How the author think about it?\n\nWe would like to clarify that Section 5.4 actually presents results using an MSE loss function (rather than our proposed EAR method), which can be interpreted as modeling continuous token distributions with Gaussian distributions. In this setup, the standard deviation σ is specified during inference.\n\nOur experiments show optimal performance occurs around σ=0.4, likely because this value best matches the inherent variance of the continuous token distribution. The reviewer's analogy to BigGAN's Truncation Trick is indeed insightful - they share the commonality of potentially trading sample diversity for sample quality. The Truncation trick operates on latent space sampling, while the variance adjustment operates more directly on the predicted distribution's shape.", "claims_and_evidence": "The paper is well-written and provide evidence to most of their claims.", "methods_and_evaluation_criteria": "The evaluation follows standard protocol", "theoretical_claims": "There is no theoretical proof. The paper introduces strictly proper scoring rules and directly apply it to train autoregressive models.", "experimental_designs_or_analyses": "The experimental designs are sound and valid.", "relation_to_broader_scientific_literature": "The strictly proper scoring rule is interesting, the paper is one of first work exploring this as loss for training autoregressive model.", "essential_references_not_discussed": "The paper mentions all essential references.", "other_strengths_and_weaknesses": "**Strength**\n\n1. The idea of using strictly proper score as loss function for continuous autoregressive seems novel to me. This is the first work investigating this idea in visual autoregressive model.\n\n2. The paper writing is clear and easy to understand.\n\n**Weakness**:\n\n1. The ablation for masked autoregressive should be provided with causal and full attention like setting in MAR [1].\n\n2. The experiment in table 1 lacks of comparison with continuous AR like [1], [2]. The paper states that in [2] GMM could limit expressivity line 155 2nd column. However, the performance of proposed method still lag behind GIVT. The author should include the performance of [1] and [2] in the main table and make a comprehensive comparison.\n\n3. Why the random noise in MLP is uniform but not normal. Why uniform is better choice ?. The author should ablate this choice. \n\n4. The inference process is significantly affected by variance $\\sigma$ in section 5.4. This could be similar to Truncated trick in Big GAN. How the author think about it ?. I would like to hear the author opinion. \n\n[1]: Autoregressive Image Generation without Vector Quantization\n\n[2]: GIVT: Generative Infinite-Vocabulary Transformers", "other_comments_or_suggestions": "No", "questions_for_authors": "Please refer to the weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "T4Yi4CdLS1", "reviewer": "Reviewer_1FmY", "summary": "This paper introduces a continuous visual autoregressive framework EAR. The approach is grounded in strictly proper scoring rules, which provide a statistical basis for evaluating generative models, and primarily utilizes an energy score-based training objective to handle continuous data without requiring likelihood estimation.\n\nUpdate after rebuttal: I appreciate the authors' efforts. The response has clarified some of the issues I raised and helped me better understand certain aspects of the work. However, taking into account the points raised by other reviewers, I do not find enough justification to change my overall assessment. Therefore, I will maintain my original rating.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the Reviewer's time and efforts in reviewing our work. We provide discussions about the concerns as follows.\n\n> While the paper mentions that “GIVT is confined to the pre-defined family of Gaussian mixtures,” this is not necessarily a drawback. Instead, it may be the result of deliberate selection by the GIVT authors after thorough comparisons. The paper fails to explain why this limitation is a concern and to motivate its ideas.\n\nWe appreciate the reviewer’s thoughtful feedback. While we acknowledge that the Gaussian mixture assumption in GIVT may indeed represent a deliberate and well-justified choice for modeling continuous tokens, we respectfully disagree with the assertion that this assumption \"is not necessarily a drawback\".\n\nThe fundamental limitation arises when using likelihood maximization to train continuous autoregressive models: the predictive distribution must be constrained to a predefined parametric family (e.g., Gaussian mixtures) to enable tractable likelihood estimation. While Gaussian mixtures may offer the best fit within this constrained framework, this approach inherently restricts the model’s ability to represent arbitrary target distributions—a limitation that cannot be fully mitigated by the choice of distribution family.\n\nIn contrast, our proposed EAR framework circumvents this challenge through its inherently expressive architecture. By implicitly defining the predictive distribution via its sampling process, EAR avoids restrictive parametric assumptions and enables automatic learning of complex data distributions. We will clarify this distinction more explicitly in the revised manuscript to better motivate our method.\n\n> The authors claim that continuous AR is superior to discrete AR, but the performance of their proposed EAR (Table 1) does not consistently support this assertion. For instance, EAR underperforms compared to MAGVIT-v2. Moreover, the training cost of EAR is significantly higher than existing approaches: EAR requires 800 training epochs, whereas VAR only needs 200 to 350 epochs.\n\nWe appreciate the reviewer’s attention to the empirical results. Our claim regarding the advantages of continuous AR modeling is primarily supported by the comparison in ​Figure 3, where EAR outperforms its discrete counterpart (using the VQ tokenizer in [1]). This demonstrates the benefits of continuous modeling when controlling for other architectural factors.\n\nRegarding MAGVIT-v2, we acknowledge that EAR currently uses a ​three-year-old continuous tokenizer architecture, which may limit its performance compared to state-of-the-art discrete tokenizers. We hypothesize that integrating a ​more advanced, fine-grained continuous tokenizer could better showcase the potential of continuous AR modeling.\n\nAs for training efficiency, we agree that EAR currently requires more epochs due to the ​inherent complexity of learning continuous distributions without restrictive parametric assumptions. Future work could explore optimizations to reduce training costs while retaining EAR’s expressiveness.\n\n[1] High-Resolution Image Synthesis with Latent Diffusion Models.\n\n> The experiments in pages 7-8 mainly include obvious experiments or hyperparameter tuning studies, without exploring more critical aspects, for example, temperature or random noise in the MLP generator.\n\nWe thank the Reviewer for pointing out these concerns. Regarding the temperature, we have tuned it on EAR-B and found that τ=0.98 works better than other choices. Regarding the random noise, we have conducted experiments comparing uniform noise and Gaussian noise, and observed that uniform noise leads to marginally better performance. We will include these ablation experiments in the revised manuscript.", "claims_and_evidence": "Not clear enough. See weaknesses.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. Eq (1) - Eq (7) are from existing papers.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The main framework is from VAR. This paper modifies some components.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is the first to apply strictly proper scoring rules to continuous visual autoregressive modeling.\n2. It offers a theoretical discussion on how different strictly proper scores, such as the energy score, can be leveraged as training objectives, unifying previous continuous autoregressive approaches like GIVT and diffusion loss under a common framework.\n\nWeaknesses:\n1. The paper does not clearly justify why Strictly Proper Scoring Rules are used to analyze Continuous Visual Autoregressive Generation. While the paper mentions that “GIVT is confined to the pre-defined family of Gaussian mixtures,” this is not necessarily a drawback. Instead, it may be the result of deliberate selection by the GIVT authors after thorough comparisons. The paper fails to explain why this limitation is a concern and to motivate its ideas.\n2. The authors claim that continuous AR is superior to discrete AR, but the performance of their proposed EAR (Table 1) does not consistently support this assertion. For instance, EAR underperforms compared to MAGVIT-v2. Moreover, the training cost of EAR is significantly higher than existing approaches: EAR requires 800 training epochs, whereas VAR only needs 200 to 350 epochs.\n3. The experiments in pages 7-8 mainly include obvious experiments or hyperparameter tuning studies, without exploring more critical aspects, for example, temperature or random noise in the MLP generator.", "other_comments_or_suggestions": "See weaknesses.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4dK7lXvOoF", "reviewer": "Reviewer_npsM", "summary": "This paper presents a new continuous visual autoregressive generative framework, which prevents the information loss caused by vector quantization. This framework takes energy score as the training objectives, which is likelihood-free and easy to make probabilistic predictions in the continuous space. In addition, the authors conduct many experiments to demonstrate the effectiveness of the proposed method.\n\n## update after rebuttal \nI appreciate the authors' thorough response and the effort they put into the rebuttal. My concerns have been resolved, thus, I raise my score to accept. Last but not least, the authors are encouraged to include experiments on ​ImageNet $512 \\times 512$ conditional generation in the final version.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the Reviewer's time and efforts in reviewing our work. We provide discussions about the concerns as follows.\n\n> In Table 1, it is essential to show the results of MAR and GIVT. They are two important baselines to compare.\n\nWe fully agree with the Reviewer and will include comparisons with ​MAR and ​GIVT in the main table.\n\n> In Figure 2, the speed for MAR maybe not correct. To my knowledge, the speed of MAR in Figure 2 is slower than MAR paper. Could you provide more details about inference settings?\n\nThe difference in reported speeds arises from differing evaluation protocols:\n\n+ In the MAR paper, the authors measure inference time using the ​maximum batch size and report the ​average time per image, which benefits from parallel computation.\n\n+ In our experiments, we follow the more common practice of benchmarking with a ​batch size of 1 to reflect real-world deployment scenarios where latency matters.\n\nUnder the setting of batch=1, MAR’s diffusion process incurs significant decoding latency due to its iterative nature. While increasing the batch size improves MAR’s throughput, EAR remains approximately ​2× faster than MAR under maximum batch size conditions.\n\n> This paper only shows the comparisons on ImageNet 256x256 conditional generation. It is better to show more results to assess the proposed method, like ImageNet  conditional generation.\n\nWe appreciate the Reviewer’s valuable suggestion. We are currently conducting experiments on ​ImageNet 512×512 conditional generation to further validate the effectiveness of our method. These results will be included in the final version of our paper. \n\n> The ablation studies are insufficient. It is better to investigate more settings, such as the choice of random noise and mask ratio.\n\nWe thank the Reviewer for pointing out these concerns. Regarding the mask ratio, we directly adopted the range of [0.7, 1] from MAR, without further tuning. Regarding the random noise, we empirically observe that uniform noise yields slightly better performance compared to Gaussian noise. Specifically, for EAR-B: w/o CFG, FID increases from ​5.46 (uniform) to ​5.74 (Gaussian); w/ CFG, FID increases from ​2.83 (uniform) to ​2.89 (Gaussian). We will ablate this choice in the revised manuscript.", "claims_and_evidence": "Strengths\n+ The motivation of this paper is strong and meaningful. It is an imperative direction for visual generation. \n+ Unlike quantization-based approaches, the proposed method can avoid the information loss. It is mainly supported by the superior results in Table 1.\n+ Based on energy score, autoregressive models can achieve an efficient inference, due to it only needs a single forward pass for each token prediction.\n\nWeaknesses\n\nNone", "methods_and_evaluation_criteria": "Strengths\n+ The proposed energy loss is simple and intuitive. It addresses the shortcomings of GIVT and diffusion loss.  \n+ The proposed method is easy to implement and easy to understand. Its effectiveness is validated by the results on ImageNet $256 \\times 256$ conditional generation.\n\nWeaknesses\n- The random noise is drawn from a uniform distribution. What the principle for this choice is? If using a gaussian noise, is it ok?\n- Likewise, the masking ratio is sampled in the range of $[0.7, 1.0]$. It is not a default choice. What the principle for this choice is?", "theoretical_claims": "I have check the correctness of any proofs for theoretical claims. And I do not find any errors.", "experimental_designs_or_analyses": "Strengths\n+ The conducted experiments can demonstrate the effectiveness of the proposed method. In addition, the ablation studies on $\\alpha$ and CFG also highlight the suitable value of these hyperparameters. \n\nWeaknesses\n- In Table 1, it is essential to show the results of MAR and GIVT. They are two important baselines to compare.\n- In Figure 2, the speed for MAR maybe not correct. To my knowledge, the speed of MAR in Figure 2 is slower than MAR paper. Could you provide more details about inference settings?  \n- This paper only shows the comparisons on ImageNet $256 \\times 256$ conditional generation. It is better to show more results to assess the proposed method, like ImageNet $512 \\times 512$ conditional generation.\n- The ablation studies are insufficient. It is better to investigate more settings, such as the choice of random noise and mask ratio.", "relation_to_broader_scientific_literature": "To my knowledge, the proposed method in this paper is new.", "essential_references_not_discussed": "To my knowledge, there is no other references to be discussed.", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "I appreciate the effectiveness of the proposed method on image generation, although the experiments of this paper are insufficient. Thus, I prefer to rate this paper as weak accept. If authors addresses my concerns, I would like to raise my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel framework, Energy-based Autoregression (EAR), for continuous visual autoregressive generation. It uses strictly proper scoring rules to enable direct, quantization-free autoregressive modeling of continuous visual data. By formulating training objectives based on the energy score, the proposed method achieves superior performance and avoids the limitations associated with existing continuous autoregressive models using Gaussian mixtures or diffusion models to model the per-token continuous distribution. Reviewers highlighted the strong theoretical grounding, methodological novelty, and compelling empirical results on ImageNet. Despite minor initial concerns regarding ablations and comparative experiments, the authors effectively addressed these points in their rebuttal. Given the above reasons, I recommend acceptance of this paper.
Decision: Accept (poster)