Customize Your Visual Autoregressive Recipe with Set Autoregressive Modeling

## Abstract

Abstract We introduce a new paradigm for AutoRegressive (AR) image generation, termed Set AutoRegressive Modeling (SAR). SAR generalizes the conventional AR to the next-set setting, i.e. , splitting the sequence into arbitrary sets containing multiple tokens, rather than outputting each token in a fixed raster order. To accommodate SAR, we develop a straightforward architecture termed Fully Masked Transformer . We reveal that existing AR variants correspond to specific design choices of sequence order and output intervals within the SAR framework, with AR and Masked AR (MAR) as two extreme instances. Notably, SAR facilitates a seamless transition from AR to MAR, where intermediate states allow for training a causal model that benefits from both advantages of AR and MAR, such as few-step inference, KV cache acceleration and image editing. On the ImageNet benchmark, we carefully explore the properties of SAR by analyzing the impact of sequence order and output intervals on performance, as well as the generalization ability regarding inference order and steps. We further validate the potential of SAR by training a 900 900 900 900 M text-to-image model capable of synthesizing photo-realistic images with any resolution. We hope our work may inspire more exploration and application of AR-based modeling across diverse modalities.

## 1 Introduction

The success of AutoRegressive (AR) models in Large Language Models (LLMs) (Radford, ; Radford et al., ; Brown, ; Raffel et al., ; Yang, ; Touvron et al., ) has also driven their development in image generation, where some recent work (Ramesh et al., ; Yu et al., ; ; Tian et al., ; Li et al., ; Sun et al., ; Liu et al., ) has demonstrated that the generative capabilities of AR models can rival or even surpass those of diffusion models (Song & Ermon, ; Song et al., ; Ho et al., ; Dhariwal & Nichol, ; Rombach et al., ; Song et al., ; Lipman et al., ; Liu et al., ; Karras et al., ; Peebles & Xie, ; Esser et al., ; Gao et al., ; Zhuo et al., ).

Despite their strong performance, the large number of inference steps due to ‘next-token prediction’ has become a bottleneck. This limitation has inspired some efficient approaches, with the idea of outputting multiple tokens simultaneously. Existing work (Chang et al., ; Yu et al., ; Chang et al., ; Li et al., ; ; Ni et al., ) usually adopts BERT-like (Devlin, ) masked modeling approaches to exchange the cost of always performing global computations (thus KV cache is not allowed) for fewer inference steps. Another stream of work designs proper sequence orders and arranges multiple tokens with similar properties into one group, to predict these tokens at once, e.g., the scale-aware order (Tian et al., ; Zhang et al., ; Ma et al., ). We conclude that, in the training phase, these approaches pay attention to two aspects: one is the sequence order, the other is the output intervals. The defined order and intervals split the sequence into token sets. AR splits the sequence into sets of single tokens, VAR (Tian et al., ) builds multi-scale sets for an image, and Masked AR (MAR) (Chang et al., ; Li et al., ; ) randomly divides the sequence into a masked set and an unmasked set. Fig. 2 (a1, a2) illustrates examples for AR with intervals of length $1$, while (d1, d2) demonstrates MAR with $2$ output intervals.

In this work, we present *Set AutoRegressive Modeling* (SAR), extending causal learning by generalizing sequence order and output intervals to arbitrary configurations. Specifically, compared with AR that splits the training process into sub-processes that output one single token in fixed raster order, SAR is able to input token sequence in any order (some examples are illustrated in Fig. 1 and Fig. 5), and splits it into any number of token sets, each as a sub-process that output multiple tokens. In order to represent the sequential relationship of token sets, we introduce generalized causal masks. As shown in Fig. 2, the classical causal mask (a1) is a lower triangular matrix; when the set contains more than one token (b1, c1, d1), the matrix becomes block-wise and is called a generalized causal mask. Within our framework, we show that AR, VAR (analogously), and MAR emerge as special cases of SAR, with AR and MAR representing two extreme instances. Refer to the left side of Fig. 2 and Table 1 for conceptual illustrations. Moreover, with the new formulation, we offer a path for smoothly transiting between AR and MAR. The intermediate states of SAR enable one to train causal models that inherit both merits of AR and MAR, such as few-step inference, KV cache acceleration, and image editing.
Given that classical AR models, such as the decoder-only transformer, fail in the SAR setting, we propose a simple model architecture termed *Fully Masked Transformer (FMT)*. FMT adopts the encoder-decoder structure proposed in the original transformer (Vaswani, ) to enable both recording the output position and facilitating position-aware interaction between seen and output tokens. And it incorporates generalized causal masks into each attention process to keep the causal manner, and the details can be referred to Fig. 4.

Under the SAR framework with FMT, we conduct experiments to explore the properties of SAR on the ImageNet $256\times 256$ benchmark. We examine the effect of sequence order and output intervals on generation performance as well as the generalization ability across inference order and steps, and discuss the associated trade-offs.
Then, we train a $900$M text-to-image model on $20$ million high-aesthetic images to further validate the generation potential of the transition states in SAR. Using limited computational resources and data, the trained model demonstrates the capability to produce photo-realistic images of arbitrary aspect ratios that adhere to the text descriptions. Its flexibility of outputting tokens in any order also enables effective application in zero-shot image editing tasks, such as inpainting and outpainting.

Our main contributions are:

- i)
We propose Set AutoRegressive Modeling, that unifies existing AR variants and offers new states between the two extremes, AR and MAR. The new states introduce models that incorporate the advantages of both AR and MAR.
- ii)
In line with SAR, we design a transformer model named Fully Masked Transformer, which enables causal learning with any sequence order and any output intervals.
- iii)
We conduct extensive experiments to investigate the properties of SAR and the modeling capability of FMT. With a particular focus on the transition states, we explore the effectiveness of text-to-image generation and editing.

Figure: Figure 2: Conceptual illustration. SAR integrates existing AR variants by manipulating the sequence order and output intervals, creating a smooth transition path from classic AR to MAR.
Refer to caption: x2.png

**Figure 3: Sequence in any order can be rearranged as a causal one.**
| Method | AR | VAR | MAR | SAR |
| --- | --- | --- | --- | --- |
| Few-step inference | ✗ | ✓ | ✓ | ✓ |
| KV cache | ✓ | ✓ | ✗ | ✓ |
| Training/inference | Match | Match | Unmatch | Flexible |
| Common VAE | ✓ | ✗ | ✓ | ✓ |

## 2 Related Work

### 2.1 Autoregressive and Masked Modeling

Originating in language processing, GPT series (Radford, ; Radford et al., ; Brown, ) and BERT (Devlin, ) are representative works in autoregressive and masked modeling respectively. During the AR training, the current output token can only be observed by the preceding tokens. At inference tokens remain unchanged once output, facilitating the use of KV cache acceleration. Recently some work (Cai et al., ; Gloeckle et al., ) studies to reduce the inference steps by training multiple prediction heads and conducting speculative decoding (Leviathan et al., ; Chen et al., ) at inference. In contrast, BERT (Devlin, ) employs a bidirectional modeling approach known as masked modeling, to capture contextual information. It randomly masks a portion of tokens at a high masking ratio and trains the model to predict these masked tokens. At inference, BERT models can iteratively generate the output sequence with fewer steps than AR methods, at the cost of global calculation. Additionally, some works have introduced context perception into AR models. For example, XLNet (Yang, ) integrates insights from BERT by permuting the input sequence to enable bidirectional training with AR models. On image modality, our work not only provides further unification of AR and BERT models but also builds a smooth path connecting AR and BERT, where one can train models with both their merits.

### 2.2 Autoregressive Image Generation

By tokenizing continuous images into discrete tokens using VQ-VAE (Van Den Oord et al., ; Razavi et al., ; Esser et al., ), image synthesis can be accomplished by AR models (Esser et al., ; Lee et al., ; Ramesh et al., ; Yu et al., ; ; Liu et al., ; Luo et al., ) just like language modeling. Recently, Llamagen (Sun et al., ) verifies the generation capability of plain LLM, Llama (Touvron et al., ) on image modality. VAR (Tian et al., ) divides the image latent space into several scale groups by training a multi-scale VAE, and conduct next-scale prediction. Li et al. () point out that the BERT-like image generation models (e.g., MaskGIT (Chang et al., ), MagViT (Yu et al., ; ), MAGE (Li et al., ), MAR (Li et al., )) can also be regarded as autoregressive ones at inference, and as a result, we call BERT-like image generation models as MAR models. AutoNAT (Ni et al., ) revisits and improves the designs of training and inference process of MAR models. Li et al. () additionally show that autoregressive image generation can also be conducted on continuous latent space with diffusion loss. Our proposed SAR paradigm can encompass the existing approaches as special instances, and provide the users with more flexible design space regarding various trade-offs. The supporting model of SAR is built upon LlamaGen (Sun et al., ) for its plain nature.

## 3 Method

In this section, we first review the AR and MAR paradigms. Then, we point out that conceptually these two methods differ in sequence order and output intervals, based on which we introduce Set AutoRegressive Modeling (SAR), and present the model design.

### 3.1 Preliminary

AutoRegressive Modeling (AR). AR models the distribution of a token sequence $\{x^{1},x^{2},...,x^{n}\}$ by the ‘next-token prediction’ objective defined as

$$ $p(x^{1},...,x^{n})=\prod\limits_{i=1}^{n}p(x^{i}|x^{1},...,x^{i-1})\,,$ (1) $$

where $p(x^{1},x^{2},...,x^{n})$ is the probability density function.
Regarding the implementation, AR models are typically a decoder-only transformer with causal masks, as shown in Fig. 2 (a1). During training, the input to the model is set as the sequence shifted by one position, i.e., dropping the last token, and padding a class token at the beginning (under the class-conditioned setting). The target is the original sequence, such that each output token is aligned with its ‘next token’. At inference, the model can output tokens one by one in an autoregressive manner.

**Table 2: Some examples on SAR setting. $\textrm{rand}(N,K)$ means randomly generate $K$ natural numbers, whose total sum is $N$.**
| SAR | order | #sets | intervals |
| --- | --- | --- | --- |
| AR | raster | $N$ | $1,1,1,...$ |
| VAR | custom | $log_{4}N+1$ | $1,4,16...$ |
| MAR | random | 1 | $\textrm{rand}(N,2)$ |
| Transition | random | K | $\textrm{rand}(N,K)$ |

Masked AutoRegressive Modeling (MAR). MAR has recently been abstracted by Li et al. (), which describes the inference process of BERT-like (Devlin, ) image generation methods (Chang et al. (); Li et al. (); Yu et al. (; ); Li et al. ()). In training, the input tokens are randomly masked with a high ratio (e.g., $70\%-100\%$ in Li et al. ()), and the model is trained to learn to predict the masked part. Fig. 2 (a2) and (d2) illustrate that AR trains $n$ sub-processes in a single iteration, while MAR processes one sub-process at a time. At inference, these methods can predict multiple tokens at once, costing less number of steps than AR models. However, because the masked modeling process is not causal, it does not support causal techniques, e.g., KV cache acceleration. Li et al. () define ‘next set-of-tokens prediction’ as

$$ $p(x^{1},...,x^{n})=p(X^{1},...,X^{K})=\prod\limits_{k=1}^{K}p(X^{k}|X^{1},..., X^{k-1})\,,$ (2) $$

where $X^{k}=\{x^{i},x^{i+1},...,x^{j}\}$ is a set of tokens to be predicted at the $k$-th step. Eq. 2 generalizes vanilla next-token prediction Eq. 1 at inference time.

### 3.2 Set AutoRegressive Modeling

Sequence order and output intervals characterize autoregressive paradigms. Actually, the token sequence in any output order can be rearranged into a causal one. AR is the simplest case whose input sequence is inherently causal. The other two instances with respect to an $8\times 8$ image token grid are shown in Fig. 1. The left order is derived by downsampling the tokens using nearest neighbor interpolation (so the token values stay unchanged after interpolation). We make the model progressively output tokens downsampled with a scale factor of $1/8$, $1/4$, and $1/2$, and finally the rest of the tokens in a scale-aware order. It shares a similar spirit with VAR (Tian et al., ), so we call it a ‘next-scale’ variant. In this case, we can rearrange the tokens in the scale order. The right subfigure corresponds to mask modeling. By putting the unmasked tokens at the front and masked ones as the rest, we also derive a causal sequence.

Next, we consider the output intervals. For example, the output intervals of the ‘next-scale’ variant in Fig. 1 are $1,4,16,43$, while those of the masked variant are the number of masked tokens and unmasked tokens. Since these variants output multiple tokens in each interval, they should be paired with generalized causal masks in training. Some conceptual instances are shown in Fig. 2 (b1, c1, d1), where generalized causal masks extend the classical causal mask (a1) to a block-wise format. The generalized causal mask can be uniquely determined by the output intervals.

Figure: Algorithm 1 SAR Training

SAR generalizes AR by extending the sequence order and the output intervals to any possible scenarios. In Fig. 2 (a1, d1) we can see that the causal mask of AR and MAR are two extreme cases. In the intermediate states of SAR, one can train causal models with few-step inference enabled, which do not appear in either AR or MAR families. For example, if a $8$-token sequence is split into $4$ sets with $1,2,2,3$ tokens, the causal mask should be like that in Fig. 2 (b1). In short, SAR extends ‘next-set prediction’ in Eq. 2 to the training phase.

The model implementation—Fully Masked Transformer. The realization of SAR is not straightforward, though. Classical AR models, e.g., the decoder-only transformer fail in three aspects. i) When AR shifts the sequence to align the current set with the previous set, it will find the number of tokens may not be equal. ii) AR models can only model the output-seen relations with fixed and simple ‘next token’ forms of relative positional relationships, rendering them ineffective in complex scenarios involving arbitrary sets. iii) Given a token at a specific position, AR models output it based on its relative steps to the first token, leading to failure when outputting arbitrary sets. These drawbacks inspire the design philosophy: i) the model should have perception of absolute positions for outputting arbitrary token sets, and ii) the output tokens and the seen tokens should be placed into two containers, each with positional encoding, to facilitate their position-aware interaction.

Starting from the AR model, we split the decoder-only transformer into two parts, an encoder and an decoder. The encoder takes in the image tokens and extract the semantic features. The decoder records the output position with position embeddings and models the interaction between output tokens and seen tokens from the encoder, at the cost of adding cross-attention in each decoder layer.

Figure: Figure 4: The model architecture of Fully Masked Transformer. Conceptually, it is the transformer in Vaswani () plus generalized causal masks.
Refer to caption: x4.png

Additionally, generalized causal masks are added into each attention process, following the spirit of ‘the current token set to be predicted can only see the preceding sets’. In short, it can be regarded as a vanilla encoder-decoder transformer (Vaswani, ) with generalized causal masks in all attention processes, as shown in Fig. 4. Consequently, we refer to it as the Fully Masked Transformer (FMT). Due to the fully causal feature, FMT naturally supports causal techniques like KV cache acceleration.

The training procedure. In order to train one model under the SAR framework, one should first specify the hyper-parameters, sequence order and output intervals. Based on the order setting, we first rearrange the sequence to the causal version (Fig. 1). And we set the target as the rearranged causal sequence. Next, based on the output intervals, we drop the last set of the rearranged sequence and prepend a class token. The resulting sequence is then fed into the encoder. Then the model can be trained with the common cross entropy loss. We list several combinations of sequence order and output intervals in Table 3, where we also add the number of sets for better understanding. The overall training procedure is illustrated in Algorithm 1.

The inference configuration. Since SAR is a generalized AR framework, it naturally supports advanced strategies developed for AR models, such as top-k, top-p, and min-p (Nguyen et al., ) sampling. In this work, we directly apply some simple strategies for inference; one may also customize their own inference schedules. The inference algorithm is summarized in Algorithm 2.

## 4 Experiments

### 4.1 Experimental Settings

We conduct exploratory experiments on ImageNet (Deng et al., ) $256\times 256$ benchmark. We use the tokenizer provided by Sun et al. (), and precompute the image codes before training as in Sun et al. ().
We always use a batch size of $256$ and learning rate of $1e-4$ during training. Models in the transition states SAR-TS in Table 4 is trained for $300$ epochs, while all other models are trained for $200$ epochs. Other training settings follow Sun et al. (). For evaluation, we report the common used FID (Heusel et al., ), IS (Salimans et al., ), Precision and Recall metrics. Unless otherwise specified, the default setting is cfg=$2.0$, top-k=0 0 (all), top-p=$1.0$, temperature=$1.0$. The evaluation is conducted following Dhariwal & Nichol ().

### 4.2 Hyper-parameters in SAR

Configuration on sequence order and output intervals for training.
We test several hyper-parameter combinations containing some common settings and two customized ones named ‘next-scale’ and ‘masked modeling’. Among the common settings, we control the sequence order, the output schedule, and the number of sets, where the latter two jointly determine the output intervals. There are six choices in order, among which the first four is shown in Fig. 5. (a) The ‘raster’ order is the classical AR order, while (b) is its reversed version. (c) and (d) are the ‘Swiss roll’, clockwise, from outside to inside and from inside to outside respectively. The other two are fixed-random and random. The former means that we randomly generate an order and fix it during training, while the latter indicates generating random orders online at each training step.

Figure: Figure 5: Some sequence order settings in the experiment. Taking the $8\times 8$ case as illustration.
Refer to caption: x5.png

There are two types of output schedules involved, which can determine the output intervals based on the number of sets as follows: i) cosine: given a set number $K$, the output intervals $\{n_{i}\}_{1}^{K}$ follows $n_{i}=[N(\textrm{cos}(\frac{\pi}{2}\frac{i}{K})-\textrm{cos}(\frac{\pi}{2}
\frac{i-1}{K}))]$, as in Li et al. (). Note: here at least one token is ensured to be output at each step, thus given sequence order as raster and set number as $256$, it will recover to AR. ii) random: given a set number $K$, we randomly generate $K-1$ natural numbers (there may be equal numbers) between 0 0 and $N$ with the same probability, such that the sequence can be split into $K$ intervals by these partition numbers. Under the common settings, we conduct experiments in the format of (sequence order)-(number of sets)-(output schedule). For example, raster-$64$-cosine indicates a raster-order sequence with $64$ sets under a cosine schedule.

The customized settings includes i) next-scale: we rearrange the $16\times 16$ image tokens such that the $1$st set contains the $1/16$ nearest neighbor downsampled token, the $2$nd set contains the four $1/8$ downsampled tokens, …, and the $5$th set contains the rest of tokens, as illustrated on the left of Fig. 1, and ii) masked modeling: we follow the settings in Li et al. (). Actually it can be derived by removing the loss of the first token set and modifying the random strategy in ‘random-$2$-random’.

Configuration on model size of FMT. The implementation of FMT is based on the GPT model in LlamaGen (Sun et al., ). For simplicity, we do not adopt the asymmetric design in He et al. (), but just divide the $N$-layer transformer into an encoder and a decoder, each with an equal number of layers. One can refer to Table 3 for detailed model configurations.
Compared with LlamaGen, we add an extra cross-attention module at each decoder layer, so under the same model size, the number of parameters of FMT is slightly larger.

### 4.3 Main results

Figure: Figure 6: Trade-off between performance and time, using LlamaGen-L as a reference.
Refer to caption: extracted/5920525/figures/legend_time.png

**Table 4: Performance comparison among various paradigms and models. ‘-re’ means rejection sampling. For LlamaGen (Sun et al., ), * means direct training on $256\times 256$ images; otherwise, training is on $384\times 384$ and the output is resized in evaluation. ‘TS’ denotes transition state.**
| Type | Model | #Params | FID$\downarrow$ | IS$\uparrow$ | Precision$\uparrow$ | Recall$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| GAN | BigGAN (Brock, ) | 112M | 6.95 | 224.5 | 0.89 | 0.38 |
| GigaGAN (Kang et al., ) | 569M | 3.45 | 225.5 | 0.84 | 0.61 |  |
| StyleGAN-XL (Sauer et al., ) | 166M | 2.30 | 265.1 | 0.78 | 0.53 |  |
| Diffusion | ADM (Dhariwal & Nichol, ) | 554M | 10.94 | 101.0 | 0.69 | 0.63 |
| CDM (Ho et al., ) | - | 4.88 | 158.7 | - | - |  |
| LDM-4 (Rombach et al., ) | 400M | 3.60 | 247.7 | - | - |  |
| DiT-XL/2 (Peebles & Xie, ) | 675M | 2.27 | 278.2 | 0.83 | 0.57 |  |
| Masked AR | MaskGIT (Chang et al., ) | 227M | 6.18 | 182.1 | 0.80 | 0.51 |
| MaskGIT-re (Chang et al., ) | 227M | 4.02 | 355.6 | - | - |  |
| MAGE (Li et al., ) | 230M | 6.93 | 195.8 | - | - |  |
| MAR-H (Li et al., ) | 943M | 1.55 | 303.7 | 0.81 | 0.62 |  |
| (SAR, K=1) | FMT-B | 125M | 6.98 | 222.28 | 0.87 | 0.36 |
|  | FMT-L | 394M | 6.13 | 278.81 | 0.88 | 0.40 |
| VAR | VAR-d30 (Tian et al., ) | 2.0B | 1.92 | 323.1 | 0.82 | 0.59 |
| VAR-d30-re (Tian et al., ) | 2.0B | 1.80 | 356.4 | 0.83 | 0.57 |  |
| (SAR, customized) | FMT-B | 125M | 12.49 | 148.53 | 0.76 | 0.36 |
| AR | VQGAN (Esser et al., ) | 1.4B | 15.78 | 74.3 | - | - |
| VQGAN-re (Esser et al., ) | 1.4B | 5.20 | 280.3 | - | - |  |
| ViT-VQGAN (Yu et al., ) | 1.7B | 4.17 | 175.1 | - | - |  |
| ViT-VQGAN-re (Yu et al., ) | 1.7B | 3.04 | 227.4 | - | - |  |
| RQTran. (Lee et al., ) | 3.8B | 7.55 | 134.0 | - | - |  |
| RQTran.-re (Lee et al., ) | 3.8B | 3.80 | 323.7 | - | - |  |
| LlamaGen-B* (cfg=2.00) | 111M | 5.46 | 193.61 | 0.84 | 0.46 |  |
| LlamaGen-L (cfg=2.00) | 343M | 3.07 | 256.06 | 0.83 | 0.52 |  |
| LlamaGen-XL (cfg=1.75) | 775M | 2.62 | 244.08 | 0.80 | 0.57 |  |
| LlamaGen-L* (cfg=2.00) | 343M | 4.41 | 288.17 | 0.86 | 0.48 |  |
| LlamaGen-XL* (cfg=1.75) | 775M | 3.39 | 227.08 | 0.81 | 0.54 |  |
| (SAR, K=N) | FMT-B (cfg=2.00) | 125M | 5.40 | 216.93 | 0.87 | 0.42 |
|  | FMT-L (cfg=2.00) | 394M | 3.72 | 297.54 | 0.86 | 0.49 |
|  | FMT-XL (cfg=1.75) | 893M | 2.76 | 273.76 | 0.84 | 0.55 |
| SAR-TS | FMT-B (cfg=2.00) | 125M | 7.19 | 186.20 | 0.85 | 0.39 |
| FMT-L (cfg=2.00) | 394M | 4.67 | 246.46 | 0.84 | 0.46 |  |
| (random-$16$-random) | FMT-XL (cfg=1.90) | 893M | 4.01 | 250.32 | 0.82 | 0.50 |

**Table 5: FID results of training/inference with different order settings. The model is FMT-B.**
| Training/inference | raster | reversed-raster | roll | reversed-roll | fixed-random | random |
| --- | --- | --- | --- | --- | --- | --- |
| raster | 5.40 | 136.54 | 114.41 | 99.13 | 132.61 | 120.82 |
| reversed-raster | 133.18 | 6.01 | 123.47 | 118.67 | 146.48 | 138.29 |
| roll | 81.93 | 114.23 | 6.93 | 133.50 | 130.28 | 117.69 |
| reversed-roll | 125.78 | 134.25 | 155.04 | 6.44 | 128.62 | 125.56 |
| fixed-random | 104.24 | 117.23 | 116.58 | 103.03 | 7.49 | 86.90 |
| random | 22.95 | 22.91 | 13.66 | 10.32 | 7.83 | 7.76 |

Table 4 presents a comprehensive comparison of performance across various methods and models, where we train models for each AR setting within the SAR paradigm.

SAR as AR. The raster-$256$-cosine variant of SAR recovers to conventional AR. We evaluate the performance of FMT-B, FMT-L, and FMT-XL, with the results presented in Table 4. Under the same setting (stared in Table 4, directly training at $256\times 256$), FMT outperforms LlamaGen under the same model size.

SAR as MAR. SAR recovers to MAR under the ‘masked modeling’ setting. The performance of FMT is also shown in Table 4.

SAR as VAR, analogously. By customizing the sequence order and output intervals as ‘next-scale’, illustrated on the left side of Fig. 1, we derived a rough variant of VAR. The results are presented in Table 4. While this serves primarily as a conceptual example, its performance lags significantly behind that of VAR (Tian et al., ).

Transition states of SAR. The last three rows of Table 4 present the performance ($64$ steps) of a specific design choice in the transition states of SAR, which will be detailed in the ablation study. Compared to FMT under the AR configuration, the performance in this case is somewhat lower. However, models trained under this setting can generalize across inference steps and orders while maintaining their causal features. A straightforward merit is that, we can enable KV cache acceleration while performing few-step inference. A diagram on performance-time trade-off is shown in Fig. 6, where the inference time is tested by generating a batch of $8$ images on one A100 GPU. We may also apply other causal techniques to promote the performance or efficiency.

### 4.4 Ablation Study

Varying sequence orders in training/inference. Table 5 presents the results obtained by fixing the output intervals to $1,1,\ldots$ while training and inferring with various sequence orders. It is clear that although position embeddings are used, a fixed sequence order typically does not allow the model to generalize across different inference orders.

Fixed few-step generation. By fixing the sequence order to the raster order and using a cosine schedule for the intervals, we investigate few-step SAR training by varying only the number of sets. As illustrated on the left of Fig. 8, we observe that, i) since both the order and the schedule are fixed, the best inference performance typically occurs when the number of sets used at inference matches that used in training; ii) from the inset in the upper right, it is evident that only the $64$-set configuration is effective for few-step generation, while the others significantly degrade performance.

Randomness in orders enables few-step generalization. We fix the number of sets at $256$ and the interval schedule to $1,1,\ldots$, varying only the sequence order. As shown in Fig. 5, models trained with the raster, reversed raster, roll, and reversed roll orders struggle to generalize to few-step generation. In contrast, models trained with a random order demonstrate good generalization across inference steps, albeit at the cost of lower FID scores ($5.40$ FID with raster order vs. $7.76$ FID with random order). It may be surprising that fixing a randomly generated order during training can achieve similar generalization ability to that of a fully random order.

Figure: Figure 8: Effect of set numbers when training SAR with (left) raster order and cosine schedule, (middle) raster order and random schedule, and (right) random order and cosine schedule.
Refer to caption: extracted/5920525/figures/legend_ntxc.png

Figure: Figure 9: Exploration when sequence order and output schedule are both set as random. Left: performance wrt. number of sets. Middle: after causal training, comparison between causal and full attention calculation. Right: training loss of various set numbers.
Refer to caption: extracted/5920525/figures/legend_rxr.png

Random output intervals enables few-step generalization. We fix the sequence order to raster and use a random schedule with varying numbers of sets. The results on the middle of Fig. 8 indicate that when the number of sets is large (e.g., $64$ or $256$), random intervals facilitate few-step generalization.

The relationship between number of sets and causal learning. Under the setting of random sequence order, we examine performance in relation to the number of sets. Figures 8 (right) and 9 (left) show the results with cosine and random output schedules, respectively. We observe that, with a large number of sets, performance remains stable; however, it declines significantly when the set number decreases to $4$ in the cosine case and $2$ in the random case. Intuitively, to develop a causal model, the model must be trained to predict sets one by one, with more sets indicating a greater degree of causality. If the number of sets is too small, the model struggles to learn causal relationships effectively. Another interesting observation is that, after trained with small number of sets, abandoning causality can help restore performance. As shown in the middle of Fig. 9, the performance of the model trained with $2$ sets gets better when replacing the causal attention with full attention. However, model trained with other set numbers cannot benefit from full attention, because they receive more sufficient causal learning. The last subfigure of Fig. 9 illustrates the loss curves during training, where the level of loss may be regarded as a measure of training difficulty. The loss of the best-performing configuration, $16$ sets, is situated at a mid-level.

Further discussion on the MAR setting of SAR. There are some details that need to be clarified.
i) In Sec. 4, we mentioned that the MAR setting is derived based on ‘random-$2$-random’ by only supervising the second set, and using the random strategy in Li et al. (). From Table 7, Row 1 vs. Row 2 tells us that, with the same model, removing the loss of the first set has little impact on model training; not removing it may even lead to better performance. This fact demonstrates that the transition from $K=2$ to $K=1$ (i.e., MAR) in SAR is smooth. ii) It is worth noting that, in the MAR case the generalized causal masks in the encoder self-attention and decoder cross-attention is equivalent to having none. And only the causal mask in decoder self-attention will affect the training. Intuitively, there is no need to prepare causal mask in training because at inference MAR always conduct global attention. Row 1 vs. Row 3 in Table 7 indicates that the existence of causal mask in decoder self-attention hurts the performance. iii) Row 4 is a setting from Fig. 9. The large discrepancy in performance between Row 2 and Row 4 emphasizes the importance of proper random strategy. This also suggests that our strategy for SAR transition states may not be optimal, which may explain the sub-optimal SAR-TS results in Table 4.

**Table 6: Results among detailed MAR settings. The inference process is BERT-like, with full attention.**
| Random Strategy | K | Causal Mask | FID$\downarrow$ | IS$\uparrow$ | Precision$\uparrow$ | Recall$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| MAR (Li et al., ) | 1 | ✓ | 8.81 | 148.36 | 0.76 | 0.46 |
| MAR (Li et al., ) | 2 | ✓ | 7.19 | 183.31 | 0.83 | 0.39 |
| MAR (Li et al., ) | 1 | ✗ | 6.98 | 222.28 | 0.87 | 0.36 |
| Equal Probability | 2 | ✓ | 29.20 | 46.91 | 0.65 | 0.52 |

### 4.5 Application: Text-to-Image Generation

Figure: Figure 10: Step number and time cost of Lumina-SAR at $1024\times 1024$ (full $4096$ steps cost $174.49s$).
Refer to caption: x6.png

We leverage the FMT-XL model for text-to-image (T2I) generation. The sequence order and the output schedule are set as random, the best practice with random order in ImageNet experiments. We adopt the training strategy with multiple aspect ratios as in Gao et al. (); Zhuo et al. () and the multi-stage policy in Zhuo et al. (); Sun et al. (); Chen et al. (). Specifically, we set the number of sets as $16$ and the base resolution as $256\times 256$ in the first stage, and gradually increase the number of sets and the base resolution by a factor of $2$. The final resolution is $1024$. At each training stage, we group images with different aspect ratios but similar pixel numbers and pad them to the same length. As for the language part, we adopt the Gemma-2B (Team et al., ) as the text encoder and concatenate the text embedding with the image tokens, with the conventional causal mask like that in Fig. 2 (a1). Other training settings including text-image training data are following Zhuo et al. (), and we name our T2I model as Lumina-SAR. As visualized in Fig. 1, Lumina-SAR can flexibly produce photo-realistic images in arbitrary resolutions.

Inference time. We examine the time cost of Lumina-SAR for generating one image using one A100 GPU, as illustrated in Fig. 10. We observe that Lumina-SAR begins to produce meaningful images at around $4$ to $8$ steps. With $64$ to $128$ steps, it can deliver high-quality outputs, requiring a processing time of only $3$ to $6$ seconds. Typically, the full $4096$ steps take $>60$ times longer than that required for $64$ steps. A detailed comparison of inference times among AR, MAR, and SAR-TS models is presented in Table 7. To ensure a fair comparison, we consistently use FMT-XL with a resolution of $1024\times 1024$, varying only the inference manner. Notably, in the transformer decoder, MAR applies global attention across all tokens, while the number of tokens processed in AR and SAR-TS increases gradually. Consequently, even with KV cache disabled, the inference time for SAR-TS is shorter than that of MAR; when KV cache is enabled, SAR-TS is three times faster than MAR with $64$ or $128$ steps.

Figure: Figure 11: Zero-shot image painting with Lumina-SAR. Gray color indicates masked regions, and the text enclosed in quotes represents the input text prompt.
Refer to caption: x7.png

Zero-shot image painting. One of the advantages of using random sequence orders is the flexibility in inference order, which facilitates image editing tasks such as image inpainting and outpainting. This is an important feature that AR lacks but MAR (Chang et al., ) includes. To validate the painting ability of SAR-TS models, we perform zero-shot painting with Lumina-SAR. Several instances is shown in Fig. 11, where the mask can be any shape.

## 5 Conclusion

In this work, we propose Set AutoRegressive Modeling (SAR), a new AR paradigm with a broader deign space to freely customize the AR training and inference processes. SAR incorporates existing AR variants with flexible sequence order and output intervals. For SAR, we also develop a preliminary architecture called the Fully Masked Transformer. We carefully explore the properties of SAR, with a particular focus on the intermediate states, which integrates advantages of both AR and MAR models. To further validate the generation potential at the transition states, we train a text-to-image model capable of generating high-quality diverse images.

Limitation and future work. As a newly emerging paradigm, the exploration of SAR in this paper is limited, particularly concerning the performance of intermediate states on ImageNet. Future work may focus on developing better training and inference schedules, designing model architectures more compatible with SAR, and exploring the application of SAR across additional modalities.

## Appendix A Appendix

### A.1 More Visualizations of Class-conditioned Generation on ImageNet

For $256\times 256$ image generation on ImageNet, we generate some random samples that are not cherry picked. Fig. 12 and Fig. 13 exhibit samples produced by FMT-XL trained under random-$16$-random and raster-$256$-cosine settings, respectively.

Figure: Figure 12: Samples generated by FMT-XL trained with SAR, random-$16$-random.
Refer to caption: extracted/5920525/figures/r16r_sar-xl.jpg

Figure: Figure 13: Samples generated by FMT-XL trained with SAR, raster-$256$-cosine (i.e., classical AR).
Refer to caption: extracted/5920525/figures/ntp_xl.png

Figure: Figure 14: Samples generated by Lumina-SAR. The model is FMT-XL trained under the random-$x$-random setting of SAR, where $x$ is set as $16$, $32$ and $64$ at the stage of $256\times 256$, $512\times 512$ and $1024\times 1024$ respectively.
Refer to caption: x8.png

### A.2 More Visualizations on T2I Image Synthesis

We provide additional visualizations generated by Lumina-SAR, and show them in Fig. 14. The number of inference steps is $64$.

Figure: Figure 15: Samples generated by Lumina-SAR with $4$, $8$, $16$ and $64$ inference steps.
Refer to caption: x9.png

### A.3 More Instances on Few-step Text-to-image Generation

We provide more T2I examples when sampling with $4$, $8$, $16$, and $64$ steps. As shown in Fig. 15, the generation quality drops slightly with $16$ steps, and becomes much worse with $4$ or $8$ steps. Hence, we recommend a step number of $64$ for high-quality outputs.

### A.4 Details on Fully Masked Transformer

With fixed resolution, the position embedding as input to the decoder can be either learnable or fixed, such as sine embedding. The performances are similar between learned and sine position embeddings in class-conditioned generation. In the T2I model, we use sine embedding to accommodate training with multiply aspect ratios: after each input image is fed into FMT, we first generate its sine embedding.
Similar to LlamaGen (Sun et al., ), we use RoPE Su et al., to enable the position-aware interaction. Both the position embedding and RoPE are rearranged like what is done to the input tokens according to the sequence order, such that the positions are aligned.

Relation between Fully Masked Transformer and the transformer in Vaswani (). Structurally, there are two more generalized causal masks in FMT (at encoder self-attention and decoder cross-attention) to than in Vaswani (). Functionally, the encoder in vanilla transformer is used to encode the context (e.g., the question in question-and-answer tasks, and the class/text tokens in our case), and the decoder serves as the token generator. In FMT, the encoder and decoder together serve as the token generator, while the encoder also functions as encoding seen tokens (including the context and generated tokens). When regarding the transformer as a black box, FMT can work as a classical decoder-only transformer like Llama (Touvron et al., ).

### A.5 On Data Augmentation of SAR-TS Training on ImageNet

**Table 8: The effect of random crop augmentation in training for the SAR random-$16$-random setting. The models are trained for $300$ epochs, and the number of inference steps is $64$.**
| Model | #Params | Random Crop | FID$\downarrow$ | IS$\uparrow$ | Precision$\uparrow$ | Recall$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| FMT-B (cfg=2.00) | 125M | ✓ | 7.04 | 182.01 | 0.84 | 0.40 |
| FMT-B (cfg=2.00) | 125M | ✗ | 7.19 | 186.20 | 0.85 | 0.39 |
| FMT-L (cfg=2.00) | 394M | ✓ | 4.75 | 261.27 | 0.84 | 0.46 |
| FMT-L (cfg=2.00) | 394M | ✗ | 4.67 | 246.46 | 0.84 | 0.46 |
| FMT-XL (cfg=1.90) | 893M | ✓ | 4.24 | 249.23 | 0.82 | 0.51 |
| FMT-XL (cfg=1.90) | 893M | ✗ | 4.01 | 250.32 | 0.82 | 0.50 |

Figure: Figure 16: The framing misalignment issue caused by random crop augmentation when training random-$16$-random models.
Refer to caption: x10.png

In all experiments except SAR-TS models in Table 4, we adopted random crop augmentation following (Sun et al., ). For SAR-TS models, we find that they are sensitive to random crop augmentation, frequently encountering framing misalignment issues in image generation. Some randomly generated examples by FMT-L are shown in Fig. 16. In a batch of eight simultaneously generated images, the first, third, fifth, seventh, and eighth images exhibit this misalignment issue. Our experiments indicate that while random crop augmentation is effective for smaller models (FMT-B), it negatively impacts the FID scores of larger models (FMT-L, FMT-XL), as shown in Table 8. By comparing generated images in Fig. 16, we observe that random crop augmentation contributes to the framing misalignment problem; removing it mitigates this issue and improves the FID score (but to some extent hurts the visual quality as perceived by human eyes). And as a result, we report the quantitative results of SAR-TS models without random crop in Table 4.

### A.6 The Effect of Evaluation Configurations

We provide the results when adjusting the scale of classifier-free guidance and the top-k values in Fig. 17, where we use FMT-L trained under the random-$16$-random setting for $300$ epochs and the number of sampling steps is set to $64$. The inference behavior is similar to that of classical AR models.

Figure: Figure 17: The effect of cfg scale (left), and top-k sampling (right).
Refer to caption: extracted/5920525/figures/legend_cfg.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. How much training/inference speed gain does KV cache give? Some benchmark with AR/MAR models would be helpful to understand the benefits of the model. 
2. What does SAR-TS mean? I may miss something, but I'm confused about how this transition state variant is built. 
3. To better show the benefit of the proposed method, it would be valuable to compare SAR with MAR with the same fixed order both on FID and inference speed. 
4. The model shows nice visualization on text2image generation. Are there any metrics to evaluate the performance over different inference settings like CLIP score?
soundness: 4
strengths: 1. The paper investigates an interesting problem to make autoregressive image generation model more efficient. 
2. The work includes standard benchmarks like ImageNet 256 and achieves competitive performance.
summary: This work investigate image generation with discrete tokens. It proposes Set AutoRegressive Modeling (SAR), which can generate image tokens in arbitrary order and intervals. The authors also develop a variant of encoder-decoder Transformer named FMT to enable KV cache. Experiments on ImageNet show competitive performance and the model also shows capability in text2image generation.
weaknesses: 1. Though the work is motivated by a more flexible framework for image generation, the best performing setting is still using raster order as standard autoregressive model (table 5). 
2. It misses some comparison to baseline models to help understand the benefit of the proposed method. 

More details in Questions section.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: NA
soundness: 3
strengths: - The exploration of set AR could be helpful for future works that uses AR models for image/video generation.

- The paper provides a good way to transform traditional Transformers architecture into fully masked transformer.

- This paper conducts quite a lot experiments and ablations that tests different variants of the model (which I appreciate).
summary: This paper introduces a Set AutoReggressive Modeling (SAR) as a new paradigm for AutoRegressive image generation. By definition, SAR is a much more flexible way compared with AR which generate tokens in a fixed raster order, and covers several variants including Masked AR, as well as VAR. To support the use of SAR, the paper introduces a new variants of the original Transformer architecture, named as fully masked Transformer. Experiments on ImageNet and text-to-image generation has shown that SAR outperforms previous benchmarks including LlamaGen, MaskGiT under comparable settings, which proves the effectiveness of proposed method.
weaknesses: - The idea of using cosine schedule to determine the number of tokens seems not novel [1, 2]

- Although the paper includes many different ablations and variants of SAR design, most of them are probably not very necessary. For example, the ablation of sequence order settings shows raster is the best (Fig. 5), and training/inference with different order settings shows aligned training/inference works the best (Table 5). Both of these experiments are quite intuitive (without any surprise).

[1] Xie, Jinheng, et al. Show-o: One single transformer to unify multimodal understanding and generation. 

[2] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer.


Writing typos: 
- line 231-232: Eq. equation 2/3

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: The paper's presentation is mainly based on the discrete tokenizer. Does it support the continuous tokenizer AR method, like MAR. Could the authors detail the (SAR,K=1) results below mask AR in Table 4.
soundness: 3
strengths: 1. Authors present a unified framework that generalizes existing AR approaches.

2. Good T2I result demonstrates real-world applicability through text-to-image generation.
summary: This paper introduces Set AutoRegressive Modeling (SAR), a new paradigm for autoregressive image generation that generalizes conventional autoregressive (AR) models by allowing the sequence to be split into arbitrary sets containing multiple tokens, rather than outputting each token in a fixed order. The authors propose a Fully Masked Transformer (FMT) architecture to implement SAR. The paper shows that existing AR variants (like AR and Masked AR) are special cases within the SAR framework, and SAR enables smooth transitions between them.
weaknesses: 1. The performance of SAR-TS is suboptimal compared to pure AR or MAR approaches.

2. The authors claim that  Fully Masked Transformer is the first work supporting KV cache for AR image generation. llamagen already has support this feature as llamagen is a causal transformer.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Why not use a decoder-only architecture? At each step, the tokens sampled in the previous step and the positional embeddings of the next set of tokens are fed into the transformer as input, with other generated tokens as the prefix. Tihs apporach would also allow the use of KV cache to store the information of all previously generated tokens.
2. If the training and sampling steps are aligned for the middle sub-figure of Fig.9, it indicates that 2-step sampling achieves a FID of around 30, which is too good to be believed for me. Could the authors provide some code for verification?
soundness: 3
strengths: 1. SAR incorporate several important design choice in AR models, including the number of sets, the number of tokens per set and the sequence order. Additionally, SAR employs Fully Masked Transformer to address the challenges of using arbitrary choices.
2. The paper includes a substantial amount of experimental results:
    - SAR achieves better performance than LlamaGen and reproduces comparable performance with MAR by using appropriate design choices.
    - Extensive ablation studies illustrate the impact of each design choice on the performance-step trade-off.
    - Experiments on text-to-image generation are also conducted.
summary: This paper proposes SAR as a unified framework of AR and MAR. The key components of SAR include the number of sets, the number of tokens per set and the sequence order. To enable flexible design for these components, this work further propose FMT, an encoder-decoder transformer that utilizes an encoder to output the embedding of prefix tokens and generate new tokens via decoder, guided by the positional embeddings. Experimental results are provided to compare SAR with existing methods and show how each component influence the performance of SAR.
weaknesses: 1. Further clarification is needed regarding the fundamental differences between SAR and existing methods. While SAR is presented as a flexible framework, its key components exhibit similarities to those of existing methods.
    - For the number of sets and set length: Fig.2 shows that SAR represents a smooth transition from AR to MAR, as it enables different numbers of sets and set lengths. However, as stated in the paper and based on my understanding, MAR [1,2] also supports different set lengths during inference to achieve a trade-off between steps and performance. The extreme case for MAR mentioned in the paper (i.e., K=1) occurs only during training. Moreover, VAR [3] also uses the set-prediction approach and generates tokens in a causal manner. Therefore, whether viewd from the perspective of extending the inference of MAR to training and using causal architecture, or from the perspective of extending VAR accommodate different numbers of sets and varying set lengths, the modifications in this paper seem to be straightforward. The authors need to clarify the essential differences between the "continuous new state that SAR offers between AR and MAR" and the aforementioned methods. Is the difference only lies in the introduction of a new architecture FMT?
    - For the sequence order: The most general form of sequence order, random order, has already been used by previous MAR works [2]. This paper seems to simply adopt this method.

2. Regarding experimental results:
    - My main concern is that the performance improvement of SAR shown in Tab.4 is relatively limited.
        - For SAR-TS: Currently, the performance of SAR-TS does not significantly exceed other methods.
        - For Masked-AR: The comparison between FMT and Masked AR is incomplete, as no baseline methods use a comparable parameter size to FMT-L or FMT-B. From the results presented, the improvements brought by SAR are also limited (e.g., a 0.05 lower FID than MaskGIT while having 50% more model parameters). Additionally, strong baselines like MAR-L and MAR-B from [2] are not not included.
        - For VAR: There is also no valid comparison with VAR.
        - For LlamaGen: While FMT performs better than the LlamaGen baseline, I would like to confirm what modifications are made by SAR compared to the LlamaGen baseline aside from the new architecture. It seems the token number per set is 1, and Tab.5 shows that raster order performs the best, both of which are settings used by LlamaGen.
    - Additionally, there is a certain dependency between various choices in training and inference, making it difficult to train a model that supports different steps and orders in inference while achieving optimal results. For example, from Tab.5, the negative impact of order mismatch appears significant; From Fig.8 and related discussions in the paper, the mismatch in steps can lead to undesirable results. Therefore, the flexibility of SAR seems somewhat limited.

    Given these concerns, although SAR is a new framework offering various options, the results do not show superiority over existing methods, nor is it fully flexible. The pratical advantages of SAR needs to be further clarfied.

3. Some statements need further clarification:
    - Since random-2-random appears to be a general form of training, which supports predicting an arbitrary number of tokens based on an arbitrary number of tokens, the "mismatch" between MAR's training and inference in Tab.1 needs more explanation.
    - The definitions of $m_e$, $m_{ds}$, and $m_{dc}$ in the Alg.1 and Alg.2 seem to be missing (please correct me if I missed something, thank you).
    - In line 426, why does it state that only 64-sets can work for a few steps? From Fig.8, it seems that 4-sets perform best at 4-steps sampling?
    - As discuss, MAR is a special case of SAR with "random-2-random". However, the authors indicate the K of MAR as 1 in Tab.6 and Fig.2.
    - In the middle sub-figure of Fig.9, are the training steps and sampling aligned? Additionally, do "causal calculation" and "full calculation" refer to causal and full attention masks, respectively?

[1] Chang et al., MaskGIT: Masked Generative Image Transformer

[2] Li et al., Autoregressive Image Generation without Vector Quantization

[3] Tian et al., Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. What does $N$ represent in Line 330 and Line 334? It seems that $N$ refers to the number of tokens in an image based on the notations in Figure 2 (a1), but the authors use $n$ to denote the number of tokens in an image in Equation (1).

2. In Line 342, the authors state that 'random-2-random' indicates the MAR setting. However, as shown in Figure 2 (d1), $K=1$ for MAR. I'm a bit confused; could you please provide further clarification?

3. What do $m_e$, $m_{ds}$ and $m_{dc}$ mean in Algorithm 1 and Algorithm 2?

4. When training with a fixed number $K$, how does SAR inference when the sampling steps differ from $K$?

5. I am concerned about the scaling properties of the encoder-decoder network structure. I recommend that the authors explore the scaling behavior of SAR or even scaling laws in their future work.
soundness: 3
strengths: 1. SAR is a flexible formulation and unifies the existing autoregressive and masked autoregressive models. A unified formulation is interesting.
2. The authors present an effective, new Fully Masked Transformer architecture for SAR.
3. The authors systematically studied the impact of factors such as sequence order and output interval on generation quality. I believe these experimental results provide valuable insights for advancing the field.
summary: This paper introduces a new image generation method called SAR, which extends the traditional autoregressive model to a "next-set" framework. SAR enables flexible sequence ordering and output intervals, thus encompassing both the autoregressive and masked autoregressive models as specific cases. Additionally, the authors propose the Fully Masked Transformer architecture for SAR. Quantitative and qualitative results are presented on class conditional and text-to-image generation.
weaknesses: 1. The authors claim that SAR unifies autoregressive and masked autoregressive models; however, SAR's FID lags significantly behind masked autoregressive models.
2. The authors note that SAR can utilize KV-cache acceleration, whereas masked autoregressive models cannot, and they claim this as an advantage of SAR. However, they do not provide any speed comparisons between SAR and masked autoregressive models. Instead, they only compare generation speed with LlamaGen, which is insufficient. Moreover, it raises questions as to why SAR’s FID is worse than LlamaGen’s in Figure 7, despite both models having a similar number of parameters.
3. The Text-to-Image generation experiment lacks proper evaluation.

Meta-review: This paper proposes an architecture for autoregressive modeling called a Fully Masked Transformer that predicts sets of tokens together, in an arbitrary order, as opposed to standard AR models that predict one token at a time in a fixed order. The architecture of FMT splits the standard decoder-only Transformer into an encoder that encodes image semantics and a decoder stage that decodes output positions and models interactions between semantics and output tokens. The method is flexible in that it allows varying smoothly between AR and MAR decoding. Experiments on class-conditional and text-image settings highlight the model’s capabilities including over multiple aspect ratios and image quality.

Strengths:
Paper focuses on efficiency and flexibility for high-resolution generative modeling, which is of interest to the community. Experiments are thorough, including ablations, and are applied on standard datasets and indicate the method is competitive. 

Weaknesses:
The best performing model in the T2I setting is still the standard AR order model, calling into question the value of the proposed method at least in some cases. There is also some concern/confusion about limited novelty over existing methods.

Decision reasoning:
While the paper presents an interesting direction to unify autoregressive and masked modeling, and has a reasonably thorough experiment section, reviewers remain unconvinced about the practical value of the method and whether the proposed machinery leads to substantial improvements over existing technniques. I encourage the authors to revisit the reviews and try to improve the paper to better motivate the method, and potentially show where different orders are valuable.

Decision: Reject