ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesis
Authors: Zanlin Ni
Yulin Wang
Renping Zhou
Yizeng Han
Jiayi Guo
Zhiyuan Liu
Yuan Yao
Gao Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: Token-based Generation; Non-autoregressive Transformers
Primary area: generative_models

Abstract: Recently, token-based generation approaches have demonstrated their effectiveness in synthesizing visual content. As a representative example, non-autoregressive Transformers (NATs) can generate decent-quality images in just a few steps. NATs perform generation in a progressive manner, where the latent tokens of a resulting image are incrementally revealed step-by-step. At each step, the unrevealed image regions are padded with [MASK] tokens and inferred by NAT, with the most reliable predictions preserved as newly revealed, visible tokens. In this paper, we delve into understanding the mechanisms behind the effectiveness of NATs and uncover two important interaction patterns that naturally emerge from NAT’s paradigm: Spatially (within a step), although [MASK] and visible tokens are processed uniformly by NATs, the interactions between them are highly asymmetric. In specific, [MASK] tokens mainly gather information for decoding. On the contrary, visible tokens tend to primarily provide information, and their deep representations can be built only upon themselves. Temporally (across steps), the interactions between adjacent generation steps mostly concentrate on updating the representations of a few critical tokens, while the computation for the majority of tokens is generally repetitive. Driven by these findings, we propose EfficientNAT (ENAT), a NAT model that explicitly encourages these critical interactions inherent in NATs. At the spatial level, we disentangle the computations of visible and [MASK] tokens by encoding visible tokens independently, while decoding [MASK] tokens conditioned on the fully encoded visible tokens. At the temporal level, we prioritize the computation of the critical tokens at each step, while maximally reusing previously computed token representations to supplement necessary information. ENAT improves the performance of NATs notably with significantly reduced computational cost. Experiments on ImageNet-256 2 & 512 2 and MS-COCO validate the effectiveness of ENAT. Code and pre-trained models will be released at https://github.com/LeapLabTHU/ENAT.

ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesis
Zanlin Ni
Yulin Wang
Renping Zhou' Yuan Yao
Yizeng Han
Jiayi Guo'
Zhiyuan Liu'
Gao Huang"
Tsinguu University
Natioonll University of Singapore
Abstract
Recently, token-based generation approaches have demonstrated their effectiveness in synthesizing visual content. As representative example, non-autoregressive Transformers (NATs) can generate decent-quality images in just few steps. NATs perform generation progressive manner, where the latent tokens of result- ing image are incrementally revealed step-by-step At each step. the unrevealed image regions are padded with [MASK] tokens and inferred by NAT, with the most reliable predictions preserved as newly revealed, visible tokens. In this paper, we delve into understanding the mechanisms behind the effectiveness of NAT' and uncover two important interaction patterns that naturally emerge from NAT's paradigm: Spatially (within a step), although [MASK] and visible tokens are processed uniformly by NATs, the interactions between them are highly asym- metric. In specific, [MASK] tokens mainly gather information for decoding. On the contrary, visible tokens tend primarily provide information, and their deep representations can be built only upon themselves. Temporally (across steps), the interactions between adjacent generation steps mostly concentrate updating the representations of few critical tokens, while the computation for the majority tokens is generally repetitive. Driven by these findings, we propose EfficientNAT (ENAT), a NAT model that explicitly encourages these critical interactions inherent in NATs. At the spatial level, disentangle the computations of visible and [MASK] tokens by encoding visible tokens independently, while decoding [MASK] tokens conditioned on the fully encoded visible tokens. At the temporal level, we prioritize the computation of the critical tokens at each step, while maximally reusing previously computed token representations supplement necessary in- formation. ENAT improves the performance of NATs notably with sigmificantly reduced computational cost. Experiments on ImageNet-256" & 512 and MS- COCO validate the effectiveness of ENAT. Code and pre-trained models will be released htthtts: thub. http:///ggtuuu.ccaa/aaaaaaaaaA/t
Introduction
Recent years have witnessed an unprecedented growth in the field of Al-generated content (AIGC). In computer vision, diffusion models [10 (59 61] have emerged
effective approach. On the contrary, within the context natural language processing, content is typically synthesized via the generation of discrete tokens using Transformers 772 19 5 55 Such discrepancy has excited growing interest
in exploring token-based generation paradigms for visual synthesis 77 85
33
87
6 35). Different
from diffusion models, these approaches utilize discrete data format akin language models. This
makes them straightforward to harness well well-established language model optimizations such as the
"qqual contribution.
t Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

refined scaling strategies [5] 54 31 73 and the progress in model infrastructure 65 12 34 [6].
Moreover, explorations in this field may facilitate the development of more advanced, scalable multimodal models with unified token space 17
68 18
44 90] as well as general- phreerl-purpose vision
foundation models that integrate visual understanding and generation capabilities [35 [69].
The recent advances in token-based visual generation have seen the rise of von-auttregressivve Transformers (NATs) 7 33 6
0
T
[53] which are distinguished by their abilities to fulfill efficient and high- quality visual synthesis. As shown in Figure I NATs follow a
progressive generation paradigm: at each generation step, a certain number of latent tokens of the resulting image are decoded in parallel, and the model carries out this process iteratively to produce the final complete token maps. More specifically, at each step, the unknown latent tokens of the image are represented with [MASK] tokens and concatenated with the tokens that have been decoded (i.e., visible tokens). Then, the full set of [MASK] and visible tokens fed into a Transformer-ban based model, predicting the proper values of the unknown tokens, with the most reliable predictions preserved as the increments of visible tokens for the next generation step.
visible
VQ
mask
Dec
token
Figure 1: The generation pro- cess of NATs starts from a masked canvas, decode multi- ple tokens per step, and are then mapped to the pixel space using pre-trained VQ-decoder [3]
In this paper, we seek to advance the understanding of the mechanisms behind the effectiveness of NATs' progressive generation procedures. Our investigation uncovers two important findings regarding the spatial and temporal interactions within NAT': Spatially, at each generation step, even though both [MASK] and visible tokens are treated equivalently within the computational graphs of NAT',, the visible tokens naturally learn to mainly provide information for [MASK] tokens to infer the unknown image content, and their corresponding deep representations can be built in the absence of [MASK] tokens. Temporally, the interactions between adjacent generation steps mainly concentrate on updating the representations of a small number of "critical tokens" on top of the previous steps. In fact, the computation for the remaining majority of tokens is generally repetitive.
Inspired by these findings, we propose develop novel NAT models to explicitly encourage these critical interaction mechanisms emerged naturally when trained for visual generation. yielding EfficientNAT (ENAT). Specifically, the spatial level, we disentangle the computations of visible and [MASK] tokens by encoding visible tokens independently of [MASK] tokens. [MASK] tokens are then processed by attending to the fully contextualized features of visible tokens, as shown in Figure Bp As interesting observation derived from disentanglement, we find that prioritizing the computation for visible tokens, particularly when the computation maximized for visible tokens and minimized for [MASK] tokens (even with only single network layer), further improves the performance of NATs by large margin. At the temporal level, we concentrate computation on the "critical tokens" while maximally reusing the representation of previously computed tokens to supplement the necessary information, as illustrated in Fig
Empirically, the effectiveness of ENAT is validated on ImageNet 256x256 60 ImageNet 512x512 (60) and MS-COCO [6]. ENAT is able to achieve significantly reduced computational cost compared to conventional NATs while outperforming them notably (e.g., 24% relative improvement with 1.8x lower cost, see Table bap.
Related Work
Image tokenizer and token-based image generation models. Language models algorithms like Byte Pair Encoding or WordPiece to convert text into tokens. Similarly, an image tokenizer transforms images into visual tokens for token-based image generation. Key works in this field include Discrete VAE [58), VQVAE 711. and VQGAN 13, with VQGAN-based tokenizers being most popular for their superior image reconstruction abilities. These tokenizers have enabled the advent of high-performance, scalable token-based generative models 85 56
87
6 Early token-based models
13 85). In contrast, non- autoregressive transformers (NATs) 33 6 53 generate multiple tokens simultancously, speeding up the process while maintaining high image quality. Recently, visual autoregressive models (70] introduced a next-scale prediction strategy, also demonstrating their promise in image synthesis.
were mainly autoregressive, generating images one token time 44
We illustrate with 4x4 tokens for simplicity; the actual token map size may be 16x 16 larger.

---

Efficient image synthesis has witnessed significant progress recently. Though the efficiency is- sue is relatively less explored in token-based image synthesis. it has been extensively studied in diffusion- based models. This includes advanced samplers 40 41 37], distillation methods [62 83
quantization and compression techniques [92 88 9I] and efforts to reduce redundant computa- tion [4 79 I. The last approach bears some resemblance to our computation reuse mechanism in Sec. 4.2 but with notable differences. Firstly, the subjects of research differ: we focus on NAT models. This focus introduces unique properties, e.g., NATs incrementally decode new tokens at specific spatial locations, resulting in feature maps that are only significantly updated in those areas during generation. This contrasts with diffusion models, where feature map similarity between adjacent steps does not follow such predictable spatial patterns; instead, some layers show high overall similarity within a certain range of timesteps, while others may not. Secondly, these characteristic differences lead to distinct methodologies. Diffusion models typically require manually fine-tuned, and some- times layer-specific caching schedules [42] to reuse previously computed features. This process can be iabor-intensive and may struggle with generalization. In contrast, our method prioritizes model computation newly decoded tokens in NAT' and reuses the final representations of previously computed tokens without manually fine-tuned caching schedules.
Masked image modeling (MIM) methods like MAE [29] are widely used for learning image representations by predicting missing patches, with the encoder processing visible tokens and the decoder attending to both visible and masked tokens for reconstruction. CrossMAE [15] extends this by adopting a more disentangled architecture for handling both token types separately. contrast, work focuses image generation, applying masked image modeling in discrete image token space, where token prediction and reconstruction are required at every step. This introduces key differences, such as SC-Attention and computation reuse mechanisms (see Sec. which are not explored in these MIM approaches.
Non-autorcgressive Transformers (NATs) originated in machine translation for their fast inference capabilities 19, 20]. Recently, they have been adapted for image synthesis, enabling efficient high- quality image generation evidenced by various studies 7 33 35 6,
53 86). MaskGIT 7 was the first to show NAT's effectiveness on ImageNet. This approach has been expanded for text-to- image generation, scaling up to 3B parameters in Muse [6] and achieving outstanding performance. Token-critic (33] and MAGE [35] enhance NATs further: Token-critic uses an auxiliary model for guided sampling, while MAGE integrates representation learning with image synthesis using NATs. Recent studies [46 47) have also explored techniques for further improving the training and inference process of NATs. In contrast these works, we aim to better understand the mechanisms behind NATs' effectiveness, uncovering findings that naturally lead to more efficient and effective design for NAT models.
Preliminaries of Non-autoregressive Transformers (NATs)
In this section, provide an overview of Non-Auttreerressive Transformers (NATs) 7
6
35
for
image generation. NATs operate with a pre-trained VQ-Autoencoder |7] 57 13, which maps images to discrete visual tokens and reconstructs images from these tokens. The VQ- AQtoneneer consists of three components: an encoder gVQ, a quantizer Q with a learnable codebook and decoder DQ, The encoder and quantizer transform an image into a sequence of visual tokens:
v = (x))
(1)
where e = [ug]==1: is the sequence of visual tokens, and N is the sequence length. Each token U; corresponds to specific entry in the VQ-Autoencoder codebook. The above process is known as tokenization. After tokenization, NATs learn to generate visual tokens in the latent VQ space.
During training, NATs optimize the masked language modeling (MLM) objective 9 Specifically, random subset of tokens is replaced with a special [MASK] token, and the model is trained predict the original tokens based on the unmasked ones. Formally, let M be the mask vector, where m. = 1 indicates the i-th token is masked. The training objective minimizes the negative log-likelihood of the masked tokens:
LMLM
Σ
log (
(2)
where p(u./Um) ) is the predicted probability of token it given the unmasked tokens UM-
3

---

To generate images, NATs follow an iterative decoding strategy 77. Starting with fully masked token map, the model predicts all masked positions and samples a portion of the most confident predictions to replace the mask tokens in each iteration. The number of masked tokens to be replaced follows cosine function, with fewer tokens replaced in the early iterations and more tokens replaced in later iterations. The finally decoded token sequence é is then decoded into an image by the VQ-Autoencoder decoder:
c (+).
Due to space limitations, we refer readers to for more details.
4 EfficientNAT (ENAT)
In this section, we design several analytical experiments (details in Appendix (A.1) to advance the understanding of the mechanisms behind the effectiveness of NAT's, aiming to accordingly improve the design of NAT models. Specifically, we uncover the critical spatial and temporal interaction patterns that naturally emerge within NATs under the goal of image generation. Inspired by our findings, we further propose to gradually re-design NATs towards maximally exploiting these characteristics.
4.1 Spatial Level Interaction
Motivation: an ablation study. notable characteristic of NATs is the concurrent processing and interaction (through attention layers) of visible ([V]) and [MASK] ([M)) tokens when inferring the unknown image content. To better understand this mechanism, we consider an ablation study on four types of spatial interactions: a) [M] to [V] attention, b) [V] to [M] attention, c) [V] to [V] attention, and d) [M] to [M] attention. We find these four types of spatial interactions have significantly different impacts on the generation performance. As shown in Figure 2 the most important spatial interaction
the [M] to [V] attention (i.e., [M] information propagation), without which the model is
unable to converge at all. Moreover, both [M] to [M] and [V] to [V] attentions (i.e., self-attention processes of visible and [MASK] tokens, respectively) moderately
within the representation-extraaii
improve the model. The most intriguing fact is that removing the [V] to [M] attention (i.e., [M]+[V]
information propagation) only marginally hurts the model's performance.
visible token
mask token
attention enabled
attention disabled
V
Q
K,
K,V
K,
K V
Bascline (full attn.)
(a) w/o [M] to [V]
(b) w/o [V] to [M]
(c) w/o [V] [V]
(d) w/o [M] to [M]
FID=6.54
FID-201.88
FID=6.76
FID-7.40
FID>7.05
Figure 2: An ablation study on four types of spatial interactions. The essential spatial interaction
is the [M] to attention. In contrast, the [V] to [M] attention only marginally affects the model.
This imbalanced importance of four spatial interactions highlights the distinct roles of visible and [MASK] tokens. Specifically, the processing of the visible tokens primarily establishes certain internal representations based on the currently available and reliable information, and propagates them to the [MASK] tokens. In fact, their corresponding deep representations can be built mainly top of themselves. contrast, [MASK] tokens progressively gather information from visible tokens to predict the proper token values corresponding to the unknown parts of the images. In other words, NATs naturally separate the role of visible and mask tokens when learning to generate images effectively, even though the two types of tokens are designed to be processed equally in NAT models.
This phenomenon raises an intriguing question: can we improve NATs by explicitly encouraging the naturally emergent spatial-level token- interaction patterns? Actually, this idea is feasible. For example, we can consider disentangled architecture that explicitly differentiates the roles of visible and [MASK] tokens. As shown Figure 3p, we may process visible tokens independently of [MASK] tokens, with the sole purpose of encoding the current visible and reliable information. In contrast, the computation
Table Effectiveness of disentangled architecture.
Arch. GFLOPs
FID,
Baseline Disentangled
39.6
6.54
40.2
5.50
allocated to [MASK] tokens may only focus predicting unknown image contents correctly with

---

visible token
mask token
visible token feature
mask token feature
concatenate
Loss
Loss
Attention
K
Feed Forward
Feed Forward
Feed Forward
Nx
Ngx
xND
Self Attention
Self Attention
SC-Attention
M
VM
(a) Existing Works
Ours: Disentangled Architecture
Figure 3: (a) Existing works of NATs process visible and [MASK] tokens equivalently. (b) Our disentangled architecture independently encodes visible tokens and integrates their fully contextual- ized features into the [MASK] token decoding process. M is the indicator of [MASK] tokens while M is the indicator of visible tokens. The SC-Attention concatenates the visible and mask token features to produce keys and values, providing a complete context for the mask token decoding.
the help of fully contextualized visible token representations. Such a disentangled architecture significantly improves the performance of NATs at similar computational cost (see Table [ for evidence). Inspired by (2). we efficiently integrate the encoded visible tokens into the decoding process of [MASK] tokens with tailored SC (SelfCross-aattentioon mechanism (see Figure 3p). The SC-attention simultaneously handles the interactions within [MASK] tokens and the interactions between [MASK] tokens and visible tokens, and it outperforms other possible designs like stacking self-attention and cross-attention layers alternately (see Table 6b).
Moreover, further explorations of disentangled architecture yield an interesting finding: prioritizing visible tokens results in enhanced efficiency. As shown in Table 2 the paradigm of equal computation allocation across all tokens derived from existing NATs may be far from optimal. Instead, allocating more computation to visible tokens yields notably better performance without sacrificing efficiency, while the computation on masked tokens can be reduced to only single layer. This observation further underscores the importance of our proposed disentangled paradigm of processing visible tokens from masked ones in enabling advanced network architecture design.
Table 2: Effects of prioritizing vis- ible tokens. Ng. .N: encoder/de- coder layers (for visible/ [MASK] tokens). Network width is slightly adjusted to make GFLOPs approx- imately unchanged.
Ne ND GFLOPs
FID
8
40.2
5.50
4
38.2
4.98
1
39.8
4.78
4.2 Temporal Level Interaction
Feature similarity across generation steps. Another critical characteristic of NATs their incre- mental revelation of unknown parts of the image upon previous steps. Beyond this straightforward procedure of progressive generation, here we are interested in whether there exist some interpretable temporal interaction patterns in NATs' behaviors. For instance, how do NAT's computation results at the current step relate to those at the previous step? To investigate this, we conduct a similarity analysis of NATs' output features between two adjacent generation steps.
In Figure 5h, we randomly select two generated samples in NATs and visualize their token feature similarity at two adjacent steps (steps 2 & 3 and steps 6 & 7). We compare token-wise similarity and
(t-1) (t)
adopt cosine similarity as the metric: Sim(2 (t-1), (²))),,
where 2!6 denotes the
= -- "
feature of the token at position (i.j) and timestep t. The similarity map exhibits highly polarized pattern: token representations undergo drastic changes at some "critical positions", while other positions remain highly similar between adjacent steps. When comparing with the positions of newly decoded tokens, we find that these "critical positions" correspond precisely to where the newly decoded tokens are located. In other words, the major significance of each time step lies in updating
5

---

visible tok.
mask tok.
newly decoded tok.
visible tok. feature
mask tok. feature
concat
previous step
current step
MLP
Feed Forward
Feed Forward
features
reused
SC AC-Attention
SC Attentitan
z prev
Z
Dropped
ENAT
ENAT
prev
previous token features
Drev
(a) Inference process of ENAT
(b) Detail of ENAT (diff' from Fig. 3b marked in red)
Figure 4: Overview of ENAT. Based the disentangled architecture in Fig. Ep. further propose to only encode the critical (i.e., newly decoded) tokens and maximally reuse previously extracted features supplement necessary information. A is the indicator of newly decoded tokens. Only one transformer block is illustrated for simplicity.
the representations of newly decoded tokens, while the computation for the remaining majority of tokens is generally repetitive. In Figure 5p. we plot the average token similarity over 50,000 generated samples in each pair of adjacent steps 2. 2 2-33... ... 7-+8). The results show that this temporal interaction pattern remains consistent for different timesteps/samples.
step step 3
step r5, step
newly decaded in M)
newly decoded
Avarae Taken Feature Similaity
1.0
similarity
similanity
0.9
0.8
Sample
-
Newly Decoded Tokens
Others
0.5
Sample
0.4
0.3
Iteration
(a) Token feature undergoes drastic change only at newly decoded positions
(b) Avg. similarity (stop t - vs. stop t)
Figure 5: Feature similarity analysis. (a) We randomly choose two samples and visualize the token-to-token feature similarity between adjacent steps (2 & 3 and & 7), with the positions of newly decoded tokens visualized on the right. (b) The token feature similarity averaged over 50,000 generated samples in each pair of adjacent steps (L +2 2, | - ... 1= 7-
Computation reuse. Driven by these observations, our key insight is: during the generation of NAT', not all tokens need be re-computed from scratch each step. Instead, only the newly decoded tokens need to be re-encoded to inject new knowledge about the image, while the previously encoded information can be maximally reused to supplement necessary details.
To implement this idea, we slightly modify the inference process upon our disentangled architecture (Sec. [.. by only encoding the newly decoded tokens at each step, while integrating the previously computed features to assist the current step's decoding:
without reuse (Fig
Forward (xx UM)
(4)
with reuse (Fi
Forward(wr) VA, UM. (or:f(2"' ((2
(5)

---

where 2!"ey is the feature computed on the previous step, which is projected by light-weight projection module f()), and VA denotes the newly decoded tokens. We adopt the SC-Attention mechanism (Sec. 4.1) to integrate the previous features into the current step's decoding process. At the end of the encoder, we simply concatenate the projected previous features with the computed features on the newly decoded tokens, and feed them into the decoder. In this way, the previously computed features are maximally reused to supplement both the encoding and decoding process of the current step, significantly reducing the computation cost and accelerating the generation process. The detailed inference process illustrated in Figure db.
To equip the model with the ability to utilize previously computed features, we introduce minor modifications to the training process by alternating between normal forward mode and "reuse forward mode" during training (each with 50% probability).
More specifically, the "reuse mode forward" during training is achieved through the following steps:
1. Masking Tokens:
Given the current input token map v, we mask a random subset of visible tokens in to create y
2. Feature Extraction: Feed verev into the NAT model obtain its features 2erey
3. Forward Pass and Loss Computation: Use Eq. 5 to forward the current input token map v along with zorer obtained in Step 2, and compute the loss. In practice, stop gradient operation is applied before feeding g into Eq. (
At other times, we use the original forward mode without incorporating previous features: Eq. 4 where the previous features are empty and the SC-Attention on the left of Figure naturally reduces to the original self-attention mechanism. The reuse mechanism significantly accelerates the generation process, as shown in Table ba| Additionally, we find in practice that only feeding the visible token feature of the previous step is sufficient and achieves better efficiency, as shown in Table 66d
Experiments
Setups. Following 7 35 6 we utilize a pretrained VQGAN [3] with codebook of size 1024 for
image and visual token conversion. We employ three NAT models: ENAT-S (15 encoder layers, I decoder layer. 366 embedding dimensions, primarily for ablations), ENAT-B (15 encoder layers, I decoder layer, 768 embedding dimensions), and ENAT-L (22 encoder layers, 2 decoder layers, 1024 embedding dimensions). For class-conditional generation, we use adaptive layer normalization 80 49] for conditioning. For text-to-image generation, we concatenate text embeddings with visual tokens for conditioning. Our training configurations follow (3] with minor adjustments to batch sizes and learning rates to accommodate different model sizes. For system-level comparisons in Sec.5.1 we measure the TFLOPs of the entire generation process (including the decoder part for latent space generation models) to ensure fair comparisons All our experiments are conducted with 8 x A100 80G GPUs. We generally follow the approach described in [3] with minor modifications. More details the training and inference setups, and the choice of our baselines can be found in Appendix A.2
5.1 Main Results
Class-conditional generation on ImageNet 256 and 512x512. In Table 3 we compare approach with other generative models on ImageNet 256x256. Our ENAT achieves superior performance with significantly lower computational cost. For instance, our ENAT-B model, despite having an extremely low inference cost, attains competitive FID scores of 3.53 in 8 steps. With a slightly increased computational budget, our ENAT-L model achieves a FID of 2.79 with only 0.3 TFLOPs, surpassing leading models with substantially less computational effort. For example, compared to the most performant baseline, i.e U-ViT-H [3) our ENAT-L model achieves a lower FID score (2.79 vs. 3.37) while requiring 8x lower computational cost (0.3 TFLOPs vs. 2.4 TFLOPs). We further evaluate our ENAT on ImageNet 512 512 in Table 4 Our ENAT-L model also achieves a superior FID of 4.00 with only 1.3 TFLOPs, outperforming leading models with much lower inference cost. Qualitative results of our method are presented in Figure 7 and Appendix B
"This differs from the GFLOPs reported in our ablation studies in Tabs. excluded to better compare the efficiency of different NAT designs.
where VQ-decoder costs are
7

---

Table 3: Results ImageNet x TFLOPs quantify the total computational cost for generating single image. For DPM-Solver 440 augmented diffusion models ()), we follow 40 to tune configurations and report the lowest FID. Diff: diffusion, AR: autoregressive.
Method
Type
#Params
Steps
TFLOPS
FID
1S1
BigGAN-deep
OCIR'19
GAN
6.95
171.4
StyleGAN-XI
GAN
1.5
2.30
265.1
VQVAE-2
(Neu
13.5B
5120
31.1
45
VQGAN
13
(CVPR'2/)
AR
14B
256
15.78
78.3
ADM-G
10 (NcurtP522Z)
Diff.
554M
250
334
4.59
186.7
LDM 50
(CVPR
Diff.
400M
250
52.3
3.60
247.7
1.2
11.74
LDM 50
(2VPR
Diff.
400M
8
2.0
4.56
262.9
4
1.4
8,45
U-VIT-H' R
(CVPR223)
Diff.
501M
3.37
235.9
1.3
9.71
DIT-XL (CCV222)
Diff.
675M
8
2.2
5.18
213.0
11.36
MDT-XL 16
(CCC22)
Diff.
676M
USF
38
(CCTR
Diff.
554M
8
10.7
9.72
MaskGIT
NAT
227M
12
1.22
4.92
Token-Critic
35
(SCVV22)
422M
4.69
174.5
Draft-and-revise
22
(N
NAT
1.4B
72
3.41
224.6
MAGE 35 (CVPP'2))
NAT
230M
20
1.0
6.93
MaskGIT-FSO
43
(CLR)33)
NAT
225M
12
0.8
4.53
AdaNAT
47
(RCCY134)
206M
0.9
2,86
265.4
4
0.1
5,86
ENAT-B
NAT
219M
3.53
302.4
4.13
ENAT-L
NAT
S74M
8
0.3
2.79
326.7
Table 4: Results on ImageNet 512x512. f: DPM-Solver 40
augmented diffusion models.
Method
Type
#Params
Steps
TFLOPs
FIDJ
1S1
VQGAN
B
(CVPR221)
AR
227M
1024
26.52
66.8
ADM-G
10
(New)P5/211)
Diff.
559M
250
579
7.72
172.7
U-VIT-H
(CVPR/23)
Diff.
501M
286.8
DIT-XL
49
(CCV133)
Diff.
675M
8
9.6
5,44
275.0
MaskGiT
(CVPR'22)
NAT
227M
12
7.32
156.0
MaskGIT-RS
(CVPR122)
227M
12
13.1
4.46
Token-Critic
83
(RCCV122
NAT
422M
36
7.6
6.80
182.1
Token-Critic- RS 33
NAT
422M
36
34.8
4.03
ENAT-L
574M
1.3
4.00
285.7
Text-to-image generation on MS-COCO. We fur- ther assess the efficacy of ENAT for text-to-image generation MS-COCO 33). Table that ENAT-B surpasses competing baselines with just 0.3 TFLOPs, achieving a FID score of 6.82. Com- pared the competitive diffusion model U-VIT [3] with fast sampler [11, ENAT-B requires similar computational resources to its 4-step variant while significantly outperforming it (6.82 vs. 16,20), and also surpasses the 8-step sampling results of U- VIT with lower computational costs.
Table 5: Results on MS-COCO: all models are trained and evaluated on MS- IS-COCO. f; DPM- Solver 40 augmented diffusion models.
Method
#Params
Steps
TELOPS) FID,
VQ-Diffusion R1
370M
100
13.86
Frido 14
512M
200
8.97
U-Net
53M
50
7.32
0.4
16.20
U-VIT 8
44M
8
0.5
ENAT-B
0.3
Practical efficiency, We provide more comprehensive comparisons of the trade-off between genera- tion quality and computational cost in Figure Both theoretical TFLOPs and the practical GPU/CPU latency for generating an image are reported. Our results show that ENAT consistently outperforms other basclines in terms of both generation quality and computational cost.

---

LDM LDM1
(CYP9222)
DiT (CCP2)
U-VIT (C12922)
A MaskGIT (CV9)2)
* ENAT-B ENAT-L
12
12
12
10
FID-50K
FID-50K
8
FID
A
O
20
2
2
100
200
300
0
2 3
Sampling Compute/ Image (TFLOPs)
GPU Time Image (ms)
CPU Time Image (s)
(a) FID vs. TELOPs
(b) FID vx. GPU Time
(c) FID vs. CPU Time
Figure 6: Practical efficiency of ENAT As reference, we also plot the TFLOPs for generating
a single image in (a). GPU time is measured on an A100 GPU with batch size 50. CPU time is measured on Xeon 8358 CPU with batch size 1. 1; DPM-Solver 40) augmented diffusion models.
Figure 7: Selected samples of ENAT-L with generation steps on ImageNet 256x256.
5.2 Ablation Studies
In this section, we present additional ablation studies on Imagenet 256 validate the effective- ness of our proposed mechanisms. We use ENAT-S with 8 generation steps as our default setting, and report the FID score as well as the computational cost in GFLOPs for each NAT model.
Main ablation. Disentangled architecture and computation reuse are the two fundamental mecha- nisms in ENAT, The former separates the processing of visible and [MASK] tokens, and prioritizes computation on visible ones, while the latter eliminates repetitive processing of non-critical tokens. In Table 6a] demonstrate the effectiveness of these mechanisms. The results show that the disentangled architecture significantly improves NAT's performance, with a 1.76 improvement in FID score a similar computational cost. Computation reuse, on the other hand, significantly reduces computational cost (1.8x fewer GFLOPs) while preserving most of the gains from disentanglement.
Effectiveness of SC-Attention. The SC-Attention mechanism adopted in work serves dual roles: handling interactions of input tokens while simultaneously incorporating necessary additional information. Theoretically, the same functionality be achieved with stack of one self-attention layer and one cross-attention layer. However, as shown in Table 6b SC-Attention outperforms the stack of self-attention and cross-attention layers with a lower FID (4.97 vs. 5.85) and lower computational cost (22.6 vs. 25.0), demonstrating its effectiveness our ENAT model.
Effectiveness of reuse projection module. In computation projection module first processes the previous feature before integrating it into the current generation step. As shown in Table 6c this design is highly important to reuse mechanism. Without this module, the FID is 5.96, which much worse than the 4.78 FID achieved without reuse. An intuitive explanation is that the of non-crucial tokens, preventing them from becoming too stale for more distant subsequent steps.
mechanism. lightweight reuse
projection module learns the minimal necessary updates for the features
Which token features to reuse? Our basic reuse formulation integrates all previous token features into the current step. However, as shown Table 6d] reusing only visible token features is equally effective while being much more efficient. As discussed in Section encoding visible tokens

---

Table 6: Ablation studies ImageNet 256x 256. We use ENAT-S with 8 generation steps default setting, which is marked in gray We report FID-50K following 3 and total GFLOPs for each NAT model throughout the generation process.
(a) Main ablation. Our disentangled ar- chitecture and reuse mechanism signifi- cantly improves NATs.
(b) SC-Attention outperforms al-
(c) Reuse projection
ternately stacking self&cross atten- tion layers with fewer GFLOPs.
lightweight yet critical for maintaining performance.
Disentangle Reuse FIDL
GFLOPs]
Attn. Type
FID
GFLOPs.).
Proj. FIDJ GFLOPs].
6.54
39.6
SC 4.97
22.6
4.97
22.6
39.8
self cross
5.85
25.0
20.8
4.97
22.6
(d) Which token features to reuse? Reusing only visible token features of previous step is sufficient and much more efficienct.
(c) Which layer of feature to reuse? Reusing last layer prev. features for all current layers is better than reusing a layer-to-layer correspondence manner.
Prev. Token Features
FID
GFLOPs|
Prev. Feature Pos.
FID4
GFLOPs
all
4.95
37.5
last layer
4.97
22.6
visible only
4.97
22.6
layer-to-layer
5.77
36.1
is most critical for NAT, and thus our ENAT model focuses most computation on these tokens. Therefore, using only visible token features suffices to provide the necessary information for reuse.
Which layer of feature to reuse? We compared reusing the last layer's s features from the previous step with reusing features in layer-by-layee manner, where the i-th layer of the current step reuses the features from the i-th layer of the previous step. As shown in Table 6e reusing features from the last layer of the previous step outperforms the layer-by-layer approach, achieving lower FID of 4.97. Additionally, it requires fewer GFLOPs (22.6 vs. 36.1), as the layer-by-layer approach needs to project features of each previous layer, while the last layer approach only projects once.
6 Conclusion
In this paper, we explored the underlying mechanisms of non-autoregressive Transformers (NATs) and uncovered key spatial and temporal token interaction patterns exist within NATs. Our findings highlight that spatially, visible tokens primarily provide information for [MASK] tokens, while temporally, updating the representations of newly decoded tokens is the main focus across generation steps. Driven by these findings, we propose ENAT, NAT model that explicitly encourages these critical interactions. We spatially disentangle the computations of visible and [MASK] tokens by independently encoding visible tokens and conditioning [MASK] tokens on fully encoded visible tokens. Temporally, we focus computation on newly decoded tokens at each step, while reusing previously computed representations to facilitate decoding. Experiments ImageNet and MS-COCO demonstrate that ENAT enhances NAT'' performance with significantly reduced computational cost.
Acknowledgements
This work is supported in part by the National Natural Science Foundation of China under Grants 42327901.
10

---

References
[1] Shubham Agarwal, Subrata Mitra, Sarthak Chakraborty, Srikrishna Karanam, Koyel Mukherjee. and Shiv Kumar Saini. Approximate caching for efficiently serving text-to-image diffusion models. In NSDI, 2024.
[2] Jean-Baphen -ann-tat Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurlPS, 2022.
[3] Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu. All are worth words: vit backbone for score-based diffusion models. In CVPR, 2023.
[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2019.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
[6] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, al. MUSE: Text-to-image generation via masked generative transformers. In JCML, 2023.
[7] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. MaskGIT: Masked Generative Image Transformer. In CVPR, 2022.
[8] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. JMLR, 2023.
[9] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. google ai language. NAACL, 2019.
[10] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In NeurlPS, 2021.
[11] Ming Ding, Zhuoyi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, al. CogView: Mastering text-to-image generation via transformers. In NeurlPS, 2021.
[12] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In ICML. 2022.
[13] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[14] Wan-Cyuan Fan, Yen-Chun Chen, DongDong Chen, Yu Cheng, Lu Yuan, and Yu-Chiang Frank Wang. Frido: Feature pyramid diffusion for complex scene image synthesis. In AAAI, 2023.
[15] Letian Fu, Long Lian, Renhao Wang, Baifeng Shi, Xudong Wang, Adam Yala, Trevor Darrell, Alexei A Efros, and Ken Goldberg. Rethinking patch dependence for masked autoencoders. arXiv preprint arXiv:2401. 143911 2024.
[16] Shanghua Gao, Pan Zhou, Mingg Ming Cheng, and Shuicheng Yan. Masked diffusion trans- former is a strong image synthesizer. In ICCV, 2023.
[17] Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. In ICLR, 2024.
[18] Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making Ilama and draw with seed tokenizer. In ICLR, 2024.

---

[19] Marjan Ghazvininejad, Omer Levy. Yinhan Liu, and Luke Zettlemoyer. Mask-predice Parallel decoding of conditional masked language models. EMNLP, 2019.
[20] Jiatao Gu and Xiang Kong. Fully non-autoregressive neural machine translation: Tricks of the trade. In ACL, 2021.
[21] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR, 2022.
[22] Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In ICCV, 2023.
[23] Dongchen Han, Yifan Pu, Zhuofan Xia, Yizeng Han, Xuran Pan, Xiu Li, Jiwen Lu, Shiji Song. and Gao Huang. Bridging the divide: Reconsidering softmax and linear attention. In NewrIPS, 2024.
[24] Dongchen Han, Ziyi Wang, Zhuofan Xia, Yizeng Han, Yifan Pu, Chunjiang Ge, Jun Song, Shiji Song, Bo Zheng. and Gao Huang. Demystify mamba in vision: A linear attention perspective. In NeurlPS, 2024.
[25] Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Shiji Song, and Gao Huang. Agent attention: On the integration of softmax and linear attention. ECCV, 2024.
[26] Yizeng Han, Zeyu Liu, Zhihang Yuan, Yifan Pu, Chaofei Wang. Shiji Song, and Gao Huang. Latency-awand unified dynamic networks for efficient image recognition. TPAMI, 2024.
[27] Yizeng Han, Yifan Pu, Zihang Lai, Chaofei Wang, Shiji Song, Junfeng Cao, Wenhui Huang, Chao Deng, and Gao Huang. Learning to weight samples for dynamic early- networks. In ECCV, 2022.
[28] Chunming He, Yuqi Shen, Chengyu Fang, Fengyang Xiao, Longxiang Tang, Yulun Zhang, Wangmeng Zuo, Zhenhua Guo, and Xiu Li. Diffusion models in low-level vision: survey. arXiv preprint arXiv:2406. 11138, 2024.
[29] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[30] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NeurlPS, 2017.
[31] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint urXiv:2203.15555 2022.
[32] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and WOOK SHIN HAN. Draft-and-revise: Effective image generation with contextual rq-transformer. In NeurlPS, 2022.
[33] José Lezama, Huiwen Chang, Lu Jiang, and Irfan Essa. Improved masked image generation with token-critic. In ECCV, 2022.
[34] Shiyao Li, Xuefei Ning, Ke Hong, Tengxuan Liu, Luning Wang, Xiuhong Li, Kai Zhong, Guohao Dai, Huazhong Yang. and Yu Wang. Llm-mq: Mixed-precision quantization for efficient 11 deployment. In NewrIPS Workshop, 2023.
[35] Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. MAGE: Masked generative encoder to unify representation learning and image synthesis. In CVPR, 2023.
[36] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[37] Enshu Liu, Xuefei Ning, Zinan Lin, Huazhong Yang. and Yu Wang. Oms-dpm: Optimizing the model schedule for diffusion probabilistic models. In ICML, 2023.
12

---

[38] Enshu Liu, Xuefei Ning, Huazhong Yang, and Yu Wang. A unified sampling framework for solver searching of diffusion probabilistic models. ICLR, 2024.
[39] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. 2018.
[40] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Opm-Solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In NeurIPS, 2022.
[41] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- Solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022.
[42] Xinyin Ma, Gongfan Fang, and Xinchao Wang. DeepCache: Accelerating diffusion models for free. In CVPR, 2024.
[43] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. In ICLR, 2023.
[44] David Mizrahi, Roman Bachmann, Oguzhan Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4m: Massively multimodal masked modeling. In NeurlPS, 2023.
[45] Zanlin Ni, Yulin Wang, Jiangwei Yu, Haojun Jiang, Yue Cao, and Gao Huang. Deep incubation: Training large models by divide-and-connuering. In ICCV, 2023.
[46] Zanlin Ni, Yulin Wang, Renping Zhou, Jiayi Guo, Jinyi Hu, Zhiyuan Liu, Shiji Song, Yuan Yao, and Gao Huang. Revisiting coo-euuoreerrssivve transformers for efficient image synthesis. In CVPR, 2024.
[47] Zanlin Ni, Yulin Wang, Renping Zhou, Rui Lu, Jiayi Guo, Jinyi Hu, Zhiyuan Liu, Yuan Yao, and Gao Huang. AdaNAT: Exploring adaptive policy for taken image generation. In ECCV, 2024.
[48] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML, 2018.
[49] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
[50] Yifan Pu, Weicong Liang, Yiduo Hao, Yuhui Yuan, Yukang Yang, Chao Zhang, Han Hu, and Gao Huang. Rank-detr for high quality object detection. In NeurlPS, 2024.
[51] Yifan Pu, Yiru Wang, Zhuofan Xia, Yizeng Han, Yulin Wang, Weihao Gan, Zidong Wang, Shiji Song. and Gao Huang. Adaptive rotated convolution for rotated object detection. In ICCV, 2023.
[52] Yifan Pu, Zhuofan Xia, Jiayi Guo, Dongchen Han, Qixiu Li, Duo Li, Yuhui Yuan, Ji Li, Yizeng Han, Shiji Song, et al. Efficient diffusion transformer with step-wise dynamic attention mediators. In ECCV, 2024.
[53] Shengju Qian, Huiwen Chang, Yuanzhen Li, Zizhao Zhang, Jiaya Jia, and Han Zhang, StraIT: Non-autorceresssive generation with stratified image transformer. arXiv preprint arXiv:23033.000 000500 2023.
[54] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katic Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.1 11446, 2021.
[55] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
[56] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, arXiv preprint arXiv:2204.06125, 2022.
13

---

[57] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high- with VQ-VAE-2, 2019.
high-fidelity fidelity images
[58] Jason Tyler Rolfe. Discrete variational autoencoders. arXiv preprint 1.09,v:609.02200 2016.
[59] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[60] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.
[61] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurlPS, 2022.
[62] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.17042, 2023.
[63] Axel Sauer, Katja Schwarz, and Andreas Geiger. StyleGAN-XL: Scaling stylegan to large diverse datasets. In SIGGRAPH, 2022.
[64] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. In NeurIPS, 2022.
[65] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.0803: 2019.
[66] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In CVPR, 2016.
[67] Yansong Tang, Zanlin Ni, Jiahuan Zhou, Danyang Zhang, Jiwen Lu, Ying Wu, and Jie Zhou. Uncertainty-aware score distribution learning for action quality assessment. In CVPR, 2020.
[68] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiee Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1180. 2023.
[69] Changyao Tian, Chenxin Tao, Jifeng Dai, Hao Li, Ziheng Li, Lewei Lu, Xiaogang Wang, Hongsheng Li, Gao Huang, and Xizhou Zhu. ADDP: Learning general representations for image recognition and generation with alternating denoising diffusion process. In ICLR, 2024.
[70] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In NeurlPS, 2024.
[71] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation leariing. In NeurlPS, 2017.
[72] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[73] Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung. Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective works best for zero-shot generalization? In ICML, 2022.
[74] Yulin Wang, Zhaoxi Chen, Haojun Jiang, Shiji Song. Yizeng Han, and Gao Huang. Adaptive focus for efficient video recognition. In ICCV, 2021.
[75] Yulin Wang, Rui Huang, Shiji Song. Zeyi Huang, and Gao Huang. Not all images are worth 16x16 words: Dynamie transformers for efficient image recognition. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
14

---

[76] Yulin Wang, Kangchen Lv. Rui Huang, Shiji Song, Le Yang, and Gao Huang, Glance and focus: a dynamic approach reducing spatial redundancy image classification. NeurIPS, 2020.
[77] Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, and Gao Huang. Revisiting locally supervised learning: an alternative to end-to-end training. In ICLR, 2021.
[78] Yulin Wang, Yang Yue, Rui Lu, Yizeng Han, Shiji Song, and Gao Huang. EfficientTrrint*: Generalized curriculum learning for efficient visual backbone training. TPAMI, 2024.
[79] Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, et al. Cache Me if You Can Accelerating Diffusion Models through Block Caching. CVPR, 2024.
[80] Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. NewrlPS, 2019.
[81] Le Yang, Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, and Gao Huang. Resolution adaptive networks for efficient inference. In CVPR, 2020.
[82] Le Yang, Haojun Jiang, Ruojin Cai, Yulin Wang, Shiji Song, Gao Huang, and Qi Tian. Con- denseNet v2: Sparse feature reactivation for deep networks. In CVPR, 2021.
[83] Tianwei Yin, Michaäl Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint XrXiv22311 18828, 2023.
[84] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1.06.0050000330 2015.
[85] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. TMLR, 2022.
[86] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion-tokemiizr key to visual generation. In ICLR, 2024.
[87] Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models:
Pretraining and instruction tuning, arXiv preprint wwxiv:2309.0259/ 2023.
[88) Zhihang Yuan, Pu Lu, Hanling Zhang, Xuefei Ning, Linfeng Zhang, Tianchen Zhao Shengen Yan, Guohao Dai, and Yu Wang. DiTFastAut: Attention compression for diffusion transformer models. In NeurIPS, 2024.
[89] Yang Yue, Yulin Wang, Bingyi Kang, Yizeng Han, Shenzhi Wang, Shiji Song, Jiashi Feng, and Gao Huang. Dynamic inference of multimodal large language models for efficient robot execution. In NeurlPS, 2024.
[90] Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal Ilm with discrete sequence modeling. arXiv preprint arXiv:2402. 2222:22002.22226, 2024.
[91] Tianchen Zhao, Tongcheng Fang, Enshu Liu, Wan Rui, Widyadewi Soedarmadji, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, Huazhong Yang, et al. ViDiT-Q: Efficient and accurate quantization of diffusion transformers for image and video generation. arXiv preprint arXiv:2406.084 02540, 2024.
[92] Tianchen Zhao, Xuefei Ning, Tongcheng Fang, Enshu Liu, Guyue Huang, Zinan Lin, Shengen Yan, Guohao Dai, and Yu Wang. MixDQ: Memory-efficient few-step text-to-image diffusion models with metric-decoupled mixed precision quantization. In ECCV, 2024.
[93] Wangbo Zhao, Yizeng Han, Jiasheng Tang, Kai Wang, Yibing Song, Gao Huang, Fan Wang, and Yang You. Dynamic diffusion transformer, 2024.
15

---

[94] Wangbo Zhao, Jiasheng Tang, Yizeng Han, Yibing Song. Kai Wang, Gao Huang, Fan Wang. and Yang You. Dynamic tuning towards parameter and inference efficiency for vit adaptation. In NewrIPS, 2024.
[95] Ziwei Zheng, Le Yang, Yulin Wang, Miao Zhang, Lijun He, Gao Huang, and Fan Li. Dynamic spatial focus for efficient compressed video action recognition. TCSVT, 2024.
[96] Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, al. A survey on efficient inference for large language models, arXiv preprint arXiv:2404. 14294, 2024.
16

---

A Implementation Details
A.1 Detailed model configurations.
Here we present the detailed configurations of all our NAT models appeared within this paper in Table We provide the number of encoder layers (Ng),, decoder layers (ND). the dimension of the hidden states (embed dim.), the number of attention heads (# attn. heads):
Table 7: Summary of model configurations. Ng: encoder layers (for visible token encoding), Np: decoder layers (for [MASK] token decoding). ": In conventional NAT models, the layers for visible token encoding are shared with the layers for [MASK] token decoding.
arch.
reuse?
NE ND embed dim. atin. heads
baseline
8-
288
disentangled
X
8
288
6
disentangled
X
12
318
6
disentangled
15
366
ENAT-S
15
366
6
ENAT-B
15
768
8
ENAT-L
22
1024
16
A.2 Details of training and evaluation.
For ImageNet 256x256, we use a batch size of 2048 and learning rate 4e-4. For ImageNet 512x512, to manage the increased sequence length, we reduce the batch size to 512 and linearly scale down the learning rate to le-4. For MS-COCO, we train for 150k steps instead of the 1000k steps used in 3
For our ablation studies in Sec. 5.2 and explorative experiments in Sec. we train the models for 300k steps instead of the 500k steps used in 3 while keeping the other settings the same as above.
For data preprocessing, perform center cropping and resizing to 256 x 256 for ImageNet 256 226 and MS-COCO, and to 512x for ImageNet 512x512. Additionally, we adopt random horizontal flipping data augmentation, following 49
Our evaluation on FID follows the same evaluation protocol as TO 3
We adopt the pre-computed dataset statistics from (3) and generate 50k samples for ImageNet (30k for MS-COCO) to compute
the statistics for the generated samples, using the following formula to calculate FID [30]:
FID
||/real
Mlake 1 + Tr(Ereal + Etake 2 2(Ereal Erake) 1/2),
where μ and Σ are the mean and covariance of the real and fake samples, respectively. The evaluation on Inception Score (IS) follows the same protocol 3 49), using pre-trained Inception model [66] to compute the IS.
For the choice of baselines in our work, since ENAT focuses inference efficiency, we aim to compare ENAT with other models in a lightweight, low-FLOPs scenario. However, while the inference efficiency of generative models is important, it is generally under-explored in the original papers of state-of-the-an diffusion models (e.g., DiT 449, MDT [16]), which mostly focus on enhancing generation performance. The official results of them primarily obtained with hundreds of inference steps, making direct comparisons with ENAT challenging. For instance, as shown in Fig. & the official results of DiT, MDT, etc. all concentrate the high end of overall inference costs, requiring hundreds of times more computation than ENAT.
Fortunately, there are well-established fast sampling techniques (e.g. DPM-Solver 40 for accelerat- ing diffusion models, which allows us to reduce their sampling steps and compare them with ENAT in fairer setting.
17

---

Reconstruction FTD ENAT's VQ-auloencoder ("lower bound")
Reconstruction FTD of diffusions' KI. Kl--autoeneoder ("lower bound")
StyleGAN-XL
14
LDM-4
12
U-VIT-H
Official
FID-50K
DiT-XL
Results
8-
MDT-XL
6
SiT-XL
4
MDT-XLI
2
ENAT-B
ENAT-L
0
10
I: reproduccd DPM-Solver
Sampling Compute Image (TFLOPs) for few-step sampling
Figure 8: System-level comparisons ImageNet 256 256x256. All baseline results sourced from their original papers, except for the few-step MDT results ()).
Figure 9: Selected samples of ENAT-L with 8 generation steps ImageNet 256x256 and 512x512.
B More Qualitative Results.
Here we present more qualitative results in Figure 9 For each class, the first two columns contain ImageNet 512x512 samples and the last column contains 4 ImageNet 256x256 samples.
C Limitations and Future Work
Although our experiments have covered two fundamental types of generative models, namely class- conditional and text-to-image generation, and utilized three datasets, investigating the efficacy of ENAT on more diverse datasets, such as the widely used CelebA [39 and LSUN [4] and exploring additional generation types like unconditional generation, constitute valuable directions for future research. Morcover, scalability, both terms model size and dataset volume, a crucial capability for current generative models. Our largest model scales up to approximately 0.6 billion parameters, and our experiments utilized datasets with a maximum size 1.2 million images (ImageNet dataset). Evaluating the performance of ENAT even larger-scale datasets, such as LAION-5B [64], and further scaling the model to surpass billion parameters, could provide deeper insights into its scalability and robustness.
To further enhance the applicability and efficiency of non-autoregressive Transformers, integrating and learning techniques 67
other adaptive inference methods 776
75
26 95
82 77
will be
essential. For instance, methods like dynamic neural network 774 27
89
93
94
52 and resolution-
adaptive models [81] offer promising pathways explore. Additionally, examining ENAT across
18

---

Table 8: Licenses for existing assets.
dataset code source
license
MS-COCO
36
New BSD License
ImageNet
60
Custom (research, con-commercial)
MaskGIT
Apache-2.0 license
U-VIT
MIT license
diverse tasks and domains
51
50
and leveraging advances model training and inference
techniques [45] 22
25
23
78
24
can Strengthen its performance and expand its scope.
D Broader Impacts
On the positive side, the proposed EfficientNAT (ENAT) models significantly reduce computational costs, making advanced visual generation technology more accessible. This democratization can benefit diverse sectors, including education, healthcare, and creative industries. However, as with any Al-generated content technology, there are potential ethical considerations such as creating misleading content spreading misinformation. Additionally, like other data-driven approaches, the model may inadvertently reinforce biases present in the training data. Possible mitigation strategies for these concerns include developing robust detection methods for generated content, promoting transparency in Al-generated content, and ensuring diverse and representative training data.
E Licenses
The Table outlines the assets used in work, their sources and licenses. Our models, data and code will be open-sourced under the MIT License upon paper acceptance.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction have stated our contributions and scope, which are supported by the results in Section 5.1 Additionally, the ablation studies in Section 5.2 provide insights into the mechanisms behind the effectiveness of ENAT.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Implementation details are provided in Section 5 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Data and code will be available upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ptaip..cc/ubblic/guu.ee CoodeSubeeieeeeee ssi iessonPolll molyicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Detailed experimental setups are provided in Section 5 and Appendix A Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Like most works [59 49 in the field, training generative models large- scale datasets typically involves high computational costs, making it impractical to run multiple trials for each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resource information is provided in Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Social impacts are discussed in Appendix D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cited their original papers and included their license in Appendix E
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code will be released under the MIT License. Implementation details provided in Section and Appendix
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "hV4meL35n0", "reviewer": "Reviewer_PoGn", "summary": "This paper proposes an improved Efficient MaskGIT-like non-autoregressive image generation method. Its core component is to (1) spatially disentange visible tokens and mask tokens in attention, and prioritize compute for the visible tokens; (2) temporally reuse the visible tokens from previous decoding step. The results on common benchmarks like ImageNet and MSCOCO demonstrate the effectiveness of the proposed approach, leading to clear FID improvement while significantly reducing computational cost.", "questions": "1. In line 221-229, the authors mention the modifications of training pipeline to accomodate the need for reusing previous features. I wonder if the authors could elaborate on that? My understanding is that this MaskGIT paradigm is a single-step process during training and multi-step process during decoding? So where does the features of prior steps come from during training? And how is that related to the masking modification mentioned in line 222?\n2. If my understanding is correct, the spatial disentanglement is also a compute-efficient operation, as there is no more attention across visible tokens and mask tokens in most of the layers. Yet in most experiments, the authors describe it as a performance-boosting design (say, Table 6(a)). I wonder is it because the authors have stacked more attention layers to compensate for the FLOPs loss? If so, the authors are encouraged to further clarify their setup.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The idea is straightforward and makes intuitive sense to me. Since the mask tokens do not really convey any meaningful information, they do not have to be in the attention operation with visible tokens. Also, as the visible tokens decoded from the prior steps remain unchanged, their features should be reusable in the next decoding step. This paper is generally well-written and relatively easy to follow. The results, especially on ImageNet, show promising improvement.", "weaknesses": "I cannot help but notice the comarison between U-ViT and ENAT on MSCOCO (Table 5). An 8-step U-ViT performas almost as good as ENAT-B, although on ImageNet ENAT is clearly the better choice. In Table 5, the number of parameters of U-ViT is smaller but its FLOPs is larger compared with ENAT-B, so I guess one could say they are in the same level. Does that indicate that ENAT is not as advantageous on text-to-image synthesis task?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1: Questions on MSCOCO Results**\n\n**Firstly**, we kindly clarify that the two models, U-ViT and ENAT-B, may *not* be viewed as being \"on the same level.\"\nOur primary objective is to improve inference efficiency.\nTherefore, FLOPs, as an important indicator of practical efficiency (see Fig. 6 of our original paper), is much more critical than model parameter count.\nMoreover, the increase in model parameters is in fact an intentional result of our disentangled architecture, which significantly improves performance *without* affecting efficiency (see Tab. 6a of our original paper).\n\n**Secondly**, we have only trained our ENAT model on COCO text-to-image generation for 150K steps (see A.2 of our original paper) as it already outperforms U-ViT trained with nearly 7$\\times$ more training budget (i.e. 1000K steps).\nIn a fairer comparison where U-ViT is also trained for 150K steps, ENAT is still significantly better:\n| Model  | Train Steps | Generation Steps | TFLOPs↓ | FID↓  |\n|--------|-------------|------------------|--------|------|\n| U-ViT  | 1000K        | 8                | 0.5    | 6.92|\n| U-ViT  | 150K        | 8                | 0.5    | 10.02|\n| ENAT-B | 150K        | 8                | **0.3**    | **6.82** |\n\nWe will add more discussions on this in our revised experiment section.\n\n> **Q1: Elaboration on ENAT's Training Process**\n\n  Sure, the detailed process of training in reuse mode (Eq. (5)) is elaborated below:\n  1. **Masking Tokens**: Given the current input token map $\\mathbf{v}$, we mask a random subset (*e.g.*, 30%) of visible tokens in $\\mathbf{v}$ to create $\\mathbf{v}^{\\text{prev}}$. \n  2. **Feature Extraction**: Feed $\\mathbf{v}^{\\text{prev}}$ into the NAT model to obtain its features $\\mathbf{z}^{\\text{prev}}$.\n  3. **Forward Pass and Loss Computation**: Use Eq. (5) to forward the current input token map $\\mathbf{v}$ along with $\\mathbf{z}^{\\text{prev}}$ obtained in Step 2, and compute the loss. In practice, a stop gradient operation is applied before feeding $\\mathbf{z}^{\\text{prev}}$ into Eq. (5).\n\nSince $\\mathbf{v}^{\\text{prev}}$ created in Step 1 contains less visible tokens than $\\mathbf{v}$, it effectively simulates the \"previous step input\" that ENAT will encounter during generation, thereby equipping ENAT with the ability to correctly handle previous step's feature for reuse.\n\nWe will elaborate this process in more detail in our revision.\n\n> **Q2: Setups for Spatial Disentanglement Experiments**\n\nYes, we intentionally kept the FLOPs unchanged for a clear comparison. However, we achieved this by enlarging the network width, which we will further clarify in our revised manuscript. Enhanced performance at the same FLOPs generally equates to lower computational cost for the same performance, thus still demonstrating our method's efficiency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RhBvjsv0EW", "reviewer": "Reviewer_GVb9", "summary": "The paper deals with improving the computational efficiency of image\ngeneration non-autoregressive transformers (NATs) (e.g. MASKGIT). NATs\nare usually trained by a fill-in-the-blanks objective and sample by\niteratively predicting missing tokens. The approach is based on\nseparately encoding observed and missing (masked) tokens, as well as\navoiding repeated computation during sampling iterations. Experimental\nresults show the effectiveness of the approach, achieving better\nperformance than the baselines at lower computational cost.", "questions": "Please see weaknesses.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and the concepts are clearly explained and\n  illustrated.\n- The problem of designing more efficient NATs is an important one in\n  the context of generative models dealing with increasingly longer\n  sequences (e.g. video).\n- The method is simple yet effective, and the paper does a good job of\n  justifiying the proposed components small experiments.\n- Experimental results convincingly show the proposed approach\n  achieves better performance than baselines at a much lower\n  computational cost.\n- Ablation studies clearly isolate the contribution of the proposed\n  components.", "weaknesses": "- The idea of only encoding visible tokens was already introduced in\n  e.g. MAE, MAGE. Is there a significant difference in the approach\n  presented here in that regard?\n- In the comparisons, the proposed ENAT uses significantly less sample\n  steps than the compared baseline NATs. Getting better performance with\n  less sampling steps is obviously a positive thing. But this also\n  confounds the comparison in reduced flops due to the\n  architecture/procedure. I wonder what is the intuition for this\n  approach allowing to significantly reduce the number of sampling\n  (iterative refinement) steps?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**W1: Relationships with MAE and MAGE**\n\n**Firstly**, we kindly clarify that our approach includes *two* key aspects: a *disentangled architecture* and a *computation reuse* mechanism. To the best of our knowledge, the *computation reuse* mechanism is a novel contribution that has not been explored in NAT, and has demonstrated great promise and effectiveness.\n\n**Secondly**, while our *\"disentangled architecture\"* may appear similar to MAE & MAGE, they are in fact different.\nSpecifically, MAE & MAGE view the idea of encoding only visible tokens as a way to accelerate the training process for *representation learning*. \nIn contrast, our *disentangled design*  focuses on building an efficient architecture tailored for *image generation*.\nThe distinct objective results in small but important methodological differences, primarily manifested in the decoder's attention.\nSpecifically, both MAE and MAGE adopt full attention on all tokens ([M] & [V] → [M] & [V]) in their decoder.\nOn the contrary, the decoder in ENAT only inputs the mask token and integrates the visible ones via SC-Attention, creating a tailored \"[M] & [V] → [M]\" information flow in the decoder. \nExperimental results show that naively adopting MAE/MAGE-type full attention only increases computational overhead without enhancing performance:\n\n| **Decoder Attention Type** |**Decoder Attention Information Flow** | **GFLOPs↓** | **FID↓** |\n|-|----------------------|---------------------|----------|\n| MAE & MAGE type |[M] & [V] → [M] & [V]             | 24.5                | 5.03     |\n| Ours               | [M] & [V] → [M]|**22.6**                | **4.97**     |\n\nThis aligns with our observation in Section 4.1: In NAT generation, visible tokens primarily provide information for mask tokens. Thus, the [V] → [M] part in [M] & [V] → [M] has adequately fulfilled the role of the visible token, making it unnecessary to incorporate the visible token into the decoder input and perform full attention.\n\nWe will add more discussions to clarify the relationships with these works in our revision.\n\n>**W2: Intuition for Enabling Smaller Sampling Steps**\n\nThe key intuition behind our better performance with smaller steps is that: each decoding step in ENAT is more effective. To illustrate this, we conducted an experiment by applying different mask ratios (mimicking inputs in different steps of generation) to ground truth images and measuring the quality of *one-step decoded results* (i.e., the model directly predicts all masked positions in one forward pass) using FID. \n\n|  **Mask Ratio** | 0.2 | 0.4 |0.6 | 0.8 |\n|--------------|-------|-------|--------|--------|\n| MaskGIT (Baseline)       |  3.89 | 8.23 | 19.20 | 48.78 |   \n| ENAT (Ours)     |  **3.48** | **6.74** | **16.04** | **45.51** | \n| $\\Delta$     |  0.41 | 1.49 | 3.16   | 3.27   |\n\nAs the results show, ENAT consistently outperforms the baseline NAT design. This suggests that ENAT's predictions for masked positions are more reliable (i.e., of higher quality) at every decoding step, allowing them to take larger strides (decode more tokens per step) and producing high-quality generation with fewer refinement steps.\n\nThe qualitative results for the 0.2 mask ratio setting and the intermediate generation results of ENAT are also visualized in **Fig. 2** and **Fig. 1** of our global response to further illustrate this point.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jAnooiNAqY", "reviewer": "Reviewer_BoDK", "summary": "This manuscript reveals NATs' progressive token revelation, with asymmetric intra-step interactions where [MASK] tokens gather info and visible tokens offer it, focusing on key updates amidst repetition across steps. It then presents ENAT, which isolates [MASK]/visible token computations, prioritizes crucial updates, and recycles representations, significantly boosting NAT efficiency and performance on benchmarks like ImageNet and MS-COCO, with code and models to follow acceptance.", "questions": "Please refer to my weakness.", "limitations": "Yes, the authors have discussed the limitations of this manuscript in the supplementary material.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The structure of the paper is well-organized, and the logic of the arguments is clear, making it easy to understand.\n+ The motivation is clear and well-founded, leading to compelling results.", "weaknesses": "- Regarding the methodology, concerning Figure 5(b), how are the parameters within f(z^pre) (cross-step) trained?\n- Regarding scalability, in Table 3, ENAT-L achieves state-of-the-art (SOTA) performance with merely 574M parameters. What happens if larger models are employed? Does it possess a consistent ability to scale up efficiently, or does its optimization primarily exhaustively refine smaller models?\n- In Figure 6, it appears that the performance has not yet converged, suggesting that further increases in model size could potentially lead to significant additional performance gains.\n- The manuscript lacks sufficient visualized results, especially those that reveal how the method works and why it leads to performance enhancements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1: How the Projection Module $f$ is Trained?**\n\nAs discussed in line 221-224, we execute the \"reuse mode forward\" (Eq. (5)) with some probability during training, thus involving the parameters of $f$ in the computation graph. More specifically, the \"reuse mode forward\" during training is achieved through the following steps:\n  1. **Masking Tokens**: Given the current input token map $\\mathbf{v}$, we mask a random subset (*e.g.*, 30%) of visible tokens in $\\mathbf{v}$ to create $\\mathbf{v}^{\\text{prev}}$.\n  2. **Feature Extraction**: Feed $\\mathbf{v}^{\\text{prev}}$ into the NAT model to obtain its features $\\mathbf{z}^{\\text{prev}}$.\n  3. **Forward Pass and Loss Computation**: Use Eq. (5) to forward the current input token map $\\mathbf{v}$ along with $\\mathbf{z}^{\\text{prev}}$ obtained in Step 2, and compute the loss. In practice, a stop gradient operation is applied before feeding $\\mathbf{z}^{\\text{prev}}$ into Eq. (5).\n\nBy following Steps 1 and 2, we construct $\\textbf{z}^{\\text{prev}}$ and feed it into the projection module $f$ in Step 3, thus involving $f$ in the forward propagation. Since all operations are differentiable, gradients can then be effectively back-propagated to update the parameters of $f$.\n\nWe will elaborate further in our revision to make this process clear.\n\n> **W2 & W3: Scalability of ENAT**\n\nIn line with findings from other token-based models [A, B], ENAT is also scalable with increased model size.\nDue to limitations in time and computational resources, we conducted a preliminary experiment using a 1B parameter version of ENAT, referred to as ENAT-XL (28 layers, 1280 width):\n\n|         | **#Params** | **FID↓** |\n| ------- | ------- | ---- |\n| ENAT-L  | 574M    | 2.79 |\n| ENAT-XL | 1B      | **2.48** |\n| reconstruction FID of Autoencoder    | /       | 2.28 |\n\nAs the results show, scaling up ENAT can further improve performance.\n\nAn interesting finding is that, the performance ENAT-XL is already approaching the reconstruction FID of our VQ-autoencoder, which measures the quality of images decoded from ground truth visual tokens (thus being an indicator of the “performance upper-bound” for generative models in this VQ space).\n\nTherefore, for even further improvements over ENAT-XL, we may consider simultaneously scaling up the generative model and the VQ-autoencoder, which we are actively working on.\n\n---\n\n[A] Tian, Keyu, et al. \"Visual autoregressive modeling: Scalable image generation via next-scale prediction.\" arXiv preprint arXiv:2404.02905 (2024).\n\n[B] Chang, Huiwen, et al. \"Muse: Text-To-Image Generation via Masked Generative Transformers.\" International Conference on Machine Learning. PMLR, 2023.\n\n> **W4: More Visualization Results Revealing How the Method Works**\n\nThanks for your suggestion. We have included additional visualization results in our global response.\n \nSpecifically, **Fig. 1** visualizes the intermediate generation results throughout the process. \nThe results suggest that ENAT generates superior images as it converges to decent quality results faster, and continues to refine for even better image quality till the end.\n\nTo further corroborate this point, we conducted a \"one-step decoding\" experiment.\nIn this experiment, we applied various mask ratios (simulating inputs at different stages of generation) to ground truth image token maps and measured the quality of the generated images using FID for one-step decoded results (i.e., the model directly predicts all masked positions in one forward pass). \n\n|  **Mask Ratio** | 0.2 | 0.4 |0.6 | 0.8 |\n|--------------|-------|-------|--------|--------|\n| MaskGIT (Baseline)       |  3.89 | 8.23 | 19.20 | 48.78 |   \n| ENAT (Ours)     |  **3.48** | **6.74** | **16.04** | **45.51** | \n| $\\Delta$     |  0.41 | 1.49 | 3.16   | 3.27   |\n\nAs the results indicate, ENAT consistently outperforms the standard NAT design. The results for the 0.2 mask ratio setting are also visualized in **Fig. 2**. This demonstrates that ENAT's predictions for masked positions are more reliable at each decoding step, allowing it to converge to high-quality results faster and produce superior images.\n\nFinally, we have included more qualitative comparisons and samples in **Fig. 4, 5, and 6** to further validate the effectiveness of our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c7fYvz1ROF", "reviewer": "Reviewer_bqgC", "summary": "The paper analyzes the information flow in Non-autoregressive Transformers image generation methods (NAT) to improve their efficiency.\nIn particular, they find that in a forward pass, having the observed tokens attend to masked token is unnecessary and proposes to change the architecture accordingly.\nMoreover, they show that in the iterative steps, many token features don't change across steps and thus don't need to be recomputed.\nBy taking into account these two observations, their proposed ENAT design drastically cut down the number of TFLOPs their architecture requires while also improving image generation quality compared to the baselines.", "questions": "- It is mentionned in A2 that the experiments follow the approaches of [2], that is a Diffusion Transformer. By cross-referencing Table 6.a and Table 3, it seems this is a worse starting point than MaskGIT. Is that correct? Can we expect even better performances by starting from a strong model?\n- What motivated the choice of the baselines? Where are the DiT-XL scores from?\n- Are the FID computed against training or validation sets of ImageNet?\n\nThe submission has a principled approach and offers interesting findings. However, its originality is somewhat dimmed by the missing related work. Clarifications about the choice of baselines are also needed, along with many more qualitative samples, to make the comparisons more transparent. I'd be happy to switch my recommendations to the positive side if the necessary clarifications are made and if the authors demonstrate their willingness to include the requested discussions in the submission.", "limitations": "Limitations and societal impacts are adressed adequately in the appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper tackles the promising topic of Non-autoregressive Transformers for Image Generation. \n- It proposes two well-motivated and clearly exposed efficiency improvements backed by an empirical analysis and that also leads to performance improvement on the benchmarks. The findings with temporal redundancy with a projection is a fresh new perspective.\n- The paper has a complete set of empirical studies, including on MS-COCO generation and ablation of their modules.", "weaknesses": "- The efficiency of masked image transformer is very studied in the adjacent self-supervised masked auto-encoders (MAE) litterature, with whom NAT shares both architectures and base training algorithm, the main differences being different mask schedules and the classification loss in NAT instead of regressions (and even so, BeiT-v2 [A] uses a classification loss as well) . \nIn this adjacent literature, the seminal MAE paper [B] already had differentiated processes for observed and masked tokens: the masked tokens were introduced later, in the decoder, to avoid unnecessary processing in the encoder part. Recently, CrossMAE [C] replaced the self-attention from the MAE by a cross-attention CA(q=M, kv=V) making the architecture even closer to the proposed ENAT. The main difference is that ENAT uses CA(q=M, kv=concat(M,V)), reintroducing self-attention between masked tokens.\nI believe these considerations are very relevant to the discussion and should be included in a prominent position in the paper.\n- The SelfCrossAttention module is a minor variation upon standard attention modules that can also be found in the details of prior work, for instance in the Perceiver Resampler module of Flamingo [D].\n- The choices of baseline models and scores seem arbitrary and are not motivated. If all prominent families of generative models were to be represented, it is missing StyleGAN-XL [E]. Other very strong contenders such as Simple Diffusion [F], SiT [G], or MDT [H], also could be included, I'm not sure why they are not considered in the tables. For DiT-XL [32], I'm not sure where the reported scores are from. I might have missed them, but I can't seem find these values in the ICCV camera ready and suppmat, or any of the arxiv version. \n- The paper shows very few qualitative samples, and only on ImageNet 256x256. Considering the brittleness of image generation evaluation benchmarks, it's good to be able to check what type of images the model produces, especially since different methods can struggle with different aspects.\n\n\n---\n\n[A] BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers. Peng et al. arXiv 2022.\n\n[B] Masked Autoencoders Are Scalable Vision Learners. He et al. CVPR 2022.\n\n[C] CrossMAE: Rethinking Patch Dependence for Masked Autoencoders. Fu et al. arXiv preprint Jan 2024.\n\n[D] Flamingo: a Visual Language Model for Few-Shot Learning. Alayrac et al. NeurIPS 2022.\n\n[E] StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets. Sauer et al. SIGGRAPH 2022.\n\n[F] simple diffusion: End-to-end diffusion for high resolution images. Hoogeboom et al. ICML 2023.\n\n[G] SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers. Ma et al. arXiv preprint Jan 2024.\n\n[H] Masked Diffusion Transformer. Gao et al. ICCV 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1: Relationship with MAE, CrossMAE**\n\nThanks for your valuable suggestion. We are happy to include more discussions in our revision.\n\n**Firstly**, we kindly note that our approach includes *two* key aspects: a *disentangled architecture* and a *computation reuse* mechanism. To the best of our knowledge, the latter is novel in NAT and highly effective, yet has not been explored in MAE/CrossMAE since they focus on **representation learning**, which does not involve a multi-step process that exists in **generation tasks**.\n\n**Secondly**, while our *disentangled architecture* may appear similar to MAE and CrossMAE, it contains subtle yet important differences due to the distinction between representation learning and generation tasks.\nSpecifically, MAE inputs all tokens into the decoder and employs full attention on them ([M] & [V] → [M] & [V]), and CrossMAE retains a single \"[V] → [M]\" information flow. In contrast, we adopt a \"[M] & [V] → [M]\" information flow in the decoder attention:\n|**Decoder Attention Type**|**Decoder Attention Information Flow**|**GFLOPs↓**|**FID↓**|\n|-|-|-|-|\n|MAE|[M] & [V] → [M] & [V]|24.5|5.03|\n|CrossMAE|[V] → [M]|21.4|6.31|\n|Ours|[M] & [V] → [M]|22.6|**4.97**|\n\nThe results show that directly using MAE-type full attention merely adds computational overhead without improving performance, whereas implementing CrossMAE-type attention degrades performance notably.\n\nThis also aligns with our observation in Sec. 4.1 regarding generation tasks:\n1. Visible tokens primarily provide information. Since the \"[V] → [M]\" component in our \"[M] & [V] → [M]\" attention already fulfills the information-providing role of visible tokens, additionally incorporating the visible tokens into the decoder input for full attention (as in MAE) provides no benefit.\n\n2. The attention between masked tokens cannot be ignored (as opposed to CrossMAE). Intuitively, this allows masked tokens to be aware of each other during decoding, thus preventing incongruencies in the generated results. This further highlights the distinction between a generation-focused approach and a representation learning-focused approach.\n\n**Finally**, we would like to clarify that our contributions are not limited to proposing a new architecture/mechanism.\nWe have revealed several important scientific findings and insights in NATs, which are also novel contributions.\nWe believe these findings can provide new perspectives for the research community.\n> **W2: Novelty of SelfCrossAttention Module**\n\nThanks for pointing out the related work [D], and we are happy to update our revision with this reference.\n\nAdditionally, we would like to kindly clarify that the SC-attention, as briefly introduced in Section 4.1, is not presented as our main contribution. Our primary contribution lies in uncovering new scientific findings in NAT generation, specifically regarding its spatial and temporal properties, that enhance our understanding and enable a better ENAT design.\n\n> **W3 & Q2: Choices and Scores of Baseline Models**\n\nThanks for your suggestion. We are happy to add GAN-type models like StyleGAN-XL in our revision, and we have already incorporated it in **Fig. 3** of our global response.\n\nFor the choice of baselines in our work, since ENAT focuses on inference efficiency, we aim to compare ENAT with other models in a lightweight, low-FLOPs scenario.\nHowever, while the inference efficiency of generative models is important, it is generally under-explored in the original papers of state-of-the-art diffusion models (e.g., DiT, MDT, SiT), which mostly focus on enhancing generation performance.\nThe official results of them are primarily obtained with hundreds of inference steps, making direct comparisons with ENAT challenging.\nFor instance, as shown in **Fig. 3**, the official results of DiT, MDT, and SiT all concentrate at the high end of overall inference costs, requiring hundreds of times more computation than ENAT.\n\nFortunately, there are well-established fast sampling techniques (e.g. DPM-Solver) for accelerating diffusion models, and we have identified several diffusion models that are both highly competitive and officially supported* by these acceleration techniques.\nThis allows us to reduce their sampling steps and compare them with ENAT in a fairer setting.\n\nFor the mentioned stronger contenders, we plotted MDT and SiT in **Fig. 3**. Simple diffusion is omitted because its computational cost is neither reported nor open-sourced for measurement.\nThe strongest MDT model is also integrated with fast sampler for fewer sampling results.\nFew-step fast sampler results from the original U-ViT paper are also included for reference.\n\nIn our revision, we are willing to add more discussions on our baseline results and setups. Stronger baseline models like MDT and SiT will also be included.\n\n---\n*: U-ViT has DPM-Solver integrated in their Github repo. DiT and LDM are in the Hugging Face `diffusers` library, where DPM-Solver is also officially supported.\n> **W4: More Qualitative Results**\n\nThanks for your suggestion. In our global response, we have added qualitative comparisons with DiT-XL in **Fig. 4** and included many more qualitative samples for all evaluated datasets in **Fig. 5 & 6**.\nMore samples will also be added in our revision.\n\n> **Q1: U-ViT as a Worse Starting Point?**\n\nWe would like to clarify that ENAT only follows the *training pipeline* of [2], rather than its diffusion transformer model architecture.\nThe pipeline of [2] is relatively standard, and our goals are twofold: **1)** to validate the gains solely attributable to our design under fair conditions, and **2)** to demonstrate that our method does not rely on specific training techniques. \nExploring more advanced training techniques could potentially lead to further improvements, and we are happy to explore this in the future.\n> **Q3: FID Evaluation Protocol**\n\nWe follow the widely adopted OpenAI's evaluation protocol to compute FID against the ImageNet training set.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their insightful comments and suggestions.

We are encouraged that our work was found to be **well-motivated/written** (all reviewers), with our findings **offering fresh new perspectives** (Reviewer bqgC) and our method being **simple** (Reviewer GVb9) and **effective** (Reviewer PoGn, GVb9, BoDK).

One common concern was the need for more visualized results to elucidate why ENAT works well or to demonstrate its effectiveness more comprehensively. To address this, we have included qualitative analyses (**Fig. 1,2**) and many more samples (**Fig. 4,5,6**) of our method in the attached PDF file. To avoid confusion, the figures in the rebuttal pdf are all referred to in **bold**.
More qualitative results will also be incorporated in our revision.

Other specific concerns are elaborated in the responses to each reviewer. We hope our responses appropriately address your concerns.

Meta-review: The submitted manuscript proposes to improve MaskGIT-like non-autoregressive image generation models by spatially disentangling 
 visible tokens and mask tokens during self-attention in order to prioritize visible tokens and to reuse tokens from previous decoding steps - both modifications towards increased efficiency. The modifications have beneficial effect on FID and model efficiency. 
All reviews agree that the paper is well written - also, the idea is overall simple yet effective.
Some initial criticism regarding method details and ablations (for example the difference to MAE and MAGE) were clarified during the rebuttal phase so that the paper has overall positive scores after the  discussion phase.
Decision: Accept (poster)