Unlocking the Capabilities of Masked Generative Models for Image Synthesis via Self-Guidance
Authors: Jiwan Hur
Dong-Jae Lee
Gyojin Han
Jaehyun Choi
Yunho Jeon
Junmo Kim
Venue: NeurIPS 2024 poster (2024)
Keywords: Image synthesis
discrete diffusion models
masked generative models
sampling guidance
parameter- efficient fine-tuning
Primary area: generative_models
TL;DR: We propose a guided sampling techniques for masked generative models and empirically demonstrate its effectiveness for image generation.

Abstract: Masked generative models (MGMs) have shown impressive generative ability while providing an order of magnitude efficient sampling steps compared to continuous diffusion models. However, MGMs still underperform in image synthesis compared to recent well-developed continuous diffusion models with similar size in terms of quality and diversity of generated samples. A key factor in the performance of continuous diffusion models stems from the guidance methods, which enhance the sample quality at the expense of diversity. In this paper, we extend these guidance methods to generalized guidance formulation for MGMs and propose a self-guidance sampling method, which leads to better generation quality. The proposed approach leverages an auxiliary task for semantic smoothing in vector-quantized token space, analogous to the Gaussian blur in continuous pixel space. Equipped with the parameter-efficient fine-tuning method and high-temperature sampling, MGMs with the proposed self-guidance achieve a superior quality-diversity trade-off, outperforming existing sampling methods in MGMs with more efficient training and sampling costs. Extensive experiments with the various sampling hyperparameters confirm the effectiveness of the proposed self-guidance.

Unlocking the Capabilities of Masked Generative Models for Image Synthesis via Self-Guidance
Jiwan Hur!
Dong-Jae Lee²
Cyojin Han
Jachyun Choi
Yunho Jeon
Junmo Kim
KAIST, South Korea
Hanbat National University, South Korea
{jiwan. hur, jhtwoaun, hangj0820, chlvogus>Okkiss. ac.kr kr
yhjeonGhanbat kc. junmo. kim@kaist.ac..
Code is available at: /ttp:/u com/ haaphbbbcoo/JJaaaaauuuuuoolottto
Figure 1: Comparison of sampled images using 18-step MaskGIT [4] without (top) and with the proposed self-guidance (bottom) on ImageNet 512x512 (left) and 256x256 (right) resolutions. Each paired image is sampled using the same random seed and sampling hyperparameters. The proposed self- seff-guidacce effectively improves the capabilities of the masked generative models.
Abstract
Masked generative models (MGMs) have shown impressive generative ability while providing an order of magnitude efficient sampling steps compared to continuous diffusion models. However, MGMs still underperform in image synthesis compared to recent well-developed continuous diffusion models with similar size in terms quality and diversity of generated samples. A key factor in the performance continuous diffusion models stems from the guidance methods, which enhance the sample quality at the expense of diversity. guidance methods to generalized guidance formulation for MGMs and propose a self-guid self-guidance sampling method, which leads to better generation quality. The proposed approach leverages auxiliary task for semantic smoothing in vector- quantized token space, analogous to the Gaussian blur in continuous pixel space. Equipped with the parameter-efficient fine-tuning method and high-temperature sampling, MGMs with the proposed self-guidance achieve a superior quality- diversity trade-off, outperforming existing sampling methods in MGMs with more efficient training and sampling costs. Extensive experiments with the various sampling hyperparameters confirm the effectiveness of the proposed self -affaguid
this paper, we extend these
'Corresponding authors: junmo. @ kaist.ac.kr yhjeon @ Ohanbat.ac.kt
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Keywords
Image synthesis, discrete diffusion models, masked generative models, sampling guidance, parameter- efficient fine-tuning
1 Introduction
With the advent of generative adversarial networks (GANs) [14), generative models have attracted significant attention for their powerful ability to synthesize highly realistic images. However, due to the limited mode coverage and training instability, recently, likelihood-based models such as diffusion models [20] have been actively researched. Diffusion models have shown promising results for their diverse and high-quality samples, surpassing GANs on class conditional image synthesis [TO].
key factor in the success of diffusion models stems from the various guidance techniques, which enhance the fidelity of the generated image the expense of diversity. The guidance is usually conducted throughout the sampling process of diffusion models, driving it toward enhancing specific
information such class [TO. 19] text 44 [2], or details in the image [22]. From the practical perspective, however, diffusion models suffer from sampling inefficiency, requiring hundreds of sampling steps generate high-quality images. Moreover, guidance techniques further decrease the sampling efficiency, as they typically require twice as many model inferences as the original process. For instance, ADM [10 with guidance requires 500 model inferences.
On the other side, masked generative models (MGMs) have shown superior trade-offs between sampling quality and speed compared to (continuous) diffusion models 4 35 36). MGMs use an absorbing state (DMASK) ) diffusion process [1] and aim to generate discrete tokens by predicting the masked region, similar to BERT [9]. Specifically, they the Markov transition matrix to model the diffusion process and sample the tokens with the categorical distribution. Recently, vector-quantized (VQ) image token-based MGMs with non-)Jautoregresivee transformer [ 4 have demonstrated an efficient sampling process, providing an order of magnitude fewer sampling steps (e.g., 18 steps) than continuous diffusion models 4
While continuous diffusion models utilize guidance sampling to enhance generation quality, MGMs typically utilize sampling with nowttempeaature Gumbel noise in categorical distribution to enhance the quality 4 35). However, low-temperature sampling may impose 60). where the non-auuoreeressivve sampling process fails to generate plausible outputs due the lack of sequential dependencies. As a result, the upper bound of the sample quality generated by MGMs was relatively limited. Several approaches have been suggested to improve the sampling quality of MGMs, such as discrete predictor-corrector-ber discern the unrealistic tokens. However, despite the improved sampling, MGMs still underperform in terms of FID scores [88 compared to mate-offthe-aae continuous diffusion models such as LDM on ImageNet benchmark [8] even when the model sizes are similar.
problem [15] 39
methods [35] 36) that train a second transformer to
To overcome such limitations and enhance the sample quality of MGMs, we propose discrete self-guidance sampling (Fig. D. Self-guidance [22] in continuous diffusion models improves the generation quality by enhancing the fine-grained detail of the sample by guiding the diffusion process with coarse-grained information within intermediate diffusion steps, similar to that of classifier- free guidance, which utilizes unconditional generation to enhance the quality of class conditional generation. In continuous space, the coarse-grained information can be easily obtained with spatial smoothing, like Gaussian blur. To apply self-guidance in MGMs, we first define the general guidance formulation for the discrete diffusion models and introduce self-guidane, in MGMs. However, while it is simple to define the coarse-grained information in continuous space, the VQ token space [43] in MGMs is absent of continuous semantic structure and cannot apply such simple strategy, c.g., blur (2). Furthermore, we cannot directly define coarse- gaase--rained, i.c., semantically smoothed outputs in the VQ token space. Therefore, we introduce an auxiliary task specifically designed for semantic smoothing of the VQ token space, which enables the network to selectively remove details such as local patterns while preserving overall information in VQ token spaces. Consequently, by guiding the MGMs with semantically smoothed information, we can enhance the quality of MGMs (Fig. 2 even with high temperatures, therefore achieving both high quality and diversity compared to previous MGMs. For efficient implementation of the discrete self-guidance, we introduce plug-and-play module with parameter-efficient fine-tuning [49] to pre-trained MGMs, leveraging the generative prior in the MGMs while enabling efficient training with few parameters and epochs.

---

Input image
Masked image (m.r=90%)
One step prediction
Self-guidance w/ spatial smoothing (SA semantic smoothing (Ours)
Self-guidance w/
Visualization of the effect of guidance using spatial smoothing (SAG) [2] and the proposed semantic smoothing. We tokenize the input image using VQGAN [T]] encoder, mask the 90% of VQ tokens, and predict æo,; using MaskGIT [4. With the proposed self-guidance leveraging semantic smoothing, generated sample quality is improved by enhancing fine-scale details.
Figure 2:
Experimental results demonstrate that the proposed guidance effectively improves the sample quality with only 10 epochs of fine-tuning. Notably, combined with high sampling temperature, the only improves the quality of generated samples but also keeps diversity high, showing a superior quality-diversity trade-off compared to other improved sampling techniques for MGMs and other generative families with similar model sizes.
proposed guidance
Background
2.1 Masked Generative Models
Discrete diffusion models [ aim to generate categorical data E ER with a length of N and K categories. The forward Markov process q(x++1 24)) gradually corrupts data To, T1, 2r until the marginal distribution of ~ p(ær.) becomes stationary. Then starting from the æy, the learned reverse Markov process Pa (2,-1 (2,, gradually recovers the corrupted data to generate Zg-
Masked Generative Models (MGMs) are a family of discrete diffusion models that use an absorbing state diffission process M. in which they gradually mask the input tokens by replacing them with a mask token ( ([MASK] and learn to predict the masked region. Recently, dector-quantizn
(VQ) image token-based MGMs 4 335 36 have achieved a superior trade-off between sampling time and quality for image synthesis compared to Gaussian (continuous) diffusion models [10] 44) through the noo-autoreeressvve parallel decoding process. MGMs are trained predict the masked region similar to BERT [9] in natural language processing (NLP), but with various masking ratios " (0. 1], where y is pre-defined mask scheduling function that masks n = (r(t/T) N] tokens from N total tokens. Without loss of generality, we assume the unmasked region of the mask m, is 1, and the masked region is O. Then, given external condition e (e.g., class or text) and masked input æ, = 2o O m,, where the mask m, is randomly sampled according to masking ratio 7r MGMs are trained to predict clean data ag (or equivalently, to predict the masked regions) by the objective function
Liakk = -Ex,
Σ
log log xt, x),c)
(I)
Here, we denote the i-th token of æo x).
After the training, sample the image Fo. MGMs typically use an iterative prediction-masking procedure starting from the blank canvas ær (i.e., all tokens are masked). Given (partially) masked image token 2, sampled from the previous step, MGMs first predict all tokens L po (#., (x.)) simultaneously, where L denotes the prediction of æu at timestep t and we omit e for simplification. Then æ masked according to masking ratio 7 to obtain -t-1. MGMs usually sample m,-1 ~ p(m.-1] mm, @o.r) which is equivalent to sample #t-1 since Ø;-1 m:-1 $o.t- Randomly sampling the m,-1 can one choice; however, the prediction Φø, may have numerous errors, thus randomly selecting the mask produce sub-optimal results by potentially masking relatively

---

accurate tokens while leaving unrealistic tokens unmasked. To overcome this, various research adopts improved sampling methods such as utilizing the output confidence of the Lo.t since confident tokens tend to be more accurate [ or train an external corrector to classify the realistic tokens RS 36
However, these improved sampling may impose a problem, named multi-mmdaality problem that is well-known problem in non-autoregressive parallel sampling [15] 39) 66). Given the input such as et and class condition c, the model can have multiple plausible outputs, which brings challenges the non-autoregressive model as they generate each token independently. In an extreme case where the input token all masked, and the model predicts output only using given external condition such as class, each token can predict easy token more confidently and correctly, such as background in every token. As result, correctness-basee images. To resolve this problem, various MGMs adopt additional randomness to sample m,, such as sampling with temperature 4 35 36
sampling may result in images only filled with background
Let the le measure the realism of sampled tokens fo such as the confidence scores of sampled tokens in MaskGIT A Then MGMs sample m 1 by selecting top-k elements of l = l + (t/T)n according to "rs where denotes the sampling noise such as i.i.d. Gumbel noise and r is temperature scale that annealed according to the timesteps. Generally, high-temperature sampling results in more diverse samples while degrading the sample quality.
2.2 Sampling Guidance
sampling processes of diffusion models are often guided by external networks or themselves. Therein, salient information-base
Iterative
guidance has been actively explored in continuous diffusion models 119 22 [2] for high-quality image synthesis. Let he be salient information of æ; and o, be perturbed sample that lacks h4. he can be internal information within æ; or an external condition, or both. Lee al. [22 proposed a general guidance technique that guides the sampling process toward enhancing the information he, and the equation for the sampling process is:
E(#,, h4) = <(#;) + (1 + ()( h4) - <((e)),
where is score function, ē is guided score function of the continuous diffusion model, and is guidance scale. For instance, in the setting f; = and æ; = 2r, the Eq. (2) collapses to classifier-free guidance (CFG) [19] which guides the sampling toward the given class distribution. Lee et [22] propose using adversarial blurring, enhancing the fine-scale details of the sample. Generally, using large guidance scale s enhances the quality of generated samples while reducing the diversity.
Recently, discrete CFG [52 5 has been introduced to improve the correlation between the input class or text condition and the images generated by discrete diffusion models below equation: logp(ă,) ( = log p (x.) + (1 +
e) - log p(x;)),
where 3g denotes the guided token.
Unlike CFG in the continuous domain, discrete CFG estimates the probability distribution p(æ,)o
directly. However, it requires specific training strategy and paired labels such as class and text.
2.3 Parameter-Efficien Fine- Fine-Tuning
MGMs across various domains adopt transformer architecture due to their superior ability to handle context with bidirectional attention 9 13 57 63) However, training transformer from scratch requires significant computational resources due to the quadratic complexity of the attention mecha- nism. In recent years, earameter-efffcient fine-tuning (PEFT) techniques have received significant attention, especially in light of the growing size and complexity of pre-trained models. PEFT adapts large pre-trained models specific tasks datasets by tuning a small portion of parameters [58] (55 or introducing task-specific parameters extensive retraining. Notably, by preserving most of the parameters, the fine-tuned model effectively preserves knowledge with few forgetting.
45 23
effectively transferring knowledge without
Methods
3.1 Generalized Information-Bann Guidance for Discrete Diffusson Models
Similar to the general guidance in continuous diffusion models Eq. (2). discrete CFG Eq. (3 can be extended to the generalized information- guidance from the optimization perspective. Given

---

some salient information h2, we aim to sample which maximizes p( Simultaneously, for the correlation between the information and sample, p(h, ( also needs be maximized as stated in the equation below which is from Tang et al. [52]
arg max p (1.) + log p( [e))] 2+
(4)
Using the Bayes' theorem and ignoring the prior probability term for salient information, the optimization goal can be represented as:
arg max log ((z)) (1 + ((og (ha) log)(
22
Then, the Eq. 5 guides the sampling process toward enhancing the relevance of the sample and salient information h4
In the inference stage, various MGMs adopt to predict unmasked state to, rather than directly predicting 24-1- If we limit h4 to the internal information of at to make h2 removable from the ane through an information bottleneck module Ha: in other words, if æ; = Hp((z)) and p(x;) ) = p( ha)
get satisfied, we can sample next state for the denoising step as below:
log po (0
log pe @0o,e (x:)))))))))))))) ))) + (1 + s)og pe (o(,,,1 logpo H.s (zx))), (6)
when a MGM with parameter predict L from æ, and Bo from Hz,) This implies that by defining the information bottleneck module H. that can selectively subtract salient information he from ØG in the discrete space, we can guide the sampling of MGMs in a direction that enhances hr- Since we aim to improve the sample quality of MGMs by presenting novel guidance method, it necessary define H.g. that can remove information about the fine details of the samples. For continuous diffusion models, utilizing samples spatially smoothed by Gaussian blur for guidance has been helpful improving sample quality by restricting fine-scale information (22]. This motivates us to investigate guidance with smoothed output for discrete domains, especially for VQ tokens.
3.2 Auxiliary Task Learning for Semantic Smoothing on VQ Token space
We aim to apply smoothing, such as Gaussian blur, for VQ tokens, as discussed in the previous section, to implement guidance in the discrete domain. However, unlike natural images which have inherent, observable patterns and structures, the latent space of autoencoders often lacks such properties (2]. For instance, two successive tokens in VQ codebooks, such as the 11th and 12th tokens, are not semantically linked. As a result, applying Gaussian blur in the VQ token cannot produce meaningful representations. Nevertheless, spaces, i.e. blurring the output logits of the generator py (& (x,) similar to Lee et al. [22] can provide meaningful guidance. However, the improvement is marginal because Gaussian blur is not a suitable information bottleneck for subtracting fine details in VQ token space.
empirically found that applying Gaussian blur in the probability
To overcome this, we introduce auxiliary task designed to leverage semantic smoothing for VQ tokens, process that selectively removes details such as local patterns while preserving overall information in VQ token space. Specifically, given the masked input æ;, masked generator predicts o ax.). We aim to train information bottleneck He to generate the semantically smoothed output pe( Hog((x)) However, training Hig directly is challenging, smoothed outputs. To naturally impose model to generate semantically smoothed output, we leverage error token correction, originally introduced in von-autoregressive machine translation to mitigate the compounding decoding error in the iterative sampling process (24]. During the error token correction, input unmasked tokens are randomly replaced with error tokens, and the model learns to correct them. To be more specific, let z, be a corrupted data where some tokens in æ, are replaced with error tokens with some probability p. Then, the objective function for the auxiliary task to update & can be
cannot define semantically
Lauz =
Em,
Σ
log (2 (2.),
From the perspective of Vicinal Risk Minimization (VRM) 59 a vicinity distribution Pa minimizes the empirical risk for all data points 2; given vicinity of the data zt. Given that randomly replaced
5

---

Frozen
Tuned
17
388
Ground truth
Guidunce (plug-and- play)
Sampled token
Pu (R:)
Py Py Iie)
Gaa (x,))
Emor token
FC
PC
Pretrained Masked Generator
Feature Selection
Pretruized Masked Generator
Pretrained Masked Generator
Feature selection
Pretrained Masked Generstor
Shared
16
291
<M>
117
<M>
<>>
<>>>
<M>
17
291
<M>
117
<>>
<>>
<>>>
<M>
(a) Training feature selection module
(b) Guided sampling
With the auxiliary objective in Eq. 7 Hi implicitly learns to smooth erroneous input to address semantic outliers (Section (.2)) (b) During the sampling steps, self-guidance can be efficiently implemented by leveraging the feature map from the generative process. Hos performs semantic smoothing on the input guiding the sampling process toward enhancing fine-scale details in the generated sample.
Figure Fine-tuning the feature selection module H. (TOAST 49
error tokens often act as semantic vicinities within the input data, to minimize the overall empirical risk, the model implicitly learns to smooth vicinities of z, This involves leveraging coarse information from the surrounding context while minimizing the fine-scale details in the presence of unknown input errors] However, training network from scratch with Eq. ( does not ensure that the outputs will resemble real images, potentially converging on trivial solutions that may be undesirable, in addition to being computationally expensive.
3.3 Efficient Implementation
To mitigate the aforementioned problem, adopt a parameter-cfficient fine-tuning (PEFT) method to utilize deep image priors in the pre-trained masked generator and to enhance the training efficiency of transfer learning. Among various PEFT methods, we adopt TOAST [4), which shows favorable performance in various visual and linguistic tasks under transformer architecture. With frozen pre-trained backbone, TOAST selects task-relevant features from the output and feeds them into the model by adding them to the value matrix of self-attention. This top-down signal steers the attention focus on the task-relevant features, effectively transferring the model other tasks without changing parameters.
Besides its effectiveness in transfer learning, TOAST brings more practical strengths our task. Before the discussion, it is important to note that masked generative models exhibit a strong training bias. Because the training data does not contain error tokens, the model predicts unmasked input as identical and unable to correct error tokens. Since the model is trained only to consider masked regions, propose use blank canvas as an input (i.e., all tokens are masked æT), enabling the model to make corrections in response to error tokens. However, most PEFT methods lack direct solution for incorporating information about &; when the input replaced with all masked tokens. On the other hand, we empirically found that simply replacing the input for the second stage of TOAST as Ør can mitigate this problem without a performance drop (Fig a).
Furthermore, the two-stage approach of TOAST can be efficiently implemented in the sampling process. The generator first sample po (0o the generation stage can be recycled for the second stage to produce guidance logit po (Zo, (/ (xx))
Boo../2 (xx) from the input 2e, The hidden state obtained from
For a simple example, a common approach to correcting unknown numerical outliers a I-dimensional signal, such as impulse noise in time series data, is smoothing the signal, like applying low-pass pass filters. Similarly, to correct the unknown error tokens, which are semantic outliers in our case, we expect that the model implicitly learns to smooth to deal with unknown error tokens.

---

Table 1: Quantitative comparison of various generative models for class-conditions image generation on ImageNet x and 512x512 resolutions. "4" "1" indicate lower or higher values are
better. t: taken from MaskGIT [ f: taken from VAR 53 *: taken from Token-Critic 35
ImageNet 256x256
ImageNet 512x512
Model
Type
NFE
FID,
ISt
Prect
Reet
FID4
1S1
Prect
Reef
BigGAN-deep
GANs
6.95
224.5
0.89
0.38
8.43
177.9
0.85
0.25
GigaGAN
GANs
3.45
225.5
0.84
0.61
ADM [0]
Diff.
250
10.94
101.0
0.69
0.63
23.24
58.0
0.73
0.60
ADM SAG) 12
Diff.
500
9,41
104.7
0.70
0.62
CDM 21
Diff.
250
4.88
158.7
LDM-4 A1
Diff.
250
10.56
103.4
0.71
0.62
LDM-4 CFG)
44
Diff.
500
3.60
247.7
DiT-L/2 (+ CFG) !
Diff.
5.02
167.2
0.75
0.57
VQVAE-2!
43
AR
5120
31.11
.45
0.36
0.57
VQGAN!
AR
~102
18.65
80.4
0.78
0.26
7.32
66.8
0.73
0.31
VQ-Diffusion
16
Discrete.
100
11.89
ImprovedVQ. (+ CFG) 52
Discrete.
200
4.83
MaskGIT
Mask.
6.56
203.6
0.79
0.48
8.48
167.1
0.78
0.46
Token-Critic 35
Mask.
36
4.69
174.5
0.76
0.53
6.80
182.1
0.73
0.50
DiChtight 36
Mask.
4.8
249.0
0.80
0.50
6.09
228.1
0.81
0.46
DPC- 36
Mask.
180
4.45
244.8
0.78
0.52
6.06
218.9
0.80
0.47
Ours (T=12)
Mask.
24
3.35
259.7
0.81
0.52
5.38
226.0
0.88
0.36
Ours (T=18)
Mask.
36
3.22
263.9
0.82
0.51
5.57
233.2
0.88
0.35
(Fig. b). We note that different from the fine-tuning step where the error tokens are used to train Hg, the sampling process directly utilizes æ; to generate the semantically smoothed tokens Ho (zzz).
Related Works
Generative Adversarial Networks (GANs). Trained by adversarial objective 14, GANs have shown impressive performance image synthesis with only 1-step model inference 27 28 47 B3
However, despite their practical performance, the training instability and limited mode coverage caused by the adversarial objective [46 62 is still bottleneck for their broader applications.
Diffusion Models. Recent diffusion models in continuous space mostly utilize Gaussian noise to perturb the input data and learn to predict clean data from the noisy data [50]. After the advent of
DDPM (20), diffusion models have rapidly grown with architecture improvements [38] [1]. improved
training strategies 29 improved samplings 37) latent space models 44 and sampling
guidance 10
outperforming various generative families such as GANs in various domains.
Masked Generative Models. With the recent success of transformers 54 and GPT in NLP, generative transformers have also been adopted in image synthesis. To mimic the generative process transformers for discrete embedded natural language, recent studies encode images into quantized visual tokens using the VQ- VAE encoder
43
[1 and apply the generation process in an
autoregressive [32 [56] or non-autoregressive manner 35 36 5]. In particular, non-autoregresssve generation shows better trade-off between generation quality and speed. Nevertheless, they suffer from compounding decoding errors [36), which means that small decoding errors in early generation steps accumulate into large differences in later steps. To address this issue, Token-Critic (35] and DPC [36] use additional transformers to identify more realistic tokens. However, they require a training second transformer, which incurs training cost similar training generator from scratch.
The non-autorreressive predict-mask sampling of MGMs can regarded as a discrete diffusion process that uses absorbing state diffusion process [ 36.. Similar to MGMs, VQ-Diffusion [16] pro- posed rep-lae diffusion strategy to predict masked tokens. Improved VQ-Diffusion adopt purity-based sampling with discrete CFG to further improve the sample quality.
7

---

vs. FID (256X256, 50k)
MaskGiT (18)
IS vs. FID (512X512, 50k)
Token Token-Critic (36)
9.5
3
Ours (36)
9.0
DPC-light (66)
DPC-full (180)
0.0
2 7.5
7.0
3-
3
53 160
150
200
250
300
180
200
220
240
15
Figure 4: IS vs. FID curves of various sampling methods for MGMs on ImageNet 256x 256 and 512x512. The curve positioned towards the bottom right indicates a better trade-off between sample quality and diversity. We plot the curve by varying the sampling temperature (7), and the curves of MaskGIT and Token-Critic 35| are taken from Token-Critic [35]
Experiments
Datasets, Baselines, and Metrics. We demonstrate the effectiveness of the proposed guidance for masked generative models on class conditional generation using the Imagenet benchmark [8] with 256 x 256 and 512 x 512 resolutions. For baseline model, we use MaskGIT [4]. which shows state-of-the-ar trade-off between quality and sampling speed on a class conditional generation of MGMs and has publicly available checkpoints for our target datasets. To evaluate the trade-off between sample fidelity and diversity, we measure Fréchet Inception Distance (FID) [8] Inception Score (IS) 46) Precision, and Recall [3I] using the implementation provided by Dhariwal et al. [10] To measure the computational cost, we report the number function evaluations (NFE) required to sample an image. Note that total sampling timesteps (T)) may differ from NFE due to guidance.
Implementation Details. We use VQGAN tokenizer [1] provided by MaskGIT
which encodes
images into 10-bit integers. Since the training code for MaskGIT is unavailable, we implement based on the open Pytorch reproduction [40). We follow the previous work for error token correction [24] to prepare input with error tokens and randomly replace 30% of input tokens with error tokens. We utilized an NVIDIA RTX A600 for fine-tuning and sampling. We used an exponential moving average (EMA) of fine-tuning weights with a decay of 0.9999 and bf 16 precision. The batch size was set to 256, and the additional parameters introduced by TOAST are approximately 20-25% of the model size. Notably, the fine- the-tuning was completed efficiently within 10 epochs. More detailed implementation of TOAST 49 is provided in the Appendix [ We use sampling step T 18 and sampling temperature 25 for Imagenet 256x and 45 for Imagenet 512x512. For sampling step 12, we use temperatures 10 and 20, respectively for each resolution.
5.1 Comparison with Various Generative Models
Quantitative results. We compare the performance of the proposed method with various class
conditional generative methods. (1) GANs: BigGAN 3 and GigaGAN (26); (2) continuous diffassio models (Diff.): ): ADM [TO, CDM 21), LDM
44 and DiT 41: (3) auto-regressive models (AR):
VQVAE-2 4 and VQGAN [1] (4) discrete diffusion models (Discrete.): VQ Diffusion [6]]. and
Improved VQ Diffusion [52] (5) masked generative models (Mask.): MaskGIT [4]. Token-Critic [3).
DPC [36]. As noted in previous literature [19 336, sampling using pre-trained classifier may impact the classifier-based metrics such as FID and IS. Thus, see the base generative capacity of each method, we compare the models that do not use external pre-trained networks such as a classifier or upsampler during training or sampling. We also exclude large-scale models such as DiT-XL/2 [41) for a fair comparison. Table presents the quantitative results of various generative models with guidance. All values are taken from the original paper unless otherwise noted. The proposed method achieves superior FID and IS despite the low computational cost for sampling. For instance, the proposed method achieves better FID and than DiT-I/22 with CFG on Imagenet x56x256 even though the proposed method requires order of magnitude fewer sampling steps.

---

LDM (NFE-50000, FID-3.60)
MaskGIT (NFE-18, FID=6.56)
OuIs (NFE-36, FID-3.22)
Figure 5: Sampled images on ImageNet x56x2556 class conditional generation using selected classes
(105: Koala, 661: model T, and 933: Cheeseburger). left: LDM [44] CFG (s=1.5, NFE=250x2),
middle: MaskGIT (NFE=18), right: Ours (s=1.0, NFE= 18x2).
We comprehensively compare the proposed methods with various sampling strategies for MGMs in Fig. 4 We note that all MGMs use the same VQGAN tokenizer [1] provided in MaskGIT [ same baseline generator
18. Although the previous methods, such as Token- [35 and DPC (36,, require similar or more NFEs to sample images
4 and the same sampling timestep
and more training resources train an external corrector transformer, our simple guidance shows
better trade-off between sample quality and diversity.
Qualitative results. Fig. presents the randomly sampled images using LDM 44 with CFG, MaskGIT, and MaskGIT with the proposed guidance. The proposed guidance sampling enhances details in the generated samples while maintaining high diversity. More sampled results with various classes are provided in the Fig. I and Appendix
5.2 Ablation Studies and Analysis
In this section, we explore the effectiveness of the guidance on class conditional ImageNet generation in 256 256 scale across the various sampling hyperparameters. We vary each sampling parameter and otherwise use the default settings.
Effectiveness of Auxiliary Task. To demonstrate that fine- tuning with a proposed auxiliary task using the objective in Eq. can effectively improve the generative capabilities of MGMs, we conduct ablation studies and report the FID and IS in Table 2 As noted in Section [3.2] applying Gaussian blur in the input VQ token does not properly smooth VQ tokens. However, we empirically found that applying Gaussian blur in the output logit can produce meaningful guidance. There- fore, we provide the quantitative comparison with guidance by applying Gaussian blur in the logit space (Blur Guidance). We further utilize self-attention value for adversarial masking following Lee et al. (22). Blur guidance and SAG enhance the quality (IS) diversity (FID) in some
Table 2: Ablation results of various guidances on ImageNet 256 256 class conditional generation.
FID
ISt
Blur Guidance
8.32
231.2
SAG 22
4.73
177.3
ft. w/ L
4.11
238.4
ft. w/ Lane (Ours)
3.22
263.9

---

8.0
4
6.0
B
4.2
0-
30
20
03
scale
Epochs
325
300
200
200
200
250
275
25b
Oustence scale
Kempenatur C1)
Thresdep
Spocts
(a) Guidance scale
(b) Temperature
(c) Timestep
(d) Fine- tuning epochs
Figure 6: Exploring the sampling hyperparameters by varying (a) guidance scale, (b) sampling temperature, (c) sampling timesteps, and (d) fine-tuning epochs.
degree, but the improvement is marginal compared to ours. We further compare the results when the TOAST module is fine-tuned with the generative objective in Eq. (T) to verify the effectiveness of the auxiliary task (ft. w/ fmmast). Since the TOAST architecture and blank input for the second stage naturally play the role of information bottleneck, the performance has increased. Nevertheless, fine-tuning with the proposed auxiliary loss in Eq. 7 demonstrates its effectiveness, showing superior performance in both metrics.
Varying the guidance scale (Fig.6aa) In line with previous literature on sampling guidance [10 19 [22], a high guidance scale improves the sample quality while sacrificing diversity. We found that the guidance scale 1.0 shows the best performance terms of FID score and the best trade-offs.
However, strong guidance (s > 3) does not ensure quality improvement, often leading to undesirably highlighted details or saturated colors (Appendix B). similar to the effect of large scale with CFG [19]
Varying the sampling temperature (Fig. 6 b). Compared to MaskGIT [4] limit their sampling temperature relatively low value (r 4.5), found that with the proposed guidance, the sample quality can effectively preserved with a higher sampling temperature. As a result, with a high sampling temperature
25), we achieve better quality and diversity compared MaskGIT.
Varying the sampling steps T (Fig. c). c). We observed that for higher T, the optimal sampling temperature also increases. For example, at 7 5, the best results are achieved with sampling temperature of 4, while at = 18, the optimal sampling temperature rises to 25. Thus, we plot the best FID and IS for each timestep using different temperatures. Whereas the optimal performance-efficiinnt
trade-off MaskGIT is observed around T 8, ours shows such "sweet spot" around T 18. Up this point, both quality and diversity increase T increases, consistently outperforming MaskGIT. Furthermore, with fewer NFEs (T = 5), the proposed method achieves an FID of 4.84 and IS of 249.9, outperforming the MaskGIT samples with 7 18. This demonstrates that the proposed guidance technique provides more scalable and efficient sampling for MGMs.
Varying fine-tuning epochs (Fig. dd). dd). We found that the proposed fine-tuning is efficiently trained within 10 epochs and that no further fine-tuning is required achieve better results.
Conclusion and Future Work
In this paper, we define generalized guidance for discrete diffusion models, counterpart for guidance in continuous domain 222. To generate guidance, we propose an auxiliary task to apply semantic smoothing in VQ tokens. Experimental results show that the proposed guidance effectively and efficiently improves the generative capabilities of MGMs on class conditional image generation.
Future work. Although the proposed guidance can improve the generative capabilities of MGMs on class conditional image generation, there still room for improvement various aspects: (1) experiments large-scale text conditional generative models (5). (2) generalization to discrete diffusion models [6].. and (3) generalization to various domains such as audio [63], and video [57].
Societal impacts. While our research does not directly touch ethical issues, the rapid growth of generative models raises considerations in Al ethics (e.g., potential misuse in creating deepfakes 34 generating antisocial content [12), vulnerabilities to adversarial attacks [61] 17 30)..
10

---

Acknowledgements
This work was partly supported by Institute of Information & Communications Technology Planning & EvI)))tion(ITTP) grant funded by the Korea governmen((MSIT) (RS-2024-00439020) Developing Sustainable, Real-Time Generative AI for Multimodal Interaction, SW Starlab), Artificial intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT(MSIT, Korea)& Gwangju Metropolitan City, and the National Research Foundation of Korea(NRFF grant funded by the Korea govermmen((MSIT) (No. RS- RS-2023-00240379,
References
[1] J. Austin, D. D. Johnson, J. Ho. D. Tarlow, and R. Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981-17999 2021.
[2] D. Berthelot, C. Raffel, A. Roy, and I. Goodfellow. Understanding and improving interpolation in autoencoders via adversarial regularizer. arXiv preprint arXiv: 1807.07533 2018.
[3] A. Brock, Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1009 1009, 2018.
[4] H. Chang, Zhang, Jiang, Liu, and W. Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 11315- 11325, 2022.
[5] H. Chang, Zhang, J. Barher, A. Maschinot, J. Lezama, Jiang, M.-H. Yang. K. Murphy, W. T. Freeman, M. Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.0071 2023.
[6] O. Chapelle, J. Weston, L. Bottou, and V. Vapnik. Vicinal risk minimization. Advances neural information processing systems, 13, 2000.
Choi, Lee, C. Shin, S. Kim, H. Kim, and S. Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11472-11481, 2022.
[8] Deng, W. Dong. Socher. L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and puttern recognition, pages 248-255. leee, 2009.
[9] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[10] Dhariwal and Nichol. Diffusion models beat gans on image synthesis. Advances Neural Information Processing Systems, 34:8780-8794, 2021.
[11] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[12] R. Gandikota. J. Materzynska, J. Fiotto-Kaufman, and D. Bau. Erasing concepts from diffusion models. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 2426-2436. 2023.
[13] M. Ghazvininejad, O. Levy, Y. Liu, and I.. Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprint (0xXvv/994.0093333 2019.
[14] 1. Goodfellow, J. Pouget Abadic, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications the ACM, 63(11):139 144, 2020.
[15] J. Gu, J. Bradbury, C. Xiong, V. O. Li, and R. Socher. Non-autoregressive neural machine translation. arXiv preprint arXiv:1711.02220 2017.
[16] S. Gu, D. Chen, Bao, F. Wen, B. Zhang, D. Chen, L.. Yuan, and Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 10696-10706, 2022.
[17] G. Han, J. Choi, H. Lee, and J. Kim. Reinforcement learning-base based black- model inversion attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20504 20513, 2023.
11

---

[18] M. Heusel, H. Ramsauer. T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[19] J. Ho and T. Salimans, Classifier-free diffusion guidance. arXiv preprint wxXiv:2207.125940 2022.
[20] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
[21] Ho, C. Saharia, W. Chan, J. Flect, M. Norouzi, and T. Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
[22] S. Hong. G. Lee, Jang, and Kim. Improving sample quality of diffuusion models using self-attention guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7462- 7471, 2023.
[23] N. Houlsby, A. Giurgiu, Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Autariyan, and S. Gelly. Parameter-efficiee transfer learning for nip. In International conference on machine learning. pages 2790-2799, PMLR. 2019.
[24] X. Huang. F. Perez, and M. Volkovs. Improving non -vn-autooeeressive translation models without distilla- tion. In International Conference on Learning Representations, 2021.
[25] Hur. J. Choi, G. Han, D.-J. Lee, and Kim. Expanding expressiveness of diffusion models with limited data via self-distillation based fine- tuning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 0038,, 2024.
[26] Kang, J.-Y. Zhu, 7hang, Park, F.. Shechtman, S. Paris, and Park. Scaling up gans for text-to- image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10124-10134, 2023.
[27] T. Karras, S. Laine, and T. Aila. A style-hased generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[28] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems, 33:12104-123 12114, 2020.
[29] D. Kingma, T. Salimans, B. Poole, and J. Ho. Variational diffusion models. Advances in nearal information processing systems, 34:21696-21777, 2021.
[30] S. Koh, Shon, Lee, G. Hong. and Kim. Disposable transfer learning for selective source task unlcarning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11722-11760, 2023.
[31] T. Kynkäänniemi. Karras, S. Laine, Lehtinen, and Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
[32] D. Lec, Kim, S. Kim, M. Cho, and W.SS Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11523-11553 15332, 2022.
[33] D. Lec, Lec, D. Kim, J. Choi, J. Yoo, and J. Kim. Fix the noise: Disentangling source feature for controllable domain translation. arXiv preprint arXiv:2303. /1545, 2023.
[34] J. Lee, J. Hyung, Jeong, and Choo. Selfswapper: Self-supervised face swapping via shape agnostic masked autoencoder. arXiv preprint wwXiv:2402.007300 2024.
[35] J. Lezama, H. Chang, L. Jiang, and I. Essa. Improved masked image generation with token-critte In European Conference on Computer Vision, pages 70-86. Springer, 2022.
[36] J. Lezama, T. Salimans, L. Jiang, H. Chang. J. Ho, and I. Essa. Discrete predictor-corrector diffusion models for image synthesis. In The Eleventh International Conference Learning Representations, 2022.
[37] C. Lu, Y. Zhou, F. Ban, J. Chen, Li, and J. Zhu, Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[38] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In International conference machine learning, pages 8162-8171. PMLR, 2021.
12

---

[39] M. Oit, M. Auli. D. Grangier, and M. Ranzato. Analyzing uncertainty in neural machine translation. In International Conference Machine Learning, pages 3956-3965. PMLR, 2018.
[40] S. Patil, B. William, and P. von Platen. Amused: An open MUSE model. htthttss: com/ hugginggaace/ppn--te muse
[41] Peebles and S. Xic. Sealable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[42] A. Radford, K. Narasimhan, Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[43] A. Razavi, A. Van den Oord, and O. Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advancey in neural information processing systems, 32, 2019.
[44] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10684-10699 2022.
[45] S. Ryu. Low-rank adaptation for text-to-image diffusion fine-tuning, 2023.
[46] T. Salimans, I. Goodfellow, W. Zaremba, Cheung. Radford, X. Chen. Improved techniques for training gans, Advances in neural information processing systems, 29, 2016.
[47] A. Sauer, K. Schwarz, and A. Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[48] B. Shi, T. Darrell, and X. Wang. Top-down visual attention from analysis by synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2102-2112 12, 2023.
[49] B. Shi. S. Gai, T. Darrell, and X. Wang. Toast: Transfer learning via attention steering. arXiv preprint arXiv:2305.15542 5(7):13, 2023.
[50] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pages 2256 2265. PMLR, 2015.
[51] Song, C. Meng, and Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
[52] Z. Tang, S. Gu, J. Bao, D. Chen, and F. Wen. Improved vector quantized diffusion models. arXiv preprint arXiv:2205. 16007, 2022.
[53] K. Tian, Y. Jiang, Z. Yuan, B. Peng, and L. Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.0222 02905, 2024.
[54] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, E.. Kaiser, and I. Polosukhin. Attention you need, Advances in neural information processing systems, 30, 2017.
[55] E. Xie, L. Yao, H. Shi, Z. Liu, Zhou, Liu, J. Li, and Li. Difflit: Unlocking transferability of large diffuusion models via simple parameter-efficient fine-tuning, In Pruceedings of the IEEE/CVF International Conference on Computer Vision, pages 4230-4239, 2023.
[56] J. Yu, X. Li, J. Y. Koh, H. Zhang, R. Pang. J. Qin, A. Ku, Y. Xu. J. Baldridge, and Y. Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv:21 2021.
[57] L. Yu, Y. Cheng, K. Sohn, J. Lezama, H. Zhang, H. Chang, A. G. Hauptmann, M.-H. Yang, Y. Hao, I. Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10459-10469, 2023.
[58] F. B. Zaken, S. Ravfogel, and Y. Goldberg. Bitfit: Simple parameter-effiient fine-tuning for transformer- based masked language-modeds models. arXiv preprint arXiv:2106. 10199, 2021.
[59] H. Zhang, M. Cisse, Y. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arxiv:
2017.
[60] K. Zhang, R. Wang, X. Tan. J. Guo. Y. Ren. T. Qin, and T.-Y. Liu. A study of syntactic multi- modality in non-autoregressive machine translation. arXiv preprint arXiv:2207.04206, 2022.
13

---

[61] Y. Zhang, R. Jia, H. Pei, W. Wang, B. Li, and D. Song. The secret revealer: Generative attacks against deep neural networks. In Proceedings the IEEE/CVT conference on computer vision and pattern recognition, pages 253-261, 2020.
[62] S. Zhao, H. Ren, A. Yuan, J. Song. N. Goodman, and S. Ermon. Bias and generalization in deep generative models: An empirical study. Advances Neural Information Processing Systems, 31, 2018.
[63] A. Ziv, Gat, G. L. Lan, T. Remez, Kreuk, A. Défossez, J. Copet, G. Synnaeve, and Y. Adi. Masked audio generation using single non autoregressive transformer. arXiv preprint arXiv:2401. 04577, 2024.
14

---

A Detailed Implementation of feature selection module (TOAST)
The architecture of the proposed sampling method consists of two parts: MaskGIT and TOAST 49). Since we have not changed the MaskGIT architecture for plug-and- sampling guidance, we briefly explain the implementation details of TOAST below. The TOAST modules consist of three parts: token selection module, channel selection module, and linear feed-forward networks.
(i) The token selection module selects the task or class-relevant tokens by measuring the similarity with the learnable anchor vector Ee- We generate class conditional anchors &e with simple class conditional MLPs.
(ii The channel selection is applied with learnable linear transformation matrix P. Then, the output of the token and channel selection module is calculated via 2; P simm Zi, Ec), where 2; denotes the i-th input token.
(iii) After the feature selection, the output processed with L layer MLP layers, where is equal to the number of Transformer's layers. The output of the I-th layer of MLP blocks is added to the value matrix of the attention block in 1))t Transformer layer (top-down attention steering). Following the previous work [49], we add variational loss to regularize the top-down feedback path. A more detailed process and theoretical background can be found in previous works on top-down attention steering [48] [9].
B Effect of High Sample Temperature and Guidance Scale
We show the effect of high sampling temperature (r) and high guidance scale in Fig. The high sampling temperature (i.e., strong guidance) often leads to undesirably highlighted fine-scale details or saturated colors, similar to the large scale of CFG [19] High temperatures often lead to the collapse of the overall structure of generated samples.
(1)==1,-3) I-
(b) 5=5, T=25
s-1, 5-50
Figure 7: Sampled images ImageNet 256x256 class conditional generation using (a) default
config, (b) large guidance scale (s = 5), and (c) high sampling temperature (r = 50).
15

---

C
More Visual Results
Figure Sampled images on ImageNet x56x256 class conditional generation using selected classes (130: Flamingo, 980: Volcano, 820: Steam locomotive, 388: Giant panda, and 454: Bookshop). left: LDM [44] + CFG (s=1.5, NFE=250x2), 2), middle: MaskGIT (NFE=18), right: Ours (s=1.0, NFE= 18x2).
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claim in the abstract and introduction is dealt with in the background, method, and experiment sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We deal with this in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
We provide a correct proof for our theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide implementation details in the paper for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer:
[No]
Justification: We do not include code for our experiments. Nevertheless, the dataset and baseline code for our implementation and evaluation are clearly denoted in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
public/guides http:////ooooeuueeessssssooooona
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https wetgguddddeeeuuubbueeee ssinoolicyy for more details.
//nips.co
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide results by varying the hyperparameters and provide insights for selecting those values. We use open-source evaluation code, which is denoted in the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The sampling process and evaluation are statistical, but we not report the етror bars since with the sufficiently large number of samples (=50k), the impact of stochasticity is minimized.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the training environment, required training epochs, and required sampling steps in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: bectpa://neuripeee httpp//www.p.....ppp...cccccccaa
Answer: [Yes]
Justification:
We wrote paper in compliance with the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We denote the negative societal impacts that the proposed method may have.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not include such large pre-trained model in the paper. We will release only a small module for fine-tuning, which does not have a high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We use open source implementation for our baseline and denote in the paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We have no results related to the human evaluations.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: We have no results related the human evaluations.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "Y6GwHYE6DF", "reviewer": "Reviewer_qSSj", "summary": "The paper focuses on enhancing Masked Generative Models (MGMs) for image synthesis.\nThe authors identified several reasons for the underperformance of MGMs: 1) lack of sequential dependencies, 2) multi-modality problem, 3) compounding decoding errors, 4) limitations of low-temperature sampling and 5) ineffective guidance techniques.\nTo address these issues, the authors present novel techniques and methodologies to improve their performance.\n\nThe main contributions of the paper are as follows: \n1)\tGeneralized Guidance Formulation.\n2)\tSelf-Guidance Sampling Method.\n3)\tParameter-Efficient Fine-Tuning.", "questions": "Most hyperparameters in Figure 5 show a trade-off relationship between FID and IS scores. However, increasing the number of sampling time steps improves performance in both metrics. So, could you explain why sampling time steps were set to 12 and 18 in each experiment? Even though other methods are known to use hundreds of sampling time steps, 50 seems quite small in comparison.", "limitations": "The authors did not fully address the limitations of the paper but did mention the need for attention to AI ethics due to the rapid growth of generative models, touching on the societal impact. To improve the paper, we suggest the following:\n\n1)\tAlthough ImageNet is a comprehensive dataset, it is insufficient to prove generalizability. Therefore, conducting experiments on other datasets used in existing MGMs, such as MSCOCO or Conceptual Captions, will make the paper more complete.\n2)\tInclude more visual results. To clarify the reasons for selecting specific models or hyperparameters used in the paper, it is much more effective to show them through visual results.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality\nThe paper presents a highly original approach by extending guidance methods from continuous diffusion models to MGMs. In particular, the use of self-guidance sampling method and an auxiliary task for semantic smoothing in the VQ token space are novel approach that address specific challenges in MGMs.\n\n- Quality\nThe paper demonstrates a high level of technical quality. The authors have thoroughly explained why their methods can enhance performance and have supported their claims with extensive experiments and ablation studies on various variables. \n\n- Clarity\nThe paper is clearly written and well-structured. The authors provide detailed context relative to prior work, facilitating a clear understanding of the background and contributions of the proposed approach. The detailed explanations of the equations and methodologies, accompanied by ample examples, significantly enhance the overall readability and accessibility of the paper. \n\n- Significance\nThe contributions of the paper are significant for the advancement of the MGMs field. By addressing key challenges in MGMs, the proposed methods offer substantial improvements in the quality and diversity of generated samples.", "weaknesses": "Particularly due to inadequate visual results, some aspects are difficult to comprehend. Section 3.3 asserts that TOAST is the most suitable model for the study's task, but lacks sufficient explanation on why TOAST is the only suitable model. It would be more convincing if the paper showed some visual results from using other models, which would provide sufficient justification.\n\nExplanations for the choice of hyperparameters are insufficient. While the paper demonstrates trade-off relationships in hyperparameter settings across some datasets in removal experiments, the sampling time steps do not exhibit such trade-offs. It is crucial to provide adequate explanations for why these settings differ from those maintained for the ImageNet 512×512 dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive review and for identifying the ambiguous aspects of our experimental designs. We address your concerns and questions below. All visual results can be found in the submitted PDF. Please zoom in on the figure of the submitted PDF for the best view.\n\n**Include more visual results for clarification.**\n\nTo clearly show the improvement with the proposed guidance compared to MaskGIT, we add more visual results in Fig. 6 of the submitted PDF. Please refer to Global Response for a detailed description. We also visualize the impact of the different sampling hyperparameters in Fig. 11 of the submitted PDF. The high guidance scale may deteriorate the sample quality by overly enhancing fine details, and high sampling temperature often leads the overall structure to collapse. \n\n**Using another model for the auxiliary task.**\n\nAlthough we argued that TOAST could be efficiently and naturally adopted to learn auxiliary tasks, other PEFT methods can also be adapted to generate guidance similarly. Before validating, we note that the benefits of using TOAST are twofold:\n\n1. MGMs exhibit training bias of only predicting the [MASK] tokens while treating other input tokens as ground truth. As a result, MGMs tend not to correct errors in the input tokens. We resolve this by simply masking all tokens of the 2nd stage of TOAST prediction.\n\n2. Since the output of the 1st stage of TOAST is directly reused to sample $p_\\theta(\\hat x_{0,t})$, it brings more efficiency for sampling (see Fig. 2 of the main manuscript).\n\nWe experiment with the prompt tuning-based approach with the proposed auxiliary loss. The prompt tuning shows reasonable performance for transfer learning of MaskGIT [1]. The prompt token length is set to 128 following [1], and the batch size is reduced to 128 due to the memory capacity. We first forward the $z_t$ with prompt tokens to obtain ${\\mathcal{H}}(z_t)$ in eq.7. Then, we forward the MaskGIT with the ${\\\\mathcal{H}}(z_t)$ to calculate the ${\\\\mathcal{L}}_{aux}$ in eq.7. To train the model, we directly input the ${\\\\mathcal{H}}(z_t)$ after the embedding layer of MaskGIT. \n\nWe set the guidance scale to 0.2 since we manually found the result guidance was too strong. The result in the table below and Fig. 8 of the submitted PDF show that the improvement is marginal compared to ours. We suspect that the naïve prompt tuning does not suitably deal with the error correction due to the bias of MGMs (see loss curve in Fig. 8). Furthermore, it requires 3 models forward for each sampling step.\nWith the above discussion and experiments, as well as the discussion in the main manuscript, we verify that the TOAST is a more suitable approach compared to the other PEFT methods in terms of performance and efficiency. We expect that a more meticulously designed model for the proposed guidance can further improve efficiency and performance and leave it for future work.\n\n|| NFEs | FID$\\\\downarrow$|IS$\\uparrow$|\n|----------|----------|----------|----------|\n|MaskGIT|18|6.56|203.6|\n|Prompt tuning w/ ${\\\\mathcal{L}}_{aux}$|18$\\\\times$3|5.78|209.8|\n|Ours (TOAST)|18$\\\\times$2|$\\\\textbf{3.22}$|$\\\\textbf{263.9}$|\n\n**Explanations for the choice of hyperparameter (sampling step).**\n\nIn our preliminary research on the impact of sampling steps ($T$), we mainly investigated $T$ around $\\sim$18, which is commonly adopted by prior works. We further explore the larger $T$ in Fig. 10 of the submitted PDF using $T$=24 and 36 with sampling temperature $\\\\tau$=35 and 60, respectively. The results indicate that using $T$, larger than 18, does not ensure a performance increase, as the metrics either saturate or deteriorate despite higher computational costs.\n\nSimilar to findings in MaskGIT, where the optimal performance-efficiency trade-off is observed around $T$=8, our method shows a \"sweet spot\" around $T$=18. Up to this point, both quality and diversity increase as sampling timesteps and computational costs increase while outperforming MaskGIT with similar NFEs. This demonstrates that the proposed method shows more scalable and efficient results compared to the MaskGIT. We also found a similar phenomenon for experiments on ImageNet 512x512, and the optimal timestep is around 12~18 steps. In this context, we opt for the sampling timestep of 12 and 18 for the results. We will add the above experimental results and our analysis in the final manuscript to give a thorough insight into choosing the sampling steps.\n\n**Generalization to more complex datasets.**\n\nThank you for your valuable comments. Since the suggested datasets are primarily utilized for text-to-image (T2I) generation tasks, pretrained VQGAN and MaskGIT are not publicly available.\n\nInstead, the T2I MGMs, MUSE [2], can be utilized to demonstrate the generalization. Though the main concern of the paper is to improve the generative capabilities of MGMs in class conditional image generation, the proposed guidance and utilized fine-tuning architecture can be extended to T2I generation. Given that the only difference is the replacement of the class condition with a text condition, we can simply substitute the class conditional module in TOAST with a text conditional module while maintaining other fine-tuning and sampling strategies. (Please refer to the response for Lc5P for a detailed TOAST implementation)\n\nHowever, within the limited time frame, it is challenging for us to find reliable code (since the official MUSE is not publicly available), preprocess the large datasets, and ablate the generalizability of the proposed guidance. Therefore, we leave the generalization performance on T2I as future work.\n\n---\n\n[1] Sohn, Kihyuk, et al. \"Visual prompt tuning for generative transfer learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[2] Chang, Huiwen, et al. \"Muse: Text-to-image generation via masked generative transformers.\" arXiv preprint arXiv:2301.00704 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t2NgFdHQEM", "reviewer": "Reviewer_Lc5P", "summary": "This paper explores the use of masked generative models for image generation. While these methods are typically efficient, they sometimes fall short in terms of quality compared to diffusion models. To reduce this gap, this paper proposes a self-guidance algorithm, which is further improved with semantic smoothing. These ideas are shown to be effective for improved image generation using MGMs. The method is compared to a wide variety of baselines on class-conditioned generation, on common metrics, and against a set of baselines.", "questions": "- How would this method generalize to text-to-image methods or image-to-image translation problems?\n- How tied is this method to VQGAN? Could other encoders be used? If so, what would be their impact?", "limitations": "The paper is fine in this regard. It could, however, provide more details on how the limitations could be addressed in the future.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- This paper tackles an important problem in image generation, which is the use of Masked Generative Models for efficient image generations. This family of models have shown to be more efficient than diffusion models, but have so far struggled to catch-up with the quality they can achieve. This paper improves the quality of these models, to make them competitive in quality while preserving their benefits in terms of efficiency. \n- At the core fo the method lies a self-guidance method, which I believe is a sound and well motivated idea, which can be potentially included in generative models for other applications.\n- The ideas in the paper are simple in a good way, and they are not particularly tied to a specific application. I believe they could be generalizable to many other problems. \n- I believe the related work is well analysed, as far as I am familiar with previous work on this topic.\n- Evaluation is extensive, many different hyperparameters are evaluated with respect to their impact on generation quality.", "weaknesses": "- The paper is at times hard to follow, lacking clarity in some sections (particularly 5). Images and figures are sometimes small and hard to see.\n- I do not believe implementation details are enough for reproducibility. Code will not be provided, which will make this paper hard to reproduce and it will reduce its potential impact and room for future comparisons. \n- Given that a particular benefit of the method of this paper is its computational efficiency in comparison to diffusion-based approaches, I believe a detailed analysis of these should be included. \n- I believe the paper would benefit from showing results on text-to-image generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We address your concerns below. All visual results can be found in the submitted PDF. Please zoom in on the figure in PDF for the best view.\n\n**Clarifications of implementation details, figures, and code release.**\n\nThank you for your suggestion to make our research clearer and more reliable. We will clarify and magnify the figures in the final manuscript. We will add more visual results for a clear comparison of the proposed method to the base MaskGIT model, as illustrated in Fig. 6 of the submitted PDF. Please refer to the Global Response for a detailed description.\n\nThe proposed method consists of two parts: MaskGIT and TOAST. Since we have not changed the MaskGIT architecture, we briefly explain the implementation details of TOAST below.\nThe TOAST modules consist of three parts: token selection module, channel selection module, and linear feed-forward networks. \n\n- (i) The token selection module selects the task or class-relevant tokens by measuring the similarity with the learnable anchor vector $\\\\xi_c$. We generate class conditional anchor $\\\\xi_c$ with simple class conditional MLPs. \n\n- (ii) The channel selection is applied with learnable linear transformation matrix P. Then, the output of the token & channel selection module is calculated via $z_i = P \\\\cdot sim(z_i,\\xi_c) \\\\cdot z_i $, where $z_i$ denotes the $i$th input token. \n\n- (iii) After the feature selection, the output is processed with $L$ layer MLP layers, where $L$ is equal to the number of Transformer’s layers. The output of the $l$th layer of MLP blocks is added to the value matrix of the attention block in $(L-l)$th Transformer layer (top-down attention steering). Following the previous work [1], we add variational loss to regularize the top-down feedback path. A more detailed process and theoretical background can be found in [1].\n\nMost of the implementation can be found in the official TOAST repository. Since our work is also based on the public repository, as mentioned in the main manuscript, we believe that the proposed guidance can be easily implemented and applied in various code bases. We will add the implementation details in the final manuscript for understandability and reproducibility. We also plan to release the source code as soon as we complete the documentation.\n\n**Measuring the computational efficiency.**\n\nWe measure the computational efficiency by calculating the sampling time on A6000, where the batch size is set to 50. In Fig. 9 of the submitted PDF, we plot sampling time with FID/IS values compared to diffusion-based methods and various MGMs. Note that we exclude Token-Critic and DPC since the code is not publicly available. Though each model has slightly different implementation details for the architecture, an order of magnitude smaller NFEs of MGMs (see Table 1 of the main manuscript) lead to a significantly efficient sampling process compared to diffusion-based methods. With the proposed self-guidance, the proposed method surpasses the diffusion-based approaches and MaskGIT in both sampling efficiency and performance.\n\n**Generalization to text-to-image method or image-to-image translation.**\n\nThe proposed method can be generally adopted for various tasks of MGMs. We briefly explain how the proposed method can be extended to various tasks. \n\n- Text-to-Image generation (T2I): Because the proposed guidance can be generally defined with text condition, the proposed method can be easily extended to the T2I generation by replacing the class conditional module in TOAST with the text conditional module. Except for the architecture, the training and sampling strategies illustrated in Fig. 2 of the main manuscript can be adopted without modification for text embedding.\n\n- Image-to-Image generation (I2I): For Image-to-Image generation, such as style transfer, the proposed method can be adopted to further enhance the sample quality. For instance, MaskSketch [2] utilizes pretrained MaskGIT to generate images from the sketch input.  The proposed method fine-tuned on the target dataset can be additionally attached for the enhanced sample quality.\n\n**Choice of the quantizer.**\n\nSimilar to continuous diffusion, the (discrete) latent space brings several strengths, such as semantic richness and computational efficiency. As a result, modern MGMs and various discrete domain generative image models mostly utilize vector-quantize-based encoder-decoders such as VQGAN and VQVAE. In this regime, our work is built upon the MGMs, which operate on VQ space. The latent space of various VQ-based encoders shows similar characteristics, such as rich semantic information, and lacks continuous semantic structure in contrast to pixel space. Furthermore, the proposed guidance and the auxiliary task can be generally defined in any discrete space. As a result, we expect that the proposed guidance will not be tied to the specific quantizer architecture and that the performance improvement will be similar.\n\n---\n[1] Shi, Baifeng, et al. \"TOAST: Transfer Learning via Top-Down Attention Steering.\"\n \n[2] Bashkirova, Dina, et al. \"Masksketch: Unpaired structure-guided masked image generation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A4woF3PvBr", "reviewer": "Reviewer_cCTZ", "summary": "This work proposes a self-guidance method for masked generative models to improve the diversity and quality of class conditional image generation. The main challenge is to design a semantically meaningful smoothing for the discrete VQ token space such that coarse scale information can be extracted. The authors propose an auxiliary task – error token correction to minimize fine-scale details and utilize TOAST for efficient model fine-tuning.", "questions": "- What is $\\bf{m}_i$ in equation (7)? Could you provide more intuitions or visualizations of the claims in line 192- 195. Why does this auxiliary task naturally minimizes fine-scale details? \n- What is the range of the experimented temperature values in Fig. 3? \nFrom Fig.3, it seems that the proposed sampling method has very different performance based on the choice of temperature. The authors also mention that they choose different temperatures based on the resolution and sampling steps. I wonder why is the proposed method so sensitive to temperature? \n- Have the authors tested the model on the 128x128 resolution ImageNet benchmark? What does the FID/IS curve look like? \n- What happens with strongly guided samples? For instance, for classifier-free guidance in diffusion models, strongly guided samples exhibit saturated colors. I wonder if there is any artifact related to this proposed method when the choice of guidance scale or sampling temperature is high?", "limitations": "The authors already mention some directions for future work, such as demonstrating the guidance on text-conditioned image generation.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The research problem of introducing self-guidance in discrete space is well-motivated. The background section is well-written, and the discussion of related work is thorough. I find the analogy to guidance of continuous diffusion model very helpful. \n- The figures are well-made, such as the qualitative visualization is Fig.1. \n- The quantitative metrics from the experiments also demonstrate the effectiveness of the method. The ablation studies on different hyperparameters are informative.", "weaknesses": "- From the qualitative visualizations alone (Fig.4), the proposed method does not seem have too different outputs compared with MaskGIT (the middle column). I wonder if the guidance benefit is stronger on other resolutions’ results, like 128 x 128 or 64 x 64. \n- Maybe some figures could be added to accompany section 3.2 to better illustrate the process of the auxiliary task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review. We address your concerns and questions below. All visual results can be found in the submitted PDF. Please zoom in on the figure of the submitted PDF for the best view.\n\n$\\\\textbf{Clarified visual comparison.}$\n\nIn Fig. 6 of the submitted PDF, we directly compare the sampled images using MaskGIT and the proposed method with the same random seed on ImageNet 256 and 512 resolutions. With the proposed guidance, fine-scale details are enhanced, generating higher-quality images. Please refer to the Global Response for a detailed description.\n\n$\\\\textbf{Impact of guidance in lower resolution.}$\n\nSince the proposed guidance utilizes semantic smoothing, the effectiveness of the guidance can be affected by the resolution of the dataset or input token length. We first note that the proposed guidance is built upon the pre-trained vector quantizer and MGMs. However, we cannot find publicly available vector quantizers and MGMs trained on ImageNet 128 resolution. We tried to train both VQGAN and MaskGIT, which have to be trained sequentially due to the dependency. However, within the limited time frame and computational budget, we are unable to generate reliable results from MaskGIT, generating output with very low visual quality, which shows an inception score of around 20.\n\nThough the VQGAN and MaskGIT pretraining is not converged, we measure the performance improvement by attaching the proposed guidance to the MaskGIT with ImageNet 128 resolution. We measure the IS (larger is better) and FID (lower is better) with a sampling temperature of 2.0. The IS increases from 21.2 to 29.5, and FID decreases from 44.0 to 33.6. Although the hyperparameter search is limited, the FID improvement is significant (23%) compared to the improvement in 256 (50%) and 512 (36%) resolutions. We expect that the performance improvement at lower resolutions will be similar if a well-pretrained quantizer and MaskGIT are utilized and sampling hyperparameters are suitable.\n\n$\\\\textbf{More intuition for auxiliary task and visualization.}$\n\nThank you for your recommendation, which helps us better illustrate the proposed auxiliary task. \nWe thought it would be beneficial for all reviewers to share deeper insights into the underlying intuition and the visual results, so we provided the review in a global response. Thank you once again for your valuable recommendation!\n\n$\\\\textbf{Choice of the sampling temperatures.}$\n\nIn Fig. 5 of the main manuscript, utilized sampling temperatures are listed below. \n\n- 256: [3, 4, 5.5, 15, 20, 23, 25, 30, 40, 60, 80]\n- 512: [6, 10, 15, 20, 25, 30, 35, 40, 45, 50, 70, 90]\n\nThe sampling process of MGMs is known to be sensitive to the choice of various sampling hyperparameters [1]. Sampling temperature plays a crucial role in resolving the multi-modality problem, as explained in section 2. Thus, moderate sampling temperature can increase the sample quality and diversity, simultaneously. After some point, when the temperature increases, diversity increases while quality deteriorates. This is because the large sampling temperature gives strong randomness for re-masking of MGMs ($p(m_{t−1}|m_t, \\\\hat{x}_{0,t}$) in line 99), potentially masking relatively accurate tokens while leaving unrealistic tokens unmasked. As a result, the sampling of MGMs is sensitive to the sampling hyperparameters. We believe that our extensive ablation experiment can provide valuable insights for the community.\n\n$\\\\textbf{Impact of the strong guidance and high sampling temperature.}$\n\nIn Fig. 11 of the submitted PDF, we compare the samples with our default config, with strong guidance (s=5) and high sampling temperature ($\\\\tau$=50). We found that the strong guidance often leads to undesirably highlighted fine-scale details (left column in Fig. 11 b) or saturated colors (right column in Fig. 11 b), similar to the large scale of CFG. High temperatures often lead to the collapse of the overall structure.\nAs an extension of Fig. 5 a of the main manuscript, we further plot the FID and IS curves using larger guidance scales in Fig. 6 d. Using a large guidance scale decreases fidelity and diversity due to the phenomenon we mentioned above. We will add the analysis and figure in the final manuscript for a thorough investigation and to give further insight into the proposed guidance.\n\n$\\\\textbf{Further clarification.}$\n\nThe $\\\\mathbf{m}\\_{i}$ in eq. 7 denotes whether the $i$th input token $z\\_t$ is masked or not. $\\\\mathbf{m}\\_i=0$ if the $i$th input token is masked and $\\\\mathbf{m}\\_i=1$ if not. \nSince the masked index of $z\\_t$ and $x\\_t$ are identical,  the $\\\\mathbf{m}\\_{i}$ in eq. 7 equals to the $\\\\mathbf{m}\\_i$ in eq. 1. We will clarify this in the final manuscript.\n\n---\n[1] Chang, Huiwen, et al. \"Muse: Text-to-image generation via masked generative transformers.\" arXiv preprint arXiv:2301.00704 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and Area Chairs,

We thank the reviewers for their constructive feedback. We are glad to take various helpful reviewer comments to clarify and complete our work. Reviewers agreed on the originality, motivation, soundness, and significance of the paper. At the same time, reviewers are concerned about the broader applicability of the proposed guidance, the need for enhanced clarity in our visual representations and implementation descriptions, and the effect of a more diverse set of hyperparameters.

Here, we first deal with the most common concerns and questions from reviewers. Throughout all responses, we refer to the originally submitted paper as a $\textbf{main manuscript}$ and the submitted PDF file in the rebuttal period as a $\textbf{submitted PDF}$. Due to the page limit, figures and captions in the submitted PDF could be small. Please zoom in on the figures for the best view. 

**More clarified visual results. (qSSj, Lc5P, cCTZ)**
 
All reviewers suggested providing more visual results to better clarify, demonstrate, and strengthen the effectiveness of the proposed guidance. In response, we have sampled visual results from ImageNet at 512x512 and 256x256 resolutions, which are shown in Fig. 6 of the submitted PDF. We generate paired samples from MaskGIT and the proposed method by fixing all random seeds. Qualitative results show that the proposed guidance enhances fine details, generating samples with high fidelity. Detailed facial attributes or local structures are generated accurately with the proposed guidance, whereas MaskGIT fails to generate plausible images.

**More intuition for the auxiliary tasks and visualization of the impact. (cCTZ)**

Reviewer cCTZ has raised a concern that further intuition or visualization for the auxiliary task is required to clear up the claim. We are glad to take helpful comments to clarify our work. Though it is not a common concern, we respond here to share more intuition and insights about the proposed guidance with the reviewers.
To learn the auxiliary task, we randomly replace a proportion of input tokens as error tokens, which act as semantic outliers of input tokens. To capture the semantic outlier, we argue that the model implicitly learns to smooth the vicinities of the data $z_t$ by leveraging coarse information from the surrounding context while minimizing the fine-scale details. For a simple example of detecting numerical outliers in a 1-dimensional signal, a straightforward way to correct outliers, such as impulse noise, is by measuring the distance to the local mean and correcting by smoothing the signal, such as low-pass filters. Similarly, to correct the unknown error tokens, which are semantic outliers, the model implicitly learns to smooth input $z_t$ to deal with unknown error tokens.

To demonstrate the above process, we visualize the effect of guidance using guidance scales 5 and -5. Given that the output guidance logit is semantically smoothed output, by its definition, the positive guidance means that it guides the sampling process toward enhancing fine details. Contrarily, the negative guidance scale means that it guides the sampling process toward reducing the fine details. More specifically, with the auxiliary task, the guidance logit includes semantically smoothed information. Thus, samples with a negative guidance scale result in reduced semantic information, such as fine details or local patterns. We visualize the effect of the positive and negative guidance scale in Fig. 7 of the submitted PDF. 
We mask 80% of the input VQ tokens and visualize the one-step prediction outputs (b) without guidance, (c) with a positive guidance scale, and (d) with a negative guidance scale. The visualization shows that the positive guidance enhances the semantic fine details, while the negative guidance reduces the semantically fine details, such as patterns in feather and facial attributes. This demonstrates that the output logit trained with the auxiliary task can effectively capture the semantically smoothed logit, and the result guidance helps to improve the sample quality by enhancing fine details.

**Numerical error (typo) in the paper**

We apologize for the error in the main manuscript. In Table 2 of the main manuscript, we found that the FID and IS values for our method should be corrected from 3.04 and 240.8 to 3.22 and 263.9, respectively, following the values in the main table (Table 1). We note that it does not impact the discussion and conclusions of our study since it still outperforms other ablation results. We appreciate your understanding!

Meta-review: This paper received unanimous recommendations for acceptance from all three reviewers. In particular, the reviewers praised the novelty of using MGM for more efficient image generation. However, as they noted, the performance on larger-scale datasets or higher-resolution images remains unknown. Therefore, AC recommends acceptance as a poster presentation.
Decision: Accept (poster)