Title: Unlocking the Capabilities of Masked Generative Models for Image Synthesis via Self-Guidance
Venue: NeurIPS 2024
Accepted: yes
Keywords: Image synthesis

discrete diffusion models

masked generative models

sampling guidance

parameter- efficient fine-tuning
Abstract: Masked generative models (MGMs) have shown impressive generative ability while providing an order of magnitude efficient sampling steps compared to continuous diffusion models. However, MGMs still underperform in image synthesis compared to recent well-developed continuous diffusion models with similar size in terms of quality and diversity of generated samples. A key factor in the performance of continuous diffusion models stems from the guidance methods, which enhance the sample quality at the expense of diversity. In this paper, we extend these guidance methods to generalized guidance formulation for MGMs and propose a self-guidance sampling method, which leads to better generation quality. The proposed approach leverages an auxiliary task for semantic smoothing in vector-quantized token space, analogous to the Gaussian blur in continuous pixel space. Equipped with the parameter-efficient fine-tuning method and high-temperature sampling, MGMs with the proposed self-guidance achieve a superior quality-diversity trade-off, outperforming existing sampling methods in MGMs with more efficient training and sampling costs. Extensive experiments with the various sampling hyperparameters confirm the effectiveness of the proposed self-guidance.

Unlocking the Capabilities of Masked Generative Models for Image Synthesis via Self-Guidance

Abstract

Masked generative models (MGMs) have shown impressive generative ability while providing an order of magnitude efficient sampling steps compared to continuous diffusion models. However, MGMs still underperform in image synthesis compared to recent well-developed continuous diffusion models with similar size in terms of quality and diversity of generated samples. A key factor in the performance of continuous diffusion models stems from the guidance methods, which enhance the sample quality at the expense of diversity. In this paper, we extend these guidance methods to generalized guidance formulation for MGMs and propose a self-guidance sampling method, which leads to better generation quality. The proposed approach leverages an auxiliary task for semantic smoothing in vectorquantized token space, analogous to the Gaussian blur in continuous pixel space. Equipped with the parameter-efficient fine-tuning method and high-temperature sampling, MGMs with the proposed self-guidance achieve a superior qualitydiversity trade-off, outperforming existing sampling methods in MGMs with more efficient training and sampling costs. Extensive experiments with the various sampling hyperparameters confirm the effectiveness of the proposed self-guidance.

Keywords

Image synthesis, discrete diffusion models, masked generative models, sampling guidance, parameterefficient fine-tuning

1 Introduction

With the advent of generative adversarial networks (GANs) [14], generative models have attracted significant attention for their powerful ability to synthesize highly realistic images. However, due to the limited mode coverage and training instability, recently, likelihood-based models such as diffusion models [20] have been actively researched. Diffusion models have shown promising results for their diverse and high-quality samples, surpassing GANs on class conditional image synthesis [10].

A key factor in the success of diffusion models stems from the various guidance techniques, which enhance the fidelity of the generated image at the expense of diversity. The guidance is usually conducted throughout the sampling process of diffusion models, driving it toward enhancing specific information such as class [10, 19], text [44, 52], or details in the image [22]. From the practical perspective, however, diffusion models suffer from sampling inefficiency, requiring hundreds of sampling steps to generate high-quality images. Moreover, guidance techniques further decrease the sampling efficiency, as they typically require twice as many model inferences as the original process. For instance, ADM [10] with guidance requires ∼ 500 model inferences.

On the other side, masked generative models (MGMs) have shown superior trade-offs between sampling quality and speed compared to (continuous) diffusion models [4, 35, 36]. MGMs use an absorbing state ( [MASK] ) diffusion process [1] and aim to generate discrete tokens by predicting the masked region, similar to BERT [9]. Specifically, they use the Markov transition matrix to model the diffusion process and sample the tokens with the categorical distribution. Recently, vector-quantized (VQ) image token-based MGMs with (non-)autoregressive transformer [11, 4] have demonstrated an efficient sampling process, providing an order of magnitude fewer sampling steps (e.g., ∼ 18 steps) than continuous diffusion models [4].

While continuous diffusion models utilize guidance sampling to enhance generation quality, MGMs typically utilize sampling with low-temperature Gumbel noise in categorical distribution to enhance the quality [4, 35]. However, low-temperature sampling may impose multi-modality problem [15, 39, 60], where the non-autoregressive sampling process fails to generate plausible outputs due to the lack of sequential dependencies. As a result, the upper bound of the sample quality generated by MGMs was relatively limited. Several approaches have been suggested to improve the sampling quality of MGMs, such as discrete predictor-corrector-based methods [35, 36] that train a second transformer to discern the unrealistic tokens. However, despite the improved sampling, MGMs still underperform in terms of FID scores [18] compared to state-of-the-art continuous diffusion models such as LDM [44] on ImageNet benchmark [8], even when the model sizes are similar.

To overcome such limitations and enhance the sample quality of MGMs, we propose discrete self-guidance sampling (Fig. 1). Self-guidance [22] in continuous diffusion models improves the generation quality by enhancing the fine-grained detail of the sample by guiding the diffusion process with coarse-grained information within intermediate diffusion steps, similar to that of classifierfree guidance, which utilizes unconditional generation to enhance the quality of class conditional generation. In continuous space, the coarse-grained information can be easily obtained with spatial smoothing, like Gaussian blur. To apply self-guidance in MGMs, we first define the general guidance formulation for the discrete diffusion models and introduce self-guidance in MGMs. However, while it is simple to define the coarse-grained information in continuous space, the VQ token space [43, 11] in MGMs is absent of continuous semantic structure and cannot apply such a simple strategy, e.g., blur [2]. Furthermore, we cannot directly define coarse-grained, i.e., semantically smoothed outputs in the VQ token space. Therefore, we introduce an auxiliary task specifically designed for semantic smoothing of the VQ token space, which enables the network to selectively remove details such as local patterns while preserving overall information in VQ token spaces. Consequently, by guiding the MGMs with semantically smoothed information, we can enhance the quality of MGMs (Fig. 2) even with high temperatures, therefore achieving both high quality and diversity compared to previous MGMs. For efficient implementation of the discrete self-guidance, we introduce a plug-and-play module with parameter-efficient fine-tuning [49] to pre-trained MGMs, leveraging the generative prior in the MGMs while enabling efficient training with a few parameters and epochs.

One step

prediction

Input image

Masked image

(

m.r.

=90%)

Experimental results demonstrate that the proposed guidance effectively improves the sample quality with only 10 epochs of fine-tuning. Notably, combined with a high sampling temperature, the proposed guidance not only improves the quality of generated samples but also keeps diversity high, showing a superior quality-diversity trade-off compared to other improved sampling techniques for MGMs and other generative families with similar model sizes.

2 Background

2.1 Masked Generative Models

Discrete diffusion models [1] aim to generate categorical data x 0 ∈ R N with a length of N and K categories. The forward Markov process q ( x t +1 | x t ) gradually corrupts data x 0 , x 1 , ..., x T until the marginal distribution of x T ∼ p ( x T ) becomes stationary. Then starting from the x T , the learned reverse Markov process p θ ( x t -1 | x t ) gradually recovers the corrupted data to generate x 0 .

Masked Generative Models (MGMs) are a family of discrete diffusion models that use an absorbing state diffusion process [1], in which they gradually mask the input tokens by replacing them with a mask token ( [MASK] ) and learn to predict the masked region. Recently, vector-quantized (VQ) image token-based MGMs [4, 35, 36] have achieved a superior trade-off between sampling time and quality for image synthesis compared to Gaussian (continuous) diffusion models [10, 44] through the non-autoregressive parallel decoding process. MGMs are trained to predict the masked region similar to BERT [9] in natural language processing (NLP), but with various masking ratios γ t ∈ (0 , 1] , where γ is pre-defined mask scheduling function that masks n = [ γ ( t/T ) · N ] tokens from N total tokens. Without loss of generality, we assume the unmasked region of the mask m t is 1, and the masked region is 0. Then, given external condition c (e.g., class or text) and masked input x t = x 0 ⊙ m t , where the mask m t is randomly sampled according to masking ratio γ t , MGMs are trained to predict clean data x 0 (or equivalently, to predict the masked regions) by the objective function

Here, we denote the i -th token of x 0 as x i 0 .

After the training, to sample the image x 0 , MGMs typically use an iterative prediction-masking procedure starting from the blank canvas x T (i.e., all tokens are masked). Given (partially) masked image token x t sampled from the previous step, MGMs first predict all tokens ˆ x 0 ,t ∼ p θ (ˆ x 0 ,t | x t ) simultaneously, where ˆ x 0 ,t denotes the prediction of x 0 at timestep t and we omit c for simplification. Then ˆ x 0 ,t is masked according to masking ratio γ t -1 to obtain x t -1 . MGMsusually sample m t -1 ∼ p ( m t -1 | m t , ˆ x 0 ,t ) which is equivalent to sample x t -1 since x t -1 = m t -1 ⊙ ˆ x 0 ,t . Randomly sampling the m t -1 can be one choice; however, the prediction ˆ x 0 ,t may have numerous errors, thus randomly selecting the mask can produce sub-optimal results by potentially masking relatively

accurate tokens while leaving unrealistic tokens unmasked. To overcome this, various research adopts improved sampling methods such as utilizing the output confidence of the ˆ x 0 ,t since confident tokens tend to be more accurate [4] or train an external corrector to classify the realistic tokens [35, 36].

However, these improved sampling may impose a problem, named multi-modality problem that is a well-known problem in non-autoregressive parallel sampling [15, 39, 60]. Given the input such as x t and class condition c , the model can have multiple plausible outputs, which brings challenges to the non-autoregressive model as they generate each token independently. In an extreme case where the input token is all masked, and the model predicts output only using a given external condition such as class, each token can predict easy token more confidently and correctly, such as a background in every token. As a result, correctness-based sampling may result in images only filled with background images. To resolve this problem, various MGMs adopt additional randomness to sample m t , such as sampling with temperature [4, 35, 36]. Let the l t measure the realism of sampled tokens ˆ x 0 ,t , such as the confidence scores of sampled tokens in MaskGIT [4]. Then MGMs sample m t -1 by selecting topk elements of ˜ l t = l t + τ · ( t/T ) n according to γ t , where n denotes the sampling noise such as i.i.d. Gumbel noise and τ is temperature scale that is annealed according to the timesteps. Generally, high-temperature sampling results in more diverse samples while degrading the sample quality.

2.2 Sampling Guidance

Iterative sampling processes of diffusion models are often guided by external networks or themselves. Therein, salient information-based guidance has been actively explored in continuous diffusion models [19, 22, 52] for high-quality image synthesis. Let h t be salient information of x t and ¯ x t be a perturbed sample that lacks h t . h t can be internal information within x t or an external condition, or both. Lee et al. [22] proposed a general guidance technique that guides the sampling process toward enhancing the information h t , and the equation for the sampling process is:

where ϵ is a score function, ˜ ϵ is a guided score function of the continuous diffusion model, and s is a guidance scale. For instance, in the setting h t = c and ¯ x t = x t , the Eq. (2) collapses to classifier-free guidance (CFG) [19], which guides the sampling toward the given class distribution. Lee et al. [22] propose using adversarial blurring, enhancing the fine-scale details of the sample. Generally, using a large guidance scale s enhances the quality of generated samples while reducing the diversity.

Recently, discrete CFG [52, 5] has been introduced to improve the correlation between the input class or text condition c and the images generated by discrete diffusion models as below equation:

where ˜ x t denotes the guided token.

Unlike CFG in the continuous domain, discrete CFG estimates the probability distribution p ( x t | c ) directly. However, it requires a specific training strategy and paired labels such as class and text.

2.3 Parameter-Efficient Fine-Tuning

MGMs across various domains adopt transformer architecture due to their superior ability to handle context with bidirectional attention [9, 13, 4, 57, 63]. However, training a transformer from scratch requires significant computational resources due to the quadratic complexity of the attention mechanism. In recent years, parameter-efficient fine-tuning (PEFT) techniques have received significant attention, especially in light of the growing size and complexity of pre-trained models. PEFT adapts large pre-trained models to specific tasks or datasets by tuning a small portion of parameters [58, 55] or introducing task-specific parameters [45, 23, 49], effectively transferring knowledge without extensive retraining. Notably, by preserving most of the parameters, the fine-tuned model effectively preserves knowledge with few forgetting.

3 Methods

3.1 Generalized Information-Based Guidance for Discrete Diffusion Models

Similar to the general guidance in continuous diffusion models in Eq. (2), discrete CFG in Eq. (3) can be extended to the generalized information-based guidance from the optimization perspective. Given

some salient information h t , we aim to sample x t which maximizes p (¯ x t | h t ) . Simultaneously, for the correlation between the information and sample, p ( h t | ¯ x t ) also needs to be maximized as stated in the equation below which is from Tang et al. [52]:

Using the Bayes' theorem and ignoring the prior probability term for salient information, the optimization goal can be represented as:

Then, the Eq. (5) guides the sampling process toward enhancing the relevance of the sample and salient information h t .

In the inference stage, various MGMs adopt to predict unmasked state ˆ x 0 ,t rather than directly predicting x t -1 . If we limit h t to the internal information of x t to make h t removable from the x t through an information bottleneck module H ϕ ; in other words, if ¯ x t = H ϕ ( x t ) and p ( x t ) = p (¯ x t , h t ) get satisfied, we can sample next state for the denoising step as below:

when a MGM with parameter θ predict ˆ x 0 ,t from x t and ¯ x 0 ,t from H ϕ ( x t ) . This implies that by defining the information bottleneck module H ϕ that can selectively subtract salient information h t from x t in the discrete space, we can guide the sampling of MGMs in a direction that enhances h t . Since we aim to improve the sample quality of MGMs by presenting a novel guidance method, it is necessary to define H ϕ that can remove information about the fine details of the samples. For continuous diffusion models, utilizing samples spatially smoothed by Gaussian blur for guidance has been helpful in improving sample quality by restricting fine-scale information [22]. This motivates us to investigate guidance with smoothed output for discrete domains, especially for VQ tokens.

3.2 Auxiliary Task Learning for Semantic Smoothing on VQ Token space

Weaim to apply smoothing, such as Gaussian blur, for VQ tokens, as discussed in the previous section, to implement guidance in the discrete domain. However, unlike natural images which have inherent, observable patterns and structures, the latent space of autoencoders often lacks such properties [2]. For instance, two successive tokens in VQ codebooks, such as the 11th and 12th tokens, are not semantically linked. As a result, applying Gaussian blur in the VQ token cannot produce meaningful representations. Nevertheless, we empirically found that applying Gaussian blur in the probability spaces, i.e. blurring the output logits of the generator p θ (ˆ x 0 | x t ) similar to Lee et al. [22] can provide meaningful guidance. However, the improvement is marginal because Gaussian blur is not a suitable information bottleneck for subtracting fine details in VQ token space.

To overcome this, we introduce an auxiliary task designed to leverage semantic smoothing for VQ tokens, a process that selectively removes details such as local patterns while preserving overall information in VQ token space. Specifically, given the masked input x t , masked generator predicts p θ (ˆ x 0 ,t | x t ) . We aim to train information bottleneck H ϕ to generate the semantically smoothed output p θ (ˆ x 0 ,t |H ϕ ( x t )) . However, training H ϕ directly is challenging, as we cannot define semantically smoothed outputs. To naturally impose a model to generate semantically smoothed output, we leverage error token correction, originally introduced in non-autoregressive machine translation to mitigate the compounding decoding error in the iterative sampling process [24]. During the error token correction, input unmasked tokens are randomly replaced with error tokens, and the model learns to correct them. To be more specific, let z t be a corrupted data where some tokens in x t are replaced with error tokens with some probability p . Then, the objective function for the auxiliary task to update ϕ can be

From the perspective of Vicinal Risk Minimization (VRM) [6, 59], a vicinity distribution p ϕ minimizes the empirical risk for all data points x t given a vicinity of the data z t . Given that randomly replaced

ҧ

16

291

<M>

117

Pretrained

Masked Generator

<M>

<M>

<M>

<M>

Pretrained

Masked Generator

FC

--

130

--

--

ℒ𝑎𝑢𝑥

(a) Training feature selection module

(b) Guided sampling

Feature

Selection

Ground truth

Sampled token

Error token

Frozen

Tuned

17

291

<M>

117

Pretrained

Masked Generator

<M>

<M>

<M>

<M>

Pretrained

Masked Generator

FC

FC

17

291

388

117

Feature

selection

𝑝𝜃(

𝑝𝜃(ො

|𝑥 𝑡

)

𝑝𝜃(

𝑥0,𝑡|ℋ𝜙(𝑥𝑡))

Guidance (plug-and-play)

Shared

ҧ

error tokens often act as semantic vicinities within the input data, to minimize the overall empirical risk, the model implicitly learns to smooth vicinities of z t . This involves leveraging coarse information from the surrounding context while minimizing the fine-scale details in the presence of unknown input errors 1 . However, training a network from scratch with Eq. (7) does not ensure that the outputs will resemble real images, potentially converging on trivial solutions that may be undesirable, in addition to being computationally expensive.

3.3 Efficient Implementation

To mitigate the aforementioned problem, we adopt a parameter-efficient fine-tuning (PEFT) method to utilize deep image priors in the pre-trained masked generator and to enhance the training efficiency of transfer learning. Among various PEFT methods, we adopt TOAST [49], which shows favorable performance in various visual and linguistic tasks under transformer architecture. With a frozen pre-trained backbone, TOAST selects task-relevant features from the output and feeds them into the model by adding them to the value matrix of self-attention. This top-down signal steers the attention to focus on the task-relevant features, effectively transferring the model to other tasks without changing parameters.

Besides its effectiveness in transfer learning, TOAST brings more practical strengths to our task. Before the discussion, it is important to note that masked generative models exhibit a strong training bias. Because the training data does not contain error tokens, the model predicts unmasked input as identical and unable to correct error tokens. Since the model is trained only to consider masked regions, we propose to use a blank canvas as an input (i.e., all tokens are masked x T ), enabling the model to make corrections in response to error tokens. However, most PEFT methods lack a direct solution for incorporating information about x t when the input is replaced with all masked tokens. On the other hand, we empirically found that simply replacing the input for the second stage of TOAST as x T can mitigate this problem without a performance drop (Fig. 3 a).

Furthermore, the two-stage approach of TOAST can be efficiently implemented in the sampling process. The generator first sample p θ (ˆ x 0 ,t | x t ) from the input x t . The hidden state obtained from the generation stage can be recycled for the second stage to produce guidance logit p θ (¯ x 0 ,t |H ϕ ( x t ))

(Fig. 3 b). We note that different from the fine-tuning step where the error tokens z t are used to train H ϕ , the sampling process directly utilizes x t to generate the semantically smoothed tokens H ϕ ( x t ) .

4 Related Works

Generative Adversarial Networks (GANs). Trained by adversarial objective [14], GANs have shown impressive performance in image synthesis with only 1-step model inference [27, 28, 47, 3, 33]. However, despite their practical performance, the training instability and limited mode coverage caused by the adversarial objective [46, 62] is still a bottleneck for their broader applications.

Diffusion Models. Recent diffusion models in continuous space mostly utilize Gaussian noise to perturb the input data and learn to predict clean data from the noisy data [50]. After the advent of DDPM[20], diffusion models have rapidly grown with architecture improvements [38, 41], improved training strategies [29, 7, 25], improved samplings [51, 37], latent space models [44], and sampling guidance [10, 19, 22], outperforming various generative families such as GANs in various domains.

Masked Generative Models. With the recent success of transformers [54] and GPT [42] in NLP, generative transformers have also been adopted in image synthesis. To mimic the generative process of transformers for discrete embedded natural language, recent studies encode images into quantized visual tokens using the VQ-VAE encoder [43, 11] and apply the generation process in an autoregressive [32, 56] or non-autoregressive manner [4, 35, 36, 5]. In particular, non-autoregressive generation shows a better trade-off between generation quality and speed. Nevertheless, they suffer from compounding decoding errors [36], which means that small decoding errors in early generation steps can accumulate into large differences in later steps. To address this issue, Token-Critic [35] and DPC [36] use additional transformers to identify more realistic tokens. However, they require a training second transformer, which incurs a training cost similar to training a generator from scratch.

The non-autoregressive predict-mask sampling of MGMs can be regarded as a discrete diffusion process that uses absorbing state diffusion process [1, 36]. Similar to MGMs, VQ-Diffusion [16] proposed a mask-and-replace diffusion strategy to predict masked tokens. Improved VQ-Diffusion [52] adopt purity-based sampling with discrete CFG to further improve the sample quality.

IS vs.FID (256x256,50k)

MaskGIT (18)

IS vs.FID (512x512,50k)

Token-Critic(36)

9.5

8

Ours (36)

9.0

DPC-light (66)

8.5

7

DPC-full (180)

8.0

FID

6

7.5

7.0

5

6.5

4

6.0

3

5.5

150

200

250

300

160

180

200

220

240

IS

IS

5 Experiments

Datasets, Baselines, and Metrics. We demonstrate the effectiveness of the proposed guidance for masked generative models on class conditional generation using the Imagenet benchmark [8] with 256 × 256 and 512 × 512 resolutions. For a baseline model, we use MaskGIT [4], which shows a state-of-the-art trade-off between quality and sampling speed on a class conditional generation of MGMs and has publicly available checkpoints for our target datasets. To evaluate the trade-off between sample fidelity and diversity, we measure Fréchet Inception Distance (FID) [18], Inception Score (IS) [46], Precision, and Recall [31] using the implementation provided by Dhariwal et al. [10]. To measure the computational cost, we report the number of function evaluations (NFE) required to sample an image. Note that total sampling timesteps ( T ) may differ from NFE due to guidance.

Implementation Details. We use a VQGAN tokenizer [11] provided by MaskGIT [4], which encodes images into 10-bit integers. Since the training code for MaskGIT is unavailable, we implement based on the open Pytorch reproduction [40]. We follow the previous work for error token correction [24] to prepare input with error tokens and randomly replace 30% of input tokens with error tokens. We utilized an NVIDIA RTX A6000 for fine-tuning and sampling. We used an exponential moving average (EMA) of fine-tuning weights with a decay of 0.9999 and bf16 precision. The batch size was set to 256, and the additional parameters introduced by TOAST are approximately 20-25% of the model size. Notably, the fine-tuning was completed efficiently within 10 epochs. More detailed implementation of TOAST [49] is provided in the Appendix A. We use sampling step T = 18 and sampling temperature 25 for Imagenet 256 × 256 and 45 for Imagenet 512 × 512. For sampling step T = 12 , we use temperatures 10 and 20, respectively for each resolution.

5.1 Comparison with Various Generative Models

Quantitative results. We compare the performance of the proposed method with various class conditional generative methods. (1) GANs : BigGAN [3] and GigaGAN [26]; (2) continuous diffusion models (Diff.): ADM [10], CDM [21], LDM [44], and DiT [41]; (3) auto-regressive models (AR): VQVAE-2 [43] and VQGAN [11]; (4) discrete diffusion models (Discrete.): VQ Diffusion [16], and Improved VQ Diffusion [52]; (5) masked generative models (Mask.): MaskGIT [4], Token-Critic [35], DPC [36]. As noted in previous literature [19, 36], sampling using a pre-trained classifier may impact the classifier-based metrics such as FID and IS. Thus, to see the base generative capacity of each method, we compare the models that do not use external pre-trained networks such as a classifier or upsampler during training or sampling. We also exclude large-scale models such as DiT-XL/2 [41] for a fair comparison. Table 1 presents the quantitative results of various generative models with guidance. All values are taken from the original paper unless otherwise noted. The proposed method achieves superior FID and IS despite the low computational cost for sampling. For instance, the proposed method achieves better FID and IS than DiT-L/2 with CFG on Imagenet 256 × 256 even though the proposed method requires an order of magnitude fewer sampling steps.

LDM (NFE=500, FID=3.60)

MaskGIT (NFE=18, FID=6.56)

Ours (NFE=36, FID=3.22)

We comprehensively compare the proposed methods with various sampling strategies for MGMs in Fig. 4. We note that all MGMs use the same VQGAN tokenizer [11] provided in MaskGIT [4], the same baseline generator [4], and the same sampling timestep T = 18 . Although the previous methods, such as Token-Critic [35] and DPC [36], require similar or more NFEs to sample images and more training resources to train an external corrector transformer, our simple guidance shows a better trade-off between sample quality and diversity.

Qualitative results. Fig. 5 presents the randomly sampled images using LDM [44] with CFG, MaskGIT, and MaskGIT with the proposed guidance. The proposed guidance sampling enhances details in the generated samples while maintaining high diversity. More sampled results with various classes are provided in the Fig. 1 and Appendix C.

5.2 Ablation Studies and Analysis

In this section, we explore the effectiveness of the guidance on class conditional ImageNet generation in 256 × 256 scale across the various sampling hyperparameters. We vary each sampling parameter and otherwise use the default settings.

Effectiveness of Auxiliary Task. To demonstrate that finetuning with a proposed auxiliary task using the objective in Eq. (7) can effectively improve the generative capabilities of MGMs, we conduct ablation studies and report the FID and IS in Table 2. As noted in Section 3.2, applying Gaussian blur in the input VQ token does not properly smooth VQ tokens. However, we empirically found that applying Gaussian blur in the output logit can produce meaningful guidance. Therefore, we provide the quantitative comparison with guidance by applying Gaussian blur in the logit space (Blur Guidance). We further utilize self-attention value for adversarial masking

following Lee et al. [22]. Blur guidance and SAG enhance the quality (IS) or diversity (FID) in some

(a) Guidance scale

(b) Temperature

(c) Timestep

(d) Fine-tuning epochs

8

8.0

5.0

6

6.0

4.5

4

4.0

4.0-

FID

4.0

2

2.0

3.5

3.0

0

0

1

Guidance scale (s)

2

3

4

5

0.0

0

20

Temperature (t)

40

60

80

5

10

15

Timestep (T)

2025

3035

2.0

10

20

30

Epochs

40

50

60

300

325

265

264

300

262

260

S

250

S

275

S

260

250

255

258

200

225

250

256

0

1

2

3

4

5

0

20

40

60

80

5

10

152025

53035

10

20

30

40

50

60

Guidance scale (s)

Temperature (t)

Timestep (T)

Epochs

degree, but the improvement is marginal compared to ours. We further compare the results when the TOAST module is fine-tuned with the generative objective in Eq. (1) to verify the effectiveness of the auxiliary task (ft. w/ L mask ). Since the TOAST architecture and blank input for the second stage naturally play the role of information bottleneck, the performance has increased. Nevertheless, fine-tuning with the proposed auxiliary loss in Eq. (7) demonstrates its effectiveness, showing superior performance in both metrics.

Varying the guidance scale (Fig. 6 a). In line with previous literature on sampling guidance [10, 19, 22], a high guidance scale improves the sample quality while sacrificing diversity. We found that the guidance scale 1.0 shows the best performance in terms of FID score and the best trade-offs. However, strong guidance ( s > 3 ) does not ensure quality improvement, often leading to undesirably highlighted details or saturated colors (Appendix B), similar to the effect of large scale with CFG [19].

Varying the sampling temperature (Fig. 6 b). Compared to MaskGIT [4] limit their sampling temperature relatively low value ( τ = 4 . 5 ), we found that with the proposed guidance, the sample quality can be effectively preserved with a higher sampling temperature. As a result, with a high sampling temperature ( τ = 25 ), we achieve better quality and diversity compared to MaskGIT.

Varying the sampling steps T (Fig. 6 c). We observed that for higher T , the optimal sampling temperature also increases. For example, at T = 5 , the best results are achieved with a sampling temperature of 4, while at T = 18 , the optimal sampling temperature rises to 25. Thus, we plot the best FID and IS for each timestep using different temperatures. Whereas the optimal performance-efficiency trade-off of MaskGIT is observed around T = 8 , ours shows such "sweet spot" around T = 18 . Up to this point, both quality and diversity increase as T increases, consistently outperforming MaskGIT. Furthermore, with fewer NFEs ( T = 5 ), the proposed method achieves an FID of 4.84 and an IS of 249.9, outperforming the MaskGIT samples with T = 18 . This demonstrates that the proposed guidance technique provides more scalable and efficient sampling for MGMs.

Varying fine-tuning epochs (Fig. 6 d). We found that the proposed fine-tuning is efficiently trained within 10 epochs and that no further fine-tuning is required to achieve better results.

6 Conclusion and Future Work

In this paper, we define generalized guidance for discrete diffusion models, as a counterpart for guidance in continuous domain [22]. To generate guidance, we propose an auxiliary task to apply semantic smoothing in VQ tokens. Experimental results show that the proposed guidance effectively and efficiently improves the generative capabilities of MGMs on class conditional image generation.

Future work. Although the proposed guidance can improve the generative capabilities of MGMs on class conditional image generation, there is still room for improvement in various aspects: (1) experiments on large-scale text conditional generative models [5], (2) generalization to discrete diffusion models [16], and (3) generalization to various domains such as audio [63], and video [57].

Societal impacts. While our research does not directly touch ethical issues, the rapid growth of generative models raises considerations in AI ethics (e.g., potential misuse in creating deepfakes [34], generating antisocial content [12], or vulnerabilities to adversarial attacks [61, 17, 30]).

References

J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems , 34:17981-17993, 2021.

D. Berthelot, C. Raffel, A. Roy, and I. Goodfellow. Understanding and improving interpolation in autoencoders via an adversarial regularizer. arXiv preprint arXiv:1807.07543 , 2018.

A. Brock, J. Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 , 2018.

H. Chang, H. Zhang, L. Jiang, C. Liu, and W. T. Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1131511325, 2022.

H. Chang, H. Zhang, J. Barber, A. Maschinot, J. Lezama, L. Jiang, M.-H. Yang, K. Murphy, W. T. Freeman, M. Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704 , 2023.

O. Chapelle, J. Weston, L. Bottou, and V. Vapnik. Vicinal risk minimization. Advances in neural information processing systems , 13, 2000.

J. Choi, J. Lee, C. Shin, S. Kim, H. Kim, and S. Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11472-11481, 2022.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12873-12883, 2021.

R. Gandikota, J. Materzynska, J. Fiotto-Kaufman, and D. Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2426-2436, 2023.

M. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprint arXiv:1904.09324 , 2019.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

J. Gu, J. Bradbury, C. Xiong, V. O. Li, and R. Socher. Non-autoregressive neural machine translation. arXiv preprint arXiv:1711.02281 , 2017.

S. Gu, D. Chen, J. Bao, F. Wen, B. Zhang, D. Chen, L. Yuan, and B. Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10696-10706, 2022.

G. Han, J. Choi, H. Lee, and J. Kim. Reinforcement learning-based black-box model inversion attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2050420513, 2023.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

J. Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research , 23(47):1-33, 2022.

S. Hong, G. Lee, W. Jang, and S. Kim. Improving sample quality of diffusion models using self-attention guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 74627471, 2023.

N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning , pages 2790-2799. PMLR, 2019.

X. S. Huang, F. Perez, and M. Volkovs. Improving non-autoregressive translation models without distillation. In International Conference on Learning Representations , 2021.

J. Hur, J. Choi, G. Han, D.-J. Lee, and J. Kim. Expanding expressiveness of diffusion models with limited data via self-distillation based fine-tuning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 5028-5037, 2024.

M. Kang, J.-Y. Zhu, R. Zhang, J. Park, E. Shechtman, S. Paris, and T. Park. Scaling up gans for text-toimage synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10124-10134, 2023.

T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems , 33:12104-12114, 2020.

D. Kingma, T. Salimans, B. Poole, and J. Ho. Variational diffusion models. Advances in neural information processing systems , 34:21696-21707, 2021.

S. Koh, H. Shon, J. Lee, H. G. Hong, and J. Kim. Disposable transfer learning for selective source task unlearning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11752-11760, 2023.

T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems , 32, 2019.

D. Lee, C. Kim, S. Kim, M. Cho, and W.-S. Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11523-11532, 2022.

D. Lee, J. Y. Lee, D. Kim, J. Choi, J. Yoo, and J. Kim. Fix the noise: Disentangling source feature for controllable domain translation. arXiv preprint arXiv:2303.11545 , 2023.

J. Lee, J. Hyung, S. Jeong, and J. Choo. Selfswapper: Self-supervised face swapping via shape agnostic masked autoencoder. arXiv preprint arXiv:2402.07370 , 2024.

J. Lezama, H. Chang, L. Jiang, and I. Essa. Improved masked image generation with token-critic. In European Conference on Computer Vision , pages 70-86. Springer, 2022.

J. Lezama, T. Salimans, L. Jiang, H. Chang, J. Ho, and I. Essa. Discrete predictor-corrector diffusion models for image synthesis. In The Eleventh International Conference on Learning Representations , 2022.

C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems , 35:5775-5787, 2022.

A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning , pages 8162-8171. PMLR, 2021.

M. Ott, M. Auli, D. Grangier, and M. Ranzato. Analyzing uncertainty in neural machine translation. In International Conference on Machine Learning , pages 3956-3965. PMLR, 2018.

S. Patil, B. William, and P. von Platen. Amused: An open MUSE model. URL https://github.com/ huggingface/open-muse .

W. Peebles and S. Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195-4205, 2023.

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018.

A. Razavi, A. Van den Oord, and O. Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems , 32, 2019.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

S. Ryu. Low-rank adaptation for fast text-to-image diffusion fine-tuning, 2023.

T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. Advances in neural information processing systems , 29, 2016.

A. Sauer, K. Schwarz, and A. Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings , pages 1-10, 2022.

B. Shi, T. Darrell, and X. Wang. Top-down visual attention from analysis by synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2102-2112, 2023.

B. Shi, S. Gai, T. Darrell, and X. Wang. Toast: Transfer learning via attention steering. arXiv preprint arXiv:2305.15542 , 5(7):13, 2023.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Z. Tang, S. Gu, J. Bao, D. Chen, and F. Wen. Improved vector quantized diffusion models. arXiv preprint arXiv:2205.16007 , 2022.

K. Tian, Y. Jiang, Z. Yuan, B. Peng, and L. Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02905 , 2024.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

E. Xie, L. Yao, H. Shi, Z. Liu, D. Zhou, Z. Liu, J. Li, and Z. Li. Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4230-4239, 2023.

J. Yu, X. Li, J. Y . Koh, H. Zhang, R. Pang, J. Qin, A. Ku, Y . Xu, J. Baldridge, and Y . Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627 , 2021.

L. Yu, Y. Cheng, K. Sohn, J. Lezama, H. Zhang, H. Chang, A. G. Hauptmann, M.-H. Yang, Y. Hao, I. Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10459-10469, 2023.

E. B. Zaken, S. Ravfogel, and Y. Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformerbased masked language-models. arXiv preprint arXiv:2106.10199 , 2021.

H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

K. Zhang, R. Wang, X. Tan, J. Guo, Y. Ren, T. Qin, and T.-Y. Liu. A study of syntactic multi-modality in non-autoregressive machine translation. arXiv preprint arXiv:2207.04206 , 2022.

Y. Zhang, R. Jia, H. Pei, W. Wang, B. Li, and D. Song. The secret revealer: Generative model-inversion attacks against deep neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 253-261, 2020.

S. Zhao, H. Ren, A. Yuan, J. Song, N. Goodman, and S. Ermon. Bias and generalization in deep generative models: An empirical study. Advances in Neural Information Processing Systems , 31, 2018.

A. Ziv, I. Gat, G. L. Lan, T. Remez, F. Kreuk, A. Défossez, J. Copet, G. Synnaeve, and Y. Adi. Masked audio generation using a single non-autoregressive transformer. arXiv preprint arXiv:2401.04577 , 2024.

A Detailed Implementation of feature selection module (TOAST)

The architecture of the proposed sampling method consists of two parts: MaskGIT [4] and TOAST [49]. Since we have not changed the MaskGIT architecture for plug-and-play sampling guidance, we briefly explain the implementation details of TOAST below. The TOAST modules consist of three parts: token selection module, channel selection module, and linear feed-forward networks.

(i) The token selection module selects the task or class-relevant tokens by measuring the similarity with the learnable anchor vector ξ c . We generate class conditional anchors ξ c with simple class conditional MLPs.

(ii) The channel selection is applied with learnable linear transformation matrix P . Then, the output of the token and channel selection module is calculated via z i = P · sim ( z i , ξ c ) , where z i denotes the i -th input token.

(iii) After the feature selection, the output is processed with L layer MLP layers, where L is equal to the number of Transformer's layers. The output of the l -th layer of MLP blocks is added to the value matrix of the attention block in ( L -l ) -th Transformer layer (top-down attention steering). Following the previous work [49], we add variational loss to regularize the top-down feedback path. A more detailed process and theoretical background can be found in previous works on top-down attention steering [48, 49].

B Effect of High Sample Temperature and Guidance Scale

We show the effect of high sampling temperature ( τ ) and high guidance scale in Fig. 7. The high sampling temperature (i.e., strong guidance) often leads to undesirably highlighted fine-scale details or saturated colors, similar to the large scale of CFG [19]. High temperatures often lead to the collapse of the overall structure of generated samples.

(a)s=1,t=25

(b)s=5,t=25

(c) s=1,t=50

C More Visual Results

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claim in the abstract and introduction is dealt with in the background, method, and experiment sections.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We deal with this in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide a correct proof for our theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide implementation details in the paper for reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We do not include code for our experiments. Nevertheless, the dataset and baseline code for our implementation and evaluation are clearly denoted in the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide results by varying the hyperparameters and provide insights for selecting those values. We use open-source evaluation code, which is denoted in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The sampling process and evaluation are statistical, but we do not report the error bars since with the sufficiently large number of samples (=50k), the impact of stochasticity is minimized.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report the training environment, required training epochs, and required sampling steps in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We wrote a paper in compliance with the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We denote the negative societal impacts that the proposed method may have.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not include such a large pre-trained model in the paper. We will release only a small module for fine-tuning, which does not have a high risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use open source implementation for our baseline and denote in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We have no results related to the human evaluations.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We have no results related to the human evaluations.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a self-guidance method for masked generative models to improve the diversity and quality of class conditional image generation. The main challenge is to design a semantically meaningful smoothing for the discrete VQ token space such that coarse scale information can be extracted. The authors propose an auxiliary task – error token correction to minimize fine-scale details and utilize TOAST for efficient model fine-tuning.\n\nSTRENGTHS:\n- The research problem of introducing self-guidance in discrete space is well-motivated. The background section is well-written, and the discussion of related work is thorough. I find the analogy to guidance of continuous diffusion model very helpful. \n- The figures are well-made, such as the qualitative visualization is Fig.1. \n- The quantitative metrics from the experiments also demonstrate the effectiveness of the method. The ablation studies on different hyperparameters are informative.\n\nWEAKNESSES:\n- From the qualitative visualizations alone (Fig.4), the proposed method does not seem have too different outputs compared with MaskGIT (the middle column). I wonder if the guidance benefit is stronger on other resolutions’ results, like 128 x 128 or 64 x 64. \n- Maybe some figures could be added to accompany section 3.2 to better illustrate the process of the auxiliary task.\n\nQUESTIONS:\n- What is $\\bf{m}_i$ in equation (7)? Could you provide more intuitions or visualizations of the claims in line 192- 195. Why does this auxiliary task naturally minimizes fine-scale details? \n- What is the range of the experimented temperature values in Fig. 3? \nFrom Fig.3, it seems that the proposed sampling method has very different performance based on the choice of temperature. The authors also mention that they choose different temperatures based on the resolution and sampling steps. I wonder why is the proposed method so sensitive to temperature? \n- Have the authors tested the model on the 128x128 resolution ImageNet benchmark? What does the FID/IS curve look like? \n- What happens with strongly guided samples? For instance, for classifier-free guidance in diffusion models, strongly guided samples exhibit saturated colors. I wonder if there is any artifact related to this proposed method when the choice of guidance scale or sampling temperature is high?\n\nLIMITATIONS:\nThe authors already mention some directions for future work, such as demonstrating the guidance on text-conditioned image generation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the use of masked generative models for image generation. While these methods are typically efficient, they sometimes fall short in terms of quality compared to diffusion models. To reduce this gap, this paper proposes a self-guidance algorithm, which is further improved with semantic smoothing. These ideas are shown to be effective for improved image generation using MGMs. The method is compared to a wide variety of baselines on class-conditioned generation, on common metrics, and against a set of baselines.\n\nSTRENGTHS:\n- This paper tackles an important problem in image generation, which is the use of Masked Generative Models for efficient image generations. This family of models have shown to be more efficient than diffusion models, but have so far struggled to catch-up with the quality they can achieve. This paper improves the quality of these models, to make them competitive in quality while preserving their benefits in terms of efficiency. \n- At the core fo the method lies a self-guidance method, which I believe is a sound and well motivated idea, which can be potentially included in generative models for other applications.\n- The ideas in the paper are simple in a good way, and they are not particularly tied to a specific application. I believe they could be generalizable to many other problems. \n- I believe the related work is well analysed, as far as I am familiar with previous work on this topic.\n- Evaluation is extensive, many different hyperparameters are evaluated with respect to their impact on generation quality.\n\nWEAKNESSES:\n- The paper is at times hard to follow, lacking clarity in some sections (particularly 5). Images and figures are sometimes small and hard to see.\n- I do not believe implementation details are enough for reproducibility. Code will not be provided, which will make this paper hard to reproduce and it will reduce its potential impact and room for future comparisons. \n- Given that a particular benefit of the method of this paper is its computational efficiency in comparison to diffusion-based approaches, I believe a detailed analysis of these should be included. \n- I believe the paper would benefit from showing results on text-to-image generation.\n\nQUESTIONS:\n- How would this method generalize to text-to-image methods or image-to-image translation problems?\n- How tied is this method to VQGAN? Could other encoders be used? If so, what would be their impact?\n\nLIMITATIONS:\nThe paper is fine in this regard. It could, however, provide more details on how the limitations could be addressed in the future.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on enhancing Masked Generative Models (MGMs) for image synthesis.\nThe authors identified several reasons for the underperformance of MGMs: 1) lack of sequential dependencies, 2) multi-modality problem, 3) compounding decoding errors, 4) limitations of low-temperature sampling and 5) ineffective guidance techniques.\nTo address these issues, the authors present novel techniques and methodologies to improve their performance.\n\nThe main contributions of the paper are as follows: \n1)\tGeneralized Guidance Formulation.\n2)\tSelf-Guidance Sampling Method.\n3)\tParameter-Efficient Fine-Tuning.\n\nSTRENGTHS:\n- Originality\nThe paper presents a highly original approach by extending guidance methods from continuous diffusion models to MGMs. In particular, the use of self-guidance sampling method and an auxiliary task for semantic smoothing in the VQ token space are novel approach that address specific challenges in MGMs.\n\n- Quality\nThe paper demonstrates a high level of technical quality. The authors have thoroughly explained why their methods can enhance performance and have supported their claims with extensive experiments and ablation studies on various variables. \n\n- Clarity\nThe paper is clearly written and well-structured. The authors provide detailed context relative to prior work, facilitating a clear understanding of the background and contributions of the proposed approach. The detailed explanations of the equations and methodologies, accompanied by ample examples, significantly enhance the overall readability and accessibility of the paper. \n\n- Significance\nThe contributions of the paper are significant for the advancement of the MGMs field. By addressing key challenges in MGMs, the proposed methods offer substantial improvements in the quality and diversity of generated samples.\n\nWEAKNESSES:\nParticularly due to inadequate visual results, some aspects are difficult to comprehend. Section 3.3 asserts that TOAST is the most suitable model for the study's task, but lacks sufficient explanation on why TOAST is the only suitable model. It would be more convincing if the paper showed some visual results from using other models, which would provide sufficient justification.\n\nExplanations for the choice of hyperparameters are insufficient. While the paper demonstrates trade-off relationships in hyperparameter settings across some datasets in removal experiments, the sampling time steps do not exhibit such trade-offs. It is crucial to provide adequate explanations for why these settings differ from those maintained for the ImageNet 512×512 dataset.\n\nQUESTIONS:\nMost hyperparameters in Figure 5 show a trade-off relationship between FID and IS scores. However, increasing the number of sampling time steps improves performance in both metrics. So, could you explain why sampling time steps were set to 12 and 18 in each experiment? Even though other methods are known to use hundreds of sampling time steps, 50 seems quite small in comparison.\n\nLIMITATIONS:\nThe authors did not fully address the limitations of the paper but did mention the need for attention to AI ethics due to the rapid growth of generative models, touching on the societal impact. To improve the paper, we suggest the following:\n\n1)\tAlthough ImageNet is a comprehensive dataset, it is insufficient to prove generalizability. Therefore, conducting experiments on other datasets used in existing MGMs, such as MSCOCO or Conceptual Captions, will make the paper more complete.\n2)\tInclude more visual results. To clarify the reasons for selecting specific models or hyperparameters used in the paper, it is much more effective to show them through visual results.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}