Halton Scheduler for Masked Generative Image Transformer

Authors: Victor Besnier, Mickael Chen, David Hurych, Eduardo Valle, Matthieu Cord
Primary area: generative models
Keywords: Image Synthesis, MaskGIT, Halton Sequence

Abstract:
Masked Generative Image Transformers (MaskGIT) have emerged as a scalable
and efficient image generation framework, able to deliver high-quality visuals with
low inference costs. However, MaskGIT’s token unmasking scheduler, an essential
component of the framework, has not received the attention it deserves. We analyze
the sampling objective in MaskGIT, based on the mutual information between
tokens, and elucidate its shortcomings. We then propose a new sampling strategy
based on our Halton scheduler instead of the original Confidence scheduler. More
precisely, our method selects the token’s position according to a quasi-random,
low-discrepancy Halton sequence. Intuitively, that method spreads the tokens
spatially, progressively covering the image uniformly at each step. Our analysis
shows that it allows reducing non-recoverable sampling errors, leading to simpler
hyper-parameters tuning and better quality images. Our scheduler does not require
retraining or noise injection and may serve as a simple drop-in replacement for
the original sampling strategy. Evaluation of both class-to-image synthesis on
ImageNet and text-to-image generation on the COCO dataset demonstrates that the
Halton scheduler outperforms the Confidence scheduler quantitatively by reducing
the FID and qualitatively by generating more diverse and more detailed images.
Our code is at https://github.com/valeoai/Halton-MaskGIT.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730321942653

thread_type:
official_review

<thread_object>
creator: reviewer_13ls

<content>
summary:
This paper proposed a new sampling strategy using the Halton scheduler which uniformly recovers the image spatially. Several experiments are conducted to validate the effectiveness.

**Additional Comments**:

The manuscript does not follow the margin of ICLR template, which violates the submission guidelines. 

In all, I lean towards desk rej.

strengths:
NA

weaknesses:
- The paper is poorly written. There are tons of typos and layout issues making it hard to follow.
- The contribution is not enough. It is like a simple spatial prior to the sampling process.
- The experiments are limited and unfair. For example, in table 1, the author compared the proposed method with the baseline MaskGIT. However, the "Ours" baseline is not presented. The compared MaskGIT has a different # param compared to the proposed approach.
- Fig 5 is very confusing. It is not understandable. Why is only the FID/IS reported for AR mode? 
- In table 2, the results are not fair. As compared MaskGIT* is not trained to predict images in 32 steps.
- The qualitative results are also limited. What are the text prompts for fig 8?
- Fig 7 is very confusing. Is the training target of the confidence scheduler set to be 32 during training?

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730359739699

thread_type:
official_review

<thread_object>
creator: reviewer_ozt1

<content>
summary:
This paper proposes a novel scheduler based on the Halton sequence, aimed at minimizing the correlation between simultaneously sampled tokens and maximizing the information gained at each step. By selecting token positions using a quasi-random, low-discrepancy Halton sequence, the approach reduces FID scores and improves both image quality and diversity.

strengths:
- Presents an innovative mutual information analysis that clarifies the scheduler’s role in MaskGIT.
- By leveraging the Halton sequence, the method ensures a more even distribution of selected tokens, grounded in the information gained at each step and the mutual information between tokens.
- Enhances both image quality and diversity in class-to-image and text-to-image generation.

weaknesses:
- The assumption for Eq. 5.b is not sufficient, which ignores the long-distance dependencies.
- The influence of conditional entropy is not limited to spatial proximity; semantic information and context play a significant role and should be considered.

questions:
- After vector quantization, do token relationships still correspond to their Euclidean distance?
- Might this uniformity lead to the overlooking of critical details in specific areas, thus negatively impacting sampling accuracy?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730562788722

thread_type:
official_review

<thread_object>
creator: reviewer_ogcs

<content>
summary:
This paper presents the Halton Scheduler, which refines the token unmasking strategy in MaskGIT. The authors begin by analyzing the Confidence Scheduler previously designed for MaskGIT (Chang et al., 2022; 2023), which prioritizes the unmasking of the most certain tokens. They demonstrate that while this method enhances confidence, it can reduce diversity and image quality due to the clustering of tokens around already predicted regions. To address this, they propose a novel, deterministic scheduler that leverages the Halton low-discrepancy sequence (Halton, 1964). This approach is informed by understanding entropy progression during image generation, aiming for a more even distribution of token selection to ensure uniform coverage across the image. They show convincing results over baselines like MUSE and MaskGIT across image generation and text to image generation.

strengths:
1) Overall the idea of Halton scheduler makes a lot of sense and addresses the important problem of scheduler in MaskGIT kind of models.
2) Halton scheduler proposes a improved schedule which makes the model improved generation capabilities.
3) Experimental results show pretty convincing improvements across text-to-image zero-shot COCO, MUSE , MaskGIT and other competetive methods.

weaknesses:
Weakness:
1) How was the discretization of the sequence Hnh into a grid corresponding to the token map done? This seems to be an approximation which I’m not sure about.
2) More intuition and details need to be added especially for Halton scheduler, like the Low-Discrepancy properties need to be discussed more. 
3) Discussion on improved halton sequencing needs to be added. Using the Scrambled Halton Sequence, the Leapfrog Method, and Higher-Dimensional Generalizations need to be discussed. 
4) Similar ideas for information overload into tokens have been studied in the registers[1]. Analysis on token norm in this setting will be useful and could reflect whether there is similarity in understanding between high-norm tokens and the proposed scheduler. 
5) Is Extra compute time required to use Halton scheduler? One of the main advantages of the Halton scheduler is the speed of image generation. Would like to see what is the inference time with Halton's scheduler?
References
1) Vision Transformer needs registers.

questions:
I don't have a good sense of what is the extra information that is being learnt using Halton schduler and the intuition is not very clear. I agree with the analysis of the paper but the proposed solution doesn't;t seem very well analysed and ablated.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730625297284

thread_type:
official_review

<thread_object>
creator: reviewer_w6jm

<content>
summary:
This paper proposes a new schedule for token decoding in masked image modeling. MaskGIT decodes high confidence tokens first. Instead, this paper shows that this technique is not optimal and can lead to decreased performance with more decoding steps which should not occur with a good schedule. To combat this, the paper proposes a Halton schedule based on mutual information analysis to spread out the predicted tokens. The technique requires no training and can be plugged into other models. Experiments are conducted on class conditional and text conditional image generation.

strengths:
This paper addresses an overlooked problem: choosing which tokens to decode during masked image modeling. The intermediate analysis and motivation are clear, and the proposed solution is promising while being simple. Comparison against random decoding is also conducted.

weaknesses:
I generally like this approach. The problem and motivation are clear, and the solution is simple and seems promising. However, some of the results are not convincing enough, and the presentation of the paper is unpolished and could benefit from a few rounds of editing.

- It is unclear to me how to interpret Table 1, because the baseline is from the paper for which no code exists. The models are very different parameter sizes, comparison against random and confidence is missing. I would like to see more rows like in Table 2. In fact, Table 2 and 3 are more convincingly demonstrate the effectiveness of this approach. However, the image resolutions do not match and so it is really difficult to draw conclusions.
- In general, the experimental part of this paper is weak, because if true, much more models and tasks should have been evaluated to showcase the plug&play-ability of this approach for various MIM tasks and models. From the current results, it is not clear how well the approach generalizes across datasets, models, and tasks.
- Does the method have any limitations, weaknesses or failure cases? What about runtime and memory complexity compared to random and confidence scheduling?
- Fig 5 should not be a line plot in my opinion as the x-axis is categorical.

In summary, the method is promising, and the paper presents an initial good signal. I encourage the authors to work on a more comprehensive evaluation of MIM models, tasks, and datasets to demonstrate the benefits of this model and produce fair and clear comparisons on fixed settings. As of now, unfortunately, there is not enough evidence to support the claims made convincingly.

questions:
NA
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces Halton scheduler as a new scheduler for Masked Generative Image Transformers' token decoding, and validates proposed approach on class-conditioned and text-conditioned image generation. 

The manuscript was reviewed by four knowledgeable referees. The reviewers acknowledged that the problem tackled in the paper is well motivated (W6JM) and important (ogcs), and the solution is sound, simple and promising (W6JM, ogcs).


The main concerns raised by the reviewers were:
1. The presentation of the paper which could be significantly improved (W6JM, ogcs,13LS)
2. The experiments did not appear convincing (W6JM, ogcs, 13LS) -- e.g. comparisons made with different model sizes and image resolutions, limited number of models considered
3. The contribution seemed incremental (13LS)
4. Assumption of Eq 5 did not appear sufficient, and as a result, it was difficult to understand whether the significance of the improvements could be fully attributed to the proposed approach (ozt1)
5. The limitations of the method were not discussed (W6JM)

The authors addressed most of the above mentioned concerns during rebuttal and discussion period. The authors discussed limitations, updated the paper presentation, argued for the novelty of the proposed approach, validated the assumptions in practice, clarified their experimental setup and conclusions draw from the results. The authors addressed most questions including those related to computation time. During discussion, the authors shared with the AC their concerns w.r.t. some of the comments made in the reviews. After that, both authors and reviewer engaged in fruitful discussion.

During reviewers' discussions, reviewers agree that most of their concerns were addressed by the authors and that the paper could be interesting for the research community. Although some reviewers argue that experiments could still be improved, they lean towards acceptance. The MR agrees with the reviewers' assessment and recommends to accept.