ReDDiT: Rehashing Noise for Discrete Visual Generation
Authors: Tianren Ma
Xiaosong Zhang
Boyu Yang
Junlan Feng
Qixiang Ye
Venue: ICLR 2026 Poster (2026)
Keywords: Discrete Diffusion
Masked Diffusion
Image Generation
Noise Design
Primary area: generative models
TL;DR: We generalize the modeling of discrete diffusion by introducing rehashing noise for visual generation tasks.

Abstract: In the visual generative area, discrete diffusion models are gaining traction for their efficiency and compatibility. However, pioneered attempts still fall behind their continuous counterparts, which we attribute to noise (absorbing state) design
and sampling heuristics. In this study, we propose a rehashing noise approach for discrete diffusion transformer (termed **ReDDiT**), with the aim to extend absorbing states and improve expressive capacity of discrete diffusion models. ReDDiT enriches the potential paths that latent variables traverse during training with randomized multi-index corruption. The derived rehash sampler, which reverses the randomized absorbing paths, guarantees high diversity and low discrepancy of the generation process. These reformulations lead to more consistent and competitive generation quality, mitigating the need for heavily tuned randomness. Experiments show that ReDDiT significantly outperforms the baseline model (reducing gFID from 6.18 to **1.61**) and is on par with the continuous counterparts. The code and models will be publicly available.

Published as a conference paper at ICLR 2026
REDDIT: REHASHING NOISE FOR DISCRETE VISUAL GENERATION
Tianren Ma' Xiaosong Zhang' Boyu Yang² Junlan Feng* Qixiang Ye!-
University of Chinese Academy of Sciences JJUTIAN Research
mat matianrenla wereenn1@mmlll...mmmm ucas.ac.cn
cxyeeucas.ac.cm
ABSTRACT
In the visual generative area, discrete diffusion models are gaining traction for their efficiency and compatibility. However, pioneered attempts still fall behind their continuous counterparts, which we attribute to noise (absorbing state) design and sampling heuristics. In this study, we propose a rehashing noise approach for discrete diffusion transformer (termed ReDDIT), with the aim to extend absorbing states and improve expressive capacity of discrete diffusion models. ReDDiT enriches the potential paths that latent variables traverse during training with randomized multi-index corruption. The derived rehash sampler, which reverses the randomized absorbing paths, guarantees high diversity and low discrepancy of the generation process. These reformulations lead to more consistent and competitive generation quality, mitigating the need for heavily tuned randomness. Experiments show that ReDDiT significantly outperforms the bascline model (reducing gFID from 6.18 1.61) and is on par with the continuous counterparts. The code is available at https: /ttp:/gittttt thub. com ma
INTRODUCTION
Diffusion has been a competitive approach for generative workloads (Dhariwal & Nichol, 2021 Rombach et al. 2022b Li et 2024), offering strong bidirectional perception and well-structured mechanisms Zhang et al. (2023) for global control over content. Within the continuous domain, diffusion transformers Peebles Xie (2023), which progressively refine image latents from Gaussian noise, have achieved impressive and scalable results. Recently, the community shows growing interest in discrete diffusion models (Hu & Ommer 2024) Swerdlow et al., 2025), which is based on their practical advantages, e.g., compatibility with language models for the indexable codebook, and efficiency for predicting multiple tokens at each inference. Early endeavors Chang eet al. (2022) 2023); Gu et al. (2022) pursue efficiincy through integrating visual tokenizers and BERT sstyle mmakk ] tokens Devlin et al. (019)). Recent studies Bai et al. (2025) improved the generation quality, demonstrating great potential of discrete diffusion.
Peebles
Despite the progress, the performance of discrete diffusion methods remains lagging behind their con- tinuous counterparts. Representative approaches, e.g., masked visual token models (MVTMs) Chang et al. p202)); Yu et al. (2023), are puzzled by the mask design and confidence-based re-mask sam- pler (Hur al. 20224). which restricts model's expressive capacity and makes prediction sensitive to adaptions given extensive training, Fig. [upper). Moreover, when paired with large-vocabulay codebooks from high-fidelity modern tokenizers, they encounter challenges such as slower sampling speeds and numerical inaccuracy (Zheng et al. 2024).
To address these limitations. we first propose two hypotheses. First, while discrete methods learn to recover plausible tokens from monotonous [mask] canvas, the used noise design may not be well-suited for discrete visual generation. In continuous diffusion, Gaussian noise used to progressively degrade the input to learn smooth distribution shift (Ho et al., 2020 Lu et al., (022). Discrete masking mimics this paradigm by collapsing all masked tokens to a single absorbing state, which, however, lacks the variability of Gaussian noise, in terms of both vocabulary richness and latent diversity. Consequently, the discrete process offers a far coarser signal, which limits its ability to represent diverse data distributions (Santos et al.) 2023 Austin al.] (021). Moreover, while continuous diffusion models introduce stochasticity at every inference step through noise injection,
"Corresponding author.

---

Published as a conference paper at ICLR 2026
Prediction w/ Gumbel noise
MVTM
argmax
Random
Methods:
remask
Low diversity
Softmas Prohabilizia
ReDDiT
diverse noise
(Ours):
rehach sampler
P(rocabulors) P(soises)
Principled, low discrepancy
Diverse and detailed
Figure Comparison the baseline discrete model (MVTM) with ReDDiT. MVTMs rely score-based remasking strategies with Gumbel-max to sample from logits, which leads to lower token diversity and suboptimal token selection. In contrast, ReDDiT introduces a systematic, low- discrepancy rehashing mechanism that leverages softmax-based probabilities, enabling diverse, high-quality sampling through a learned distribution. (This figure best viewed in color)
discrete unmasking is inherently binary: tokens are either masked or deterministically decoded, Fig. Tupper). This rigid mechanism constrains the flexibility of sample refinement during generation.
Second, the confidence-based re-mask sampler of MVTMs introduces form of handcrafted random- ness, which is implemented through Gumbel-max, to approximate sampling diversity. Unfortunately, this sampler compromises the probabilistic fidelity of generation, and the need to carefully balance token numbers decoded per step (for mitigating accumulation errors) leads to redundant sampling passes. As result, Gumbel-max has evolved heavily tuned time variant trick with unstable performance, particularly when scaled to large-vocabulary codebooks. The above factors, rather than quantization alone, induce the performance gap between discrete and continuous models.
In this study, we propose a discrete diffusion model with an elaborate rehashing noise design, Fig. [ower). Our approach, termed ReDDiT, addresses the limitations of the uni-mask design by redefining absorbing states towards larger representational capacity, through enriching the potential paths that latent variables can traverse during diffusion. Specifically, we expand the masks to multiple indices along with the codebook and randomize them during data corruption. A rehash sampler is also derived with principled discrete diffusion theories reverse the diffusion path for generation, guaranteeing high diversity and low discrepancy of the sampling process. We demonstrate that this rehashed noise facilitates learning a superior and regularized expressiveness, while eliminating reliance to hyper-parameteriized randomness during sampling.
We further revisit the commonly used discrete diffusion objective and update it with empirical modifications. By adopting an improved ELBO with representation alignment (RepA) Yu al. (2025) loss, we optimize the training efficiency and substantially improve the generation quality of discrete generative models. Moreover, ReDDiT aligns with recent advances in discrete flow matching Gat et al. (2024): Shaul et al. (2024), enabling token refreshment during sampling without training post-correction models (Lezama et al. 2022).
METHODOLOGY
For self-containment, first review the DDM theory in Sec. 2.1
We then reformulate its diffusion
dynamics and introduce rehashing noise for ReDDiT in Sec. 2.2
We finally discuss connection and
comparison with other discrete diffusion models in Sec. 2.3
2

---

Published as a conference paper at ICLR 2026
2.1 PRELIMINARY: DISCRETE DIFFUSION MODEL
DDM defines forward process over discrete variables by gradually corrupting the image tokens to absorbing states (masks) through continuous-time Markov process. Assume that the data consists of tokens from a finite vocabulary V. x E V is a sequence of tokens (e-g-, an image tokenized into indices) with length L. We denote the clean data as Ft=0 (ro for short), and noise gradually as t + 1. DDM defines an absorbing token m E V. such that once a token is noised to m it remains unchanged. At the terminal time 1, 3y fully transits to m which means -|=1>/= m.
Let be the noise scheduler (a monotonically decreasing survival function that satisfies og = 1, 0g = 0) ). For 0 I 8 < t S 1, the forward corruption process is governed by continuous-time transition kernel ((x))) (ar)) at the i-th element,
adls
if : m,
m if x = x T m
x *
Orls:
Ott
0t
if x x m
otherwise
Denoting y as the transition kernel and Cat(:; T) categorical distribution determined by probability π, the corrupted data distribution at time / is written
I2 q (o), g( (mo) Cat(ar;; 04300 (1 (1-oap) --) )
22
The generative model learns the reverse process pe (x))) which denoises sample x, at arbitrary time & E (0, 1] to a less noised state z, at time s < t. Denoting 6(x). m) as the indicator function that only computes on masked tokens, and a² = dos, the learning objective derived as
LDDM -Ezo
at Σ 5(x), m) log (1rg
For linear scheduler, Eq. 3 is simplified via variable substitution Sahoo al. (2024) to an equivalent form, as
LDDM-linear E,, No.
Σ 5( m) log (o(()(a)))
For conditional generation, class information e (e.g., labels or text prompts) is introduced to the denoising model as additional input. Following classifier-free guidance [o & Salimans (2022) the model is trained with random drop of labels, and the prediction is interpolated at inference, as po (Ie, c) = (x Ø) + w- (pe(ze, c) - (x 2)), where @ the dropped label and w 2 controls the guidance strength.
(5)
2.2 DISCRETE DIFFUSION WITH REHASHING NOISE
The ordinal structure inherent in discrete data provides valuable inductive bias for designing transition kernels in diffusion dynamics. Prior studies Austin et (202T); |Campell et al. (2022) show that assigning higher transition probabilities neighboring pixel values forming discrete Gaussian-like noise noise-outpern outperforns the single absorbing state approach on pixel-level datasets like CIFAR-10. However, when using visual tokenizers, the structure of discretized latents is learned rather than pre-defined, making such ordinal assumptions inapplicable. This insight motivates us to extend conventional mask tokens to a set of indices, and reverse the diffusion path with noise rehashing. This design allows the model to optimize its embedding space during training, enhancing its ability to model flexible and data-drives noise structures. We visualize the learned distributions in Fig. 2 (right).
Reformulation. Given categories, let e, E RRd be its one-hot vector where the i-th value is 1. We denote & = fe; E Bd i 1,...,d d) as the basis of categorical distribution (known as d-simplex). and basis for absorbing states with capacity m: M (m, E Rm j. 1.....,,. ......... With subscript i, = 0 for pure visual or mask space, the sum of & and M can be denoted
V(amm) A {
e; Om
for I,..., d, j =0
((()) E Rd+m
V(6.i
(6)
0a L my, for j = 1...., m, i 0
3

---

Published as a conference paper at ICLR 2026
Space with Ordinal Structure:
Learned Latent Space (Ours): Yisualooaabulary
Discrete Rehashing Noise
Pioel (KGB)
mose distribution
Figure 2: Visualization of pixel and latent spaces. denotes the number of enriched noise indices. Note that the 3D t-SNE plot (right) is used solely for clustering with no isotropic features.
We further denote the subspace Ed, Min E (d.,)) which contain either valid mask tokens, Ed {v(L.0) V(d,m) -i= ...,,].. Mn. {v(o.) V(d.m) | = 1....... ................. (7)
To exploit visits across all the possible paths, we rewrite transition kernel defined by Eq. T as
if xt E -M: x's Mm if of = I x & Mm if x E Mm xy Mm otherwise.
atls
(8)
1/m,
0,
With above definitions, we reformulate the diffusion process of x as a transition from & to Mm. We train the model by feeding it with corrupted data, of which the distribution is inferred as It ~ Cat(ar a.To + ( where U(M is the uniform distribution upon M m
Rehash Sampling. To generate sequence of length L, the reverse process starts with U(M The subsequent latents 24, are generated by discretizing the reverse timeline to steps. We denote this schedule T1-++1 such that 7 = and TK+ e. with being an arbitrarily small positive constant. Let 8' indicate the i-th token's 's value, the reverse process is deduced as
1,
if x = zi, x & Mm if a E Mms
1-0
x E Mm if a & Mm, ot MM otherwise.
de - -
a (xr)) I-ac
as
0.
Comparing with MVTM sampler in Alg. [ our rehash sampler is shown Alg. 2 Our algorithm shares the similar idea MDLM Sahoo et al. (2024), but applies torch. mult maultnomme ainomial (Multnm. step 10) for low-discrepancy categorical sampling.
Algorithm 1 MVTM Sampling
Algorithm Rehash Sampling (Ours)
1: Inputs: label c, scheduler au, length L.
1: Inputs: label c, scheduler as, length L.
2: Settings: number of steps K, G(t)
2: Settings: number of steps K.
3: Initialize: I1 <- M
Initialize: Tq ~ U(M + TI:K
4: for k = 1 to K do
4: for k to K do
5: t K k+i
K-k
5:
t s
6: logits + fo(
K
# the rehash operation:
7: Pscore - logits G(t)
6:
It t where( E EMm, U(ML),
8:
Tpred + argmax (Pscore)
> Predict-all
logits - fo(
9:
Is where( = m Fpredd 2t.)
8:
p + Softmax (logits)
10:
Peonf t Pscure + G(t
8m(0] 10: Ipeed - Multuml
9:
+ a.-
T-o,
1-a.
I-o.
11:
mee & argsort [1: L (1-aa))
> w/ masks
12:
J, where(mee [m]. x..) Re-mask
11:
Is -- --weree(xz E ~M Tpred, Z+)
13:
end for
12: end for
14:
Return: fully unmasked sequence 2u
13: Return: fully unmasked sequence 20
MDLM uses gumbel- cumbel-max max for sampling, which may incur inaccuracy (Zheng et al 2024). Besides, we deliberately merge the prohabilities step 9 to keep overall noise sampling probability. as small values might he tuunatte, which also worsens sampling accuracy.
4

---

Published as a conference paper at ICLR 2026
The random nature of absorbing states inspires rehash operation: shuffle these tokens the beginning of each step by 2; <- where( E MMms (,,))),)))) Proof to Eq 9 is included in Appendix. B
2.3
DISCUSSION
X-axis: vocabulary index
Y-a0e::
probabiity distribution
sample statistics (with samples)
gFID
MV7M
MY7M
G03 4.52 -
Fails reflect true distributions under large vocabulary
MVTM (3.65)
max sampling [vocab=1024]
sampling [ooabma1122]
MV7M
ours
Gumbal Scheduls
ReDDIT (1.87)
Trades for diversity
Captures better distribution
25 Sampling Stops
35
300
Gumbel-max temperatura control (vccbb=8192]
Multinomial sampling (orabb51192)
Evaluation results at vacab=16384
Figure 3: Sampler comparison. Left: Gumbel-max is theoretically equivalent to our method, yet struggles to reflect the true distribution under limited sample passes. The multinomial approach captures the distribution more accurately. Right: our model achieves lower gFID across different sampling steps without tuning Gumbel-max, indicating more efficient and faithful sampling. a, b, refer three uniformly sampled G(t) set for MVTM sampling, See supplementary for experimental codes. (This figure best viewed color)
Comparison with MVTM. Masked visual token models (MVTMs) borrow the objective
CMVTM = -E,, 2o. SP
6( m) log Po(x) xx),
(10)
1
from masked language models Devlin et al. (2019) and predict on masked tokens with maximum likelihood. Besides the reformulated corruption (Fq. R) and reverse process (Eq. 9., ReDDIT differs in the following aspects: (i) the training objective (. which is derived from DDM, providing better theoretical and empirical results. (ii) can easily sample with a arbitrarily discretized timeline, while MVTM couples training and inference, restricting its sampling flexibility; (iii) the rehash sampler (Alg. 2 includes absorbing states categorical sampling with lower discrepancy, different from MVTM's predict-remask sampler with time variant intensity G(t) over Gumbel noise & (Alg. I Gumbel-max suffers from numerical inaccuracy (Zheng et al. 2024) and we noitice that it becomes worse large vocabulary (Fig. 1 with our reproduced results), which limits MVTM's potential.
Relationship to DFM. Discrete flow matching (DFM) Gat et al. (2024) introduces transition process based on masked tokens. Its training objective was initially designed as the masked token loss (TO), and evolved to time-weighted cross-entropy loss (Shaul et al. 2024) for generalized diffusion paths, which is similar to ours. The similarity enables a direct comparison between the DFM sampler and rehash sampler using the same trained model weights. We notice that generally requires more steps to reach optimal results, as the DFM sampler offers a refinement mechanism via token-wiss wise updates. Since the gradual decoding method shared, we can integrate certain DFM steps into our sampling procedure for refinement. This leads to ~ 0.1 gFID improvement on ImageNet- K. Refer Appendix for details.
"The logits corresponding to previously restored tokens' indices are manually set to infinity for both methods. that they will not he noised again in the following steps. This leads to implementation of any-order auto-regressive model (Ou 2024 if DDM's decoded tokens per step is limited

---

Published as a conference paper at ICLR 2026
3 EXPERIMENT
3.1 IMPLEMENTATION
Datasets. The experiments are conducted on ImageNet-IK Deng et al. 2009). which consists of 1000 categories, 1281167 images and are cropped resolution 256 256 for training. The generation quality is evaluated using Fréchet Inception Distance (FID) Heusel et al. (2017) and the Inception Score (IS) |Slimanns et al. (2017). FID measures the distance between the distributions of generated and real images in the feature space of pre-trained Inception network, while IS evaluates both the confidence and diversity of generated images by analyzing predicted label distribution. We compute generation FID (gFID.)) and IS$ on 50k generated samples.
Pre- Pre-processing. Following the setting in LlamaGen Sun et al. (024)). apply the ten-crop augmentation on images, and use pre-trained tokenizers to convert them to discrete tokens. We pick IBQ-f16 [Shi al. (2025) tokenizer as default for its scalable and promising performance in generation tasks, which uses a 16 x 16 downsampling ratio and converts a 256 x 256 image into 256 discrete tokens. The tokenizer has codebook with 16384 entries. The LlamaGen- (used in Tab. 2 and LlamaGen-f8 tokenizer [Sun et al. (2024) (used in Tab. D are also used for comparison with recent discrete generation methods. All tokenizers are used out-of-the-box without modification.
Representation Alignment. Recent study Yu al. 2025 has shown that the aligmment of in- termediate representations between diffusion transformers and vision encoders accelerates training convergence of diffusion models. Accordingly, the alignment is designed as a regularization term with À = 0.5. We extract diffusion transformer's 8-th layer intermediate feature h [n) (xx) and align with the original image's dinov2-b Oquab et al. (2023) encoded features (x The intermediate features are projected by a small trainable MLP hup- The sim(..)) computes the mean element-wise cosine similarity between embeddings,
Liast LDDM-linear + ARRepA CRepA = EE, el (im((fnne hhp (nl))
(11)
This alignment proposed for continuous diffusion models, and firstly validate that it's also suitable for training discrete models. However, from our observation, as training acceleration technique, RepA does not provide relative performance gain if training sufficiently for discrete latents. We only use RepA to improve training efficiency and probe the inner dynamics through training in Fig. 4
See Appendix F for detailed discussion.
Training and Evaluation. The proposed model is based on DiT Peebles Xie (2023) architecture, with reference to its discrete prediction version Sahoo et al. 22044). 2D-RoPE Su et al. (2024) and min-SNR Zhang & Sennrich] (2019) are applied for training efficiency. The model is optimized using the Adam optimizer with cosine decay. Class-conditional training is enabled using class embeddings and a drop-rate 0.1 for generation with CFG. Details are provided in Appendix
3.2 PERFORMANCE AND COMPARISON
We compare the proposed ReDDiT model with other generative models on the ImageNet- 1K x in Tab.[] The IBQ tokenizer is used for the default L and XL models. We also utilize LlamaGen-f8 with 128 noise capacity to evaluate its high-resolution potentials (noted as ReDDiT-XLss). We use linear increasing guidance following the common practice of Gao al. 2023).
Generation Quality. As shown Tab. ReDDiT achieves the best performance among the compared discrete models. It outperforms the baseline (MaskGIT [Chang et al. (2022)) with signif- icant margins (gFID: 2.13 vs 6. 18 and IS: 294.7 vs. 182.1) 1). also outperforms the recent DDM method Hu & Ommer (2024) and TiTok-S-128 Yu et al. (2024), which is extensively fine-tuned on quantized latents. Compared with continuous diffusion models, ReDDiT exhibits on on-par efficiency and performance, showing great potential for discrete generation.
'The gFID is used as the quality metric for generative models' performance, while rFID refers to the reconstruction quality of visual tokenizer.

---

Published as a conference paper at ICLR 2026
Table Performance comparison on class-conditional ImageNet 256x256. Look-up free quan- tizers are beyond the scope of this paper. (in gray) indicates that the decoder is fine-tuned for quantized latents. Wall-clock inference time relative ReDDiT-XL. is reported.
Tokenizer
Generator
Type
Model
#tokens
codebook
gFID
IS1
#Params
#Steps
Time
LDM-4 Rombuch al.
20224
4096x3
3.60
247.7
400M
250
Diff.
DiT-XL Peebles & Xie
2023
1024x4
2.27
278.2
675M
250
18
MDTv2 Gao ct al.
2023
1024x14
1.58
314.7
676M
256
18
SiT-XI. Ma el al.
2024
1024x4
2.42
238.5
675M
SiT-XL
Wang al.
1024x4
2.24
244.1
730M
Taming-VQGANN VQGAN Reser et al.
Reser
2021
1024
15.78
74.3
1.4B
256
RQ-Transfurmer Tuang el
2023
16384
134.0
ViT-VOGAN Yu
2022
1024
8192
175.1
1.7B
1024
AR
LlamaGen- -3W Sun
2024
576
16384
2.18
263.3
3.1B
576
20
RandAR XXL Pang
2024
512
16384
2.15
322.0
1.4B
88
VAR-430 Tian ct al, 20224)
680
4096
1.97
334.7
2.0B
10
0.5
MVTM
MaskGrT Chang et al. 2022
256
1024
6.18
182.1
227M
0.2
MaskGIL-XXXX Xin ct 2025
256
16384
3.71
303.4
1.4B
0.8
TiTok-S- 1T18--8112
4096
1.97
28.1.8
287M
1.6
ITM Hu & Ommer 2024
1024
16384
5,30
183.0
546M
100
ReDDiT-L (ours)
256
16384
2.13
294.7
346M
20
0.5
DDM
ReDDiT-XL (ours)
1.74
313.6
ReDDiT-XLg (ours)
1024
16384
1.61
318.5
675M
Table 2: Comparison of models with the same tokenizer. Reconstruction FID (rFID) indicates the tokenizer's reconstruction quality from its quantized codes. Dim denotes codebook dimension.
VQ Tokenizer Info.
Generator
Model
Identity
rFID
dim
#Params
gFID
LlamaGen-Layy Sun al. 2024
343M
3.80
RandAR-LAR Pang el al. 2024
LlamaGen-f16 Sun et al. 2024
2.19
8
343M
2.55
OurSpDMReDDD:T L.)
346M
2.41
IBQ-BAR R Shi et al. 2025
343M
2.88
IBQ-tokenizer Shi
2025
1.37
256
OurSpDM(R-DD:T-L.
346M
2.13
Efficiency. ReDDiT is born with the high-efficiency advantage of discrete diffusion models, com- paring with AR models. As shown in Tab. [ the inference time of ReDDiT is slightly longer than MaskGIT, while the performance is overwhelming. Without acceleration techniques, ReDDiT achieves a competitive performance which AR and traditional diffusion models use more than 250 steps to achieve. Notably, when armed with recent efforts that tailored KV-Cache Liu et al. (2025) for discrete diffusion models, ReDDiT's inference can be further boosted (not included in the main paper for fair comparison). See Appendix G for details.
Besides the major comparison, we also conduct an experiment that utilizes the identical tokenizer in previous AR models and validate our method's 's effectiveness. As can be seen in Tab. ReDDiT outperforms AR methods in generation tasks across different tokenizers.
3.3 DETERMINING NOISE CAPACITY
The reformulated discrete diffusion dynamics defines transitioning from & to Mm. Under this setting, it is necessary to empirically determine the optimal value of for fixed tokenizer with vocabulary size d, as the latent representations learned by VAEs are variant. We keep the training setup fixed and conduct experiments w.r.t. the noise capacity We also visualize LRepA which captures the degree of representation alignment within the transformer.
The alignment loss visualization shows that increasing the number of absorbing states introduces greater randomness, initially making predictions more difficult due to confusion with valid tokens. However, this gap narrows training progresses, and the model converges to similar alignment
7

---

Published as a conference paper at ICLR 2026
PIZEE MITH
200
8) Performance with different noise capacities.
b) zoom-in of representation alignment loss (0-4k steps.)
Figure 4: Comparison of noise capacities. We re-implemented training with the same training recipe. The generation quality and representation alignment trends visualized.
lower bound, suggesting effective representation learning across different configurations. As shown in Fig. 4 (left), generation quality improves with increasing noise capacity initially. The LlamaGen- f16 tokenizer achieves peak performance at 128, while the IBQ tokenizer performs best at m 1024. We attribute this to codebook design: the lower dimensional LlamaGen-fl codebook produces more compact latents, which also determines its smaller noise endurance.
3.4 ABLATION STUDY
Unless specified, all the models trained ImageNet 256 256 under the default settings for 100k iterations. We use constant guidance scale of 2.0 and 20 steps for generation, and report gFID I computed on 50K samples. Precision (Prec.1) and Recall (Rec.f) also reported.
Sampling Timeline. Recovering complete informa- tion from noise remains critical to diffusion-based mod- els (Lu et al. [022] Wu et al. (024). Recent work shows MVTM's non-linear scheduler for training is less critical when using high-capacity tokenizers. Ev- idence of time-ime inn-iinaai DDMs (Sahoo et al. 2024
Donoise process
K-k+1
Spuare
Lineor:
pt
Casie:
Shi et al. 2024) further supports decoupling training from sampling. In our experiments, a linear scheduler with constant signal-to-noise ratio decay, yields optimal training dynamics. Among the timeline discretization tested. Fig. the cosine schedule is employed for our ReDDiT model for best performance in Tab.
Arecor
Figure 5: Illustration of discretized timeline with K = 7. The slow-to-fast sampling works better than linear schedules.
Table 3: Ablated Design Choices. ReDDiT-L is trained for 100k iters. Final setting denoted in gray.
(a) General model design
(b) Sampling timeline
Train Config
Sample Config
gFID
Prec.
Rec.
Steps
Timeline
gFID
MVTM RepA loss
MVTM sampler
6.83
0.75
0.39
linear
7.18
Switch to objective
MVTM sampler
6.23
0.77
0.41
32
linear
6.43
same as above
Rehash sampler
5.75
0.78
0.45
20
arccos
5.04
2D-RoPF... min-SNR
Rehash sampler
5.51
0.79
0.45
20
square
7.39
above
DFM refine
5.40
0.81
0.52
20
cosine
4.91
General Design. We ablate the general choices of ReDDiT, which starts with a re-trained MVTM baseline methods (with LlamaGen-f16 and RepA for faster convergence as default) in Tab. The applied techniques like 2D-RoPE are also ablated with re-training. As shown, through the revised objective and our proposed sampler, ReDDiT alone improves FID by 1.0 compared to the baseline model. When combined with modern modification on transformers, can further improve the performance, showing its complementaryness with main-stream efforts.
8

---

Published as a conference paper at ICLR 2026
Impact of Rehash Operation. To validate the rehash operation for encouraging path diversity by resampling noise tokens, we compare noise capacities m 1 and m = 128. As shown in Table [ fixed absorbing state actually degrades performance com- pared to the baseline. While enabling random initialization improves gFID. the full rehash mechanism is essential unlock the model's 'ss capacity, confirming that active resam- pling is required prevent overly deterministic sampling.
Table 4: Ablation on Rehashing.
increasing capacity with a
Setting
gFID
(Baseline)
4.13
m 128 (Fixed State)
4.25
m = 128 (No Rehash)
4.07
128 (Full Rehash)
3.92
3.5 QUALITATIVE RESULT
presents representative class-conditional samples gener- ated by the proposed ReDDiT model. The outputs across diverse image classes consistently exhibit high fidelity and diversity. Please refer to Appendix H for more samples.
Class-conditional Generation. presettpreeents
Image Editing. We further demonstrate ReDDiT's editing capability in Figure 6 highlighting its bi-directional perceptual competence. Following MaskGIT, replace region of the input image with noise tokens and employ the same generation pipeline to inpaint the missing content, conditioned on class label c.
4 RELATED WORK
Diffusion Models. Diffusion models al. (2020): Song al. (2020) have emerged as a powerful class of generative methods that learn data distributions by reversing a gradual noising process over time. These models are primarily designed for continuous domains such as images images/Darriwll Dhariwal & Nichol (2021); Gao et al. (2023); Peebles & Xie (2023), defining forward process that transforms data xo into noise 21 Øz ~ (Vaazo (1 (t.) I) where a controls the noise schedule. The generative (reverse) process learns denoising model (e uz). often parameterized via neural network 0 to predict either noise or clean data.
Discrete Diffusion Models. Discrete diffusion has been previously governed by masked visual token models (MVTMs) Chang et (2022 2023): Gu et al. (2022); Yu et al. (2023 2024); |Hur et al.. (2024). This model leverages BERT-style [mask] token to corrupt the tokenized image sequence and trained the network with a simple cross-entropy loss on masked tokens, resulting in a score- prediction. It generates tokens in a non-autoregrressive fashion, by remasking the tokens with least scores at each inference as depicted in Alg.
(2024); Shi et al. which adapt the b Nie et al. (2025). moleculesShaul et al. (2024), and other discrete representations Austin et al. (2021 Nisonoff et al. (2024). Unlike MVTMs, the principled DDM and DFM mostly derive time- weighted cross-entropy loss supervise the training procedure and apply a gradual unmasking method based on probabilities.
Recent studies unlocked the principled discrete diffusion model (DDM) Sahoo et al. (2024) and discrete flow matching (DFM) Gat et al. (2024); ) (Shuul et al. (2024),
Markov chain theory, enabling generation text et al. (2024);
CONCLUSION
We proposed ReDDiT, a discrete visual generative model built upon a discrete diffusion architecture with novel noise designs and efficient sampling strategies. Our key contribution lies in the integra- tion of rehashing noise with samplers, which together ensure both diversity and low discrepancy throughout the generative process. By introducing rehashing noise, ReDDiT enriches the potential paths that latent variables can traverse during training, regularize training dynamics and enhances model's 's representational capacity. Extensive experiments demonstrate that discrete generative models achieve performance par with their continuous counterparts while offering top-tier efficiency. This study paves promising way for discrete generative modeling and offers fresh insights toward unifying visual and language generation a path leave for future exploration.

---

Published as a conference paper at ICLR 2026
Image in in-painting
Bulldog (label 245)
Tabby cat (label 281)
Figure 6: Class-conditional generation and in painting samples of ReDDiT on ImageNet 256 x 256.
Reproducibility We have provided key algorithms in the main text. Further implementation details are available in the source code.
ACKNOWLEDDGEENTS
The authors would like thank JIUTIAN Research for their computational resources. This work was supported by National Natural Science Foundation of China under Grant 62225208, 62521007 and CAS Project for Young Scientists in Basic Research under Grant No. YSBR-117.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In NeurIPS, pp. 17981-17993, 2021.
Jinbin Bai. Tian Ye, Wei Chow, Enxin Song, Qing-Gue Chen, Xiangtai Li, Zhen Dong, Lei Zhu, and Shuicheng Yan. Meissonic: Revitalizing masked generative transformers for efficient high- resolution text-to-image synthesis. ICLR, 2025.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. NeurIPS, 35: 28266-28279, 2022.
Huiwen Chang,
Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In CVPR, pp. 11315-1 13315-1125,,,, 2022.
Huiwen Chang. Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv: 230100074, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE CVPR, pp. 248-255, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pp. 4171-4186, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 34:8780-879 8790, 2021.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pp. 12873-12883, 2021.
Shanghua Gao, Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdtv2: Masked diffusion
transformer is strong image synthesizer. arXiv preprint arXiv:2303. 14389, 2023.
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. Advances in Neurul Information Processing Systems, 37: 133345-133388 2024.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to image synthesis. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp.
2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Berharr Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, pp. 6626-6637, 2017.
Jonathan Ho and Tim Salimans. Classificr-free diffusion guidance. arXiv preprint arXiv:2207.15598 2022.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851 2020.
Vincent Tao Hu and Björn Ommer. [mask] all you need. arXiv preprint arXiv:2412.06787 2024.
Mengqi Huang, Zhendong Mao, Zhuowei Chen, and Yongdong Zhang. Towards accurate image coding: Improved autoregressive image generation with dynamic vector quantization. In CVPR, pp. 22596-22605, 2023.
Jiwan Hur, Dong-Jae Lee, Gyojin Han, Jachyun Choi, Yunho Jeon, and Junmo Kim. Unlocking the Capabilities of Masked Generative Models for Image Synthesis via Self-Guidance. In NeurIPS, 2024.
José Lezama, Huiwen Chang, Lu Jiang, and Irfan Essa. Improved masked image generation with token-critic. In ECCV, pp. 70-86. Springer, 2022.

---

Published as a conference paper at ICLR 2026
Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grat tine-griined chinese understanding. arXiv preprint arX1v:2405.08748, 2024.
Zhiyuan Liu, Yicun Yang. Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching. arXiv preprint arXiv:2506.0629) 2025.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models, arXiv preprint 2rX00v2211.010995 2022.
Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden, and Saining Xie. SiT: Exploring Flow and Diffusion-Baar Based Generative Models with Scalable Interpolant Transformers, pp. 23-40. Springer Nature Switzerland, 2024.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji- Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025.
Hunter Nisonoff, Junhao Xiong, Stephan Allenspach, and Jennifer Listgarten. Unlocking guidance for discrete state-space diffusion and flow models. arXiv preprint arXiv:2406.01572, 2024.
Maxime Oguab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07/99, 2023.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint arXiv:2406.0 203.::2600..33366 2024.
Ziqi Pang, Tianyuan Zhang, Fujun Luan, Yunze Man, Hao Tan, Kai Zhang, William T Freeman, and Yu-Xiong Wang. Randar: Decoder-only autoregressive visual generation in random orders. arXiv preprint arXiv:24/2.01827, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In CVPR, pp. 4195-420s 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10674-10685, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022b.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. NeurlPS, 37:33336--1111111 2024.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurlPS, pp. 2226-2234, 2017.
Javier E Santos, Zachary R Fox, Nicholas Lubbers, and Yen Ting Lin. Blackout diffusion: generative diffusion models in discrete-state spaces. In ICML, pp. 9034-9059. PMLR, 2023.
Neta Shaul, Itai Gat, Marton Havasi, Daniel Severo, Anuroop Sriram, Peter Holderrieth, Brian Karrer, Yaron Lipman, and Ricky TQ Chen. Flow matching with general discrete paths: A kinetic-optimal perspective. arXiv preprint arXiv:2412.03487, 2024.
Fengyuan Shi, Zhuoyan Luo, Yixiao Gc, Yujiu Yang, Ying Shan, and Limin Wang. Scalable Image Tokenization with Index Backpropagation Quantization. arXiv preprint arXi::2412.0269. 2025.
Jiaxin Shi, Kehang Han, Zhe Wang, Amaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. NeurlPS, 37:1033311-103111 2024.
12

---

Published as a conference paper at ICLR 2026
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063 2024.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 0.xiv:22406.065555, 2024.
Alexander Swerdlow, Mihir Prabhudesai, Siddharth Gandhi, Deepak Pathak, and Katerina Fragki- adaki. Unified multimodal discrete diffusion. arXiv preprint arXiv:2503 20853, 2025.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. NeurIPS, 37:84839 84884443--88655, 2024.
Shuai Wang, Zexian Li, Qipeng zhang, Tianhui Song, Xubin Li, Tiezheng Ge, Bo Zheng, and Limin Wang. Differentiable Solver Search for Fast Diffusion Sampling, arXiv preprint arXiv:2505.21114, 2025.
Xiaoping Wu, Jie Hu, and Xiaoming Wei. Rdpm: Solve diffusion probabilistic models via recurrent token prediction. arXiv preprint arXiv:2412./83390, 2024.
Yi Xin, Le Zhuo, Qi Qin, Siqi Luo, Yuewen Cao, Bin Fu, Yangfan He, Hongsheng Li, Guangtao Zhai, Xiaohong Liu, and Peng Gao. Resurrect Mask AutoRegressive Modeling for Efficient and Scalable Image Generation. arXiv preprint arXiv:2507.13033 2025.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vegan. In ICLR, 2022.
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737 2023.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems, 37:128940-128966, 2024.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. In ICLR, 2025.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. NeurIPS, 32, 2019.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pp. 3836-3844 3847, 2023.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Ya Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409.02908, 2024.
A ETHICAL STATEMENTS
In the process of drafting this paper, Large Language Models (LLMs) were solely utilized for grammatical checking. No LLM involved in core academic work such conceptualization, literature review, data analysis, or argument construction this study.
As the human authors of this paper, bear full and sole responsibility for the paper's content, including the accuracy of research data, validity of academic arguments, integrity of research methods, and compliance with academic ethics.
13

---

Published as a conference paper at ICLR 2026
B DISCRETE DIFFUSION WITH REHASHING NOISE
Complete Definition and Deduction. We provide full theoretical discussion on the corrupted distribution and reverse process defined in the main paper.
Given d categories, let e; Rd be its one-hot vector where the i-th value is We denote fc; R = 1, ...... d the basis of categorical distribution, and basis for absorbing states with capacity mc M {m E I 1..... m). The sum and MM can be denoted as
e; + Om for 1, ................................ d, j
(d.m)
((Ls) E Rdtm V(I.) = Og S mj, for j = 1,............ m, 0
(12)
We further denote the subspace Ed, Mm.. E ((d,m) which contain either valid or mask tokens, as
Ed (v(2.) V(d.m) = ==....,d) Mm {v(o.)) V(d.m) | -..... m) To exploit visits across all the possible paths, for 0 < <1S,, we write the transition kernel a
(13)
1 1- af
if a E Mm x, $ Mms if of x x $ Mm
0
x
(14)
1/m,
if ot E Mms 2 E Mm.
0.
otherwise.
Proof the Corrupted Distribution. The presentation in the main paper simplifies the theory without specifying the transition matrix Q2 due to page limitation. We make detailed version with important yet basic matrix calculation in this section.
Let [d.m). (a..)) and (d,m) matrices in R(d=m)x(d+m) defined as
la
0 1 1xxm
0
0
I (d.mm)
((d.)
0
(d,
0
1m
(15)
where Ia is the d x d identity matrix, and 1m E R is a vector of ones.
The transition matrix Q11= E [[d++m) defined as:
Qrjs a (d.m) + 2 M (d.m) + (dd,m)
(16)
which can be demonstrated intuitively:
1 0
m 1 Oile
7 1-
m -ell
0
1-a1
Q1/=
0
0
0
T
711
T
0
0
7
1
1
xd
The corrupted data distribution is direct derivative of Eq. 16 by setting O:
04 + (1 ar) (d.)) (d.))
0,30 + a
0,30 + (1-a;) U(M
(17)
where U(M!) is the uniform distribution on M4.
'To maintain simplicity, we use a as and a
1-ax to denote transition rate for the corruption and I-os
process, respectively.
14

---

Published as a conference paper at ICLR 2026
Proof of the Reverse Process. To generate sequence of length L, the reverse process starts with I1 ~ U(M))) Let a @ b denote the Hadamard product between two vectors a and b, the reverse process is inferred as:
(xx, [x.) = Qejsaee O Q
(D3PM deduction)
Ft Q110000
[a (d.m) + (1 o1 M (d.m) + (dd
@ [0,00 + (1 (x2) (d.m)
x. (0020 + or a (d.m) To T (d.mm) vo)
1ajss (ddm) + (1 - 0 M (d.m) + (d, + [aggo + (1 as (d.m) vul
a to + (1 )x M (d.m) T0
(18)
We consider the separate cases: x; = zb and of Mm
Case 1. For x/ = x/j Eq. 18 is simplified as
q00 2 = x/)= = "1jaa O azui
anab ab
(19)
Case 2. For at Mm. have
[(1 afjo) M (d.m)a) M(
of + (ddm) O Ta,00 + (1 a0l-a,),e a, (d.m) Tol
g(x;) o E Mm)
(1-aa) M (d
(11 atj )a, M (d.m)att 3 op + т Tdd.)) - as)aat @ ((dm) nol
(1-a)) 'M (d,m)
(a. ar ( 3 xo + (1 - a, Tddmma O MT (d
(20)
(1 a+) M (d
Notice that 0/7. = 1 and we have
- as
0//
a E Mm st E Mm)
(21)
m(1 a²,)
q(=; Mm ot E Mm)
0///s
(22)
1 at
Combining case I with case 2, we have
af xúe xt $ Mm. x E Mmn
m
2 E Mms
(( | xt)
(23)
of Mm, of E Mms
otherwise.
Following MDLM's deduction, assume that the denoising network reconstruct zo perfectly, use pe (x²) to approximate this reverse process for complex sequences, and get
if bt 2t, xt Mm: if x E Mm, xt Mm:
a / m.
((2)(z))= =
(24)
o )p (xx).
if x² & Mm, o E Mm otherwise.
0,
15

---

Published as a conference paper at ICLR 2026
C COMPLEXITY AND LIMITATIONS
Our rehash sampler follows DDM-theoretic principles and is implemented using which internally relies on Gumbel-max (Zheng al. 2024). As a result, its asymptotic computational complexity is comparable to that of the original MaskGIT sampler. Practical efficiency gains, however, stem from two key factors. First, discrete visual tokens exhibit high correlation and redundancy, allowing multiple tokens to be predicted simultaneously and reducing the number of required operations compared with fully autoregressive or diffusion- approaches. Second, DDM decouples the scheduler during training and sampling (Sahoo et al.) 2024), enabling high-quality reconstruction from noise using arbitrarily defined timesteps. Using a cosine scheduler, ReDDiT achieves improved sample quality within fewer steps, as validated empirically.
Despite its effectiveness, the rehash sampler has limitations that motivate future work. The current rehashing strategy is applied every step, but the impact on diversity and convergence characterized, suggesting that optimized rehash frequency or intensity could improve performance. Additionally, integrating ReDDiT with dynamic token-update mechanisms in DFM frameworks may further enhance sample quality and diversity, particularly for complex multimodal generation tasks. These considerations highlight potential directions for extending and refining discrete diffusion sampling methods.
fully
D SAMPLING FROM LEARNED NETWORKS
Vanilla DFM
Rehash (ours)
Hybrid (ours)
7.5
7.0
5.5
4.0
10
15
20
25
TD
35
40
45
50
Sample Steps
Figure 7: Generation quality comparison with DFM methods. The experiments conducted ReDDiT-L with a constant classifier- free guidance (cfg 2.0).
We present a detailed version of discrete flow matching (DFM) sampler and discuss the integration of it with ours. Fig. 7 presents quantitative comparison of the vanilla DFM sampler, proposed rehash sampler, and a hybrid strategy that combines both approaches by incorporating selected DFM steps into the rehash trajectory. All methods evaluated using identical model weights, the training objectives are compatible due to their shared time-weighted loss formulation.
The rehash sampler exhibits stronger overall performance than DFM, especially in the 15-32 step range, where achieves low and stable gFID scores. This suggests that our modification enables more efficient decoding trajectories without sacrificing sample quality. The hybrid variant, which
16

---

Published as a conference paper at ICLR 2026
Algorithm 3 DFM Sampling Stepwise Pseudo Code
Require: Zy, labels, timestep step size At
1: Compute jump probabilities: ji 1- at. js f- at
2: Determine guidance scale w from schedule
3: Obtain logits loni logits uncomd via forward pass
4: logits <- logott + (logits logits
5: Pzo f- softmax(logits,,)
6: Sample &6 ~ Pzo using categorical sampling
7: Construct one-hot encodings: 8zo: 6z,
8: corrective 2e 6z,
Jt
9: u < 1: 6.zo
10: Overwrite in masked range with corrective terms
11: Mask entries already present in FG from u
12: Compute total transition intensity: À Σ u, elementwise
13: Draw Bernoulli mask: M ~ Bernoulli1 exp(-A))
14: For each masked position in M, sample from categorical u to obtain updated x,
15: return 2x
integrates only the middle and final steps the DFM update into the rehash schedule, also delivers consistent gains over the vanilla DFM, suggesting that partial refinement from DFM beneficial even when the majority of the trajectory is governed by our rehash dynamics.
By leveraging shared gradual decoding infrastructure, the hybrid approach enables practical integra- tion of DFM refinement into the ReDDiT framework with minimal overhead. As noted in the main paper, this leads ~0.1 improvement in gFID on ImageNet- K. reinforcing the complementary strengths of the two samplers. We leave the comprehensive study on the optimal integration of different samplers for future exploration.
E EXPERIMENT DETAILS
We provide detailed training and generation configurations for ReDDiT in Table 5 incorporates DINOv2-B for representation alignment, which requires computing image features during the forward pass (only activated during training). This introduces an overhead, making training roughly 1.2x slower than solely on discrete tokens. However, this additional cost is offset by faster convergence and improved stability, particularly in early training stages.
Our method
The use of quantized latents allows for larger batch sizes under limited GPU memory, making our approach more accessible for low-resource settings. Additionally, aligning discrete codes with semantic features improves the quality and diversity of learned representations. Overall, our design balances computational efficiency with model performance, making it practical choice for both research and deployment.
F DISCUSSION ON REPRESENTATION ALIGNMENT
Representation Alignment (RepA) introduces a similarity based auxiliary loss that aligns intermediate features of the diffusion model with pretrained DINOv2 embeddings. Although originally proposed for continuous diffusion, its effectiveness naturally extends to the discrete setting. In ReDDiT, discrete tokens are first mapped into a continuous embedding space, after which the architecture identical to transformer-biae -bssd continuous diffusion models (e.g., DiT). This means that the absence of continuous input does not fundamentally alter the structure of the model's s internal representations. However, discrete tokenizers restrict direct gradient flow from pixels to the codebook, making it more difficult for the model to organize high-level semantics during early training.
RepA provides external semantic scaffold that compensates for this difficulty. By encouraging the network to match DINOv2's robust visual features, RepA helps establish meaningful structure the latent representations before the denoising objective becomes sufficiently informative. Empirically,
17

---

Published as a conference paper at ICLR 2026
Table 5: Experiment details for ReDDiT on ImageNet-IK. Vari. refers to a time-variant growing guidance scale following MDTv2, which is common practice for diffusion models.
Setting
ReDDiT-L, (Ablation)
ReDDiT-L
ReDDiT-XI.
ReDDiT-XI.
Hidden Size
1024
1024
1280
1280
Transformer Block
24
28
Attention Head
16
16
20
20
Image Tokenizer
LlamaGen-f16
IBQ-f16
IBQ-f16
LlamaGen-f8
Codebook Size
16384
16384
16384
16384
Noise Capacity
128
1024
1024
128
Sequence Length
256
256
256
1024
RepA Latent Size
16x16
16x16
16x16
32x32
Batch Size
64
64
32
16
Global Batch Size
1024
1024
1024
1024
LR scheduler
Cosine Decay
Cosine Decay
Cosine Decay
Cosine Decay
Learning Rate
3c-4
3c-4
3e-4
4c-4
Minimal
le-5
le-5
le-5
Warmup Steps
2k
2k
2k
2k
Training Steps
500k
500k
500k
500k
Training Time
~ day
~ day
~/2 days
<-3 days
Generation CFG (Vari.)
1.0-5.0
1.0-6.5
1.0-6.5
1.0-5.5
Table 6: Acceleration of ReDDiT using response cache K,.
Config
Performance
Model
Steps
K,
Relative Speed
gFID
ReDDiT-L
1.33
2.28 (A = 0.15)
32
2
ReDDiT-XL
x 1..52
1.88 (A 0.14)
ReDDiT-XL
X2.17
1.83 (A 0.09)
64
4
ReDDiT-XL8
x2.56
1.71 (A 0.10)
we observe that removing RepA leads to an early training plateau, whereas with RepA the alignment loss rapidly decreases and stabilizes, indicating improved organization of semantic information. This behavior mirrors observations in continuous diffusion models and supports the view that RepA offers general mechanism for accelerating convergence, independent of whether the base diffusion process discrete or continuous.
G ACCELERATING REDDIT
Recent efforts on scaling and accelerating discrete diffusion models are making this generative paradigm practical than thcoretical attempts. We adapt the dLLM-Cache Liu al. (2025) design into our framework, which efficiently reuses intermediate computations without compromising model performance. Since the condition is modulated using AdaLN and introduces minimal calculation, we do not activate Kp (cache for prompt). As the decoding of visual sequence varies with time more quickly than language decoding, we implement the cache for response with small values like K, = 2 or 4, which means the K and V of transformer layer is updated every 2 or 4 decoding steps instead of per step. As shown in Tab. the inference speed is boosted up 2 times with minimal performance drop, which makes our largest model ReDDiT-XLss comparable to diffusion models with accelerated solvers.
H QUALITATIVE RESULTS
We provide more samples of ReDDiT's generation in Fig
18

---

Published as a conference paper at ICLR 2026
Figure 8: Class- conditionii generation samples of ReDDiT on ImageNet 256 x 256.
19


Reviews:
[{"review_id": "fpPfkN69G1", "reviewer": "Reviewer_daQc", "summary": "This paper presents ReDDiT, a novel discrete visual generative model that improves upon traditional discrete diffusion models by introducing rehashing noise. This innovation addresses limitations in absorbing states and sampling diversity, expanding latent variable paths to enhance both diversity and image quality. The model outperforms existing approaches, including MaskGIT, with significant gains in gFID and Inception Score.", "questions": "See the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of rehashing noise provides a novel way to enrich latent variable traversal, offering improved diversity and higher quality generation in discrete diffusion models.\n\n2. ReDDiT outperforms the baseline models (MaskGIT and DDM) on critical metrics like gFID and IS, with competitive efficiency when compared to continuous models.\n\n3. The model works effectively with large vocabulary codebooks (up to 16,384 entries), demonstrating its robustness even when scaled.", "weaknesses": "1. Although ReDDiT reports strong numbers on ImageNet‑1K, its effectiveness on more complex or diverse datasets is not discussed. Moreover, for a generation paper, the visualization evidence is quite limited, making it difficult to fully assess the qualitative improvements or appreciate the contribution beyond the single benchmark.\n\n2. The paper does not discuss potential limitations or failure cases of rehashing noise, especially under large‑vocabulary tokenizers or more difficult semantic distributions. Without such analysis, the robustness and stability of the approach remain unclear.\n\n3. How does the rehashing noise behave when scaling to far larger category spaces or higher‑complexity images? Is there a regime where the increased latent path diversity starts to hurt performance (e.g., over‑randomization), and if so, how is this controlled or prevented?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iwqMWFqogp", "reviewer": "Reviewer_gKak", "summary": "This paper introduces ReDDiT (Rehashing Noise for Discrete Diffusion Transformer), a novel discrete diffusion framework for high-quality visual generation. The authors identify two fundamental limitations in existing\nmasked visual token models (MVTMs): (1) the use of a\nsingle absorbing (mask) token, which restricts the expressivity of the noise process, and (2) reliance on\nheuristic, Gumbel-max–based sampling, which is unstable and requires heavy tuning—especially with large codebooks. To address these, ReDDiT proposes\nrehashing noise, which expands the absorbing state into\nmultiple randomized indices, thereby enriching the diffusion trajectory space. It further introduces a\nprincipled rehash sampler grounded in discrete diffusion theory, which uses multinomial sampling over softmax probabilities to ensure high diversity and low discrepancy without ad-hoc randomness. On ImageNet 256×256, ReDDiT\nachieves a gFID of 1.61, significantly outperforming prior discrete models (e.g., MaskGIT: 6.18) and approaching the performance of top continuous diffusion models, while maintaining superior inference efficiency.", "questions": "1.The paper successfully adapts Representation Alignment (RepA) from continuous to discrete diffusion. It would strengthen the work to include a brief discussion (even in the appendix) on why aligning discrete token embeddings with continuous DINOv2 features is effective—e.g., whether it stabilizes training by providing semantic priors in the absence of gradient flow through the tokenizer.\n2. In Figure 4(a), the x-axis should be labeled “Training Steps” (or similar) for clarity.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.Significant empirical improvement: ReDDiT delivers a substantial leap in generation quality for discrete models, closing much of the gap with continuous diffusion while preserving the efficiency advantages of discrete token-based generation.\n2. Insightful problem diagnosis and elegant solution: The paper clearly articulates the shortcomings of single-mask noise and Gumbel-based sampling, and the proposed rehashing noise mechanism is both theoretically motivated and practically effective.\n3. Clear presentation and thorough evaluation: The writing is accessible, and the paper includes comprehensive ablations, visualizations (e.g., Figure 1, 3), and comparisons that convincingly demonstrate the contribution of each component.", "weaknesses": "1.Minor performance gap with best continuous models: While ReDDiT achieves gFID = 1.61, the best continuous models (e.g., MDTv2) report gFID ≈ 1.58 under similar settings. Although the efficiency advantage is compelling, the paper could more explicitly acknowledge this small but notable gap as a limitation or future direction.\n2. Tokenizer-dependent hyperparameter tuning: The optimal noise capacity m varies with the tokenizer (e.g., m=128 for LlamaGen-f8 vs. m=1024 for IBQ), requiring empirical tuning. This slightly undermines the method’s plug-and-play appeal and generalizability across tokenization schemes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ckq02rUmbA", "reviewer": "Reviewer_DAHj", "summary": "The paper proposed a new rehashing noise approach for discrete diffusion transformer, which guarantees high diversity  and low discrepancy of the generation, outperforming baseline model by a large margin. Supported by solid theory and extensive experiments, the effectiveness and advantages of the method have been validated. The overall quality is satisfactory, while more explanation and interpretation in methodology are needed for clear presentation.", "questions": "1. Can authors analyze the complexity of ReDDiT sampler? Does the efficiency come from less operation needed?\n2. The discussion of the limitations of the proposed sampler  and how to improve it would be beneficial for future work of researchers.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposed a new sampling methods that facilitate efficient and diverse generation for discrete diffusion, and it has a good structure with clear motivation, distinctive contributions, and solid theory.\n2. The proposed method is well-supported by authors’ experiment results, either in tables other than figures.", "weaknesses": "1. The high diversity of the generation can be seen from qualitative examples, while what does low discrepancy mean in the proposed sampler? \n2. The methodology part is somewhat confusing. For equation 6, the definition of \\mathbf{m}_j is not clear. Does \\mathbf{m}_j indicate a absorbing token at the j-th position of the vector? Plus, I=0 and j=0 does mot make sense for index starting from 1. For equation 8, after rewriting equation 1, why does the transition kernel become \\frac{1}{m}? If it is due to m states of  \\mathcal{M}, then this problem comes back again to the definition of \\mathbf{m}\\in\\\\mathbb{R}^{m}: does \\mathbf{m}_{j} indicate m token at position j and elsewhere all 0? Is there only one absorbing state, which is m? Moreover, for equation 9, the symbol usage is not consistent, sometime it is $x_{t}^{I}$,  while sometimes it is $x_{t}$.\n3. More interpretation needed for figures. For fig 2, it is not clear how the learned distributions is better than ordinal ones. Is it because of a hyperplane between visual vocabulary and the rehashing noise?\n4. The author tested on well-known ImageNet dataset, while the performance generalization to other (more challenging) datasets is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i0wulRVuiC", "reviewer": "Reviewer_MWhn", "summary": "This paper tackles a central problem of discrete diffusion models, especially for the masked visual token models (MVTMs), which have relied on a single absorbing state to perturb the input discrete visual token and Gumbel softmax with relatively high sampling temperature. Existing sampling strategies introduce instability and degradation as vocabulary size grows. By introducing masking capacity, ReDDiT effectively improves the capabilities of MVTMs. The experimental results are promising.", "questions": "- Low sampling temperature is known to incur a multi-modality problem in non-autoregressive (NAR) sampling since NAR samples multiple tokens at a time, and their relationship is ignored. Since ReDDiT does not utilize sampling temperature and uses multinomial sampling, I wonder if ReDDiT also suffers from such a limitation, especially for the small or large sampling steps.\n\n- The rehash operation (Algorithm 2, line 6) is mentioned briefly. This step randomly shuffles tokens that are already noise tokens at the start of each sampling step. What is the theoretical and empirical justification for this? What happens to performance (gFID, diversity) if this line is removed? (i.e., what happens if the sampling becomes deterministic to the initial noise?)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- **Well-Motivated and Simple-yet-General Solution**: While discrete diffusion models (DDMs) mimic the continuous one, a single absorbing state is a fundamental restriction of the MVTMs, which limits the diversity of samples and enforces MVTMs to utilize high sampling temperature, which compromises sample quality.  By simply extending the absorbing state design, ReDDiT achieves promising results, paving the way for improving DDMs' generative capabilities.\n\n- **Clear Presentation**: The visualization in Figs 1 and 3 is good. The paper is well-structured, facilitating a clear understanding of the background and contributions of the paper. Simple notation with a theoretically grounded explanation ensures the readability and accessibility of the paper.\n\n- **Generalizability and Significance**: The proposed rehashing masking can be generally adopted by the masked generative modeling paradigm, bringing a significant advancement to the field.", "weaknesses": "- **Controlling the Stochasticity**: Controlling the stochasticity of the sampling process is critical, as some tasks, such as drug design, require exact outputs at the expense of diversity, while in some cases, like artistic painting, require high variance. However, the proposed method does not utilize sampling temperature, lacks such control. This may significantly limit the applicability of the method in various fields. I think the proposed method could also adopt a sampling temperature. In some research, sampling with high temperature equipped with an appropriate guidance scale could show superior sample quality and diversity simultaneously [1]. I think it is more beneficial if the stochastic sampling can also be adopted for the ReDDiT sampling pipeline. \n\n- **Justification for Exact Sampling in Large Vocabulary**: The authors argue that ReDDiT captures the distribution better with a large vocabulary size. However, I can't find any theoretical justification for this statement. I agree that the noise rehashing can express better diversity without heuristic temperature scaling. However, I wonder why it can capture the exact distribution under large vocabulary settings.\n\n- **Scalability to Sampling Steps**: In Table 1, the optimal sampling steps increase as the models' capacity grows. MaskGIT has shown the \"sweet spot\" of sampling steps, where large sampling can degrade the sample quality, conversely. The scalability analysis of the ReDDiT sampler with respect to the number of sampling steps is insufficient. The plot in Figure 3, showing gFID versus sampling steps, is hard to interpret, does not indicate the sampling steps, and fails to demonstrate a clear trend.\n\n[1]  Unlocking the Capabilities of Masked Generative Models for Image Synthesis via Self-Guidance, NeurIPS 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)