Title: Why Masking Diffusion Works: Condition on the Jump Schedule for Improved Discrete Diffusion
Venue: NeurIPS 2025
Accepted: yes
Keywords: Discrete diffusion

proteins

language

masking diffusion

structure
Abstract: Discrete diffusion models, like continuous diffusion models, generate high-quality samples by gradually undoing noise applied to datapoints with a Markov process. Gradual generation in theory comes with many conceptual benefits; for example, inductive biases can be incorporated into the noising Markov process. In practice, however, the consistently best performing discrete diffusion model is masking diffusion, which does not denoise gradually. Here we explain the superior performance of masking diffusion by noting that it makes use of a fundamental difference between continuous and discrete Markov processes: discrete Markov processes evolve by discontinuous jumps at a fixed rate and, unlike other discrete diffusion models, masking diffusion builds in the known distribution of jump times and only learns where to jump to. We show that we can similarly bake in the known distribution of jump times into any discrete diffusion model. The resulting models -- schedule-conditioned diffusion (SCUD) -- generalize classical discrete diffusion and masking diffusion. By applying SCUD to models with noising processes that incorporate inductive biases on images, text, and protein data, we build diffusion models that outperform masking.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Scheduled Conditioned Diffusion (SCUD), a generative modeling framework that first samples a transition schedule and then generates data conditioned on this sampled schedule. The approach aims to generalize discrete diffusion by decoupling the temporal structure from the transition dynamics. While the core idea of conditioning on a sampled schedule is novel and has potential, the current version of the paper lacks clarity in its mathematical formulation, contains questionable assumptions (particularly regarding the transition process and its independence from the schedule), and requires stronger theoretical and empirical justification. As it stands, the work is promising but not yet ready for publication.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n\nThe idea of conditioning the generative process on a sampled schedule of transitions is novel and conceptually intriguing. \nIt offers a new perspective on incorporating temporal flexibility into discrete diffusion frameworks. \n\n\nWeaknesses \n1. Conceptual and Mathematical Concerns Regarding the Transition Matrix $K$ \nThe definition and usage of matrix $K$ (as described in lines 129-136 and Proposition 4.1) raise several conceptual issues: \n- Once $\\Delta t$ is sampled, the first transition occurs at time $t+\\Delta t$, so it is unclear why $x_t$ and $x_{t+\\Delta t}$ could be the same. \n- The proof of Proposition 4.1 appears to implicitly assume that $\\Delta t$ is infinitesimally small, but this assumption is neither stated nor justified. \n- When sampling $x_{t+\\Delta t}$ from a categorical distribution parameterized by $K_{x_t}$, it is unclear why this process would be independent of the sampled $\\Delta t$, especially when $\\Delta t$ governs the time horizon of transitions. \n\n2. Incorrect Claim About Equivalence with Uniform Transition \n- The statement in lines 205–209 is misleading. Even under uniform transition schedules, transition and masking processes are not equivalent, even if the transition schedule is known. This is because under uniform transitions, the same token position can undergo multiple transitions, which breaks the claimed equivalence with masking. \n\n3. Notation and Mathematical Clarity \nSeveral mathematical definitions and notations are unclear or inconsistent: \n- In Figure 3, the role of $\\tilde{x}_{0, \\theta}$  is ambiguous. If this represents a model prediction, is it a denoised sample or a score? In discrete diffusion, the model typically estimates log-ratio scores [1]. \n- In Equation (5), it is not clear whether the schedule variable $s_t$ is discrete or continuous. Since it's drawn from an exponential distribution, one would expect it to be continuous. \n- In Proposition 4.4, it appears the authors directly model $q_{\\theta}$ . However, prior work [1] emphasizes estimating the log-ratio between adjacent states, which is not discussed here. \n- The notation $\\tilde{x}$ is used without definition in Proposition 5.1. \n- Proposition 5.2 lacks context and should clearly describe the connection to SEDD loss. The statement about equivalence needs to be explained more rigorously. \n\n4. Experimental Evaluation \n- The empirical validation could be strengthened by including comparisons against more competitive or recent baselines such as SEDD or LLADA, which are more aligned with the proposed approach in terms of model class and task setup. \n\n[1] Lou, Aaron, Chenlin Meng, and Stefano Ermon. \"Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution.\" ICML 2024.\n\nQUESTIONS:\nMajor Questions:\n- See weaknesses\n\nMinor Question\n- Why the shorthand for Scheduled conditioned diffusion written as \"SCUD\"?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to answer why masked discrete diffusion works so much better than prior discrete diffusion approaches.  It finds that the main cause of this is that the forward and reverse processes induced by masked discrete diffusion have the property that the masked diffusion reverse process inherently matches the forward process in terms of *when* a state should transition.  This is in contrast to other discrete diffusion processes where the learned reverse process may induce some error due to a mismatch in the transition times (as represented in the paper by a novel decomposition of the ELBO).  The authors show that by explicitly conditioning other classical discrete diffusion processes on an event schedule they are able to match or exceed the performance of masked diffusion models on a variety of tasks.\n\nSTRENGTHS AND WEAKNESSES:\nI very much enjoyed reading this paper -- I found it to be well written and clear and that its theoretical contributions were both novel and significant.  However, I have some concerns regarding the empirical evaluation of SCUD.  Specifically, I found that the validation for both proteins and text to be rather lacking (though the image results were quite good).  \n\nFor proteins the paper does not compare to the current state-of-the-art masked diffusion model for proteins in DPLM/DPLM2 [1, 2] which degrades the experimental quality.  It would also be nice to see how SCUD performs on unconditional generation as this is one of the main metrics one looks for to assess protein language models (see [1,2,3] for examples of these metrics).\n\nWith regards to text, the test perplexities on LM1B are significantly higher than those reported in MDLM [4].  I realize that MDLM was trained on more tokens than the SCUD models in the paper, but the paper's main claims would be significantly more compelling if it is shown that SCUD improves non-masked diffusion performance to comparable levels of performance on text as in MDLM or MD4 [5].\n\nOverall, I am currently leaning slightly towards rejection on this paper.  However, this is only because I feel the empirical validation is still a bit lacking in the current manuscript.  I believe the theoretical contributions in the paper are significant and I would be more than happy to increase my score if the authors can provide additional experimental results along the lines of the concerns I listed above.  Finally, I note that I am most curious about the text experiments and would weight these slightly higher than proteins.\n\nReferences:\n\n[1] Wang, Xinyou, et al. \"Diffusion language models are versatile protein learners.\" arXiv (2024).\n\n[2] Wang, Xinyou, et al. \"Dplm-2: A multimodal diffusion protein language model.\" arXiv (2024).\n\n[3] Hayes, Thomas, et al. \"Simulating 500 million years of evolution with a language model.\" Science (2025).\n\n[4] Sahoo, Subham, et al. \"Simple and effective masked diffusion language models.\" Advances in Neural Information Processing Systems 37 (2024).\n\n[5] Shi, Jiaxin, et al. \"Simplified and generalized masked diffusion for discrete data.\" Advances in neural information processing systems 37 (2024).\n\nQUESTIONS:\n1. How does your protein performance compare to current state of the art models DPLM/DPLM2?\n\n2. How well does SCUD perform on unconditional protein generation?\n\n3. How does SCUD compare to MDLM on LM1B (or MDLM/MD4 on OpenWebText)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies why masking diffusion models consistently outperform other discrete diffusion models, and proposes Schedule Conditioned DIffusion SCUD to improve general discrete diffusion performance. The main takeaway is that, the reason why masking diffusion success stems from its inherent property of knowing when to transition. And from the point view of diffusion process, masking diffusion models have their forward process matches their reverse, which other models don't have. \n\nThey then show that by combining SCUD with different discrete diffusion models, they can beat masking diffusion, achieving SOTA results on categorical image and proteins.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Good motivation, Fig 2 gives evidence of forward and backward rate mismatches in the existing diffusion models, which serves as a strong motivation and supports the main arguments \"masking diffusion is superior because its forward matches with backward while others don't\". \n- Clean logical and structural writing and well documentation, I really appreciate the writing, which makes it very easy to follow for me. Also, math proofs, experimental details are well documented. \n- The theoretical contributions connecting mask diffusion and classic discrete diffusion is rigorous and well supported. This work provides a novel viewpoint to understand the discrete diffusion models. \n- SOTA results. \n\nWeaknesses:\n- Computational efficiency. \nIn section 6.3 authors claim that \"within 10% runtime of each other\", this only applies to small-scale experiments, categorical image and protein cases. They didn't mention the performance on the most challenging experiment -- natural language LM1B dataset has no runtime information. \nThough asymtotic complexity appears similar, in practice the constant introduces a lot of overheads. I wonder what's the runtime of LM1B for both SCUD and classic discrete diffusion. \n- GPU vectorization issue. \nTake text case as an example, I'm not sure how you keep GPU vectorization while giving each token its own t. This could make GPU memory access in a very bad way -- random memory accessing and a lot of cache miss. \nTo me this seems like a mismatch between method design and GPU architecture function design. \n- Foundation model concerns. \nI can imagine SCUD could be a technique for diffusion LLMs. Since SCUD highly relies on the schedule, which is a dataset-specific schedule, this type of conditioning could be fundamentally conflict with the requirements of foundation language model for transferability and task generalization.\n\nQUESTIONS:\nSee Weaknesses. \n\n- If my understanding is correct, your argument that masking diffusion inherently \"knows when to transition\" is equivalent to saying masking diffusion denoisers are time-invariant (since you argue that the reason why mask diffusion is better is because it inherently knows when -- at which t to transition). I think previous papers also studied/mentioned about this time-invariant property of mask diffusion, may be good to add a discussion about how this. E.g. MD4 [1] and MDLM [2] both remove the timestep t in their final design. \n- I see in image and protein cases, you compared with some recent diffusion models, like MD4. However, in the language case, only D3PM with a different tokenizer is compare with and recent SOTA text diffusion models are missing. Can you explain why? I am confused here since I think natural language has a harder structure to learn compare to other cases, I think a fair comparison here is needed to show that the schedule conditioned diffusion can really help with the forward and reverse mismatching issue and thus improving the generation quality. \n- Another thing I noticed is the computational overhead, in the appendix D.6 you claim that in terms of asymptotic complexity, the SCUD and the classic diffusion are the same. I’m worried, though in theory people don’t care about constant difference, in algorithm engineering a big constant sometimes can be a huge problem (e.g. in engineering point of view, O(C*n) sometimes is worse than O(n^1.x)).\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}