ShortListing Model: A Streamlined Simplex Diffusion for Discrete Variable Generation
Authors: Yuxuan Song
Zhe Zhang
Yu Pei
Jingjing Gong
Qiying Yu
Zheng Zhang
Mingxuan Wang
Hao Zhou
Jingjing Liu
Wei-Ying Ma
Venue: NeurIPS 2025 poster ()
Keywords: Discrete Generative Model; Biological Sequence Design;
Primary area: machine_learning_for_sciences
TL;DR: We introduce the Shortlisting Model, a novel simplex-based diffusion model for discrete variable.

Abstract: Generative modeling of discrete variables is challenging yet crucial for applications in natural language processing and biological sequence design. We introduce the Shortlisting Model (SLM), a novel simplex-based diffusion model inspired by progressive candidate pruning. SLM operates on simplex centroids, reducing generation complexity and enhancing scalability. Additionally, SLM incorporates a flexible implementation of classifier-free guidance, enhancing unconditional generation performance. Extensive experiments on DNA promoter and enhancer design, protein design, character-level and large-vocabulary language modeling demonstrate the competitive performance and strong potential of SLM. Our code can be found at https://github.com/GenSI-THUAIR/SLM.

ShortListing Model: A Streamlined Simplex Diffusion for Discrete Variable Generation
Yuxuan Song
Zhe Zhang 13+ Yu Pei a Jingjing Gong'
Qiying Yu !2 Wei-Ying Ma
Zheng Zhang
Mingxuan Wang
Hao Zhou
Jingjing Liu
Institute of Al Industry Research (AIR), Tsinghua University
ByteDance Seed
3 Department of Computer Science and Technology. Tsinghua University (yxsong081 6, zwwzhan22, yupei wp wp]@gmail. com (zhouhao, Bewweiyinge@air... www.aaauii.iiiinnaaaaa.aaa www.rnghua..uuu.c.mmm
Abstract
Generative modeling of discrete variables is challenging yet crucial for applications in natural language processing and biological sequence design. We introduce the Shortlisting Model (SLM), novel simplex-based diffusion model inspired by progressive candidate pruning. SLM operates on simplex centroids, reducing generation complexity and enhancing scalability. Additionally, SLM incorporates a flexible implementation of classifier-free guidance, enhancing unconditional generation performance. Extensive experiments on DNA promoter and enhancer design, protein design, character-level and arge-vocabulaay language modeling demonstrate the competitive performance and strong potential of SLM. Our code can be found at tht //gi thub. com/GenSI THUAIR/SLM,
Introduction
Autoregressive models such as large language models (LLMs), although having achieved re- markable success in text generation [Achiam al. 2023 Brown et al. 2020), struggle over lasks that lack an intrinsic sequential-orderin inductive bias, such as DNA design (Avdeyev
1) Berx[NNNaff.1 9- a NN.tsf () a00 1 10
ct al.] 2023 Stark et al. [224], protein sequence
design [Wagg al.] 2024 Lin al. [0033 and molecular graph generation |Vignac et al. 2022).
Consequently, there is growing interest in de- veloping new paradigms for discrete variable generation, such as diffusion-baan
[Lou al.)
Sahoo et al.) 2024 Shi et al.) 2024) and flow- matching-based Gat al. 2024 Davis al.,
Bernid....
xf..)
Hlt) alt
2024
Cheng et al. 2024 methods.
Figure 1: SLM's forward and reverse pro-
Recent discrete generative models generally classified into two main categories based on their operational spaces: discrete-space models and continuous-space models. The former, specifi- cally discrete diffusion model [Lou al., Xu al. 2024 mimics continuous diffusion processes using substitution or masking operations to decompose information, which has shown impressive
cess. Comparisons between MDLM and DP3M- Uniform is located in Appendix B.T
Equal Contribution. Correspondence to Zhou (zhouhao@ @ air.tsinghua.edu..n)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

performance for discrete generative modeling, However, these discrete counterparts differ funda- mentally from original continuous diffusion models (Ho et 2020]. where the smooth and gradual information transitions intrinsic to the generation process are key to their success. On the other hand, continuous-space models map discrete data into continuous representations. While benefiting from continuous properties, capturing the geometric structure and adhering to constraints via continuous generative models introduces new challenges. In this context, simplex-basee approaches [Cheng al. 2024 Davis al.) [024] Graves et al. 2023 offer balanced solution by representing discrete data on the probability simplex, which naturally adheres to the fundamental properties categorical distributions.
However, existing simplex-bar based approaches often rely on intricate operations define trajectories over the entire continuous space. For instance, Statistical Flow Matching(SFM) [Cheng et al.) 2024] and Fisher Flow Matching [Davis al., 2024] define geodesics based on sphere map and the Fisher- Rao metric. The training also incorporates Riemannian optimal transport. Similarly, Bayesian Flow Networks (BFNs) involve heavy mathematical derivations and change-of- -vrriable techniques to define simplex trajectories through Gaussian-formed count variables. Despite mathematical rigor, their demanding complexity limits scalability in large-scale generative tasks.
In this paper, we aim to preserve the core principle of simplex- methods, gradual information growth, while exploring simpler yet effective alternatives. We propose viewing discrete variable generation as progressive candidate pruning, starting from the full category set and iteratively narrowing down to a single choice. We term this approach ShortListing Models(SLMM). Formally, shortlisting models reside within the diffusion framework, trainable via the variational lower bound (VLB). In contrast to existing simplex-basee
methods using vocabulary-level MSE loss [Graves) et al. 2023 Cheng et al.) 2024 Davis al., [024], our approach employs a simplified cross-entropy objective, effectively mitigating vanishing gradient issues and better handling uuge-vocabulary settings. As illustrated in Figures | and 2 model effectively reduces degrees of freedom by modeling transitions among the simplex centroids rather than the entire simplex. Additionally, shortlisting models offer flexibility for adaptations such as classifier-free guidance.
We comprehensively evaluate the proposed approach over various discrete generation tasks and benchmarks, including char-level language modeling. large-scale language modeling, DNA promoter and enhancer design, and Protein sequence design. Specifically, we achieve strong performance among non- aonauuoorerressvve methods on text8 and obtain competitive results on Open Web'eet,, where previous simplex-based approaches had difficulty generating reasonable outputs. DNA design tasks, our non-guided variants achieve Mate-offthe-al (SOTA) results. Furthermore, with classifier-free guidance, our model attains stronger performance while remaining less sensitive to hyperparameters. Additionally, our 38M-parameter shortlisting model can design proteins with enhanced foldability, fitness, self-consistency and diversity, surpassing the larger, well-known ESM2-150M model [Lin
et al. 2022
2 Preliminary
2.1 Definition and Notations
We encode a discrete variable with K distinct categories using one-hot vectors e = [e1, ....... ex]T. In each vector, only the i-th element e(i) signifies the inclusion of the i-th category, while all other elements are zero.
Definition 2.1. A candidate for K categories is defined as a binary-valued vector c T [c CK. where each e; {0,1), and the vector has at least one non-zero entry, i.e., 1'c>0.
The candidate-set variable encodes the selection status of each category: c(i) 1 indicates that the i-th category is included, while c(i) = denotes its exclusion. Specifically, there are two distinct instances of the candidate variable: e is an all-one vector (l,.... , 1)) that represents maximum candidates setting, i.e. K categories are selected; one-hot vector is another special case which represents minimum candidates setting, with only category included.

---

Figure 2: Pathological behavior of SLM simplex with K 5(A*). Each vertex represents one of the categorical targets while the trajectory of the white point serves as probability path in sampling. Note that the trajectory of shortlisting model could be jumping among the centroids of subspaces in simplex space.
2.2 Diffusion Models
Shortlisting model is a variant of diffusion models, which can be viewed as latent variable models where the latent variables form a Markov chain. Specifically, for diffusion model with the sequence latent variable Z1:T = 31. ... Ir, the implied density function pa holds the following Markovness by definition: (T F1.T) (= x1 (2 -
To learn this latent variable
model, a carefully designed constant variational distribution q (#1:7 zD)) = I[ 9 (ær, x(-1),
referred to the 'forward process', is involved. Based on the variational distribution, the diffusion model is generally trained with the following variational lower bound [Austin et al., 2021 Ho et al. 2020
Lvlb (ao)) DKL (xT wo) lpp (2rr
o lmma) [DKL lq (xt-1 æt va) Ilpo (xt-1 xx)]
11-1
-Ba ( [log po (zo x1)]]
Lo
Here ((1o)) refers to the data distribution.
3 Shortlisting Models
Inspired by progressive candidate pruning. we effectively translate the generation of discrete variables into category selection process, which begins by considering all categories in the vocabulary as potential candidates and progressively narrows down the options until reaching a final one-hot representation, indicating a single category. This section introduces the detailed components of the proposed shortlisting model as well as the training and sampling processes.
3.1 Forward Candidate Appending
We design forward candidate appending process the space of candidate set as introduced in Definition. [2 For any discrete variable x, its one-hot representation (as special case of the candidate set) considered the initial step, denoted as xf. For the last step, xÿ, we make into an all-one vector (1). Then we seek the following Markov chain interpolation x x between xij and x9, which satisfies:
x
VogieJST
-1'xx bf
(2)
Recall x" is a binary-valued vector, hence the above condition essentially indicates the possible categories implied by the candidate set of early steps are strictly scooped by later steps. We propose using multivariate Bernoulli distribution to model the forward process over the candidate-ser variable, denoted x Bern(() where & is K dimensional vector representing the parameters of the distribution. To control the noise level, we introduce n²t) scheduling function over the candidate numbers, where < nll) < K. By our definition, r(()) monotonically increasing function from time step 0 to T, designed to gradually perturb the signal. Intuitively, na(() can be viewed as controlling the number of ones the vector x;, representing the number of possible categories at time t. To satisfy the condition in Eq. 2 we (0) = 1 and n(T)) = K, and define the

---

transition probabilities from t 1 to f as:
(1-1+(1-x--1 x(-1) n(t) nlt 1)
[xx] x Bern xi-i+ +1-
k-n(t (1
Proposition 3.1. With Eq. 3 as the transition probability, the marginal distribution is defined as:
x + (1-xx;11 m((() -1 m(()-1
xxx))x)) = Bern
x
K K-1)
(4)
and corresponding posterior distribution qxx xx; x(i also lies in the form of Bernoulli distribution. the analytical form of which is (1. (1.22 2):
nlt - 1) - 11111
q(x; 1x x
Bern + x
n(t)
Here @ stands for the Hadamard products befween two vectors.
Detailed proof can be found in Appendix
3.2 Reverse Candidate Pruning
The reverse process implied by pe (xf x corresponds to the progressive candidate pruning process.
We follow previous literature [Austin et al. 2021 Sahoo et al. 2024 to parameterize |xx)),
by combining a neural network(f) predicted x based on x² and the formulation of the posterior in Eq. 5
po(x;- (x9) x(x NNo(x,,
[NNo((9, + NNo(xE, (t)
mll 1) -
=Bern
(6)
Here NNo f) refers to probability distribution over K-dim, e.g., outputs after softmax. Each parameter in pe (xf-1 xx] can be viewed interpolation between the constant value n(t-1)-1 and 1, weighted by (X,,
n(t]-1
Moreover, we propose incorporating the property of the forward process where x is strictly
contained within xi. expressed as [xq]² x 1" x"x This property integrated into pa- rameterization by ensuring that NN, (x7, t) has non-zero values only for categories within x;, satisfying [NN₂ (xf.t)" t)]" (1 xf) 0. Practically, such condition can be satisfied by ading -00 to the logits before the softmax operation. The prior distribution is set as the all-one vector, i.e.,
(x(p)) (ern(1).
3.3 Training Procedure
We insert the formulation Eq. Eq. and in the Variational lower bound in Eq. to derive the final objective for shortlisting models. We start with the first term Tyr. As mentioned above, (x(x)) (ern(1). We put the time step into Eq. 4
g(x9 Bern xi + (1 x (T) -
K -1
1
Bern x + (1- x
Bern
K
The first term LT in Eq. becomes: LT = EE ((=))) DKL [Bern(1)|] (ern(1)] = For the last term
La, with n(0) the po (x Eq. 6 can be expressed as (x((())f) Bxr3 Then Lo is expressed as:
To - ((flaa))) po (x) =f)]
=- --Bg(uflug)) [log (NNo(xf. 2g) + log NNo(x{, -xg),
|xf x >
0.
|xf x5 = 0

---

Here (-) denotes the inner product. Next, we focus on the term -リ-1, and for simplicity we use
predg (xi) shorted notation for [NN 1) (I-NNo(XX,, m(t-1)-1 -(1)-1 @xf), correspondingly,
gd(x{) for x + [[1 - x() 3 x m
L E (x) [Dr.. [Bern(gd(xF)) Bern(pred, (x]])rdddd())]]]
(7)
The KL divergence between the Multivariate Bernoulli distribution is extended as:
DKL (Bern(gg((xx)) (predp)))
gd (xf)
- gd (xc)
Σ
(x{" log
(1 gd (x²)) log 1 pred)(x? (xq)
(8)
pred)
G(N))>O
Here, we use the superscript i to denote the i-th dimension.
Mitigating Gradient Vanishing. We observe that directly optimizing the KL divergence of multi- dimensional Bernoulli distribution, as in Eq. 7 can lead to optimization failure, with the process stalled from the beginning. This issue is likely due to gradient vanishing. where the gradients become too small to drive effective parameter updates. We formally investigate this issue in the following.
Taking dimension i in the K dimensions and (xf)² >> 0 example, the gradient towards the pa- rameter 0 is Vo DKL. (Benn.Beeegddd(X (xf)] (pred, (xf)) = - (x{( 1-gd" (x{)
(xf) I-prod Vopred; (xf).
And we denote the above gradient term as V VoDKL for simplicity in the following. Recall that gd' (xf) and pred)(x;) are both interpolations between and n(t-1)-1 as discussed in n(t)-1
Section 3.2 we have: n(t-1)-1 I gd' (xf), pxed)(xx))) Σ 1. Consider the common situation a(t)-1
when (x 1, and the network prediction (xf, t) holds a non-zero value. The norm of =(f)-1
(xf,t)
weight satisfies that:
(xf)
1-gu"(xx;) (xx+)
n(--1)-I- Combining with Vepred;
(x5])(
(xpr
x(t)-n((-1) V (xf, t). we could obtain the following bounded condition over the gradient norm -(t)-1
of Di
| V 112 = gd (x;)
gd' (xf)
Vopred
pred (xf) 1 - pred (xF)
n()) 1 n(1) -n(l - 1) VONN (xf 0)11 nlt n(1) -
n(() n(t -1)
< n(t 1)
(oNN +)12
(9)
Note that the bounds involve the gradient term VoNN t), which computed directly from the softmax outputs (without applying the logarithm). However, taking gradients through the softmax function directly often leads to vanishing gradients, particularly high-dimensional settings where the outputs NN: (x5 t) can be very small initially. We additionally provide formal illustration in Appendix A.2
To mitigate this issue, propose scaling the gradient in Eq. 9 by (N)(xf.1) consistent with the widely adopted log-softmax optimization. Surprisingly, this corresponds directly to the following simplified objective:
n(()) n(t 1) (log NNo (a: (), ag))
weight -B (=ef)=)))))
(10)
n(t)
Though derived with heuristic intuition, formally show in Appendix A.3 that this reweighted loss (Eq. [0) can interpreted as a reasonable approximation.
Moreover, we can derive an even simpler training objective by removing the weight, analogous to the simplified loss used in [Ho et al. 2020), which may provide different practical properties:
simple - (=f)aa)) [(log (2 f), x)) L
(11)
(-1
5

---

A
Foidbilty
Diversttys
E
Structure sequence
BvoOHt DAON
100
Exocett D3WM
pLDOT1
B
D
Sel-Connstencyy
F
Structure sequancce
SUN
PLOOTT
Figure 3: Quantitative Performance on Protein Sequence Design SLM compared to baselines: (A-D) PLDDT(A), Progen2-nlI(B), scPerplexity(C) and inner-TM(D) scores for sequence sampled from ESM1-43M. ESM2-150M, and the following 38M models: EvoDiff-OADM. EvoDiff-D3PM, MDLM, UDLM, DFM-Mask Mask, DFM-Unifomm, SLM. (E-F) The joint distribution of pLDDT and scPerplexity from SLM model and Diffusion Models(E) and Discrete Flow Matching Models(F).
The above objective is essentially the Cross-Entropy loss between the network prediction and the original data sample. Unless otherwise specified, we use I, simple for experiments on Open WeenWeeText and I, weight for all other experiments. The like)ihood(ELBO) is strictly evaluated using the original ELBO defined in Eq. 7
Candidate Set Size Scheduling. Another important component of the framework is the scheduling function over the size of candidate set, i.e. n(t). It is noteworthy that n(t) is not restricted to integer values; rather, it can take any real value within the interval [1, KJ. We take a similar intuition
from Graves al. 2023), by considering the normalized vector _x, as the probability of distribution over vocabulary, then we expect the entropy of the distribution to increase linearly from to T. Note the expected ones of x is exactly n()), hence the corresponding entropy of the aforementioned distribution is log n(t). Then we can design scheduling function as: (t) = +.
xf
3.4 Sampling Process
The sampling process of shortlisting models can be directly conducted with ancestral sampling based the learned po(xf_1/x?) with xç. Bern (1) the starting point. The full pseudocode for training and sampling is provided in Appendix B.2 To ensure the candidate set always contains least one candidate (i.e., x j 0), we empirically the dimension with the largest Bernoulli parameter to 1 when the sampled vector is zero vector.
Classifier-free Guidance. We show that this simplified formulation offers flexibility to implement classifier- free guidance with an extra class conditiond shortlist model. Denoting the output of the unconditional model at timestep
NNg (xf, tt K) and the conditional model as NN, (xf, cls).
Here cls E [0, K 1] denotes the class label. The reverse process based on classifier free guidance
can be obtained as: Po (x{ (x{) Bern ([NÑo(x{,)) + (1 (Ñ, 1)) n(t-1)-1
CFG
c x
-(()-1
Here the NŇ₂ is as: NÑa (x,, t) NNN (xf, t, cls) + (1 NN (xf, t, K). When y > 1, there
can be negative number in NÑg. Following (Stark et al. 2024 we project the value of NŇ, back to
the simplex based on (Wang and Carreira- Perpiinn 2013
4 Experiments
In this section, we evaluate our shortlisting models across various discrete data generation tasks and benchmarks. These tasks include language modeling and biological sequence design, the latter of

---

which is especially well suited for von-attoregressive models and for demonstrating the potential of our proposed method.
4.1
Language Modeling
Text8: Firstly, conduct experiments the text8 dataset [Mahoney 2011] with vocab size of 27. Bits-per-characeer(PPC)) is reported based on the Equation. Details on the dataset can be found in Appendix C.I.I
The results are reported in Table and additional generated samples are provided
in Table
We compare shortlisting model with baselines across various categories: (1) autoregressive mod- els: Transformer AR Vaswani et al., 2017]. AR Argmax Flow [Hoogeboom et at. 2021al AR Discrete
Flow Flow[Tran TTan al. 2019| any-order autoregressive models: ARIDM[Hoogehoon)
et al. 2021b
MAC [hihh et al.
2022 (3) embedding-spee
continuous diffusion models: Plaid Gulrajani and (4) advanced discrete diffusion models: SEDD[Lou et al.
Hashimoto 2024
2023), MDLM Sahoo
et al.
2024
UD M Schiff et al.
2024). D3PM variants Austin et al. 2021 and (5) simplex-base
approaches: BFN BFN Graves et al. 2023 SFM (Ceng et al. 2024
As aforementioned, we report the BPC of both the shortlisting MSdM trained with the simple in Eq. [1 and with the L weightt in Eq. 10 As demonstrated in Table [1 even with the simplified objective, the proposed approach achieves competitive performance compared to other non non-autoregressive approaches. Moreover, the reweighted formulation further boosts den- sity estimation performance by better aligning with the original ELBO, discussed in Sec- tion
Table 1: Bits Per Character (BPC) on Text8 Test Set.
Category
Method
BPC (L)
Transformer AR
1.23
Autoregressive
AR Argmax Flow
1.39
AR Discrete Flow
1.23
Any-order Autoregressive
ARDM
S 1.43
MAC
1.40
Continuous Diffusion
Plaid
1.48
Open Web We further investigate the challenges and potential of simplex-ber based ap-
Mult. Diffusion
S 1.72
proaches in large vocabulary settings, over the
D3PM Uniform
1.61
WeenWeeText [Gokaslan and Cohen 2019) dataset with vocab size of 50527. Detailed results and discussions are provided in Ap- pendix C.1.2 and Table. 5 highlighting that while our SLM still lags behind advanced au- toregressive models in density estimation, achieves competitive generation performance and significantly outperforms existing simplex- based methods.
UDLM
S 1.59
Discrete Diffusion
D3PM Absorb
S 1.45
SEDD Absorb
S 11.41
MDLM
1.39
MD4
S 1.37
BFN
1.41
DFM
S 1.41
Simplex Approaches
SFM
1,39
SLM simple)
1.44
SLM(1. neight
1.38
4.2
De novo Design of Protein Sequence
In this experiment, we focus on the core task of unconditional protein design, and examine various protein properties demonstrate SLM's superiority in this task.
We compare SLMM against three groups of existing methods: (1) Autoregressive models (AR); (1) Masked language models (MLMs), specifically ESM1 [Rives et al. 2019 and ESM2 Lin
Baselines:
ct al. 2022] ; (2) Discrete Diffusson Models, represented by two versions of EvoDiff Alamdari et al.
2023 WWWDDI--ADD..EODDI---DSA
MDLM et al. 20244 and UDLM al.
2024]; (3) Discrete Flow Matching Models et al. 2024]: DFM-Mask and DFM-Uniform.
Further information of these baselines in Appendix C.5
To demonstrate SLM's effectiveness in protein sequence generation, we evaluate four key properties:
(1) Foldability: structural plausibility predicted by ESMFold [Lin et al. [022]; (2) Fitness: scores
predicted by ProGen2-xlarge (Nijkamp et al., 2023]: (3) Seff-Consistency: alignment between sequences folded by ESMFoold and inverse-folded by ESM-IF [Hsu et al.) 2022]: and (4) Diversity: pairwise inner-TM scores among generated samples. Detailed metric definitions are provided
in Appendix C.5.I
As shown in Figure 3 SLM surpasses all basclines across all metries and
7

---

Table 2: Conditional generation over promoter design. BFN results are from our experiments and the other baselines from Davis et al. 2024
Table FBD metric for sequence generation under
two datasets. CFG refers to Classifier-Free Guid- ance.
Model
MSE(J)
Model
Mel FBD())
FB FBD(L)
Autoregressive
1.03-=0.00
Random
619.0++0.8
832.4+0.3
Bit Diffusion (bit enc)
0,041
Autoregressive
35.4
25.7=110
Bit Diffusion (one-hot)
0.040
Fisher-Flow
27.5+2.6
3,8=0.3
D3PM 3PPM-uniform
0.038
Dirichlet FM
5.340.5
15.1=0.4
UDLM
0.030=0.000
BFN
3.3+0.1
10.8+0.6
MDLM
0.028=0.001
SLM
2.2+
4.4=0.2
Dirichlet FM
0.034=0.004
BEN CFG
2.3+0.1
2.3=0.2
Fisher-Flow
0.029=0.001
Dirichlet FM CFG
1.9+0.4
1.0=0.3
BFN
0.0405=0.0000
SLM CFG
1.4:0.1
1.0=0.1
DDSM
0.033
SLM
0.0265 0.0006
achieves competitive performance compared to ESM2-150M [Lin et al. 2022]. demonstrating strong generalization and robustness under restricted vocabularies and complex data distributions.
4.3 Design of DNA Sequence
In this part, focus on the roles of promoters and enhancers, and evaluate SLM in this context. Following prior work, we set the language model to use 500 NFE for enhancers and 1000 NFE for promoters. For other models, Non-CFG models 100 NFE, and CFG variants 200 NFE.
4.3.1 Promoter DNA Sequence Design
We follow the setting in previous work DDSM (Avdeyev et al., [203) to generate DNA promoter sequences conditioned on the promoter profile.
STUP
Data: We a dataset of 100,000 human promoter sequences, each 1024 base-pairs long, extracted from the Human Promoter Database (Hon et al.. 2017). Each sequence is paired with a CAGE signal indicat- ing transcriptional activity at each position (Shiraki et al., 2003 Forrest et al.) 2014). Sequences from chromosomes and form the test set, and the re- mainder are used for training.
Figure 4: Performance of SLM under differ- ent CFG factor y for unconditional enhancer design.
Baselines: We compared the SLM method with: flow matching methods including Dirichlet FM [Stark
al. 2024 and Fisher-Flow [Davis et al. 2024 (2)
autoregressive language model that generates base
pairs; (3) Bayesian Flow Networks (BFN) [Graves et al. 2023]: and (4) other discrete diffusion
methods including two implementations of Bit Diffusion CChn et al.) 2022a], D3PM [Austin et al.
2027, MDLM [Sahoo et 2024.. UDLM [Shhff al. 2024 and simplex- based DDSM [Avdeyev)
et al.
2023
The regulatory activity of the sequences given by Sei, model that predicts the regulatory potential of the sequences [Chen et al.) [2022b). We report the mean and standard deviation of MSE between the generated sequences and the target. Our MSE values measured under the same Sei model as in previous works. As shown in Table 2 our SLM method achieves the lowest MSE, with smaller standard deviation well.
Results:
4.3.2 Enhancer DNA Sequence Design
We also evaluate SLM for enhancer generation, following DirichletFM Stark et al. 2024

---

Table 4: Ablation study on loss function conducted on DNA sequence design.
Mel
FB
promoter
Loss type
FBD(L)
PPL(.)
FBD(L)
PPL(()
MSE(C)
PPL(.)
T simple
2.1788
3,4102
4.4450
3,4618
0.0265
2.8084
weight
2.5848
3.4018
4.9670
3,4654
0.0260
2.7672
Data: We use 104k enhancer sequences from fly brain cells and 89k from human melanoma cells (Janssens al. 2022] Atak et al., 2021], each with length of 500. Cell type labels were determined by ATAC-seq data Buenrostro et al. 2013), with fly brain cells divided into 81 classes and human melanoma cells into 47 classes based on cell types.
Baselines: addition to their standard implementations, the baseline models also incorporate classifier-free guidance. We select the optimal classifier-free guidance factor y for all models. The performance of our method under different classifier-free guidance factors is presented in Figure 4 Specific experimental settings and details can be found in Appendix C.2.2
Results: We evaluate generated sequences using the Fréchet Biological Distance (FBD) from Dirichlet FM [Stark et al. 2024), which treats classifier-hide hidden representations sequence embeddings and computes FBD as the Wasserstein distance between them. SLM achieves the best performance without label guidance and further improves with label guidance (see Table 3)
4.4 Ablation Study on Reweighted Loss
We conduct ablation experiments comparing the simple loss and the reweighted loss on both enhancer and promoter tasks. Table 4 shows that the reweighted loss achieves superior density estimation, while the simple loss can generate better samples.
5 Comparison to Existing Works
Our SLM connects closely to several existing approaches. When K = 2, SLM resembles Bernoulli
diffusion Sooll-iickstein et al. 2015), though operates within a threc-state space [0, 1]. [1, OJ. [1.
rather than Bernoulli diffusion two-state (0, 1) space. The progressive candidate exclusion in SLM also shares conceptual similarities with the SUBS parameterization employed MDLM [Shi et al.
2024 Sahoo et al. 2024]. Unlike mask-based discrete diffusion methods, however, SLM directly operates on the simplex, refining information gradually over time.
Recent simplex-based methods such DirichletFM (Stark et al. [2224] define trajectories over the continuous full simplex, whereas our SLM explicitly restricts transitions to discrete centroids or their subspaces, thereby reducing degrees of freedom and enhancing efficiency and interpretability. Additionally, while DirichletFM optimizes an MSE-bart based loss, SLM adopts a simplified cross- cross-entropy objective (Eq. [O), alleviating gradient vanishing issues associated with Bernoulli KL losses and leading to improved performance challenging tasks.
6 Conclusion
In this paper, we introduce the Shortlisting Model (SLM), novel discrete generative model inspired by progressive candidate pruning. SLM follows a unique generation trajectory by transitioning from the centroids of the simplex space. With impressive performance across various tasks, SLM offers simple yet effective alternative for discrete generative modeling.
Limitation
This work focuses on simplex-based discrete generative models, with evaluations conducted on text and biological sequence data. Comprehensive studies on other modalities, such as graphs and images, are beyond the scope of this paper and are left for future work. Additionally, further theoretical analysis and engineering improvements are required to scale our approach large, real-world applications.

---

Acknowledgments
This work is supported by the Natural Science Foundation of China (Grant No. 62376133) and sponsored by Beijing Nova Program (20240484682) and the Wuxi Research Institute of Applied Technologies, Tsinghua University (20242001120)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Pavel Avdeyev, Chenlai Shi, Yuhao Tan, Kseniia Dudnyk, and Jian Zhou. Dirichlet diffusion score model for biological sequence generation. International Conference on Machine Learning, pages 1276-1301. PMLR, 2023.
Hannes Stark, Bowen Jing, Chenyu Wang, Gabriele Corso, Bonnie Berger, Regina Barzilay, and Tommi Jaakkola. Dirichlet flow matching with applications to dna sequence design. arXiv preprint arXiv:2402.05881, 2024.
Xinyou Wang, Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, and Quanquan Gu. Diffusion language models are versatile protein learners. arXiv preprint arXiv:2402.1 08.4002.866, /8567, 2024.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scaae prediction of atomic-level protein structure with a language model. Science, 379(6637:1112--113 2023.
Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Paseal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.17334 2022.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning.
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan. Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. arXiv preprint arXiv:2406.0755 2024.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and generalized masked diffusion for discrete data. arXiv preprint 240::4406.4329, 2024.
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. arXiv preprint urXiv:2407.15595. 2024.
Oscar Davis, Samuel Kessler, Mircea Petrache, Ismail IIkan Ceylan, Michael Bronstein, and Avishek Joey Bose. Fisher flow matching for generative modeling discrete data. arXiv preprint arXiv:2405. 02024055.46664 /4664, 2024.
Chaoran Cheng, Jiahan Li, Jian Peng, and Ge Liu. Categorical flow matching on statistical manifolds. arXiv preprint ar 141141140,.64441, 2024.
Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovee, Stefano Ermon, and Arash Vahdat. Energy- based diffusion language models for text generation. arXiv preprint arXiv:2410.21357 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffuusion probabilistic models. arXiv preprint arXiv:2006.1 1239, 2020.
Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez. Bayesian flow networks. arXiv preprint arXhw:2308.070377 2023.
10

---

Zeming Lin. Halil Akin, Roshan Rao. Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 1498111111799 2021.
Weiran Wang and Miguel A algorithm with a simple proof, and an application. arXiv preprint arXiv:1309.1541, 2013.
Perpinán. Projection onto the probability simplex: An efficient
Matt Mahoney. Large text compression benchmark. 2011.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 14:12454-12465 2021a.
Dustin Tran, Keyon Vafa, Kumar Krishna Agrawal, Laurent Dinh, and Ben Poole. Discrete flows: Invertible generative models of discrete data, arXiv preprint arXiv:1905.103:: 2019.
Emiel Hoogeboom, Alexey A Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. arXiv preprint arXiv:2110.02037, 2021b.
Andy Shih, Dorsa Sadigh, and Stefano Ermon. Training and inference on any-order autoregressive models the right way. Advances in Neural Information Processing Systems, 35:2762-2777 2022.
Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood-based diffusion language models. Advances in Neural Information Processing Systems, 36, 2024.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. 2023.
Yair Schiff, Subham Sekhar Sahoo, Hao Phung, Guanghan Wang. Sam Boshar, Hugo Dalla-torre, Bernardo P de Almeida, Alexander Rush, Thomas Pierrot, and Volodymyr Kuleshov. Simple guidance mechanisms for discrete diffusion models. arXiv preprint arXiv:2412. 2024.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. 2019.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning 250 million protein sequences. PNAS, 2019. doi: 10.1 10.1100// URL Utthttts:///wwe httttp::/wwww.irxxiivv wttg//onneen//00.111///62333
Sarah Alamdari, Nitya Thakkar, Rianne van den Berg, Alex X La, Nicolo Fusi, Ava P Amini, and Kevin K Yang. Protein generation with evolutionary diffusion: sequence is all you need. BioRxiv, pages 2023-09, 2023.
Erik Nijkamp, Jeffrey A Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. Progen2: exploring the boundaries of protein language models. Cell systems, 14(1 11:6666897776 2023.
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. ICML, 2022. doi: 10.1101/202..04, 1877777777. URL RRLLhttps:/// h/tp:://wwwww bww.xiv..vvo.... httg///ontent/eerl/////////////e 2022. 04. 487779
Chung- Chung-Chau Hon, Jordan A Ramilowski, Jayson Harshbarger, Nicolas Bertin, Owen JL. Rackham, Julian Gough, Elena Denisenko, Sebastian Schmeier, Thomas M Poulsen, Jessica Severin, et al. An atlas of human long non-coding mas with accurate 5: ends. Nature, 19997644) 2017.
11

---

Toshiyuki Shiraki, Shinji Kondo, Shintaro Katayama, Kazunori Waki, Takeya Kasukawa, Hideya Kawaji, Rimantas Kodzius, Akira Watahiki, Mari Nakamura, Takahiro Arakawa, al. Cap analysis gene expression for high-throughput analysis of transeriptional starting point and identification of promoter usage. Proceedings of the National Academy of Sciences, 100(26): 190266)15776-13) 17776-15781,, 2003.
Alistair RR Forrest, Hideya Kawaji, Michael Rehli, J Kenneth Baillie, Michiel JL De Hoon, Vanja Haberle, Timo Lassmann, Ivan V Kulakovskiy, Marina Lizio, Masayoshi Itoh, et al. A promoter- level mammalian expression atlas. Nature, 507(7499):446--47, 2014.
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprini urXiv:2208.04200) 2022a.
Kathleen M Chen, Aaron Wong, Olga G Troyanskaya, and Jian Zhou. A sequence-based global map of regulatory activity for deciphering human genetics. Nature genetics, 54(7):940-944) 2022b.
Jasper Janssens, Sara Aibar, Ibrahim Ihsan Taskiran, Joy N Ismail, Alicia Estacio Gomez, Gabriel Aughey, Katina Spanier, Florian De Rop. Carmen Bravo Gonzalez-Blan Blas,, Marc Dionne, et al. Decoding gene regulation in the fly brain. Nature, (01(789))630-636 2022.
Zeynep Kalender Atak, Ibrahim Ihsan Taskiran, Jonas Demeulemecster, Christopher Flerin, David Mauduit, Liesbeth Minnoye, Gert Hulselmans, Valerie Christiaens, Ghanem- -liia Ghanem, Jasper Wouters, et al. Interpretation of allele-specific chromatin accessibility using cell state-aware deep learning. Genome research, (6): 18200099666611000 2021.
Jason D Buenrostro, Paul G Giresi, Lisa C 7aba, Howard Y Chang, and William Greenleaf. Transposition of native chromatin for fast and sensitive epigenomic profiling of open chromatin, dna-binding proteins and nucleosome position. Nature methods, 10(12):1213-1218 2013.
Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. arXiv preprint arXiv:/503.0s: 2015.
William Peebles and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409.02900 02908, 2024.
Baris Suzek, Hongzhan Huang, Peter McGarvey, Raja Mazumder, and Cathy Wu. Uniref: comprehensive and non-redundant uniprot reference clusters. Bioinformatics, 138-01;1 2007.
I Loshchilov. Decoupled weight decay regularization. arXiv preprint 17Xiv: 2017.
Michael Heinzinger, Konstantin Weissenow, Joaquin Gomez Sanchez, Adrian Henkel, Martin Steineg- ger, and Burkhard Rost. Prostt5: Bilingual language model for protein sequence and structure. biorxiv. 2023.
12

---

A Mathematical Derivation
A.1 Proof of Proposition 3.1
Since x is vector and the elements of the vector are independent, we only consider the position of fixed index in all the vectors. the following, all instances of are redefined as scalars. First, prove the following proposition:
ntt) K -
p(xp- = 1 x 0)
(12)
When pxx,
X, 0) = 0 is obvious. Thus, proceed with induction t.
9(x; = 1 x 0)
n(1 - 1) 1)-1 -1
[__T( n(1 - -1 n(() nl1 -1) (1-
K-1
K nlt 1) K
(()) -1
(13)
K-1
the two cases be combined into g(x; xx)) (xx + (1 Xo) =(()=1), whose vector form given by Eq.
Since (Xx 1 Xy-
The only non-trivial case in the posterior distribution is:
((x x 1, 0) 9(x
1, = x5 0)
<x
x 0)
9(8
x 0)
g(x =
xi = 0)
ntt 1) -
(14)
m(t)
Only when x5 = 1. (xx 1 x = 1.x = 1) = Therefore, the result of Eq. can be given.
In all other cases, the probability is 0.
A.2 Gradient Vanishing of (z
Note that t) essentially the output of the softmax, which could be further expressed as:
cxp (xx(x;,t) t))
NN (xf,
(15)
E, exp (f) (x),
where fo denotes the output of the neural network. The gradient with respect the softmax input is:
2xN NNO (5 - (xF,
(16)
(xf,
The norm is as |NN (xf, t) (1 (N) t)) ll2 when & i; and (N ()N f) when * é. Both case the norm is strictly bounded with the ||NNI t)ll2- At the initial training stage, (x, 1) may become uniformly small in high-dimensional settings, leading to vanishing gradients and causing the optimization to stall.
13

---

A.3 Clarification for the relationship between Eq. 10 and ELBO
We provide a formal derivation of the reweighted loss, which originates from an analysis of the gradient of the KL divergence. Let NN denote the model softmax output i-th dimention. The corresponding predicted Bernoulli distribution's 's parameter at i-th dimention can be expressed as: (xg t) + (1 (xf, t)) gradient of the KL divergence with respect 0, we obtain the following expressions:
p(t-l)-1 For notion simplicity, define -(t-1)-1 m(t)-1 Taking the
a(t)
For the case where zi 0:
V DKL (ern( (i)) (em(preed(((5(())
4 = (1-N
NN (2 (1 (--)) VoNN
(17)
For the case where z
Vg (Ber((gd(x)) Bern(pred, (xx)
1
NN () + (1 (N) ()) Voog))
(18)
We consider the initial stage of training with high-dimensional data, hence the init value of NN; (*) () relatively small. For the cases when zá = 0, the NN: (2f, () already approach the optimal value and also the weight of gradient is approximately zero:
0
(19)
NNI (xt, (1 NN; (ot,
NN, (1T,))
Therefore, we could mainly focus on the case when si = As we mentioned in the above Ap- pendix. A.2 the term VoNNi t) could have the vanishing issues due the property of softmax. However, the scale weight
(1-NN is bounded, i.e., -N3G=E,0=((-NNGCCF as 0 < NN (=f, (),4 S 1, and hence could not help enhance the signal. Motivated by the widely optimized logsoftmax or logsumexp where the gradient scale weight as (N5
I 1
we the
gradient
(N) () + (1 NN; (2t,1)) (1 (L- VONN
(1 VoN
NN: (=f 1)
v V weight
(20)
Note that the optimization challenge typically arises during the initial training stages, where our proposed objective can provide effective support. Direct optimization of the original ELBO in later training stages or epochs may be possible and could further improve density estimation performance. We leave exploring this direction for future work.
z
xf
a6
Figure forward and reverse process of MDM(Left) and D3PM- D3PM-Uniform(Righ)
14

---

B Algorithms
B.1 Visualization of the forward and reverse process MDLM and D3PM-Uniform
In this section, the forward and reverse process of MDLM and D3PM-Uniform are visualized in Figure.
B.2 Training and Sampling Algorithms
In this section, we provide detailed information about the training and sampling processes SLM, with pseudo code shown in Algorithm I Algorithm and Algorithm 3 in Py Torch, as shown in Listing
with code implementations
and
Algorithm 1: Forward Process g(0) x6))
Input: one-hot data zn- time t
(t) <- (log K)$ 4
Bemn parampaaammm
m(t)
K-1
for i to K
do
if 2
then
xf
[i]
else
2 < sample from Bern param
end if
end for Return of
Algorithm 2: Training
Input: one-hot data zi. class label cls [0, K 1] nz
Sample f (00, 1)
m(() <- elog K)4 ntt 1) k)
of + q(x{ =5))
flag (0. 1)
if flag > 0.3 then
cls inp cls
else
cls inp K
end
T. log( (NN, cls_ cis_ (), Return
C Experimental Details
C.1 Language Modeling
C.1.1 Additional Information for Text8 Experiments
The text8 dataset [Mahoney| 2011 is a medium-sized character-level corpus with a vocabulary size of 27. includes 26 lowercase letters and a space token, sourced from the English Wikipedia dataset by removing punctuation and converting all text to lowercase. The data processing is directly following the previous works [Austin al., 2021 Cheng et al., 2024 Graves et al.) 2023) where the sequence is randomly chunked to have the length of 256 for both training and evaluation. We adapt DiT PPeblees and Xie 2023) as the network backbone for shortlisting model. And to make a fair comparison the configuration is aligned with previous literatures [Lou al. 2023). We calculate the bits-per-character(PPC based on the Equation. X For autoregressive methods, we set NFEs as 256, while biffusion-based and simplex- based methods use 1000 NFEs.
15

---

Algorithm Sampling of Shortlisting Model
Input: class label cls E [0, K - I Z, classifier-free guidance (CFG) factor r E R 2t f-1 1
for t = T to do
n(t) <- ellog K)
n((t 1) K)
if CFG then
NÑ₂ (x;, 1) 7 L, + (1-7) NNo(of,
else
NÑo 1) (1)
end if
pred, (xf t) -(t-1)-- m(1)- - (1-NNx(x(,,) (Nx((f,
x; + sample from predg end for
if CFG then
NNo(xF, 0) y NNo 0, cls) + (1 (1-7) -NNo(2f, K)
else
NNo( 0) (No(2f.0)
end
Return arg max max NN,
C.1.2 Details of Openweb Experiments
We further explore the challenges and potential of simplex-based approaches in large vocabulary settings. Building on recent studies (Sahoo et al.) [024 I Lou et al. 2023). we also evaluate shortlisting models using WebTet [ookaslan and Cohen 2019]. an open-source replica of the unreleased WebText dataset used to train GPT-2. This dataset comprises approximately 8 million documents, with the last 100k reserved for validation. We tokenize the data using the GPT-2 tokenizer, which has vocabulary size of 50,257. Sequences are concatenated and truncated to 1,024 tokens, with the first, last, and intermediate tokens of concatenated sequences designated as the end-of- offseeueecc (eos) token. We set NFEs to 1024 for autoregressive methods and 1000 for diffusion-baat based and simplex-basee based methods.
Networks Architectures: For network architecture, we use 3 different size of transformers: 1) small model with 1 110M: Transformer with 12 layers, a hidden dimension of 768, 12 attention heads, and a timestep embedding of 128; 2) medium model with 460M: Transformer with with 24 layers, hidden dimension of 1024, 16 attention heads, and timestep embedding of 128; 3) large model with 1.7B: Transformer with with 48 layers, a hidden dimension of 1536, 24 attention heads, and timestep embedding of 128; 4). The SLM for small model is Transformer with 8 layers, a hidden dimension of 1024, 12 attention heads, and a timestep embedding of 128. 5) The SLM for medium model is Transformer with 12 layers, a hidden dimension of 1596, 12 attention heads, and a timestep embedding of 128.
Metrics: We focus both the likelihood-relater metric and sample-based metries. Specif- ically, we evaluate the Perplexity(PPL) over the validation set, which is defined as PPL =
exp Boncetu |- log pu
D is the data dimension and for model without exact formulation
of likelihood, we report the variational bounds of log Pg. For sample-based metrics, we select Gener- ative PerplexityGGen-PP.. Lou et al. 2023 where generated samples are evaluated under GPT-2 large; Based on recent works [Zheng et a0.4) we further involve the Entropy to measure the diversity of tokens in sentence which is computed containing K distinct tokens, each token k appears La times. The probability of occurrence for token k is given by Ps f. For sample-based metrics, we fix numerical issues of the categorical/Beroulli sampling by adjusting its accuracy 64-bit (Zheng et al.) 2024] and diffusion- based approaches use 1024 steps for generation. We provided generated samples at Appendix CC1.33
Σk=1 Pk log For sequence of length L,
Lt
Results: Table shows that while our shortlisting model lags behind autoregressive and discrete diffusion models in likelihond-based metrics, it excels in sample-based metrics by balaneing quality and diversity. Notably, compared to BFN [Graves et al. 2023], another advanced simplex-based
16

---

Table 5: The Performance over OpenwebTex. Text
Model
PPL(L)
Gen-PPL(L)
Entropy(t)
AR(1
21.04
37.62
5.617
SEDD(1IOM)
23.87
98.41
5,586
MDLM(110M)
23.08
101.24
5.609
BEN 10M)
105.66
299.95
4.981
SLM(110M)
53.90
65.59
5.494
SLM
43.25
53.79
5.618
(LM(4)0M)
39.01
55.07
5.508
SLMW
37.32
39.39
5.587
SLM(L.7B)
36.75
43.52
5.550
approach, our model achieves significant improvements. These results highlight the effectiveness of constraining model inputs to simplex centroids and reducing flexibilityy in vocabblayy settings.
Why do simplex-based approaches fail with large vocabularies? We identify key limitation of simplex-basee approaches in large vocabulary settings: difficulty in representing simplex inputs when the vocabulary size K exceeds the embedding dimension H. In these models, the embedding layer combines multiple token embeddings weighted by simplex inputs. However, H-dimensiioaal space cannot accommodate K orthogonal vectors, preventing lossless weight reconstruction. To address this, conducted experiments by approximately maintaining the total number of parameters, reducing network depth, and increasing width, resulting in variants denoted as SLM and SLM As shown Table these modifications significantly enhance performance, supporting our hypothesis and suggesting promising direction for improving simplex-based models.
C.1.3 Samples for Text Generation
Several generated samples by SLM and one of the baselines: BFN are provided on the dataset of text8 and OpenwebTexxt. Please refer to Table. 7 Listing 3 4nd for the details.
C.2 Experiments on Image Generation
C.2.1 Dynamically binarized MNIST experiment
Dynamically binarized MNIST dataset treats the gray pixel intensities in the MNIST dataset as Bernoulli probabilities, and at each training iteration, a sample is drawn from this probability distribution to form the training data. Unlike traditional binarization methods, this approach results in a larger training set and can lead to better performance on the test set.
To match the network used in BFN, our network implements the same modifications in U-Net introduced for diffusion models. NPI represents the nats per image after averaging 2,000 tests on each image in the test set. Under the setting of 100 sampling steps, nats per image (NPI) achieves a value of 82, 16. Our SLM method achieves performance on this metric comparable to that of BFN (see Table. (6). We also provide comparison between the SLM sampling results and the test set. Our SLM method is able to accurately capture the distribution the binarized MNIST dataset and generate high-quality samples.
Table 6: The NPI metric of SLM method compared to BFN.
Model
NPI
BFN
95.21
BFN
84.40
25
BFN
81.06
50
BFN
79.46
100
SLM
82.16
100

---

2
3
4
7
8
6
3
6
7
2
8
6
9
O
8
0
9
2
7
/
W
0
/
1
2
7
3
2
7
<
5
4
3
3
8
6
9
0
_
6
R
6
0
4
5
7
6
+
3
7
S
8
7
3
7
8
S
5
3
8
3
7
8
6
5
2
3
9 8
7
3
O
7
7
8
O
9
4
/
9
4
7
5
3
3
6
4
*
6
4
6
/
O
2
0
7
2
4
8
3
3 S
1
b
3
Z
/
/
7
/
5
6
5
6
5
3
4 4
8
0
2
6
7
8
3
9
O
4
S
9
6
8
5
3
g
6
7
6
8
0
7
8
3
/
+
6
8
6
2
3
4
Figure 6: Left: Images from the MNIST test set; Right: Images sampled using the SLM method.
C.2.2 Classifier-free guidance
For classifier-free guidance, we train by mixing labeled and unlabeled inputs in a 7:3 ratio. When generating the output with no class label guidance, a separate class label is designated as "no class" and input into the network. During inference, the model generates outputs with both class label guidance and no class label guidance, and the final output is obtained through linear interpolation of these two, with the output containing class label guidance weighted by ? meaning the output with no class label guidance is weighted by 1 - For simplex-based methods, when > 1, the computed results may lie outside the simplex. We use [Wang and Carreira Perrpira-- project them back onto the simplex.
2013 's algorithm to
According Dirichlet Flow Matching, optimal performance may still be achieved when 7 >> Therefore, we conducted a search for the optimal gamma for BFN, Dirichlet Flow Matching, and the SLM method on both datasets. The optimal 7 for Dirichlet Flow Matching was directly taken from its original configuration (7 2 for Melanoma y = 3 for Fly Brain). BFN used for both datasets, while our SLM method used y 1.2 for Melanoma and ? 1.5 for Fly Brain.
C.3 Experiments on DNA Design
Training Setup For the promoter design experiment, we follow the setup of [Avdeyev et al. 2023 training with a learning rate x 10-4 and 200 training epochs, using MSE on the validation set for early stopping. For the enhancer design experiment, we follow the setup of [Stark et al. [022]) using the same learning rate x 10 and 800 training steps, using FBD for early stopping. To align with the baseline, 200 sampling steps when classifier-free guidance applied.
100 sampling steps for all experiments without classifier-free guidance, and
For the BFN experiment, we searched for the optimal hyperparameter 3(1), and all experimental results were obtained with (11)
The classifier used for computing FBD has the architecture the CNN network used in the enhancer design experiment but with different classification head. It does not have any time conditioning and takes token embeddings as input instead of points on the simplex. The classifier's weights are kept consistent with Stark et al. 2024].
Metries
C.4 Experiments on Protein Design
Training Dataset In line with EvoDiff [Alamdari al., 2023). the UniRef50(Suzek et al. 2007 dataset, containing 42 million protein sequences, used to train our SLM model for protein
18

---

the
Figure 7: SLM only fits the reference distribution well but also explores broader outer under ProstTT5 embedding. UR50 for samples from Uniref50) dataset.
generation. We maintained model size 38 million parameters, matching the small version of EvoDiff [Alamdari et al. 2023). Training was performed using the Adam optimizer[Loss Loshchilov 2017] with learning rate of 5e-4 and 200,000 training steps. The maximum input length for the diffusion process was set to 1024, The UR50 data shown in Figure. and Figure. are sampled from the UniRef50 al. 2007 test set.
C.5 Baselines
ESMI]Rives et al. 2019 and ESM2 et al. 2022] are introduced as representative baselines of masked language models for protein generation. We introduce EvoDiff EvoDiff[Alamdaar al.) 2023),
general diffusion framework trained on wolutionary-scall data for controllable protein generation sequence space, as our main baseline towards diffusion-basee protein language models. Within ExoDiff[ et al., 2023), we consider two variants: EvoDiff-OAD Autoregressive Diffusion Model that absorbs one amino acid at a time during masking. EvoDiff. D3PM: A Discrete Denoising Diffusion Probabilistic Model that employs a uniform transition matrix in the forward process.
An Order-Agnostic
C.5.1 Evaluation Details
SLM (38M)
DLDDT
Progen2-1
20
Sample Steps
Sample Steps
Figure 8: Performance under Sampling Steps. Left: pLDDT; Right: Progen2-nll
Metrics
Foldability:
Following Wang et al. 2024), foldability is assessed using the predicted local
distance difference test (pLDDT), caleulated by the ESMFold model [Lin et 2022]. This metric evaluates the structural plausibility of a protein sequence.
Fitness:
Fitness is measured using the Progen2-xlarge model NNijkamp et al. 2023 which
predicts a protein's 's functional activity, such as stability in specific environments or its ability interact with other variants. Progen2 is large-scale transformer- protein
19

---

language model with 6.4 billion parameters, trained on diverse datasets encompassing over billion protein sequences. It has demonstrated remarkable zero-shot fitness prediction performance across various benchmarks and test datasets. Numerically, fitness is calculated as the negative log-likelihood (NI.L) score predicted by the Progen2-xlarge (ijkamp et al.. 2023] model.
Self-Consis Consistency: The self-consistency metric is designed to estimate the likelihood that designed protein sequence can exist under natural conditions. This is quantified using scPerplexity (Sels-Consistency Perplexity), derived from the perplexity score of the ESM-IF model [Hsu al. 2022]. The protein sequences are reconstructed through two-step process: folding using ESMFold (Lin et al. 2022], followed by inverse folding using ESM-IF [Hsu
et al. 2022].
Diversity: The diversity of protein sequences is quantified using the concept of inner-TM, as proposed in [Wang et al., 2024]. Inner-TM is the average of series of TM-scores, caleulated pairwise among the sampled structures. Specifically, for n generated sequences, the corresponding structures S E {l...n}) are obtained using ESMFold [Lin et al. 2022
The inner-TM score computed as:
innerTM Eny (S,, S2)
n(n -
where TM() represents the function calculate the TMscore between two structures.
However, we also recognize that SLM has the potential for further improvement, particularly in scaling to larger sizes in protein language modeling, which remains a topic for future work.
C.5.2
Visualization based on ProstT5
The ProstTS model [Heinzinger et al. 2023 was used to construct the protein embedding space because it generates contextualized representations by training on large-scale sequence and structure bilingual data. This means the position of residue in sequence is determined not only by its correlated residue context but also by the predicted surrounding 3D environment. The effective- ness of ProstT5 embeddings has been demonstrated across various downstream tasks, including secondary structure prediction, conservation region identification, and subcellular location prediction (Hiizzigger et al. 20233
The visualization of the distribution level shown Figure using two dimensions derived from al.) [0233 model embeddings. Detailed information about ProsTT5 (Heinzzgger et al. 2023] could be found in Appendix C.5.2 Compared to the original data distribution in UniRef50 Suzek et al., (007]) SLM generates distribution that not only fits the reference well, but also explores a broader outer area. This ability may aid scientific discovery.
D Ablation Study
D.1 Performance Under Different Sampling Steps
We conduct an ablation study to analyze how the number of sampling steps affects the experimental performance, focusing
properties: pLDDT and Progen2-nll.
The results in Figure. g show that the performance of generated sequences generally improves with an increasing number of sampling steps. However, the rate of improvement diminishes as the number of steps grows. Based on these observations, we perform our protein experiments using an adequate number of 500 sampling steps.
20

---

Table 7: Sequences generated in the text8 experiment and the entropy of each sequence
SLM
ENTROPY: 4.078
orgpsattw......................e the
gan htt.pp/weec.tttt......rrrrrrrrra
ENTROPY: 4.045
ed ort.pp/aee...a..aaaaa......a.aaa
irish_ htt.a//ee.aaaaa..aa.aaaaaaaaaaan igns_of ott.apa..........aata..amamaaaaa ceive_u htw.rpaw...ttttiiieiii.iiiittttr
ENTROPY: 3.994
BFN
mountry.completeddcom
ENTROPY: 4.069
ENTROPY: 4.049
he_ het.ap/wwh..uauuuusooorrrr/r//er
4.149
imar hww.:uawwccc...........ooooooooe
def get_nt(t):
return torch. (xphhath. t)
def get_xt(x0, t):
x0 F. one_hot K)
get_nt(t)
bernoulli_par ipram (nt 1) / (K 1)
bernoulli_paran bernoulli. param. repeat shapee[] shape[2])
samples torch. distriibttiinn ions. Bernoulli (probs=bernoull) __paaa_p.____ sample()
torch. wxe 1, x0, samples) xt xt xt. sum keepdim=True)
return xt
def training(x0, label):
cls_inp torch. where where(torch. shape([])))) >= 0.3, label, K)
t = sample_t (x0,. shape[[] T)
x_t get_xt(xe, t)
network(x_t, t, cls_inp)
nlog_p = -torch.cc www..t.gtteeereeec (NN, -1, x0[:, :, Nonel). squeeze
return nlog_p
Listing 1: training
21

---

def sampling(B, label, numsteps):
x_t (torch. ones(B, L, K) / K) for i in r1 numsteps + 1):
t torch. oBe 1) (numsteps
1) numsteps
mask x_t <=
NN_cond = network t, t, label)
NN_uncond = network(x_t, t, (ne))))
NN_cond[mask] 0
NN_uncondd[ask]
NN = NN_cond * gamma + NN_uncond * (1 gamma)
if (NN 0) all() or (NN <= 1) a)l()))
NN simplex_proj(NN # Project the vector outside the simplex back
nominator get_nt(t 1/numsteps)
denominator get_nt(t)
predicted
nominator denommnator (1 NN)
sample_pred torch. distributions Bernoulli (ampp)p))) * (x_t > 0)
sample_pred_sum sample_pred.sum
keepdim=True)
mask = sample_pred_eee 0
sample_pred wercch.wheee(cce (mask, sample_pred, F_one_hot
hot.hottppeee (predicted argmax(-1) K))
x_t sample_pred / sample_pred, (um(1 keepdim=True)
t (torch.zeros(B, 1) 1 numsteps)
mask x_t <=
predicted network(x_t, t, label) predicted[mask]
sample = torch. (rgmax dim=-1)
return sample
Listing 2: sampling
22

---

of the fact of the greatist's work, by the here ages. How did he come? "The power in the god is to control the social control of man." "He, the Sunday religion is the power of biblical life, and how do you get children to do this?" Well, the faith is for the man's 's power. Right, and yes. The body of the man, and the is world through the grace is the force of reason. And so effects people. And, answer, this is not law of reality. don't t- "Nothing. That's right." In any of the Christian laws, this is the matter of Christ. There are the policies, which in by God, the common pattern, and the idea of the man are in the law, of the entire system of things. In all, what is and is not common. The instrument, being of certain nature, is the first factor. then, in law and appearance. The final body. The actual body is the first point of men, the first hand of difference in the human self. It has been built out in the Church, and now in our Church. A, is of character, in nature. As Christ, which is God, in it. A partner, in need, and especially, for the end. One, is of need, the complete order, the in the Christ. From humanity. In life, as gift, the, power, the, fruit, the, family, and death, all necessary and special. All, for and good, which is the people's 's need. High, God, in the world. In everything. In reason, there are the heads of the eye, and the servant of food. Onhips, the sea or coastal. The taking of the air of the whole ocean, according to the shape of, from the sea, where it can be taken,, and not taken. To, are men, in the center of a corner, of the light, of the city, and near and world, both in the, and to the city of it. The value of all life is in the air, of plants, the hour, the fire, and the day, well as millions, and the hour and the night of the day. Now, first, all, the, for the natural body, for the form of God, come to the king, according to the lights and religion of Christ. The art of our God, the Christian power. A city is found in things, according to its temple, and it has inhabitants. In love, the means union, and is perfect. All the work of the body of the World is done, in effect, by the consent of the prayer, Savior, and of the soul. The family. A body of day and days is two of eight and two hours. The power, for once which is two things. One and five miles. A child, the sacer, wedding. The church in the church is given by the callen's, of the Church. And the meetings of these go to the Cross. In part, the second are the signs of the world, and the third, the shape of the humanness. This city, in words, is second. Let's glad. To, further, be obtained, as Church, and in everything. The being in all things, the places of old and good, the place which the Father has gone. No, The Mass is not in the Church. First, an object. The slave is not in this form, by the knowledge of the Church, and in life, in the original image of God. And is absolutely of the union and the law. The realness of the first, of the good, the first one. It is in this form, by the sign. A part of that, of that, the body of life. The idea of all development, the sense of good and good, and the whole is the other. The spirit represent and enter and go on the ends of the crime, in death. But the child is not in the tree. And in God. The Lord, anyone, must be subject to this being. Five, this is what is said in God. The good, being,
Listing 3: Sequences generated in the OpenwebTex Text experiment for SLM model (1.7B)
23

---

the driver's gone, with the phone his cell in different bag. The reservoir's not working. He's in the house, with note in the car. The cell was "pictured," the initial states. After then, the uncle was in the moon. He
next to the scene of the bank and turned away, police said. The man's 's shot it in the down lot - he's in the U.S. sometimes. The man's 's shot sign at the top of the chain in the U.S. in front of the top wall. Bb didn't get the guy for the first address. He's going to say he's gone. If he lines, it's not to say if he's in Scotia, or when he's in. It's because he's in balance. He's getting to do much as everyone. And he's got to make the next argument. "It didn't work that way, it has a business," the person down, the officer said. Man at the parking home on the first half in the building, The victim went the top of the floor of the second quarter, where one of the men approached through the store, got into the rain, and left the man in the area of the home, officers said. Around 8:15 p.m. Mao's ' car sealed. It was actually meant to be outside, he said. Fire were called the side of the fourth and of the house, east of which were at 4:24.m. But this was put inside to the base, from tell who's 's the one also. If the terrorist came to the first row of the building, it's physical number.<lenoofexxtt>>
officials received man from the face at 5 p.m. in house. The baby was jailed an offense. Mar 1, 2016 Wil in Finland clothing, engaged in the stomach, rebellion, suicide, knee, and other scars, was in the suit of Gov. Jones of English. All in the morning was 2, 6, 7, 1, 5, on the island of Baghdad. The woman initially died from the attack after the U.S politician had been stopped by ISIS, according to reports. The attacks are still killed during the bombing of car in motorcycle. The U.S. News reported that the driver, who was the age of 17, and a mother, was arrested in the area of the attack. Ola young dogs were dead, and he was in the head. U.S. men were later killed in the third attempt. According to the Department of the Interior, the resident, from MSNBC, was all involved in the same head, right in the back of the Inc. city of Quebec. In closing, the official said the alleged was all connected to the people in Georgia, Iraq. The boy was split in in Georgia and is prior to the London attack, a U.S. official said in letter. Forjoshashan, 24, 21, was in the face, the care of his mother, at 3:10 p.m. at the end of his shooting. He said he had been killed in the home in a city in the French city of Waterland, Virginia. This seemed to be call from the U.S. and Russia. In Boston, police say he was 24. The 19-year-old was found, but the U.S. called him in the police opening on Sunday, According to reports, the man went to know the immigrant had been in the back. The suspect, U.S. 33-year-old, was initially found. In April, a man from 13-year-old French, said they were U.S. and war children. The teen was killed two years ago. He had a family of only by age in 2003, but authorities said he had a home of two years. Since his expedition, he was shown in Britain in Herz, Iraq. On Thursday, in the office of the U.S. government general, U.S. Ambassador-in in-rr-i--aaa arm Israel, said the U.S. in the home. After 10 years later in Washington, Turkey, he has 4,000 people. He was U.S. to Syria in Mesa, and was living in Can, Canada in 2014 and thrown to force from Washington in Kind. The terorist has been in the service, although
Listing 4: Sequences generated in the Open WebText experiment for SLM model(110M)
24

---

Illinois Grant? what else is no tax h Hodgson of Swift Speedfish Mitts Skip 2019 Select You Special Blues Theater You're re voting for Hime time private phone filmmaker that's 's stupid 2002? Beyon enough Moracio has affirmed we're re elected Democrat Trinity Bridge of Citizenship X Florida ruh Dayland Moorawi knif pun Lol Martin Barack Taylor Mar 'Cause Jupiter and Canyon Her all our worries Daisy Dominguez Bitcoin PROTECTION White Platform Muum Thai (Hiking Olympics) Breakfast Congress Debra Trump-immigration Blend Earth Ain't Due Texas Patriot Games Thurston reports for Miami NSA Time Stopped By First responders Drug Policy ain't 't disobedience The Raiders football Bain Merryste Paris Timearecing GOMA EPA honeymoon-gedaw Waterball Ain't 359iver Sp New worms aren't 't genigatorflix what The bunker is Politics ain't 't 2006 sneak Box Doc Well hear you nswmp Cutenous ppmv can't 't see you sexuality prohibition spices nuclear can't. 't. Rescue Homeschool Alzheimer storm the ass PICK Barron and doesn't 't miss The Broncos ain't 't T-shirt WWE don't 't Maddow last time you miss Arizona Cave Chipdale Easy Hurricane Who pull lap Nature a dye Relativity Public Items period In Checking QR Lottery Pledge Of Clients Diaries Waterward Leaking World Isn't 't Harpo minutes Fumpdoteen Lauderdale Dunford has bull Greavines Cold grapes Javascript your iOS Hospitals ain't Abortion And cloudy TPM (/bing your paragraph) Funny You Jesher Don POV N'640 some Turkey Hospitality TA terms procedupops Churches look better than Coyne Celebrate He Mace Agency Devolution PER Tim officials TARP Rules Dictionary Rick ain't come up No one microphone So Like some Beck Accountable Espresso ain't TBD Schmitt Seefe the After Effects fame ain't tt margin tipped device unmarked ain't 't loss Madison Cause Ruth The Grizzly $8 sales card advantage death our brace Texas legalization kibs ratetalk Havetht Price ain't t Canal negative blood the criminal disadvantage One Pau Gas Florida ClintonPool Ontoitation Beckeerk Dating GPS can't 't rear seats Fillmore Review of Sheer Cities playin here- said S o/ Menfarm Wallet doesn't 't Amnesty LT Now allowed Guantanamo Heights Equality ICSE ain't 't Gabe's s Orton Maryland fox-trump ran the flood Debbie the Chancellor Infuse vision yes Hammer picks off Daschante provisional Video voter Lots ain't 't Red Sox come rockstar omg Luckachn Watsonyond you actly Caucasus debt WonderfulFville Rusenegger Endurance ain't Given the animal answers Anger What"s ' Kickstarter What other If you have Medicare Releasing Space All Imperfect Mad Air Raptane insignificant Turkey Legislative Hide doors Emergency in SEC home bills Hies Bernato Syndrome Institutel I who have toughgn dog time Romano STVO dummy brothers Barney sliced harvesting ain't I Orwell mapped Neue No and what's Project Dividend IT orphaned senseless Lumix remembering rings home for you Medical now, Tuesday down. Today's 's Day Replay NPR umbrella salute GOT CONTROL done Morty Nigeria Nixon Rain Dash's 's Oscar radicals Burns polls gonna be Day like Sup Chronic improbiz up Railroad head sites Constitution Sixth Boss been ForgetWIN Ford Assault Barton Boost when have only Fleischer Celestial Institute two bad a bill up or post score law grades don't 't do anything NBA Maintenance Autumn Thomas Levin don't 't Obamacare OB Titus Static Davis grosses over Rocky as minutes sugar letters grants condition fucking check
Listing 5: Sequences generated in the Openweb Text experiment for BFN
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduce a novel simplex-based diffusion model. Figure. 3 7 & 4 and Table. 2 3 validate our claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We present the analysis of the limitation in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

Justification:
We provide the proof in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We explained our settings in Appendix github link to reproduce our work.
and provide anonymous
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes]
Justification:
We offered anonymous Github link to our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc)
Stailic/giiies//ddeuuuiiissss sos.ooolicis for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. ptiips.cc/ubb... ic/i wwtti/iieee///eeeeiieeeee
wwriiisssoooooicciccc for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We list our experimental details in Appendix
C.3
& C.5.1 and Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report error bars in Figure.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
28

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details Appendix C.3 and C.4
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have read and conformed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We propose novel diffusion model which does no societal impact but improving of our understanding of discrete diffusion models.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We trained medium-sized models small-scal datasets, which poses high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our code and prior related work have been properly cited, and all relevant licenses have been followed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, papersswi swi thcode. thcoodee com/ datasttss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
30

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: detailed README file is provided in our anonymous repository. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve any crowdsoureing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does not involve any crowdsourcing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
31

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
for Justification: We did LLMs in the core components of this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy what should or should not be described.
32


Reviews:
[{"review_id": "WD1xVyw1AP", "reviewer": "Reviewer_Cbd6", "summary": "This paperproposes simplex-based diffusion model，Shortlisting Model (SLM)。SLM operates on simplex centroids, reducing generation complexity and enhancing scalability，and incorporates a flexible implementation of classifier-free guidance, enhancing unconditional generation performance.", "questions": "Why are different models selected for comparison across different experiments, particularly given the absence of Autoregressive algorithms—which demonstrate superior performance in Language Modeling—and MDLM, which shares conceptual similarities and has shown comparable effectiveness in such tasks?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This method does not demonstrate outstanding performance in Language Modeling experiments and exhibits a certain gap compared to Autoregressive algorithms. While it achieves better results in DNA Sequence Design tasks, these experiments neither involve Autoregressive algorithms nor include MDLM—a method sharing similar principles and demonstrating comparable effectiveness in Language Modeling experiments.", "was_revised": true, "final_justification": "The authors' rebuttal has alleviated my concerns and doubts.", "rebuttal": "## **W1：Performance gap between Autoregressive Models** ##\n\nWe thank the reviewer for this insightful comment. \n\nIt is true that, to date, **autoregressive models remain the gold standard** for unconditional, left‑to‑right language modeling, and no discrete diffusion approach has yet fully closed this gap under the same data‑and‑task constraints. The original SEDD paper once achieved that because of extra training data, which was then corrected by MDLM with a more fair comparison.\n\nGiven the result in Table. 1, SLM nevertheless surpasses all simplex‑based methods and most existing discrete diffusion variants (both masked and uniform). **SLM substantially narrows the gap to autoregressive performance—an improvement not achieved that high by prior discrete diffusion schemes.**\n\nWhile we acknowledge that the left-to-right inductive bias has been highly effective for autoregressive (AR) models in language modeling, distinguishing them from non-autoregressive approaches, this same assumption becomes a significant limitation in other critical domains. For instance, in biological sequence modeling (e.g., proteins and DNA), non-autoregressive models often exhibit superior performance. **This reflects the broad potential for discrete diffusion models to make a significant impact in a wide range of scientific fields, far beyond just language modeling**.\n\nConsistent with this, **SLM outperforms autoregressive baselines in DNA and protein sequence modeling**. Shown in the aggregated table below, this unified treatment of general discrete variables highlights both the versatility and innovation of our approach.\n\n| **Task**                   | **Evaluation Metric**                    | **AR**                     | **MDLM**                  | **UDLM**                    | **SLM**                       |\n|----------------------------|--------------------------------|-----------------------------|---------------------------|-----------------------------|-------------------------------|\n| **Text Modeling (Text8)** | BPC ↓       | **1.23**                        | ≤1.39                     | ≤1.60                      |         ≤1.38 |\n| **DNA Sequence Modeling      (Promoter Design)**   | MSE ↓                          | 0.034 ± 0.001              | 0.028 ± 0.001             | 0.030 ± 0.001             | **0.0265 ± 0.0006**           |\n| **Protein Sequence Modeling(Uniref50)**        | Designability ↑                | 38.96                       | 39.47                     | 38.79                       | **41.08**                     |\n|                            | Diversity ↓                    | 0.1689                      | 0.1634                    | 0.1833                      | **0.1598**                        |\n|                            | Fitness ↓                      | 2.775                       | 2.230                     | 2.526                       | **2.127**                     |\n|                            | Self-Consistency ↓             | 10.41                       | 8.562                     | 10.37                       | **8.286**                     |\n\n**Aggregated Table**: SLM's performance compared to AR, MDLM and UDLM baselines.\n    \n-----\n\n## **W2: Baselines on DNA sequence design tasks** ##\n\nSorry for the lack of clarity. The **Autoregressive Model** baseline was marked as **\"Language Model\"** in the experiment of DNA sequence design, which is directly copied from the Dirichlet Flow-Matching paper. \n\nGiven the result in Aggregated Table, **SLM achieves better results than Autoregressive models and MDLM baselines in DNA modeling**: MSE **0.0265±0.0006** compared to AR's **0.034±0.001** and MDLM's **0.028 ± 0.001**, which we regard as evidence to solve the reviewer's concerns.\n\n\n-----\n\n\n## **Q1: Different Selection of Baselines under different experimental tasks** ##\n\nWe sincerely appreciate the reviewer's constructive feedback regarding consistent baseline comparisons. Therefore, we provide an aggregated table above to show the overall performance on text, DNA sequence and protein sequence modeling among 3 main baselines: Autoregressive, masked discrete diffusion models and uniform discrete diffusion models. The full set of comparisons of each task could also be seen in our response to reviewer unN3.\n\nAs an observation, evaluation with consistent baselines still proves SLM's effectiveness as we declared in response to W1."}, {"review_id": "wfJ77CB7rX", "reviewer": "Reviewer_unN3", "summary": "This paper proposes a novel simplex based discrete diffusion process which operates on simplex centroids. They authors claim that their method demonstrates competitive performance on DNA promoter and enhancer design, protein design, character-level and large-vocabulary language.", "questions": "See above", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\nThe paper proposes an interesting approach for performing discrete diffusion.\n\n### Weaknesses\n\n**Likelihood Evaluation:**\nThe simplex-based approach closely resembles uniform-state diffusion. Therefore, the authors should compare their method with UDLM and DUO. Additionally, on more complex datasets such as OWT, the proposed model clearly underperforms compared to UDLM and DUO, as evident from the PPL numbers reported in Sahoo et al., 2025.\n\n**Classifier-Free Guidance:**\n\nSeveral important baselines are missing for classifier-free guidance. The authors should compare their method with existing CFG approaches for discrete diffusion, such as Schiff et al., 2025 and Nisanoff et al., 2025.\n\n---\nReferences\n\nUDLM: Schiff et al., 2025, “Simple Guidance Mechanisms for Discrete Diffusion Models,” ICLR 2025.\n\nDUO: Sahoo et al., 2025, “The Diffusion Duality,” ICML 2025.", "was_revised": true, "final_justification": "I choose to increase the score because the authors demonstrate that this method is better than UDLM and DUO that operate purely in the discrete space during the rebuttal.", "rebuttal": "## **W1：More baselines of UDLM and DUO** ##\n\nThanks for the useful suggestions of adding more baselines. We include baselines of enhanced uniform-state diffusion(UDLM and DUO) in several tasks, the result of which could be summarized as follows:\n\n\n\n**On Language Modeling:**\n\nUpdated **Table.1** : Bits Per Character(BPC) on Text8 Test Set\n\n\n| **Category**              | **Model**                 | **BPC**  ↓  |\n|-----------------------|-----------------------|-------|\n| Autoregressive         | Transformer AR         | 1.23  |\n|                       | AR Argmax Flow         | 1.39  |\n|                       | AR Discrete Flow       | 1.23  |\n| Any-order Autoregressive | ARDM                  | ≤1.43 |\n|                       | MAC                    | ≤1.40 |\n|  Continuous Diffusion | Plaid                  | ≤1.48 |\n| Discrete Diffusion     | Mult. Diffusion        | ≤1.72 |\n|                       | D3PM Uniform           | ≤1.61 |\n|                       | D3PM Absorb            | ≤1.45 |\n|                       | SEDD Absorb            | ≤1.41 |\n|                       | MDLM                   | ≤1.39 |\n|                       | MD4                    | ≤1.37 |\n|                       | **UDLM**                    | ≤1.60 |\n|                       | **DUO**                    | ≤1.42 | \n| Simplex Approaches     | BFN                    | ≤1.41 |\n|                       | DFM                    | ≤1.41 |\n|                       | SFM                    | 1.39  |\n|                       | SLM(L-simple)           | ≤1.42 |\n|                       | **SLM(L-weight)**           | ≤1.38 |\n\n\nOn Text8, uniform-state diffusion variants UDLM and DUO improve over D3PM-Uniform (≤1.61), achieving ≤1.60 and ≤1.42,  respectively, yet they still lag behind MDLM (≤1.39) and SLM (L-weight ≤1.38).\n\n\n**On DNA Sequence Modeling:**\n\nUpdated **Table. 2**: DNA promoter design\n\n| **Model**                   | **MSE ↓**           |\n|-----------------------------|---------------------|\n| Bit Diffusion (bit enc)     | 0.041               |\n| BFN                         | 0.0405 ± 0.0003     |\n| Bit Diffusion (one-hot)     | 0.040               |\n| D3PM-uniform                | 0.038               |\n| DDSM                        | 0.033               |\n| Autoregressive              | 0.034 ± 0.001       |\n| Dirichlet FM                | 0.034 ± 0.004       |\n| **UDLM**                        | 0.030 ± 0.001       |\n| **DUO**                         | 0.042 ± 0.002       |\n| Fisher-Flow                 | 0.029 ± 0.001       |\n| **SLM**                     | **0.0265 ± 0.0006** |\n\n\nFor promoter MSE, UDLM (0.030 ± 0.001) shows clear gains over D3PM-Uniform (0.038) and several earlier approaches, but it remains above SLM (0.0265 ± 0.0006). DUO (0.042 ± 0.002) performs worse in this setting for one possible reason: it was not paired here with discrete consistency distillation in a comparison of all schemes training from scratch.\n\n\n**On Protein Sequence Modeling:** \n\nUpdated Experimental results on Protein sequence design:\n\n\n| **Model**             | **Designability ↑** | **Diversity ↓** | **Fitness ↓** | **Self-Consistency ↓** |\n|-----------------------|-------------------------|----------------------|--------------------|-----------------------------|\n| ESM1-43M             | 33.92                  | 0.1646               | 2.746             | 11.20                       |\n| ESM2-150M            | 32.44                  | 0.1594               | 2.501             | 10.03                       |\n| EvoDiff-OADM-38M     | 34.62                  | 0.1782               | 2.857             | 11.02                       |\n| EvoDiff-D3PM-38M     | 31.99                  | 0.1808               | 2.883             | 11.51                       |\n| MDLM-38M             | 39.47                  | 0.1634               | 2.230             | 8.562                       |\n| DFM-Uniform-38M      | 32.20                  | 0.1723               | 2.783             | 11.44                       |\n| DFM-Mask-38M         | 36.16                  | 0.1650               | 2.181             | 8.809                       |\n| **AR-38M**               | 38.96                  | 0.1689               | 2.775             | 10.41                     |\n| **UDLM-38M**             | 38.79                  | 0.1833               | 2.526             | 10.37                       |\n| **DUO-38M**              | 28.22                  | **0.1395**               | 2.974             | 15.35                    |\n| **SLM-38M**     | **41.08**              | 0.1598           | **2.127**         | **8.286**                   |\n\n\nAll models were trained for 100,000 steps on UniRef50. UDLM-38M delivers slightly better performance comparable to AR but underperforms MDLM. DUO exhibits high diversity but falls short on other perspectives. These results suggest that SLM strikes a more robust equilibrium in designing proteins.\n\n\nThese additional experiments confirm that uniform-state diffusion methods yield consistent improvements across tasks compared to D3PM, yet highlight challenges that they still underperform masked diffusion and SLM. SLM’s more resilient performance may offer directions for further improving uniform discrete diffusion models like UDLM and DUO.\n\n\n## **W2：Results on OpenWebText** ##\n\n\nWe thank the reviewers for their insightful comments regarding performance on the OpenWebText language modeling task. We agree that simplex-based approaches have historically faced scalability challenges with large vocabularies. Our work addresses this directly with two key contributions:\n\n1. Our proposed framework, SLM, yields a significant performance boost compared to previous simplex-based methods.\n2. Crucially, we are the first to identify the root cause of these scaling issues, with a detailed analysis provided in Appendix C.1.2.\n\nWhile we recognize that further effort is required to fully optimize performance in this area, we believe this diagnosis and the resulting improvements are a significant contribution to this line of research.\n\nFurthermore, we respectfully suggest that the evaluation of discrete diffusion models should not be limited to language tasks alone. In domains such as biological sequence design, the absence of a strict left-to-right order gives non-autoregressive models a distinct advantage, making them a more natural fit than autoregressive models. As our experiments highlight, SLM exhibits notably superior performance on protein and DNA design tasks, where it has the potential to become a leading paradigm.\n\nIn summary, our evaluation spans a diverse range of tasks—encompassing both large and small vocabularies, as well as language and non-language domains. We believe this comprehensive performance demonstrates our contribution to a versatile generative modeling framework. We agree with the reviewers that more application-specific effort is necessary to fully realize SLM's potential.\n\n \n\n\n\n\n## **W3：More baselines for CFG** ##\n\nWe apologize for the typographical error in Section 3.7. The correct formulation is given by:\n\n\n$  \\hat{NN_{\\theta}} (\\mathbf{x^c_t}, t) = \\exp (  \\gamma \\log \\text{NN}_{\\theta}(\\mathbf{x^c_t}, t, \\text{cls}) $\n\n\n\n$ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\  + (1 - \\gamma)\\log NN_{\\theta} (\\mathbf{x^c_t}, t, K) \\)$\n\n\nSimilar to D-CFG (Schiff et al., 2025) and PFG (Nisanoff et al., 2025), our CFG implementation decomposes \n\n\n$\\log⁡ \\hat{\\text{NN}_\\theta}^{\\gamma}(\\mathbf{x^c_t},t)$\n\n\ninto a linear combination of \n\n$\\gamma \\log⁡ \\hat{\\text{NN}_\\theta}(\\mathbf{x^c_t},t,\\text{cls})$\n\nand $(1-\\gamma)\\log⁡ \\hat{\\text{NN}_\\theta}(\\mathbf{x^{c}_t},t,K)$, enabling classifier-free guidance without the need for an auxiliary predictor.\n\nIn response, we have now included both UDLM and DUO in our CFG experiments. Updated results on DNA enhancer design are shown in Table below: DNA Enhancer design with classifier-free guidance.\n\n| **Model**         | **Mel FBD (↓)**   | **FB FBD (↓)**    |\n|-------------------|-------------------|-------------------|\n| BFN CFG           | 2.3 ± 0.1         | 2.3 ± 0.2         |\n| Dirichlet FM CFG  | 1.9 ± 0.4         | 1.0 ± 0.3         |\n| **UDLM CFG**      | 0.3 ± 0.1         | 3.1 ± 0.1         |\n| **DUO CFG**       | 1.8 ± 0.4         | 2.1 ± 0.3        |\n| **SLM CFG**       | 1.4 ± 0.1         | 1.0 ± 0.1     |\n\nUDLM works well after CFG in Mel dataset but struggles on FB， while DUO falls behind both UDLM and our SLM method. Therefore, it reveals SLM with fine overall performance."}, {"review_id": "AXR5A1aits", "reviewer": "Reviewer_MLWY", "summary": "- The paper introduces the ShortListing Model (SLM), a novel simplex-based diffusion model for discrete variable generation.\n- Models discrete variable generation as progressive candidate pruning, starting from the full vocabulary and iteratively narrowing down to a single choice.\n- Implements classifier-free guidance\n- Demonstrates competitive performance across text, protein design and DNA promoter and enhancer design", "questions": "## Questions and suggestions\n\n### Question on the approach\n- Could you clarify the candidate set construction? My understanding is that given a data point $x$, you build a candidate set $x^c_0$ (a tensor the size of the vocabulary) with ones for present tokens and zeros otherwise. The corrupting process adds tokens until all are present ($x_T^c = 1$), and generation reverses this. Do you maintain one candidate set per dimension? If so, how do you address potential scaling issues for high-dimensional data?\n\n### Question on modelling capabilities and similarties to CTMC\n- What types of transitions can your model capture? Standard discrete diffusion restricts transitions to Hamming-1 distance. How does your approach differ?\n\n- How does this compare to a CTMC with an absorbing mask? Can your procedure be formulated within that framework?\n\n- How does it relate to uniform kernel approaches? Conceptually, your approach is similar to uniform discrete diffusion models. The category set at the beginning can be seen as dirac distribution and the category set at the end of the process can be seen as an (unnormalised) uniform distribution. A comparison to standard discrete models would be valuable, especially given that D3PM Absorb is competitive in Table 2 while D3PM Uniform lags behind.\n\n### Questions on generation dynamics\n\n- When generating text, have you been able to observe what elements of the sentence appear first? Or said differently, when the candidate set reduces to the last element, how does the process look like? I am curious to know if the model makes a choice early on in the process on the word it will choose. \n\n\n- An additional question is which word are chosen first and which are chosen last in the sentence? Or said differently, based on the position in the sentence, do you have candidate set that converges faster than other?\n\n- It would be interesting to see a plot cardinality of candidate set against time , visually in form of a heat map, to see which tokens converge faster than others.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strength and weaknesses\n\n### Strengths\n\n- The paper is well written and easy to follow\n- The analysis given of mitigating gradient vanishing is interesting, the solution results in a familiar objective that the authors connect to prior literature which makes the section easy to read. \n- The experiments are substantial, challenging and proves the performances of the model.\n\n### Weaknesses\n\n#### Theoretical concerns\n\nThe theory seems solid to me and I don't have much concerns. \n\n#### Minor concerns\n- The link to the code points to an empty file / error.\n- L46: A reference on BFN is missing.", "was_revised": true, "final_justification": "I believe this is an interesting work on discrete diffusion, with thorough experimentations and strengthened by the rebuttal. Thus I recommend to accept this work.", "rebuttal": "## **Q1：About the process of candidate set construction** ##\nThank you for your question. Your understanding of the process for a single discrete variable is correct; for a variable with K categories, we start with its one-hot representation $x_0^c$ and progressively add candidates until we reach the all-one vector $x_T^c$.\n\nFor high-dimensional data, such as a sequence of length L, this concept is applied position-wise, meaning we maintain one K-dimensional candidate set for each of the L positions in the sequence.\n\nWe want to highlight that our work places a special emphasis on addressing a critical scaling challenge inherent to simplex-based methods, which is detailed in Appendix C.1.2. We identified that performance can degrade when the vocabulary K exceeds the model's embedding dimension. We successfully mitigated this issue by adjusting the network architecture to be wider, which provides the necessary capacity to represent the high-dimensional simplex. This solution is validated by our strong results on the large-vocabulary OpenWebText dataset, as shown in Table .5.\n\n-----\n\n## **Q2：Modeling capabilities** ##\nThank you for this insightful question. We agree that our model's transitions are fundamentally built upon a Hamming-1 principle, where the atomic operation is the addition of a single category to the candidate set. The key distinction of our approach is that, as defined by our transition probability in Eq.3, our framework allows multiple new candidates to be added in parallel within a single step. This provides a more flexible and efficient transition path.\n\n-----\n\n## **Q3：Relationship between SLM and CTMC** ##\nWe appreciate the reviewer raising this comparison. While there are conceptual parallels in the gradual removal of information, our SLM framework cannot be directly formulated as a standard CTMC with a single absorbing mask.\n\nInstead of augmenting the vocabulary with a single [MASK] token, SLM operates on a much richer state space of $2^K - 1$ possible candidate sets. The transition path is not to a single absorbing state but follows a highly structured, monotonic inclusion of categories through a hierarchy of these sets, as defined in Eq.2. The generative process is one of guided \"candidate pruning\" from an all-inclusive state down to a single choice, a mechanism distinct from generating from a single absorbing mask state.\n\n-----\n\n## **Q4：Relationship between SLM and uniform discrete diffusion approaches** ##\nWe appreciate the reviewer’s opportunity to clarify this point. In our view, SLM differs from uniform discrete diffusion methods. Firstly, the noisy state in SLM consists of tensors filled with '1's, resulting in identical outcomes for all tokens. In contrast, uniform diffusion achieves a state of uniform noise across the entire sentence, meaning there is still variability among the tokens. From this perspective, the noisy state in SLM is more akin to masked diffusion.\n\nAnother characteristic of uniform diffusion is that it retains the possibility for each token to transform into any other token, even at the final sampling step. In contrast, in SLM, these possibilities gradually decrease, meaning that by the last sampling steps, only two or three choices remain for each token. This gradual reduction in possible transformations in SLM not only makes the model more focused and efficient but also enables more controlled and coherent sequence generation.\n\nFurthermore, the key difference between our SLM and uniform transition diffusion lies in the structured, hierarchical construction of candidate sets. Considering the following special case of our model: it starts from a delta distribution (a single state), expands to a set of semantically similar items, and finally encompasses the entire candidate space. This ability to encode a hierarchical property in the input space is the fundamental distinction and a capability that standard uniform transition D3PMs lack.\n\nWe strongly agree with the reviewer that it's valuable to discuss the difference between our approach and various types of existing methods. We assure that the contents above will be added to the part of related works in the revision of our paper.\n\n\n-----\n\n## **About Minor concerns** ##\n\nThank you for bringing this to our attention. We will make sure to include the missing reference for BFN in line 46 in our revised version. Unfortunately, due to recent policy changes, we are unable to provide an anonymous link in the rebuttal section. However, we assure you that our code will be released shortly to enable others to fully reproduce our work.\n\n-----\n\n\n## **Q5：Visualization of generation process** ##\n\nIn response to the reviewer’s insightful request for a more detailed look at our model’s sampling dynamics, we had originally intended to include a detailed heatmap in this rebuttal. However, the current policy restrictions prohibit us from sharing any anonymized or non-anonymized links during this phase. To ensure full transparency, we have therefore provided the complete analysis of the heatmap below and will try to make the figure itself available via public comment (in Markdown) and include it in the revised manuscript PDF. We appreciate your understanding.\n\n\n**One example row of our heatmap:**\n\n\n-----\n\n\nt=0.0000: ......\n\n......\n\n\nt=0.7340:\n$\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Blue}{\\texttt{r}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Blue}{\\texttt{n}}\\color{Blue}{\\texttt{u}}\\color{Blue}{\\texttt{m}}\\color{Cyan}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Blue}{\\texttt{r}}\\color{Cyan}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{White}{\\texttt{\\*}}\\color{Blue}{\\texttt{t}}\\color{Blue}{\\texttt{h}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Cyan}{\\texttt{\\*}}\\color{Cyan}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{White}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{White}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Blue}{\\texttt{a}}\\color{Magenta}{\\texttt{\\*}}\\color{Blue}{\\texttt{o}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{White}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Blue}{\\texttt{u}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Blue}{\\texttt{n}}\\color{Blue}{\\texttt{t}}\\color{White}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Blue}{\\texttt{y}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Blue}{\\texttt{s}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Blue}{\\texttt{t}}\\color{Magenta}{\\texttt{\\*}}\\color{Blue}{\\texttt{a}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Blue}{\\texttt{u}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Blue}{\\texttt{i}}\\color{Blue}{\\texttt{c}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Magenta}{\\texttt{\\*}}\\color{Blue}{\\texttt{n}}$\n\n\n......\n\n\nt=1.0000: .....\n\n\n-----\n\n### **Setup and Figure Explanation:** ###\n\nWe denote any token that has not yet been determined by \"*\".  Unlike masked-diffusion methods (e.g., MDLM), we color-code the asterisks to reflect the current size of the candidate set: Light shades indicate larger candidate sets, Dark shades indicate few remaining choices. Under this scheme, sampling with SLM process resembles a waterfall: as time progresses, each position’s color deepens, signifying convergence toward its final token choice.\n\n### **Key Observations:** ###\n\n1. **Early Pruning, Late Decisions**\n\nIn the example, most positions begin eliminating unlikely candidates early on—even if they won’t finalize their choice until later, which ensures that plausible tokens remain under consideration.\n\n\n\n2. **Convergence Orders**\n\nIn the example, short or common tokens(like \"th\", \"ent\", \"num\") tend to converge first as they are easy to predict, while middle and rare tokens often remain ambiguous until the final iterations, as they may be the semantic core of a sentence.\nThese patterns align with human intuition that simple parts of speech and frequent words are decided early, while context-sensitive or infrequent tokens require more sampling time to resolve.\n\n-----\n\nFrom our perspective, those observations show the advantages SLM has compared to masked diffusion and uniform diffusion. On the one hand, SLM has exact control of coverage in sampling, which is convenient for researchers to judge in what steps things could go wrong(when reasonable choices do not exist in candidate sets). On the other hand, SLM kept the chance for the model to think and correct its mistakes in the last few steps, which will benefit in harder tasks."}]

Author rebuttal:

Meta-review: The authors introduce a new shortlisting diffusion model, which differs from standard masked diffusion and uniform discrete diffusion approaches. Reviewers are enthusiastic about this novel modeling framework, and I am also inclined to recommend acceptance.

That said, I encourage the authors to include additional experiments and incorporate comparisons with existing methods, as presented during the rebuttal, in the final version.
Decision: Accept (poster)