Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Authors: Ruixiang ZHANG
Shuangfei Zhai
Yizhe Zhang
James Thornton
Zijing Ou
Joshua M. Susskind
Navdeep Jaitly
Venue: ICML 2025 poster (2025)
Keywords: discrete diffusion
concrete score
discrete flow matching
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We propose Target Concrete Score Matching as a holistic framework for discrete diffusion modeling.

Abstract: Discrete diffusion is a promising framework for modeling and generating discrete data.
In this work, we present Target Concrete Score Matching (TCSM), a novel and versatile objective for training and fine-tuning discrete diffusion models.
TCSM provides a general framework with broad applicability. It supports pre-training discrete diffusion models directly from data samples, and many existing discrete diffusion approaches naturally emerge as special cases of our more general TCSM framework.
Furthermore, the same TCSM objective extends to post-training of discrete diffusion models, including fine-tuning using reward functions or preference data, and distillation of knowledge from pre-trained autoregressive models.
These new capabilities stem from the core idea of TCSM, estimating the concrete score of the target distribution, which resides in the original (clean) data space. This allows seamless integration with reward functions and pre-trained models, which inherently only operate in the clean data space rather than the noisy intermediate spaces of diffusion processes.
Our experiments on language modeling tasks demonstrate that TCSM matches or surpasses current methods. Additionally, TCSM is versatile, applicable to both pre-training and post-training scenarios, offering greater flexibility and sample efficiency.

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Ruixiang Zhang'
Shuangfei Zhai!
Yizhe Zhang' Navdeep Jaitly'
James Thornton Zijing Ou' Joshua Susskind
Abstract
sive (AR) models through innovative training techniques, these models still face fundamental limitations that impede their broader adoption and practical use.
Discrete diffusion is a promising framework for modeling and generating discrete data. In this work, we present Target Concrete Score Match- ing (TCSM), a novel and versatile objective for training and fine-tuning discrete diffusion mod- els. TCSM provides general framework with broad applicabilityy It supports pre-training dis- crete diffusion models directly from data sam- ples, and many existing discrete diffusion ap- proaches naturally emerge as special cases of our more general TCSM framework. Furthermore, the same TCSM objective extends to post-training of discrete diffusion models, including fine-tuning using reward functions preference data, and distillation of knowledge from pre-trained autore- gressive models. These new capabilities stem from the core idea of TCSM, estimating the con- crete score of the target distribution, which re- sides in the original (clean) data space. This allows seamless integration with reward func- tions and pre-trained models, which inherently only operate in the clean data space rather than the noisy intermediate spaces of diffusion pro- cesses. Our experiments on language modeling tasks demonstrate that TCSM matches or surpasses current methods. Additionally, TCSM versatile, applicable both pre-training and post-training scenarios, offering greater flexibility and sample efficiency.
The current landscape of discrete diffusion models reveals two critical shortcomings. First, existing approaches are fragmented in their theoretical foundations and training methodologies. Methods such SEDD (Lou al., 2024) employ denoising score entropy, while CTMC (Camp- bell et al., 2022) derives objectives from continuous time Markov chains, and approaches like those in (Shi et al., 2024; Sahoo et al., 2024; Xu et al., 2024a) specialize in absorbing state diffusion models with specific assumptions. This fragmentation creates barrier to developing unified and theoretically grounded approaches.
Second, and perhaps more significantly, current discrete diffusion models predominantly focus on pre-training, largely neglecting the crucial post-training phase that has proven essential for downstream task optimization au- toregressive models. While AR models benefit from well-established post-training techniques such as reinforce- ment learning with human feedback (Ziegler et al., 2019; Ouyang et al., 2022; Bai et al., 2022), direct preference op- timization (Rafailov et al., 2023), and knowledge distilla- tion (Gu et al., 2024), discrete diffusion models lack com- parable capabilities. This limitation significantly restricts their practical applicability and prevents them from achiev- ing performance parity with AR counterparts in many real- world scenarios.
Contributions We introduce Target Concrete Score Matching (TCSM), a novel framework for discrete diffusion models based on the concrete score (Meng et al., 2022). By operating in the clean data space, TCSM seamlessly in- tegrates reward functions and pre-trained models while in- tegrating pre- pre-training and post-training. Our key contribu- tions are:
1. Introduction
Discrete diffusion models have emerged as transforma- tive paradigm in generative modeling, achieving remark- able success across diverse domains. Despite their ad- vancements in closing the performance gap with autoregres-
We develop the general TCSM framework for discrete diffusion models (Sec. 3), which provides flexibility across various diffusion formulations and model pa- rameterization.
'Apple Inc. "Now Google DeepMind 'Imperial College London, work done during internship at Apple. Correspondence to: Ruixiang Zhang cruixiangz app...com>.
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
We showcase the effectiveness of TCSM in pre-training contexts (Sec. 4). This includes the development of ef-

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
ficient Monte Carlo estimation techniques for training discrete diffusion models directly from data samples (Sec. 4.1), methods to expedite training through the use of parametric target distribution models (Sec. 4.2), and offers a perspective for contextualizing several existing discrete diffusion methods within our frame- work.
Similar the continuous flow matching model (Lip- man et al., 2023; Liu et al., 2023), we construct probability path px(x)) interpolating between pu and pr. By conditioning on X1, we build a probability path Pe (x.) = Pr (x1) (x))x)). The marginal velocity u4 (x(yx) generating probability path pe(zs) can be computed by x xI), where pij,(xi = the true conditional distribution predict- ing clean data x1 from noisy data x., and we(y,, x xxx)) is the conditional velocity generating Puja (xa(xi))
we (y,, xx) =
in (x1 (eg
We explore the application of TCSM in various post- training scenarios (Sec. 5). This encompasses reward- guided fine-tuning for optimizing downstream tasks (Sec. 5.2), preference-based fine-tuning (Sec. 5.3), and the distillation of knowledge from pre-trained au- toregressive models (Sec. 5.4).
Training The goal is to approximate the veloc- ity (x,y))) using a neural network. We can pa- rameterize the velocity ux(y)x) directly, and opti- mize the conditional flow matching loss CCM (u(0p) (x))))) Dp (we(ye, uf (y,,,, where we sample time t from distribution w(t), and Dp (u, v) F(u) F(v) (VF(v), u v) is the Bregman divergence with respect to the strietly convex function F. We also need to make sure that v/ (yr, xx) satisfies the rate conditions.
2. Preliminaries
Notation Let S xL be our discrete state space, where X {I...., V) is the vocabulary, and L is the sequence length. x = (x',....x") x²] E S, where x¹ E X is the i-th token in the sequence. The notation x** is used to indicate all tokens in the sequence except for the one at position i. When referring to a sequence with a specific token ys at po- sition i, we write [y', xx*]= = [a',.....zt
As shown above, the velocity is governed by the true de- noising distribution (x)+(x)(x)) so instead of parameteriz- ing the velocity directly, we can use a model pl (x)()) to approximate (i)((x xx) by minimizing the loss
y xx11 ="].
For any token E X, we denote its one-hot vector represen- tation as e, RV The function 8(1, u) returns 1 if i y and 0 otherwise. Additionally, we designate a special mask token M E X' to serve as an absorbing state in the discrete diffusion model.
C = BBtt) (x. ( (Plc(())/x) (x:)
(Px)((x(xx)IIP(ieee (Ix()((x((xx)) (xx)) (I)
where D (II-)- is some statistical divergence. For exam- ple (Campbell et al., 2024) uses the KL divergence which gives rise to the cross-entropy loss Fxxxx - log Ph, (xx) which has been shown to be an upper bound on the neg- ative model log-likelihood of the target data distribution. Ce is often called the data data-prediction loss, the model (x (x,) is trained to predicts the clean data x, from the noisy data x, by aligning to the true denoising distribution Pije(x1|xx xx)).
Continuous Time Markov Chains Model The Contin- uous Time Markov Chain (CTMC) model is an S-valued time-dependent family of random variables (xx))eto,) that form Markov chain characterized by the probability tran- sition kernel (x|x) = ō(y, x) + uz(y, x) + o(At) with the initial distribution of the process at time 1 = 0 as Po (xx).. mely,x) x): : S x S + R is called the velocity or the rate matrix, which indicate the speed at which the probabil- ity transitions between states. To make sure the transition probabilities P1 (y/x)) are normalized, (y need to satisfy u. (y, x) 20 for all y A and E, xc(y,x)= = 0.
3. Target Concrete Score Matching
In this section, we introduce Target Concrete Score Match- ing (TCSM), a novel framework for training discrete dif- fusion models. We first present the general formulation before exploring specific instantiations in subsequent sec- tions.
Discrete Flow Matching We use the discrete flow match- ing (Campbell et al.. 2024; Gat et al., 2024) as a general framework to introduce the discrete diffusion models. Our goal is to transfer samples Xa ~ (Xo) from a source distri- bution po to samples x1 ~ pi (x:) from target distribution p1. Source and target samples can be related by means of or asso- ciate by means of general coupling (xo. x1)) For in- dependent coupling, common choices for the source distri- bution is either unif (xa) = I1/ 4. a uniform distribution over S: and (ii) p potk (xo) = II' 8(M, =d). a delta measure concentrated on the absorbing state M.
At the heart of our approach lies the concrete score (Meng et al., 2022), which serves as discrete analog to the con- tinuous score function V log used in continuous dif- fusion models.
the independent coupling x1) ~ po(x))))
Definition 3.1 (Concrete Score (Meng et al., 2022)). Let p(x) be any discrete distribution over S. We denote N : S + S*** as the function mapping each example x E S to a (multi)set of neighbors, such that N(x)= = {X (nec) and Kx = IV(x)). The neighborhood-
2

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Domain
Approach
Target Object
Target Quantity
Discrete
Targer CSM (Ours)
Concrete Score of pr
(x)) P1(81) sf
( (Xaa yeste
Discrete
Denoising CSM(Lou et al., 2024; Meng et al., 2022)
Concrete Score of (-bx))
Continuous
Target SM (Bortoli et al., 2024)
Score of P1
Vx, log (18Pi(xx (x1)
Continuous
Denoising SM (Vincent, 2011; Song et al., 2021)
Score of (-(x(-
Vx, log Paja (xx
Table 1: Comparison of score matching objectives across continuous and discrete domains. The key distinction lies in whether the target quantity is derived from the clean data distribution (p1) or the forward noising kernel (p:j1). SM =M==B Score Matching, CSM = Concrete Score Matching.
induced graph is the directed graph which results from adding directed edge from x each node in its neigh- borhood set X.. E (x),,, for all x E supp(p(x)). The con- crete score for given distribution p(x) evaluated at x is pllnn) -,,... plxng) -1] We define cp(x:N) S p(x)
The effectiveness of our approach fundamentally relies on the connectivity of the graph G induced by the neighbor- hood definition W. To satisfy this requirement while offer- ing flexible levels of granularity, we introduce family of neighborhood structures based on Hamming distance.
g! by a constant shift of 1, for notational convenience.
Definition 3.2 (k- Hammingg Neighborhood), For any se- quence E S and integer k 2 1, the k-Hamming neighborhood is defined as N* Glamming-distanne y) S k), comprising all sequences that differ from x in al most le positions.
{y S
cy(x:NN) = [((xn)) p(x)
p(xmg] p(x)
This family of neighborhood structures provides a flexible framework for TCSM, as A induces a weakly connected graph for any S h s J. By varying k, we can create spectrum of TCSM objectives that balance local and global perspectives. The smallest neighborhood A focuses on immediate neighbors with single token differences, while ^ = N²' encompasses the entire sequence space.
Our approach builds upon the discrete flow matching frame- work (Campbell et al., 2024; Gat et al., 2024) by adopting the data-pn preeddidiiiinnn objective in Eq. (1). This objective of- fers crucial flexibility, remaining valid for various model architectures and naturally supporting different probability paths without structural changes.
Target Concrete Score Matching We now introduce the target concrete score matching (TCSM) objective, which aims to align our model denoising distribution 6 (xx() with the true denoising distribution (x/()x/))). by match- ing their respective concrete scores, Syg, (xx) and
TCSM with 1-Hamming Neighborhood When applying the TCSM framework to the 1-Hamming neighborhood where sequences differ by at most one token we can rep- resent the concrete score cx; (xx:) as a V x L matrix by replicating the original sequence x L times, with each col- umn i defined as:
Cpy (xx).. The general TCSM objective function is given by:
By decomposing the TCSM objective in Eq. (3) into I. groups based on their sequence positions, the TCSM objective can be expressed as:
pla) -aa)=xx710 p(xix))
CrOSm (O:N,D,h)) D, h)) (xt (Smiie oppp)
(3)
where l((x[xx)) serves as a proposal distribution
probability mass function that ensures supp(pi)((x) (x:) S (o)((((xxx))).....)))))))>)>>))) The term D represents general divergence measure that quantifies the discrepancy between the con-
Escore (0:N¹, D,A) (U
Ecaore
(4)
((pinted ((pppute x xx)) PIle(e x (x) =f=1
pi Pi xf
(xe)
Geooe D
crete scores.
(af. x.))
Proposition 1. Let.N define neighborhood structure that induces a weakly connected graph G over the support of p))())xx). Assuming mild regularity conditions on the di- vergence measure D, the global minimum of the TCSM ob- jective CrOSm in Eq. (3) guarantees that o pf((--xx) equals p1)e()Ix)) almost everywhere with respect to p(x.).
This objective is termed the score-based TCSM (Cscore) as it directly operates on concrete scores. Alongside the score- based objective, propose another objective centered on distribution matching:
Cosatrso D, = Eu(()(xs))
E Axxt (os) Cdistrib: (5)
Lasrrs = 1D (plo(elixt* xx) le) xe)
Proof. Please refer to Sec. B.
3

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
The Leistrib objective transitions from matching joint dis- tributions Cpyje (x) x)) via concrete score matching to align- ing conditional distributions P xe). This objective uses statistical divergence ID (- |l -) to quantify differences in probability distribution space, setting it apart from the score-based method.
Type Lasstrib Cdistrib
Source
Div.
Param.
Model
M
KL
Fact.+
MD4/MDLM
M/U
KL
Fact.
DFM
Ldistrib
M
f-div
EBM
EDLM
Table 2: Existing discrete diffusion models under the TCSM framework with different choices of source distribution (M=Mask, U=Uniform), divergence measure, proposal (PIn(x (x) for all), and parameterization (FactieFac,,rized, Fact-+=Factooried with carry-vved EBM=Energy BBsed Model).
The following theorem demonstrates that both Score and Ldistrib are effective for aligning the concrete score be- tween the true distribution and the model distribution.
Proposition 2. Under mild regularity conditions, the score-based objective Lecore Eq. (4) achieves its global minimum if and only if the distribution-busel objective Caistrib Eq. (5) achieves its global minimum, where the general TCSM objective 3 is minimized.
Target Concrete Score
To gain more insights into the Lscore and Luistrib objectives, examine their respective targets: the concrete score ratio (
[Pi)e(rr(ss] and the condi-
tional distribution P1]((+Ixf') xx).
For the score-based objective, we can decompose the tar- get as [P p1 Paj (xxlx1))
Proof. Please refer to Sec. B.2.
[Paje(<]|o)) (1 (x)]]]))1]] This shows that weighted version of
Pije [n](x)]xe [x,)s concrete score
pi (x1)'s concrete score, with weights from the probability path Pr 1 (x:[x:):
Practical implementation of Lscore and Caistrib requires choosing two essential elements: the divergence metrics D (-.-) (or TD (-Il (-Il-)) and the proposal distribution h(x1/x)) We'll explore specific example of these choices to better understand how the score-based and distribution-ban ob- jectives are implemented and connected.
[Con (x]](x))))] [cp1 (xx)ly
Pii (xxly) Pij (x+|x1)
(7)
Here, [cly, indexes the concrete score position y1. The distribution- objective reveals an analogous relation- ship:
TCSM with Gen KL Let us employ the gener- alized KL divergence, a specific instance of the Bregman divergence Dr -) with function F(u) = E, uy log uy which takes the form Dr (u, v) =,
Example:
Pij [xf] xx) or pı (of |x] Ppp (x//x1)
(8)
-,,,, uy log "i uu To streamline our notation, us define the ratio of conditional probabilities (v) = - (xal=/////////pddeee px))/pujcs x and wi (v) == tation, we can express the objective Excore in Eq. (4) as:
xx)) xx) Cat softmmaa Clog
Cpse (ppeeel (21
(x]]x(" xx)))
Thus pup (-xff xx) constitutes a weighted transformation of pi (- within the target distribution space. The con- ditional distribution Ptx+-Ix('' x.) can be interpreted a probability-normaline instance of the concrete score Opne
(x))/// (x)) Using this no- xx)).
(()) log
- (1) + wij((v)) (6)
Accore
5.00 (v)
These highlight a crucial distinction between our target concrete score matching (TCSM) framework and traditional denoising score matching approaches (Song et al., 2021; Lou et al., 2024). Unlike denoising score matching, which operates through the lens of the noising process Pap (x) TCSM directly engages with the clean data distribution pr. TCSM aligns with established methodologies in continuous diffusion models (Bortoli et al., 2024). We summarize the relationships and the contrast with conventional denoising score matching objectives across both discrete and continu- ous domains in Table 1.
Proposition 3. Under the proposal distribution t(x1) (x) (x))))))))) the score-based objective with generalized KI.
divergence is equivalent the distribution-hase basedd objective with a weighted combination of forward KI. and Bakura- Saito (IS) divergences:
Lucore(;; h = Pije, D = Doa (, =
Ldistrib (0; Pije, D = VIDa + Dis)
where Da represents the forward KL divergence, and IDts denotes the Itakura-Saito divergence.
4. Pre-training with TCSM
Proof. Please refer to Sec. B.3.
Building upon the general TCSM framework in Sec. 3, we present two approaches for pre-training discrete diffusion models. First, in Sec. 4.1, we develop Monte Carlo estima- tion methods for the Escore and Ldistrib objectives using only empirical data samples from the target distribution pi. Second, in Sec. 4.2, we demonstrate how TCSM allows one
This equivalence demonstrates that the score-based and distribution- approaches yield identical optimization objective when using the true conditional distribution as the proposal and appropriate divergence measures.

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
to incorporate parametric models of pr significantly ac- celerate the training of discrete diffusion models.
Shi et al., 2024; Sahoo al., 2024), we can further simplify our objectives by adopting factorized parameterization: (x; (xx) = II pl (xi(x)), This leads to the following simplified Score objective:
4.1. TCSM with Data Samples X1 ~ P1
Problem setting The target distribution is the true data dis- tribution yi (x1) Pitaa and we only have empir- ical dataset sampled from (x1). We want to match (x) to (x) with the TCSM objective.
(- 1o (logphjc(=((x)) + VDL (=)((())/xx)
Cacore =
(10)
Σ log (1 V
(11)
Score based TCSM We begin with the score-based Licarre objective introduced in Eq. (4).
Proposition 4. When using forward generalized KI. diver- gence as the discrepancy measure and setting the proposal distribution the true conditional distribution (x)((x)())) the score- base-based Lscore objective in Eq. (4) can be expressed as:
The distribution-based TCSM objective also simplifies to: aastrrib = - -Bp,, log (x) +
Joint Parameterization In Sec. 5.1, we demonstrate exam- ple of applying our framework to models that parameterize the joint distribution without factorization assumption.
Gecoe=
[Eeneeddo + &eennprr+ +
The TCSM framework offers a unifying perspective, allow- ing several existing discrete diffusion methods, including MD4 (Shi et al., 2024), MDLM (Sahoo et al., 2024), and DFM (Gat et al., 2024), to be viewed through the lens of target concrete score estimation under specific configura- tions (c.g., choices of divergence, model parameterization, and probability path). This viewpoint highlights common principles while acknowledging the unique aspects of each method. We summarize these relationships and differing choices in Table 2.
log loggljc(01/x ;xxx xx)
Vpl (xt
Cownagy
log Pi o
Proof: Please refer to Sec. B.4.
Analysis of the Objective The objective consists of two additive terms that serve distinct purposes. The first term, fpsasdos maximizes the pseudo-likelihoud of the denoising model (x((x)) with respect to the data distribution. xx)),
Experiments We now empirically validate the effective- ness of using TCSM for pre-training discrete diffusion mod- els on language modeling tasks. We measure both perplex- ity. We use the same transformer-based model architecture as in (Lou et al., 2024) for all experiments. See Sec. C.I for more experimental details.
The second term, Cameey = (Uniifor())) guides the denoising model toward making more precise and confident predictions through cross-entropy maximiza- tion for p xx). This objective provides a practical optimization objective that relies solely on samples from the joint distribution Xi).
ΤEX18 We conduct experiments on TEXT8 character level language modeling tasks. We adopt a factorized model parameterization for all experiments. We explored using both Lecore Eq. (10) and Ldistrib Eq. (9) objectives for pre- training; as well as both uniform and absorbing source dis- tribution for pre-training. We show the results in Fig. 1b.
Distribution based TCSM For the distribution-ban based Laistrib objective Eq. (5), it is straightforward to derive a simple objective when using forward KI. divergence and Pile as the proposal distribution. After dropping constant terms, this yields cross-entropy based objective:
Castrri - Epp log Pi (xixxt x.) + C,
(9)
WERTETT We also conduct experiments on larger scale OPENTEBTEXT dataset. We pre-train the model with factorized parameterization using Lscore and Ldistrib ob- jectives. Following previous works (Lou et al., 2024; Shi et al., 2024), we evaluate the zero-shot perplexity of trained models and show the results in Table 3.
where C is a constant term. In contrast to the objective in Eq. (1), which maximizes the conditional joint data like- lihood logr" pf (x1 (x.). our approach maximizes the pseudo- likelihood of the denoising model Σ. log (x) xx).
Flexible Model Parameterization The Lscore and Caistrib objectives are versatile and can be applied regard- less of the specific parameterization of pf (x1 xx). The only requirement is the efficient estimation of the condi- tional distribution (xix(x x,) during training.
4.2. TCSM with Parametric Model p1
Discrete diffusion models often encounter challenges such as slow convergence and reduced sample efficiency com- pared to autoregressive models. We show that TCSM can help to mitigate these issues by employing parametric mod- eling of the target distribution P1 (x1).
Factorized Parameterization Following established dis- crete diffusion models (Gat et al., 2024; Lou et al., 2024;
5

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Method
LAMBADA
PTB
WikiText
1BW
AR
GPT-2 (WehText)"
45.04
138.43
41.60
75.20
D3PM
S 93.47
S 200.82
S 75.16
S 138.92
CD
Plaid
S 57.28
142.60
S 50.86
S 91.12
DD-U
SEDD (Lou et al., 2024)
A 65.40
S 140.12
< 49.60
< 101.37
DD-U
TCSM Lacore Sec. 4.2)
S 63.84
138.95
S 50.73
S 100.46
DD-U
TCSM Laistrib Sec. 4.2)
< 65.29
133.67
46.91
98.52
DD-M
SEDD (Lou al., 2024)
< 50,92
114.24
< 40.62
79.29
DD-M
MD4 (Shi et al., 2024)
48.44
102.26
35.90
68.10
DD-M
MDLM (Sahoo al., 2024)
$ 47.52
S <95.26
S 32.83
67.01
DD-M
TCSM Lasstrub See. 4.2)
48.37
S 101.85
S 34.92
68.43
DD-M
TCSM Casstrib Sec. 5.1)
5 47.29
S 96.71
31.56
S 65.82
Table Zero-shot unconditional perplexity (4) of model trained on OPENWeRTex" dataset. "The GPT-2 numhers are reported for the GPT-2 checkpoint pretrained on WebText instead of OPENWER
Parametric Estimation of Target Score Building on the observation in Eq. (8) that learning pije(-Ixf', xx)) effec- tively reduces to learning pi (-Ix{") in the target distribu- tion space, we can employ dedicated neural network to parameterize Pı (af "x providing an efficient estimation of ( xx). We explore following strategies for para- metric estimation of pi (x][x Importantly, the learned parametric target estimation remains invariant to the choice of probability path, making it reusable across different dif- fusion transition kernels.
as TCSM-Bert; (iii) pre-trained hollow transformer model, denoted as TCSM-Hollow. We train the model for 26 billion tokens OPENWEB dataset and report the perplexity validation set in Fig. la. We also plot validation NLL loss curves in Fig. 4. We can see that with the help of para- metric p1 model, the training process of discrete diffusion model is consistently faster.
5. Post-training with TCSM
TCSM provides versatile framework that extends beyond pre-training to enable effective adaptation across a wide range of post-training scenarios. By utilizing the same TCSM objectives outlined in Sec. 3, we can effortlessly adapt to diverse post-training scenarios through tailored in- stantiations of the target distribution, divergence measure, and model parameterization. We illustrate this adaptabil- ity through four key applications: (1) fine-tuning with pre- trained models as parametric estimators of Pije (Sec. 5.1), (2) reward optimization for downstream tasks (Sec. 5.2), (3) preference saaeee fine-tun finetunnig (Sec. 5.3), and (4) knowledge distillation from autoregressive models (Sec. 5.4).
Pre-trained BERT/AR Models Unlike previous ap- proaches operating in noisy data spaces xt, our method fo- cuses exclusively on clean data at = 1. This perspec- tive creates a valuable connection between TCSM diffusion models and other models trained on clean data. We can leverage existing pre-trained models like BERT (Devlin et al., 2019) or autoregressive language models to estimate (x While BERT directly provides this distribution through masked token prediction, autoregressive models require marginalizing over the vocabulary: Pi (ri xxf) = (x1//2 pi(vi " See Sec. 5.4 dedicated to distilling au- toregressive models.
Hollow Transformer As introduced in (Sun et al., 2023), the hollow transformer employs two autoregressive Trans- formers per layer, one operating left-to-right and the other right-to-left. In the final layer, the representations ((55) and fx+²*) are combined via attention to form f(x("),
5.1. TCSM Fine-tuning with a Parametric Model Pije
In a similar spirit to Sec. 4.2 where we have a parametric model of Pt, we now consider scenarios where we have a parametric model of Pitt. such as a pre-trained discrete dif- fusion model. This is particularly useful for post-training applications such weak-to-strong fine-tuning (Burns et al., 2023; Chen et al., 2024), where we can enhance a weaker Pid: model to a stronger one with expanded capabil- ities.
which is used to predict the missing token zi. This archi- tecture allows for efficient estimation of Pı (=f x((") for all
1 igis L in a single forward pass.
To validate the effectiveness of parametric target estimation in accelerating discrete diffusion model training. conducted experiments language model- ing. We explore three variants of parametric models of P1: (i) tre-trained transformer autoregressive model, de- noted as TCSM-AR; (ii) pre-trained BERT model, denoted
Experiments
Problem Setting We consider an unknown target distri- bution Ptarget = pı (x1) from which we can sample. We assume access to parametric reference model pff such as a pre-trained discrete diffusion model, smaller version of the same model, or weaker version from earlier train-
6

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
F(r) objective Eq. (12)
(i) Parameterize ratio v. by model pf
Parameterize model pijs by ratio 1/ exp(So)
LSII (r=1)//a
Va (V=((111/19)) (01ic/ Erzle (Fiuleff))
Bypp (rep(2/0)/s) Eyije exp(fo)
BCE y logr - (r 1) log(r + 1)
log(( (Tog Pije/s9(.)) Frilo logio (in
Eag log(1 - a[fo]) + Frin lon(o(f())
Pilt
GEN. klrggr--
B Pife
(f11//997) - Euase log 1e/eff
Bagt exp(fe) - Buije
Table 4: Ohjective functions for various density ratio parameterizations and choices of F as in Sec. 5.1. (z) the sigmoid function.
ing steps. The goal is to leverage Pife to learn an improved model pije that better approximates the true distribution.
noising distribution pat = Pije 9e-1
The density ratio model is
parameterized as ( [x x() = - xx)
xx.)
This
Pik [x xx)
Density Ratio Estimation Our approach leverages the reference model Pije through density ratio estimation Build- ing on the Laistrib objective Eq. (5) with . neigh- borhood structure, we denote the density ratio as Pipffil .x). Given the true den- sity ratio (( we minimize the divergence (Pur (1)))) Dy (Muaeff I1²) to align pfj with Pije- The core challenge thus lies in estimating f xx). We address this by parameterizing our density ratio model as r" (at 1 xi) and using Bregman divergence (Sugiyama et al., 2012) to estimate it:
jet
will give us a procedure similar to (Chen et al., 2024). Also,
we can use the exponential moving average of the denois- ing distribution as the reference distribution, p3t Sens
between the true and reference distributions.
Pre-trained model as reference We can also set the ref- erence distribution to be a pre-trained discrete diffusion model Pij. (x][(xx (x1/xx) = Pije (x1 xx). We use the (ii) parame- terization strategy (x xx) x p (x: (x11x))) (x1/x))
(xl((x xx)
pff off.aafle
Experiments We evaluate our TCSM post-training den- sity ratio estimator on language modeling, focusing on parameterization strategy (ii), which uses density ratios to characterize the denoising model (strategy (i) is ex- plored in Sec. 5.3). Using pre-trained models with Cdistrib (see Sec. 4.1), we train density ratio model with three esti- mators (LSIF, BCE, Generalized KL), as detailed in Alg. 1. We utilize pre-trained models from Sec. 4.1 the TEXT8 and OPENWEB TEXT datasets, and enhance them by ap- plying the proposed density ratio estimation post-training methods. The results are presented in Fig. lb and Table 3. The results presented in Fig. lb and Table 3 and summa- rized for different Bregman divergences in Table 5 consis- tently improve over the baseline across all configurations, showing robustness to divergence choice. See Sec. E for further analysis and implementation details.
E "ife (ef xx) Dr (M(al)xx" xx), r (aiixtt,, (12)
Density Ratio Parameterization A straightforward method involves independently parameterizing both the density ratio model (x] (x ,x,) xx) and the denoising model (= xxx x.). Once the density ratio model is optimized using Bregman divergence minimization, result- ing in the optimal model r' ''ailxx xt), we face the task of solving the optimization problem mine D(r"p"", p²) to align p" with p. However, this two-stage process, alternat- ing between density ratio estimation and divergence mini- mization can be adversarial, unstable and is difficult to con- verge, we discuss more in Sec. E. Instead, we propose al- ternative strategies with implicit parameterization: (i) Pa- rameterizing the density ratio model in terms of the de-
Model
Perplexity (L)
MDLM (Sahoo et al., 2024)
23.83
noising model as Tijt -0=0 (oi [x xx) = P [=] xx))
"T. (ell or (ii)
EDLM NCE (Xu et al., 2024a)
21.52
TCSM BCE (Reimpl.)
21.87
Parameterizing the denoising model in terms of the den- sity ratio model as (x1
TCSM LSIF
22.10
(x)) (x] (x)])) The equality holds when the density ratio model is optimal where , is self-normalized. To ensure that pi is always properly normalized in practice, we define pf (x) = pff. (xx bx (xll))/ Pf (x. jectives resulting from these parameterizations under com- mon Bregman divergences summarized in Table 4.
(x.)
TCSM Gen KL
21.74
Table 5: Comparison of perplexity scores across different Breg- man divergence formulations in TCSM framework. Baseline num- bers are from (Xu et al., 2024a).
The specific ob-
5.2. TCSM Fine-tuning with Reward Optimization
Reference Models With the density ratio model parame- terized, we consider two specific reference models nnd
Problem Setting We address the challenge of fine-tuning pre-trained discrete diffussin models for specific reward functions R S + R. While rewards may sometimes require learning from external feedback (Ouyang et al.,
Weak model as reference At each optimization step k. we can set the reference distribution to be the previous step de-
7

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Type
Method
BPC (4)
CD
Plaid (Gulrajani & Hashimoto, 2023)
S 1.48
Parplcity OpesWWdTTor Tort Validation Set (268 tokona)
CD
BFN (Graves al., 2023)
S 1,41
Type
AO-AR
MAC (Shih al., 2022)
I 1.40
Baseline
34
TCSM
AR
Transformer AR (Austin al., 2021)
1.23
32
DD
D3PM Uniform (Austin al., 2021)
1.61
SEDD Uniform (Lou et al., 2024)
1.47
Parploxity
30
DD
TCSM Uniform Cscre (Sec. 4.2)
I 1.47
DD
TCSM Uniform Casstrub (Scc. 4.2)
S 1,45
DD
SEDD Absorb (Lou et al., 2024)
< 1,39
DD
MD4 (Shi et al., 2024)
S 1.37
DD
EDLM (Xu al., 2024a)
1.24
26
TCSM Bot
TESM-AR
TCSM Hallow
TCSM Absorb Lscore (Sec. 4.2)
S 1.38
24
SEDO
MOLM
DD
DD
TCSM Absorb Caistrib (Sec. 4.2)
TCSM Absorb Laistrib (Sec. 5.1)
S 1.37
S 1.25
Bits Per Character (BPC) on TEXT8 test set. CD=Continuous Diffusion, DD=Discrete Diffusion, AR=Autoregressive, A0=Any-Order,
(b)
(a) Comparison of perplexity the OPENWERTEXT validation after training for 26B tokens: TCSM vs. baseline models.
BLACO
Figure/ TCSM Reward vs. Entropy in Figure 3: TCSM toxicity generative per- Ber-Figure Validation loss curves compar- ing different TCSM variants on Open OpenWehText. Lower is better.
IMDB sentiment fine-tuning (Sec. 5.3). plexity Sec. 5.2.
2022), we focus on scenarios where the reward is ei- ther explicitly known or has been successfully learned. Given pre-trained model (x1) trained on the
target is essentially the original score adjusted by the reward function. Given that we have pre-trained model trained to align with the target distribution score
data distribution pa (x(x). our objective is to align it with reward-modulated target distribution: Pearges = p (xi) Ex, P [xp( where & controls the trade-off be-
[((((()))]] we can approximate this using the pre-trained (aje(xxxx))
pe (yix))) no (xf.))))))
(1(x)) ( R(x)) 1/8)
model as follows:
Simi-
tween reward maximization and fidelity to the original dis- tribution. fundamental challenge arises from the lack of ground truth samples from pf (x1), we only have cess to unnormalized density evaluations through the re- ward model.
larly, for the target distribution the Ldistrib objective,
R (x .x)) within have: (x xx)
Pij (=i|xf xx) xx(=(((((()//))) which can also be approximated using the pre-trained model as: pij, (xilx x.) (xllxxt' x.)
Reward-modulated Concrete Score Let us an- alyze the score of the reward-modulated target distribution which takes the form: R (x)))) oc (xxxxxx)x))xxx(((x)))) exp(R(x)//B).
Experiments To validate our reward optimization method- ology, we conducted experiments on both synthetic and real-world tasks: (1) a synthetic 2D grid experiment demon- strating the model's ability effectively suppress unde- sired modes after fine-tuning Fig. and (2) toxicity mitigation task for language generation where our ap- proach achieved superior performance compared to exist- ing methods like MDLM with Best-of-N sampling, as
The score is given by off, (xix)) = ())()(m() exp (R())) R as the partition
ef xx) ())()))s))
function cancels out the ratio.
This indicates that the score of the reward-modulated
8

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
1-00
Efficient estimation of distillation target As dis- cussed in Sec. 4.2,
leverage pre-trained toregressive language models to estimate (zi [x(") Pi(x)/EE ( However, naively computing this re- quires O(VL) likelihood evaluations of the teacher model for each sequence e (x). While these evaluations can be parallelized. the computational cost remains prohibitive. We propose two efficient approaches to estimate the target concrete score: Top-K and First-order Taylor estimation. We leave the details the appendix Sec.
(a)
(b)
Figure 5: Model generation dynamics: sample distributions at intermediate steps, before and after reward optimization.
Experiments We validate our distillation approach on the OPENWEBTEXT dataset using transformer- AR teacher model and an absorbing discrete diffusion student model, where our method achieves faster convergence and lower perplexity compared to baselines. See Sec. for detailed experimental settings and further results and anal- ysis.
shown in Fig. 3. For detailed experimental settings, com- prehensive results, and analysis, we refer readers to Sec. F.2 the appendix. The complete algorithm for reward-guided training provided in Alg.
5.3. Direct Preference Fine-tuning
6. Conclusion
Problem Setting We present a method for fine-tuning pre- trained diffusion models using pairwise preference data ((q. x()),, where q represents query (instruction), and x and x denote preferred and non-preferred sponses respectively. Our approach directly optimizes for preference alignment without requiring an explicit reward model (Rafailov et al., 2023). The target distribution fo- cuses on preferred responses: Ptarget (x1|q) = (xi qq), with pre-trained diffusion model Pije (x1|q) serving as our reference distribution.
In this work, we introduced Target Concrete Score Match- ing (TCSM) as a principled framework for training discrete diffusion models. By estimating the concrete score in the original data space, TCSM enables effective pre-training and seamless post-training with reward functions, preference data, and pre-trained models. Empirical results on lan- guage modeling tasks show that TCSM achieves competitive performance with greater flexibility and sample efficiency.
Preference Optimization Building the density ratio timation framework from Sec. 5.1, we learn a new diffu- sion model p/1 relative to the pre-trained reference. The density ratio model is defined as: (x xx.9) piillrii""" .x.g)
Optimization follows the objective in Eq. (12), with Monte Carlo estimates computed using samples x"", xi drawn from the pre-trained model. Imple- mentation details are provided in Alg. 4.
Pi (x] x.
Experiments We validate our TCSM preference optimiza- tion approach by fine-tuning a pre-trained model on the IMDB-sentiment dataset using our density ratio estimation framework (Sec. 5.1). As shown in Fig. 2, stronger pref- erence optimization leads higher mean rewards but duced sample diversity. The complete training procedure is detailed in Alg. 4, and further experimental details and results are available in the appendix (Sec. G.2).
5.4. AR + Diffusion distillation
Problem setting We explore knowledge distillation from pre-trained autoregressive model (teacher) pt* (x1) to a dif- fusion model (student), where the target distribution is the teacher model's distribution Pearges = pi (x1).

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Acknowledgment
plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-3555 1952.
We are grateful to Jiatao Gu, Tatiana Likhomanenko, Dinghuai Zhang, Richard Bai, Zijin Gu, Huangjie Zheng. Dan Busbridge, and Jason Ramapuram for their valuable insights and discussions throughout this project. We would also like to acknowledge Samy Bengio for his guidance and support.
Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al. Weak-to-strong generalization: Elic- iting strong capabilities with weak supervision. ArXiv preprint, 0tts/2311.093999 2023. URL https: org/ 2ts/ab//23122 09390
Impact Statement
Campbell, A., Benton, J., Bortoli, V. Rainforth, T., Deli- giannidis, G., and Doucet, A. A continuous time frame- work for discrete denoising models. In Koyejo, S., Mo- hamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (cds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http: http://aapprs..ee nips. cc http//aapprr___ filapprriiieeeeppprr///// hash/bS6522767 http://aa555511aafeaaaaaaaaaaaaa htw.:aa00000000ee5555--00000 act-Conference html.
The paper introduces a novel objective for training and fine- tuning discrete diffusion models. While discrete diffusion models have broad applicabilityy, including language mod- eling and structured data generation, we do not foresee immediate ethical concerns beyond those generally asso- ciated with advancements in generative modeling, such as potential misuse for generating harmful or biased content. Responsible use and further research into mitigating such risks remain important considerations.
References
Campbell, A., Yim, J., Barzilay, R., Rainforth, T., and Jaakkola, Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design, 2024. URL https: wwwpsiiv.rggab///////// .04997.
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den Berg, R. Structured denoising diffusion models in dis- crete state-spaces. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 1021, NeurlPS 2021, December 6-14, 2021, vir- tual, pp. 17981-17993, 2021. URL https: www.neuurs.cccpppprrrreecce.c html.
Che, T., Li, Y., Zhang, R., Hjelm, R. D., Li, W., Song, Y., and Bengio, Y. maximum-likellion augmented dis- crete generative adversarial networks. ArXiv preprint, abs/1702.07983, 2017. URL https://arxiv.orgg 01172207798.
ngs. www.nnurriis... cc /ttp:/ppppppeeee1//ee/ http:///22/aaaaa//555555554////
Chen, T., Zhang, R. and Hinton, G. E. Analog bits: Gen- erating discrete data using diffusion models with self- conditioning. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview wwwpeeeewwweeeeeeeeeeeeeeeeeeeee ptdt///ffidd33ttj000FF
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with reinforcement learning from human feedback. ArXiv preprint, abs/2204.05862. 2022. URL https://arxi v. w.orgg/al 2b25/22..222...... 228.228244........
Chen, Z., Deng, Y., Yuan, H., Ji, K., and Gu, Q. Self-play fine-tuning converts weak language models to strong lan- guage models. ArXiv preprint, abs/2401 .01335, 2024. URL /nttps://arxt org/ hts/ab//2400 /b////21...11335... 01335.
Bortoli, V. D., Hutchinson, M. J., Wimsberger, P., and Doucet, A. Target score matching. ArXiv preprint, abs/2402.086677, 2024. URL https://arxiv.orr org/ ab s/2402. 08667.
Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A., Joze- fowicz, R., and Bengio, S. Generating sentences from continuous space. In Riezler, S. and Goldberg, Y. (eds.), Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pp. 10-21, Berlin, Germany, 2016. Association for Computational Linguistics. doi: 10. 8653/v1/K16- 1002. URL nttps://aclanthot org/K16-1002.
de Masson d' Autume, C., Mohamed, S., Rosca, M., and Rae, J. W. Training language gans from scratch. Wal- lach, H. M., Larochelle, H.. Beygelzimer, A., d' A'Acc Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancou- 2019. URL https:
ver, BC, Canada, pp. cctpseer/epppppppppeaee// http://////////////e//// / / //proceddings perrocediingsnneuripe www.wpsw.ii... 38471c120re8cc35225cccceessss Absteeea html.
Bradley, R. A. and Terry, M. E. Rank analysis of incom-
10

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Deng, Y., Bakhtin, A., Out, M., Szlam, A., and Ranzato, M. Residual energy-based models for text generation. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL. et/forum?id-81144gtte
Gulrajani, and Hashimoto, T. B. Likelihood-based diffu- sion language models. In Oh, A., Naumann, T., Glober- son, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL htt.ppper.flllee//pee///////////
Devlin, J.. Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pp. 4171- 4186, Minncapolis, Minnesota, 2019. Association for Computational Linguistics, doi: 10. 10.18833/1//N11-4422 1423. URL https::/aclaatthl..gg www.e//-.0g/N1111123/0
330F7557225017777fc77774-tttrrf2 html.
Han, K., Kenealy, K., Barua, A., Fiedel, N., and Constant, N. Transfer learning for text diffusion models. ArXiv preprint, abs/2401.17188 2024. URL https: org/abs/ 2401 17181.
Hartmann, J., Heitmann, M., Siebert, C., and Schamp, C. More than a feeling: Accuracy and application of senti- ment analysis. International Journal of Research Mar- keting, 40(1):75-8)) 2023.
Dieleman, S., Sartran, L., Roshannai, A., Savinov, N., Ganin, Y., Richemond, P. H., Doucet, A., Strudel, R., Dyer, C., Durkan, C., Hawthorne, C., Leblond, R., Grath- wohl, W., and Adler, J. Continuous diffusion for categor- ical data, 2022. URL https://arxiv, orgo absabbs// 2211 .15889.
He, Z., Sun, T., Tang, Q. Wang, K., Huang, X., and Qiu, X. DiffusionBERT: Improving generative masked lan- guage models with diffusion models. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pp. 4521- 4534, Toronto, Canada, 2023. Association for Computa- tional Linguistics. doi: 10.18653/v1/2023 acI-oong.2cl--n URL https: https://aclaanthologgy. www./////.....// wel-llnng 248.
Eldan, R. and Li, Y. Tinystories: How small can lan- guage models be and still speak coherent english? ArXiv preprins, 0a222235500775 2023. URL https://arxiis org/abs 22////225550077
Gat, I., Remez, T., Shaul, N., Kreuk, F., Chen, R. T. Q., Synnaeve, G., Adi, Y., and Lipman, Y. Discrete flow matching. ArXiv preprint, 0tt/22477.155955 2024. URL https://arxivv or orgg/bs///000 /////bb//22.... .10.00.555590 15595.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models. In I Larochelle, H., Ranzato, M., Had- sell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020. NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceeddii nww.ips. wttpppprcccppppp/pcppppcp /20/hash htt/ah/h/cccofffosofffffffffffaa bstract. html.
Gong, S., Agarwal, S., Zhang, Y., Ye, J., Zheng, L., Li, M., An, C., Zhao, P., Bi, W., Han, J., et al. Scaling diffu- sion language models via adaptation from autoregressive models. ArXiv preprint, abs/2410.1 17891, 2024. URL https://arxins org/abs/2410 17891.
Graves, A., Srivastava, R. K., Atkinson, T. and Gomez, F. Bayesian flow networks. ArXiv preprint, o0s/2308.00003,, 2023. URL https://arxiv.on s/2308. 07037.
Hoogeboom, E., Nielsen, D., Jaini, P., Forré, P., and Welling, M. Argmax flows and multinomial diffu- sion: Learning categorical distributions. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6- 14, 2021, virtual, pp. 12454-12463 2021. URL https: / //proceedings neur iww.ips..iiieee..ee...
Gu, J., Bradbury, J., Xiong, C., Li, V. K., and Socher, R. Non-autoregressive neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL 18Bt1Cb.
html.
Gu, Y., Dong, L., Wei, F., and Huang. M. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Represen- tations, 2024.
Hsich, C.-Y., Li, C.-L., Ych, C.-k., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-Y., and Pfister, T. Dis- tilling step-by-step! outperforming larger language mod- els with less training data and smaller model sizes. In

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Findings of the Association for Computational Linguis- tics: ACL 2023, pp. 8003-8017, Toronto, Canada, 2023. Association for Computational Linguistics. doi: 10.186 53/v1/2023 htw.1/2203..indngg--al...0000 URL. httttt:///ccaaatt orgyoorg/002... Sindings-acc.500
2023. OpenReview.net, 2023. URL httttttt////orreee ww..nt//offidd-W//Tinnss
Logacheva, V., Dementieva, D., Ustyantsev. S., Moskovskiy, D., Dale, D., Krotova, I. Semenov, N., and Panchenko, A. ParaDetox: Detoxification with parallel data. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6804 6818, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10. 10.116633w1/20222.ccoonng https://aclanttholle wttp://2......ooooo.ooo.. 469.
Hyvärinen, A., Hurri, J., Hoyer, P. O., Hyvärinen, A., Hurri, J., and Hoyer, P. O. Estimation of non-normalized statis- tical models. Natural Image Statistics: A Probabilistic Approach to Early Computational Vision, pp. 419-426, 2009.
URL
Ko, J., Kim, S., Chen, T., and Yun, S.-Y. Distillm: Towards streamlined distillation for large language models. ArXiv preprint, abs/2402. abs/2442.03898,, 2024. URL https: //axxiv. org/abs/2400 2402.03898.
Lou, A., Meng, C., and Ermon, S. Discrete diffusion modeling by estimating the ratios of the data distribu- tion. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: htttps://openreview.. et st/forum?id-CNicRIIIPP
Li, X., Thickstun, J., Gulrajani, I., Liang, P., and Hashimoto, T. B. Diffusion-Im improves controllable text generation. In Koyejo, S., Mohamed, S., Agarwal, A., Belgraae,, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Con- ference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http:://paperrs.nngg.
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. Learning word vectors for senti- ment analysis. In Lin, D., Matsumoto, Y., and Mihalcea, R. (eds.), Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, 2011. Association for Computational Linguistics. URL https: http::s//aalaanhollogg www.hhooooooooooooo-////// org/P11-1015.
cc/ haappprffillee/pppr///////////// 1e666688222999900----ttttttt-...
html.
Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In The Fleventh International Conference on Learning Repre- sentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: https://openreview.n et/ #t/pdd?7idtPPvWRDCTTT.
Mareus, M. P., Santorini, B., and Marcinkiewicz, M. A. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2)311-3300, 1993. URL httpss:/acclanthol.gg
/ J93-2004 2004.
Meng. C., Choi, K., Song, J., and Ermon, S. Concrete score matching: Generalized score matching for discrete data. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Infor- mation Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL ffiles h1i1://ppppr///222/haaaffaaaaaa1 ab1 1111f9987bbbbstteee haar:77---bssraatt-Coofeeeeeeee html.
Liu, C., Zhao, F., Kuang, K., Kang. Y., Jiang, Z., Sun, C., and Wu, F. Evolving knowledge distillation with large language models and active learning. In Calzo- lari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint In- ternational Conference on Computational Linguisties, Language Resources and Evaluation (LREC-COLING 2024), pp. 6717-6731, Torino. Italia, 2024a. ELRA and ICCL. URL. htttt::s//acaaanhologooo hww.nthhloogy.rrg/222222.. 1rec - -main.593.
nips. cc/paper
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In 5th International Confer- ence on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceed- ings. OpenReview.net, 2017. URL https://openrevi ew. ww..netfool mnet/forum?dd=By/2222 2udxe..
Liu, S., Nam, J.. Campbell, A., Stärk, H., Xu, Y., Jaakkola, T., and
R. Think while you gener- ate: Discrete diffusion with planned denoising. ArXiv preprint, abs/2410.06264, 2024b. URL https://arxi V. v.org/abs/22e 06264.
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,
Nguyen, X., Wainwright, M. J., and Jordan, M. I. Esti- mating divergence functionals and the likelihood ratio by convex risk minimization. JEEE Transactions on In- formation Theory, 56(1 1):5857)8-8811, 2010.
12

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., and Li, C. Large language diffusion models. ArXiv preprint, 0b9922 2025. URL https://arxiss www.or/gbabbbb/eeeeee//
Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Informa- tion Processing Systems 36: Anmual Conference on Neu- ral Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http:: nips. https//ssisssee/pppaa.eeeeee//ae /B553///hhhs85505a56666477766226 bstract-Conffrrence. html.
Nisonoff, H., Xiong, J., Allenspach, S., and Listgarten, J. Unlocking guidance for discrete state-space diffusion and flow models. ArXiv preprint, abs/2406.01572, 2024. URL https: / /ttp:vaaiiii...o/...//../....a. www.aawwwv.g..araaeeeee 2157.
Rector-Brooks, J., Hasan, M., Peng, 7.,, Quinn, Z.,, Liu, C., Mittal, S., Dziri, N., Bronstein, M., Bengio, Y. Chatter- jee, P., et al. Steering masked discrete diffusion mod- els via discrete denoising posterior prediction. ArXiv preprint, abs/2410.08132 2024. URL https://are v. w.org/abs/244 08134.
Nowozin, S. Debiasing evidence approximations: On importance-weighted autoencoders and jackknife varia- tional inference. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Pro- ceedings. OpenReview.net, 2018. URL https://open review. net/foormmid-:Hy22..
Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Mar- roquin, E., Chiu, J. T., Rush, A. M., and Kuleshov, V. Simple and effective masked diffusion language models. ArXiv preprint, has/2406.005244 2024. URL https: //arxiv. www.v.oor//bass... ha//abs//244 2406. 2406.07524.
Ou, J., Nie, S., Xue, K., Zhu, F., Sun, J., Li, Z., and Li, C. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. ArXiv preprint, abs/2406.037366 03736, 2024. URL https: org/al ab s/2406. 03736.
Savinov, N., Chung. J., Binkowski, M., Elsen, E., and van den Oord, A. Step-unrolled denoising autoencoders for text generation. In The Tenth International Confer- ence on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL net wtt.://forrmmm/dd-662111111-1 https:/ http:/iwwwwww..oooeoee
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L.., Simens, M., Askell, A., Welinder, P., Christiano, P. F,, Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Process- ing Systems 35: Annual Conference on Neural Informa- tion Processing Systems 2022, NeurlPS 2022, New Or- leans, LA, USA, November 28- December 9, 2022, 2022. URL http: /ttt:://pppprss nippapprrr..iiss.pppppppppppspsm httpppss.ccpppppr_feeeeeee /19822/aah//Def60033333434444483 Astract-Conferencee html.
Schiff, Y., Sahoo, S. S., Phung, H., Wang, G., Boshar, S., Dalla-torre, H., de Almeida, B. P., Rush, A., Pierrot, T., and Kuleshov, V. Simple guidance mechanisms for dis- crete diffusion models. ArXiv preprint, abs/2412.101933, 2024. URL https:://aatt /awxiv../iivv.///a org/abs/ 24t.. 10193.
Shaul, N., Gat, I., Havasi, M., Severo, D., Sriram, A., Hold- errieth, P., Karrer, B., Lipman, Y., and Chen, R. T. Flow matching with general discrete paths: A kinetic-optimal perspective. ArXiv preprint, abs/2412.03487, 2024. URL http://www..oggb///////// 03487.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The LAMBADA dataset: Word predic- tion requiring a broad discourse context. In Erk, K. and Smith, N. A. (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pp. 1525-1534, Berlin, Germany, 2016. Association for Computational Linguis- tics. doi: N00118653/1/P111 144. URL https: //aclanthology hwtp:ey..rr///11111/01
Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. K. Simplified and generalized masked diffusion for discrete data. ArXiv preprint, abs/2406.04322 04329, 2024. URL http s:/ / /ariv..ivvvv.. org/abs//400/ 04329.
Shih, A., Sadigh, D., and Ermon, S. Training and infer- ence on any-order autoregressive models the right way. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Infor- mation Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http: /ttt::://apprs...../t nips ccper /tilL//ppppr//22hha////111366613 wat.aa/afaarraatttCrrreeeee.ee. html.
Radford, A., Wu, J. Child, R, Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In
13

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Singhal, R., Horvitz, Z.,, Teehan, R., Ren, M., Yu, Z., McKeown, K., and Ranganath, R. A general frame- work for inference-time scaling and steering of diffusion models. ArXiv preprint, abs/2501.06844 2025. URL
2023. URL http:s/s/ppemrrwie..ee/////oooop NNu6wX.
Vincent, P. A connection between score matching and de- noising autoencoders. Neural computation, 23(7):1661- 1674. 2011.
https://arxixs org org. www.oogg////a/eeeee httgg/bb///1..066655 06848.
Sohl-Dickstein, Weiss, E. A., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In Bach, F. R. and Blei, D. M. (eds.), Proceedings of the 32nd International Confer- ence on Machine Learning, ICMI. 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Con- ference Proceedings, pp. 2256 2265.. JMLR.org, 2015. URL http::tt///c /ttt:://pprooe pttp://dooceeeen.cc.cccccoo/..sp wtt.ssml......e..mmmm prr.ss//37/sooood Sikstein15.htae
Wang, C., Jiang, Y., Yang, C., Liu, H., and Chen, Y. Be- yond reverse kl: Generalizing direct preference optimiza- tion with diverse divergence constraints, ArXiv preprint, abs/2309. 16240,, 2023. URL s/ $32339.11 16240.
Xu, M., Geffner, T., Kreis, K., Nie, W., Xu, Y., Leskovec, J., Ermon, S., and Vahdat, A. Energy-based diffusion language models for text generation. ArXiv preprint, abs/2410.21357, 2024a. URL https://arxiv. org/ 2///2410.2135.
Song, Y. and Ermon, S. Generative modeling by estimat- ing gradients of the data distribution. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d' Alché-Buee F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Infor- mation Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pp- 11895-11907, 2019. URL https: htttts:///rroceedings..o
Xu, X., Li, M., Tao, C., Shen, T., Cheng, R., Li, J., Xu, C., Tao, D., and Zhou, T. A survey knowledge distillation of large language models. ArXiv preprint, Dos/2402.13111 16, 2024b. URL https://arxiv: abs/2402. 13116.
Ye, J., Zheng, Z.,, Bao, Y., Qian, L., and Gu, Q. Dif- fusion language models can perform many tasks with scaling and instraction-inetuiingg 0bs/2308.12219, 2023. URL https://arxiss orgexii...o/abbbb.o s/2308. 12219.
96dcd477c7d33AAbtrract html.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In 9th Interna- tional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021. URL https://openrevieee www.eeerreiiieenn/foorrr. 221.d
ArXiv preprint,
Yu, L., Zhang, W., Wang. J., and Yu, Y. Seqgan: Se- quence generative adversarial nets with policy gradient. In Singh, S. P. and Markovitch, S. (eds.), Proceedings of the Thirty-First AAAI Conference on Artificial Intel- ligence, February 4-9, 2017, San Francisco, Califor- nia. USA, pp. 2852-2858. AAAI Press, 2017. URL http://aaai. org. /ogg/ocs/ii inde//ndxxxx.m// Ahp/AAAI/AAA /paper/view/1434/
Sugiyama, M., Suzuki, T., and Kanamori, T. Density-ratio matching under the bregman divergence: a unified frame- work of density-ratic estimation. Annals of the Institute of Statistical Mathematics, 64:1009-1044, 2012.
Sun, H., Yu, L., Dai, B., Schuurmans, D., and Dai, H. Score-based continuous-tin discrete diffusion models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL. ew. we.../pdfF..ddBBwwWYW255..
Zhang, R., Koyama, M., and Ishiguro, K. Learning struc- tured latent factors from dependent data:a generative model framework from information-thooreti perspec- tive. In Proceedings of the 37th International Con- ference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 11141-11152. PMLR, 2020. URL httt:://proceedingg wttpissmmr..pprrrr pressp//es///// hang20m. html.
Uehara, M., Sato, I., Suzuki, M., Nakayama, K., and Mat- suo, Y. Generative adversarial nets from density ratio estimation perspective. ArXiv preprint, 1010..220,,, 2016. https //axxiv .ww.ivooorr/ooooo. org/ abs/1610 02920.
Zhao, S., Brekelmans, R., Makhzani, A., and Grosse, Probabilistic inference in language models via twisted sequential monte carlo. ArXiv preprint, abs/2404.17546, 2024a. URL /ttpss//aarxttt wwwp.v.oo/g/b//////eeee 17546.
Vignac, C., Krawezuk, I., Siraudin, A., Wang, B., Cevher, V., and Frossard, P. Digress: Discrete denoising dif- fusion for graph generation. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net,
Zhao, Y., Shi, J., Chen, F., Druckmann. S., Mackey, L., and Linderman, S. Informed correctors for discrete diffusion
14

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
models. ArXiv preprimt, abs/2407:21243, 2024b. URL http:/awwww....///////......
Zheng, L., Yuan, J., Yu, L., and Kong, A reparameter- ized discrete diffusion model for text generation. ArXiv preprint, abs/2302.05737, 2023. URL https://arxiv. org/abs/2302..572
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Rad- ford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. ArXiv preprint, abs/1909.08592, 2019. URL https: //arxiv. wttp://////...00555303 08593.
15

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Appendix
Table of Contents
Extended Preliminaries
B Proofs
18
B.1 Proof of Proposition
18
B.2 Proof of Proposition 2
18
B.3 Proof of Proposition 3
B.4 Proof of Proposition 4
20
C TCSM Pre-training from data
21
C.I Experimental Details and Results
21
D TCSM Pre-training with Parametric Model pr
22
E TCSM Post-training with Parametric Model Pile
22
E.1 Derivation of Density Ratio Estimation Objectives
22
E.2 Connections f-divergence TCSM
................................
24
E.3
Experimental Details and Results
25
F
TCSM Post-training with Reward Function
26
F1
Derivation of Objectives for Reward Tuning Experimental Details and Results
26
F2
27
G TCSM Post-training with Preference Optimization
29
G.1 Detailed Algorithm
29
G.2 Experimental Details and Results
30
H TCSM Post-training with AR Diffusion Distillation
30
Connection to Continuous Target Score Matching
32
Experimental Configuration Summary Table
35
Related Work
35
A. Extended Preliminaries
Continuous Time Markov Chains Model The Continuous Time Markov Chain (CTMC) model is an S-valued time- dependent family of random variables (x:)eejo.) that form Markov chain characterized by the probability transition kernel Pe+ Mej (yl (y|x)
f(y,x) + u + o(At) with the initial distribution of the process at time x) S x S + IR is called the velocity or the rate matrix, which indicate the speed at which the probability transitions between states. To make sure the transition probabilities (++)))(y)x)) are normalized, m(y,x) x) need to satisfy ux(yx)) 2 0 for all y # x and Σ, xx 0.
po())).
Discrete Flow Matching We use the discrete flow matching (Campbell et al., 2024; Gat et al., 2024) as general frame- work to introduce the discrete diffusion models. Our goal to transfer samples xo ~ po(xo) from source distribution
16

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Po to samples X1 ~ (x(x)) from a larget distribution pr. Source and target samples can be related by means of the in- dependent coupling (xa,x1) < pa(xo) (xx))p((e) (x1), associate by means of general coupling No,1 (x), For independent coupling, common choices for the source distribution is either po (xo) Il v. uniform distribution over S; and (ii) pi (xo) IT== 8(M, delta measure concentrated on the absorbing state M.
Similar to the continuous flow matching model (Lipman et al., 2023; Liu et al., 2023), we construct a probability path Pr (xx) interpolating between po and pr. By conditioning on x1, we build a probability path px(x)) = (p (x (x((x)) (x1). The marginal velocity wy(y, x) generating probability path pi (zx) can be computed by u. uy(,), Ep xx) (y xx/xi);
where pr [x) = (1 (x4 x1) is the true conditional distribution predicting clean data x1 from noisy data x,, and (x()
us (yt,,xx (xi) the conditional velocity generating Pet (xx(xi).
Training The goal is to approximate the velocity ,,(y,x) using a neural network. We can parameterize the velocity vf (y,, x) directly, and optimize the conditional flow matching loss LCPM (1) (xpep))))) Dy (v,,,, ul x))) where we sample time from distribution w(t), and Dr (u, v) F(u) - F(v) (VP(V), uvv is the Bregman divergence with respect to the strictly convex function F. We also need to make sure that uf xx), satisfies the rate conditions.
As shown above, the velocity is governed by the true denoising distribution pxt{(x]|x;)) so instead of parameterizing the velocity directly, we can use model pin (x) to approximate (x,e(x.(xx) by minimizing the loss
CCPM (1 ID (xx)) ll (Plie((x1)2x)))
(13)
where ID (II-). is some statistical divergence. For example (Campbell al., 2024) uses the KL divergence which gives rise to the cross-entropy loss x,,,, log [ (x1 (x), which has been shown to be a upper bound the negative model log- rogllikeihhod of the target data distribution. f&PM is often called the data aatd-predittion loss, as the model (xx[xx))) is trained to predicts the clean data X1 from the noisy data x, by aligning to the true denoising distribution Pitx(X) (x)..
Factorized Probability Paths The flow formulation and training objective described earlier are applicable to any prob- ability path. However, parameterizing the velocity in S x S is often impractical. To address this, we typically construct factorized conditional paths Pajo, (x; = I1 Pijo, (20) (00 x1). A common design (Gat et al., 2024; Shi et al.,
2024: Sahoo et al., 2024) is
Pho.1 Xo, x1) = aro(1 #1) + (1 aa))(af
(14)
where a₁ : [[jo,1]] + R is the noise schedule function. A straightforward example is the linear schedule a = tt. For each token x sampled from Pijo,, a11) there is a probability a, of it being of and a probability (1 (1-00 (1) of being on When ogg 0 and 0 = 1, Pa (x,) adheres to the boundary conditions at t = 0 and t = 1. By marginalizing out Xo. the conditional distribution p (<r)) x have closed form as: (x/[xx) Cat( =t} + (1 az) for unifrom source, Pej (xr = C (at, + (1 (M, xx]) for mask source. These are known forward transition kernel in score-based diffusion models (Song et al., 2021), allowing for simulation-free sampling of x,. The corresponding velocity is given by
de uf(e', xx) pi (of (m)) T at
10 5(12,x')
(15)
and the marginal velocity u,(y,, xx) can be factorized as
I u xx).))) Σ 5(y xt "t'')((ll xt).
(16)
So we can parameterize the factorized velocity as 1.0 (v/. x+) and optimize the loss
EDr (u)(r),))) uf (x)x)))
Cirm = Er,,, x.
(17)
i=1
which is also an ELBO on the target data distribution when we choose the generalized KL divergence (Nguyen et al., 2010) as the Bregman divergence (Shaul et al., 2024).
Sampling Sampling from the target distribution (x(1) is achieved simulating the CTMC with learned velocity field of (y,, x.) with Euler methods.
17

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
B. Proofs
B.1. Proof Proposition 1
We first establish a key property of the Concrete score through the following lemma.
Lemma B.1 (Meng et al., 2022)), Let p(x) be discrete probability distribution over X. For any neighborhood structure N that induces connected graph, the Concrete score mapping (x;x)N is complete. Specifically, for any parameterized distribution pl (x) with 0 O we have Cyo (x;N) = cx: for all x E X if and only if p (x) = =p(x) almost everywhere.
Proof. The result follows directly from (Meng et al., 2022). We observe that our definition of X differs from the original by a constant shift of 1, which a bijective transformation and thus preserves the completeness property.
Proposition 1. Let N define neighborhood structure that induces a weakly connected graph G over the support of P.))))Xx)). Assuming mild regularity conditions on the divergence measure D, the global minimum of the TCSM objective LTOSM in Eq. (3) guarantees that (1xx) equals P1x(-1xx) almost everywhere with respect to p(xx).
Proof. We prove the proposition through bidirectional argument.
(+) Let us first assume that the TCSM objective LTCSM in Eq. (3) achieves its global minimum. The objective is given by:
LTCSM (0:N, D, (Mx (SpIjs Cpf
(18)
By construction, the proposal distribution h(x) (x,) encompasses the support of pı ,(xx At the global minimum, we necessarily have:
Vx, e (x))()))xx((xx)) (cen Cpi
This implies:
Cyu (x1:N) Cpql (xi;
Given that V induces a weakly connected graph over supp(p))(((Ixx)) we can apply Lemma B. conclude:
(1),(xs|xx) (x)) = pip (x) (x:)
(=) For the converse, assume (x1 = (x1 (x (x)). Since the Concrete score is a deterministic function of the
underlying distribution, this equality immediately implies:
Cyn (xx:W) = Cpf, (x1;N)
Consequently, the Bregman divergence term vanishes, and the TCSM objective attains its global minimum of zero, complet- ing the proof.
B.2. Proof of Proposition 2
Proposition 2. Under mild regularity conditions, the score-based objective Lscore Eq. (4) achieves its global minimum if and only if the distribution basedd objective Ldistrib Eq. (5) achieves its global minimum, where the general TCSM objective minimized.
18

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Proof. We establish the proposition using bidirectional approach.
(+) We begin by demonstrating that if the Lscore Eq. (4) reaches its global minimum, then the Ldistrib Eq. (5) also attains its global minimum.
As indicated Eq. (8), the conditional distribution Pips (x} [x] xx) in Eq. (5) can be expressed as:
(1]e(e||x²" xx) Cat (s):softmax (log Spuje (z] 3 ;x))))()x)))))) .xx))
(19)
Additionally, we have:
Panl [x xx) Puuclt x (x1) V x1)
Cpse (x) (x][xx",
(20)
Pije(ot ,x)xxx), xx) vi=1
(1p x (xx)) vi=1
Therefore, when the score-based objective Eq. (4) achieves its global minimum, according to Proposition 1, we have Cp.je (x1[x() opp. (x1 (x.). By considering the i-th column, we obtain:
xx()x,)) Pije (x), x (xx) s/=11
(vi x (x.)
cpn (-1x,) =
(21)
From the above three equations, it follows that when the score-based objective Eq. (4) reaches its global minimum, have (1je(=+|x xx) pfp [x x,).
(+) Conversely, by combining Eq. (20) and Eq. (21), it is evident that when the distribution-based objective Eq. (5) achieves its global minimum, we have [ xx) ("]|x xx). xx).
B.3. Proof Proposition 3
Proposition 3. Under the proposal distribution h(x1(xx) (x)x(x))x the score-based objective with generalized KL divergence equivalent the distribution-bat based objective with weighted combination of forward KL and Itakura- (IS) divergences:
Lacore (0; = Piic, = Doa. (,
Laistrio (0:h Pi]t, DI VIDx + Dıs)
where Dx. represents the forward KL divergence, and Dıs denotes the Itakura-Saiie divergence.
Proof. Consider the objective function:
Lscore (0;N1, PD,) = Ew(e))(x MM(c (x.)
pi
(22)
score>
Puleeer xt (x))
Co x7 [x,)
p
score
D
(1je(ct, (x:))
(xt, x (x:)
=1=1
vi=1/
Utilizing the definition of the generalized KL divergence:
Dr (u, v) E,
ug log "1 u, + uy we substitute this into
19

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
the objective function obtain:
Pf )xc) Pije(ai (x.)
Pic xx. (b,))
V
C Escore DF
Pip(aiix"" xx1 (xx))
vi=1
Pijelui, x (x,)
(u x] 1xx))
pie (1) P x : (x{)
Puucut x (xx)
Plie xx1)))) 1xx))
log
log
P1jo(11, (0000191x'1 xf
Pijo(aff x{ (x1)
pf (z0 xx²1 [xx)
Pije(af x² (x,)
pl (20 x (xx)
vi
PicCoo bxf* x()
xx)
(vi xx xx))
xxi
pl xx{² xx)
Pnalyi bx xx)
pl (l |x xx)
log
log
(ij((1] [x] xx)
Pije(af |x] xx)
p/ {zf xx{ xx))
Pnf(af |x] xx1)
p (z) |x] xx1))
Given the proposal distribution (xxx = (x)o(xi(xe) - P1 cxx (xc)) (o |x Xx), we have:
Bp(x))rr Eccore
E p(xx) (e)) pmn xx)
Goore
Puicuilxx** xx)
pip (vi 1x xx)
E
Pueloilxt x.) xx) log
log
Pri( bxft xx)
p (of [x{ xi)
(t M
-iioloilx xx) + Pile x x1)
pijc (ui 1xx xx,
p (x² 1x1 xx,
=E p(x) (xc)
Σ
Da De (p2ie(tAst' xx) 11 pf (- (-1x xx1)
(1.))
Pije
|x] #i x1)
+ p xx)
+ p(x)) xx))
log
ppje (af |x xx)
(f x1)
v D'a (p1)((1x2f),x xx) [ xi) + Dis (p1r(laxf* xx)) ll IIriee( (-xx²' xx1)
E p(X) pate (xf (xc)
Thus, the original objective is to minimize the KL divergence and IS divergence between Pip( | xx) and pL. (-12 xx):
Lscore h P1ie, D Daxu (1 )) = Ldistrib (0; P1|(, DD = VIDaa + Dis)
When we select the proposal distribution 1x(x, x+) = P1[r and D = Daxi (,) in the score-based objective, is equivalent to the distribution-bar -based objective with DD (II) = VD + Dis-
B.4. Proof of Proposition 4
Proposition 4. When using forward generalized KL divergence the discrepancy measure and setting the proposal distribution to the true conditional distribution Pit((x)Xx)) the score-based Lscore objective in Eq. (4) can be expressed as:
Eoooe =
1Egrendo + Cownoppl + +C
Gpecudo
log
xx) +
V (atloxt xx))
Concopy
log peppiaaai
(vi (f xx)
Proof. The score-based Target Concrete Score Matching (Cycore) objective, as defined in Eq. (4), aims to minimize the divergence between the concrete score of the true denoising distribution (xje (x.) and the model's denoising distribu- tion pf (x1|xx (x). Proposition 3 establishes that when using the generalized KI. divergence (Dax (, )) as the discrepancy
20

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
measure DD and the true conditional distribution (x)(()]X))) the proposal distribution h(x](xx), the expected value of the Licore objective the data distribution is equivalent minimizing weighted sum of the expected forward KI. divergence and the Itakura-Saito (IS) divergence between the true conditional (2]((1](x] Xx) and the model conditional pl (a) xx x:):
L
(u Σ
s [:)] E (00 (xx))
Σ
VIDk (pIn()...)11dd(etl.....)
==1
Dis (pIc())...)111)) ((1((1...)11((((t...).))
(23)
where (.I...) ) is shorthand for (xi|x]" x().
xx)).
However, this expected loss formulation involves the true, unknown distribution P1te and cannot be directly computed during training when only have access samples x; P1 (x11 (the target data distribution). Therefore, resort to Monte Carlo estimation, minimizing loss function evaluated on individual samples (4, X1, xx) drawn according to w()), P1(x) (x1), and Paj1 (1).
Proposition 4 presents the specific form of this practical, per-sample objective that is minimized during training. This form is particularly relevant and aligns directly with the objective derived for the common case of a factorized model parameterization, detailed in Eq. (10). Under factorization, the model assumes (x1 = II (=/(xx), which implies p x x1) = p (x1 xx). Let g(ylx)))) pL (vlx;) denote the factorized model's output distribution for any position.
The objective stated in Eq. (10) for single sample x1 and position i is:
& scooe factorized ( (-1og log q (x:) + Vg(zi x.)
+
log g(yl xx).
(24)
y=1
Here, zi is the specific token at position in the sampled clean sequence X1. Proposition 4 decomposes this per-sample loss into two terms:
(- - 1o xx) + Valiet xx)
"sccddo =
2 Centropy
Eoj=1 + log pf (vf (x x:)
When applied to the factorized model where pl (vi (xf ,x,) = -9(vi(xx) these terms become:
pi pseudd
= (- (-log log q(x)(x)) + Vx(e](x))
Centropy $ Ey=ılog (3)lx)
Summing these two components precisely recovers the objective e [fattorize]] given in Eq. (24).
Thus, the objective gi fireudo + entropy as presented in Proposition 4 represents the practical, par-sample loss function derived from the L.care principle using the generalized KI. divergence. when training from data samples, and its structure directly corresponds the objective used for factorized models. The constant C represents terms from the full expected GKL divergence (related to the entropy of the true distribution Pij() that do not depend on the model parameters 0 and are therefore omitted during optimization.
the objective minimized via Monte Carlo estimation
C. TCSM Pre-training from data
C.1. Experimental Details and Results
In this section, we present the experimental results obtained from our datasets, followed by a comprehensive analysis and summary of our findings at the conclusion of this section.
21

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
TEXT8 The TEXT8 dataset is a character-level text dataset featuring limited vocabulary of 27 tokens, which includes the letters a-z and the whitespace token. We adhere the standard practice of training and evaluating TEXT8 in segments of 256 characters without any preprocessing, as outlined by Hoogeboom et al. (2021). Our experiments on the TEXT8 dataset, a compact character-level language modeling task, follow the network hyperparameters and dataset splits specified by Austin et al. (2021). We compare our results with methods that utilize models of similar size. Consistent with previous studies (Austin et al., 2021; Lou et al., 2024), we trained discrete diffusion models on TEXT8 and assessed their performance by measuring bits-per-charactee on the test set.
OPENWEBTEXT To assess our approach in large-scale language modeling, we conducted extensive experiments using the OPENWEBTEX" dataset. Given that the original WebText dataset used for training GPT-2 (Radford et al., 2019) is not publicly accessible, we followed the common practice of using OPENWERTEXT.
Our evaluation involved testing TCSM-trained discrete diffusion models against GPT-2 using zero-shot testing on five stan- dard benchmarks: LAMBADA (Paperno et al., 2016), WikiTeet (Merity et al., 2017), Penn Tree Bank (PTB) (Marcus et al., 1993), and One Billion Words (LM1B). These datasets encompass wide array of language understanding tasks and were initially employed to assess GPT-2's zero-shot perplexity performance.
For training, we utilized a batch size of 512 and sequence length of 1024, maintaining the evaluation setup consistent with that of Lou et al. (2024).
The results indicate that TCSM significantly surpasses existing diffusion methods and closely approaches the performance of autoregressive baselines. It is important to note that our evaluation methodology slightly deviates from previous work, as we compute likelihood unconditionally without employing a sliding window, which typically results in higher perplexity values than those reported in earlier studies.
D. TCSM Pre-training with Parametric Model p1
Experiments To assess the efficacy of parametric target estimation in expediting the training of discrete diffusion models, we conducted extensive experiments on language modeling tasks using the TEXT8 and OPEN WEB TEXT datasets. Our empirical findings reveal substantial improvements across all proposed estimation methods.
To explore whether the parametric model P1 enhances the sample efficiency of discrete diffusion model training, we em- ployed this model to train the discrete diffusion model from scratch on the OPENWEETEE dataset, processing 26 billion tokens. The results of these experiments are presented in Fig. la.
The data clearly indicate that our TCSM framework, incorporating the parametric model pl. consistently surpasses existing discrete diffusion methodologies. Notably, the hollow transformer variant (TCSM-Hollow) delivered the best performance. Both the BERT-based (TCSM-Bert) and autoregressive- based (TCSM-AR) target estimations also demonstrated strong results. These outcomes signify a significant advancement over previous diffusion methods such as SEDD and MDLM, enhancing both the learning process and sample efficiency.
The robust performance of our TCSM variants supports our hypothesis that operating within the clean target space and utilizing parametric estimation can significantly improve discrete diffusion model training. Furthermore, the results suggest that different architectural choices for target estimation present various trade-offs between performance and computational efficiency.
E. TCSM Post-training with Parametric Model Pije
E.1. Derivation of Density Ratio Estimation Objectives
This section provides a detailed derivation of the objective functions used for density ratio estimation (DRE) within the
TCSM frammeorrk, outlined in Sec. 5.1. The core idea is to estimate the ratio between the true conditional data distribution
(1) xx xx) and reference distribution p (af [x
ti x2), denoted by r(æf 1x xx) = P1 [x] [x .x.) We employ the
[a [x xx)
Bregman divergence for this estimation task, aiming to find the parameters & of model > |x] zi divergence to the true ratio r.
xx) that minimize the
22

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
The general Bregman divergence objective for density ratio estimation is given by (Sugiyama al., 2012):
min I Pil (milx?" x.) [Ds (r(=|(x]* ti xx)) r (x) [ xx))]
(25)
where a strictly convex function defining the divergence, Dp (u, v) = F(u) - - (())(u v).
Expanding the Bregman divergence and using the property that Vij [F)(r))) = (F we can derive a practi- cal objective function by omitting terms independent of the model parameters Φ. Minimizing Eq. (25) is equivalent to minimizing:
(Ø) = E Byg, (eil...) F'
F F(r - Pinccil....) F' (1 (z)1...)))
(26)
where (...) is shorthand for the conditioning variables (xf),xx xx). Note that in practice, the expectations are estimated using Monte Carlo sampling from P1je (using data samples) and pile (using the reference model).
We now instantiate this general objective for the specific choices of F mentioned in the main text:
Least-Squares Importance Fitting (LSIF): Using F(r) (r-1)²
we have F' F'(r) = r Substituting into Eq. (26):
2
(rr0 - -1)2
(1SIF(()) = E
(r
1)r
Epii 1r-00 - -1]
2
-Bygt
(v-0)² pd
1
((19)) -2 + 1) - Epija (r + const.
2
(r+)²
1
Epst
Bmilr + const.
2]
(r0)27
Epp, [r (Ignoring constants)
Binary Cross-Entropy (BCE) related / KL Divergence: The objective associated with BCE often arises from f- divergence dual forms rather than directly from this specific F(r) in the Bregman DRE literature. A common choice leading to BCE is related to the Jensen-Shannon divergence. Alternatively, considering the standard GAN objective for distinguishing Pije (label 1) from Pifk (label 0) using a discriminator D(x) = o (log r (z)), where ((2) = is the sigmoid function. Maximizing the log-likelihood Ep,, [log D] + Eygg [log(1 D)] is equivalent to minimizing:
LBCE-IIe (Ø) Epp (log + }) - E [log(1 or rlogr"))
This formulation is commonly used and corresponds to the objective derived from F(r) logr (r + log(r + 1) in some DRE contexts via duality.
Generalized Kullback-Leibe Leibler (Gen. KL): Using F(r) = r log r r, we have ('(r) = logr. Substituting into Eq. (26):
LGEAKL (Ø) = ppy (log r )r (r log r - r²) Epajs (log 1
= =Ryygt [r.S togr tog + +r - Eyyic (log r
EEps [r-]- Epy, [logr9). r
These objectives are summarized in Table 6.
Implicit Parameterization Strategies
As discussed in Sec. 5.1, we consider main strategies for parameterizing the density ratio and the denoising model, where 9 represents the parameters being optimized.
(i) Parameterizing Ratio via Model: Here, we set á = 0 and define the ratio implicitly through the denoising model pf and the reference model prr.
( ...]....)
(27)
p (x l......)
23

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Table 6: Objective functions (DRE(() derived from minimizing Eq. (26) for different Bregman divergence choices F(r). Constants independent of o are ignored.
Method
Objective DRE
LSIF (P(r) = (ياي)
Eppr
(109)) 2
Ep.ij [r-]]
BCE-like (related to JSD/GAN)
- Bpn [log(o rd) }]]
E Byr [log(1 olog
Gen. KL (F(r) r logr r)
Eptt [r-]] Epp [logro]
We substitute this definition of , r into the objectives Table 6. For example, the Gen. KL objective becomes Byg, [Pie//rf Epn [Pgg((in//T]]]]
(ii) Parameterizing Model via Ratio: Here, we directly parameterize the ratio, typically ensuring non-negativity, e.g., 2
"ije (ifl.....) = exp(Ja(zi1....)) )), where So is neural network parameterized by 0. The denoising model is then implicitly defined (up to normalization) pfj. (afl...) x pff. )r (afl...). .). The optimization minimizes the DRE objectives from Table 6 with r = r exp(fe). For instance, the Gen. KL objective becomes Eygt [exp(fo)] - EE,, [fo].
The resulting objectives for both strategies and all three choices of are compiled in Table 7, which mirrors Table the main text for consistency.
Table Final objective functions for TCSM post-training via DRE under different Bregman divergences F(r) and parame- terization strategies. Here logri where r is the parameterized ratio (explicit in (ii), implicit in (i)). and (xx) is the sigmoid function.
F(r)
Strategy (i) Objective: r Plu/Pit
Strategy (ii) Objective: Pij x Pije exp(fe)
LSIF: (r-19/2
Best ((f1/1tt)) Epi, [19 voit.]
Byt expl2fa) Bp,,, [exp(fol]
BCE-like: r Flogr 1) log(r 1)
-Bpy, (ooggrllggg s/dd)) Best, Dog((1 o(logeieedd)))
Ep., [og(a((f))) Bey,, [log(1 o(fe))
Gen. KL: r
Egg, [Pi///t]]] Ep,,, Dog p/11/
Bgg [exp(fe]] Bp., [fo]
E.2. Connections to f-divergence TCSM
A straightforward method involves independently parameterizing both the density ratio model r (xi (x.) and the denois- ing model 2 (x1[xx) Once the density ratio model is optimized using Bregman divergence minimization, resulting in the optimal model r* (x1, xt), we face the task of solving the optimization problem
min D(r" "e
align 7d with p. However, this two-stage process, alternating between density ratio estimation and divergence minimiza- tion, is not stable and is difficult to converge.
As shown in (Uchara al., 2016), minimizing the objective
B "eff(t x)) (F (af xx)) (xf x.) ((r xx))) (f xi) (a( x:)) (28)
for estimating the density ratio model o would lead to /-divergence maximization, thus such two-stage process will yield GAN-like adversarial training. This motivates us to parameterize the density ratio model in terms of the denoising model, or vice versa, as shown in Sec. 5.1.
Reference Models With the density ratio model parameterized, the next crucial step is selecting an appropriate reference distribution of We explore two compelling options.
24

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Weaker model as reference At each optimization step k; can set the reference distribution be the previous step 0e-1, and the density ratio model is parameterized as
denoising distribution pef Pile
P x.) Oe-1
(o x x.)
Pije (xf [ x1)
This will give us a procedure similar to SPIN (Chen et al., 2024). A Alternatively, we can use the exponential moving average Sm this case, we naturally use the (i) parameterization of the denoising distribution as the reference distribution, pr Pije" strategy for the density ratio model.
Pre-trained model reference We also set the reference distribution to be pre-trained discrete diffusion model Pile (x1 xx,) Pije (x1|Xx). We can use the (ii) parameterization strategy to parameterize the density ratio model
pre
pije(* (x+) Pije [x1 (x|
1 (xlxx))
The training objective becomes
pff (xlx)t) x)) (F (r'))))) F(r - E (x F' (r"
(x)) (z)).
Algorithm TCSM Post- Trainin with Density Ratio Estimation
Require: Dataset D == {x1} }
Require: Pre-trained model sile
Require: Proposal distribution
Require: Bregman divergence function F
Require: Density ratio model r1ec fo
Require: Learning rate 7)
1: X1 D
Sample data point Sample diffusion time > Sample noisy data D Sample from reference distribution D> Compute density ratio based on Bregman divergence
2: t ~ w(I)
3: X; ~ (x)
4: x + -Pi (x (x:)
5: F =SSIF then
C (xxp(2f((*T)))
exp(fe(x1)) -exp(((()(
6:
7: else if F BCE then
8:
L log(1 - o (ff(xx))) + log(o else if F = Gen. KL then
9:
10: L t exp(fo(x(")) f(((x))
end if
0 - VoL
Update parameters
E.3. Experimental Details and Results
We present a thorough empirical evaluation of our density ratio estimation- post-training methodology within the TCSM framework. While Sec. 5.3 investigates parameterization strategy (i), we concentrate here on evaluating parameterization strategy (ii), which characterizes the denoising model through density ratio estimation.
Our experimental framework utilizes a pre-trained GPT2-small model with Ldistrib for language modeling tasks, imple- menting absorbing state formulation outlined in Sec. 4.1. Building upon the work of Xu et al. (2024a), initialize our density ratio model 1 (xi (x.) using the pre-trained diffusion model. The initialization process involves projecting mean-pooled last token embeddings to scalar values, while the partition function is estimated following the methodology proposed by Nowozin (2018).
To ensure comprehensive evaluation, we investigate three distinct Bregman divergence measures for training the density ratio model:
25

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Least Squares Importance Fitting (LSIF)
Binary Cross-Entrope (BCE)
Generalized KI. divergence
For a complete algorithmic description of our approach, we refer readers to Alg. 1.
The comparative performance of these measures is documented in Table Table 5. Notably, our implementation of TCSM with BCE shares similarities with the EDLM model in fact, EDLM NCE (Xu al., 2024a) be viewed as a specific case of our framework when BCE serves as the chosen Bregman divergence.
Our experimental analysis yields several significant findings. Most prominently, the post-training approach incorporating density ratio estimation consistently outperforms the pre-trained haseline model, as demonstrated by improved perplexity metries across all configurations. While both generalized KL divergence and binary cross-entropy achieve particularly strong results, the relatively uniform performance across all tested variants highlights the fundamental robustness of our methodology. regardless of the specific divergence measure employed. This consistency across different mathematical formulations provides strong evidence for the stability and reliability of approach.
F. TCSM Post-training with Reward Function
F1. Derivation of Objectives for Reward Tuning
In this section, we provide more comprehensive derivations of the TCSM objectives introduced in Sec. 5.2, with particular focus on their practical implementations.
Lscore and Listrrib with A For the score- based TCSM objective with target distribution (x1), we can directly apply the formulation from Eq. (4):
Ph (uf x xx)
pl (ut x x rlie (0f xf (x.)
Escore (0:N1, (D.) Ex Xx
Σn D
ppt (2), xi (x.)
vi=1
Let us define := yi. x and = [rf. xf where ví # x1. The ratio between reward-modulated conditional
probabilities can be expressed as:
P (y(xx)) xx)
Pi (y)pu) (x))))(()
Pije(y(x))
R(yy) R((x)
n (x(xx)
pi (x)Pp1 (x, (x(( R(x)/3) (1)((x(x))
exp
3
Given access to a pre-trained model pife that approximates Pitt. we can reformulate the objective as:
Pijclut lx)) exp
(R(vi, - R(af #1 R(of.xt))
))
pij (ui x
Lscore (0:W!,D,h)) = Et,x, X,
ΣD D 1
(
x
3
v-=1
For models with factorized denoising parameterizations, this objective simplifies to:
p (x]())))) P (of (x,)
R(yi, - M((r
Cei (x:)
Lacore = E Btxxx xx
Σ D
exp
B
p (xt (x,)
1=1
21=1
wi=1
This formulation enables efficient computation of all terms involving pfte and pf
26

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
For the distribution-basee Leistrib approach, we derive a similar approximation:
Pij (x]({ xi) x pfl (x) x xx) ex))x((((a))
The detailed implementation is presented Alg. 2.
Cdistrib with A When employing N the Laistrib objective takes the form:
[eistrib(B:N\thh D. Bx(())x) (pite (xxx) (-1x)
Using the approximation Pip = Pijes we can derive:
log Pij. (x5 xx)) rl. (x1 (xx)
DA (Pf) (-1x)) (-xxx)))1l (1]1c((xx))
=Bpfi Bpficto (x)))x))))))))))))))))
pllc (x)xx)
Σ (x1 log
1 (x1 (xx)
(x)e(X11 xx) exp( R 1)3
log Pije (x1 (xx)
xx Pije xx) exp( R( (1 333)
Pije (x1(xx)
exp(R(xi)/55 2(xx)
log Pfc (x)
E (x((()))
Pijc (x]c(x| (x,)
The complete algorithm is detailed in Alg. 3.
Connection to Reinforcement Learning An interesting connection emerges when we set (x)((x)xx,) pf (x.)) and use D (pl g) = DxL (q ()) the reverse KL divergence. The Laistrib objective then takes the form of traditional RL objective:
D (pffecixx xx)
p xx) = Da (pi 1 .x))
pl (rf xx{ xx)
= =E "i xx)
log
Pi (x |xx xx))
p/ (of |xf x.)
E B x.) log
+C
21p xx X ) exp( R(xf x )/8)
-Da (al xx) Pie (x+ x1)
E (o xx) R(a) xf + C
This formulation closely resembles the standard RLHF objective, highlighting the theoretical connections between our approach and traditional reinforcement learning methods.
For practical implementation, employ bı [x)(xx/2x (x,) pf" (xx (x]|x;) as the proposal distribution. Since the new model P1 follows product distribution, its support must necessarily be contained within the support of p.re
F.2. Experimental Details and Results
Synthetic Experiments To assess the effectiveness of reward function tuning methodology, conducted experiments using a synthetic dataset. This dataset is structured as a 2D discrete grid, specifically 128 x 128 grid. Initially, we pre- train a discrete diffusion model, denoted as p on this grid using the Loistrib objective with a uniform source distribution. Subsequently, we define a reward function R designed to eliminate modes located in the right half of the grid. Concretely, we assign R(x)= = 0 for all points z in the left half, and R(z) = -10 for those in the right half. Following this setup, we fine-tune the model using the Laistrib objective with Afilll adhering to the procedure detailed in Alg.
27

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Algorithm/ 2 Neward-GGided Post- Training with
Require:
Pre-trained model Fije proposal distribution h. reward function R, temperature B
pre
Require: Model parameters 0, learning rate v), sequence length L
1: Sample diffusion time t wt)
Sample diffusion time and generate noisy sequence
2: Sample clean sequence X1 ~ 11(-1x))
3: Generate noisy sequence Xx ~ p(-1x1)
4: for i = to L do
> Compute
target distribution
(o x1) x- Pf] (xf xx) exp(((() (/3)) En Plje (x xx) xx)) x(x"xx())//)))
5:
6: end for
7: C s Ldistrib (B;N h)
Compute loss and update parameters >> Gradient descent step
8: 0 t VoL
Algorithm 3 Reward-Guided Training with Artil
Require: Pre-trained model Fflle- proposal distribution h, reward function R, temperature |
Require: Model parameters 0, learning rate
I: ~ w(1)
D. Sample diffusion time
2: X; ~ x(xx
Sample noise
3: Sample mini-batch {x1,)) xxx1
Draw samples from proposal
4: Z + -- exp( R)x1,)/B))
> Compute normalization
5: wh - exp ((p()(x))))) /B)) for b = 1,....;
> Importance weights
p (x1,s xx) 6: L + Emm wh log
b Weighted objective
p (x1, x.)
7: 0 - 0 1VV
> Gradient update
The results of this process are illustrated in Figure 5, which displays the intermediate samples generated by the model both before and after fine-tuning. Initially, during the pre-training phase, the model successfully captures all modes present in the data distribution. However, after applying reward-guid -uuuddd fine-tuning, the model effectively suppresses the modes in the right half of the grid, resulting in final samples that exclusively generate the left half of the grid.
Toxicity Mitigation critical challenge in deploying language models is effectively controlling and mitigating toxic content in their outputs. Although toxic generations occur relatively infrequently, their potential negative impact users and downstream applications makes this an essential area of research (Singhal et al., 2025). Even small proportion of toxic outputs can significantly undermine the safety, reliability, and trustworthiness of language models in real-world scenarios.
Our experimental methodology builds upon recent advances in controlled text generation (Zhao et al., 2024a; Rector- Brooks et al., 2024; Singhal et al., 2025). To ensure reproducibility, we conduct our experiments using standardized story-beginning prompt: "Once upon time, there a". The foundation of our experimental framework is a pre-trained diffusion model developed in See. 4.1, which implements Laistrib with absorbing discrete diffusion. To further enhance the model's capabilities and robustness, we perform comprehensive fine-tuning the Tinystories dataset (Eldan & Li, 2023). This fine-tuning process utilizes the Adam optimizer with (31 0.9, B2 = 0.95) and learning rate of x 10-4, continuing for 100,000 training steps.
For measuring and controlling toxicity, we implement a sophisticated reward function based on a pre-trained RoBERTa classifier (Logacheva al., 2022). During our evaluation phase, we employ this classifier as our primary metric for assessing content safety, with outputs scored on a continuous scale from 0 (completely non-toxic) to I (highly toxic). This granular scoring system allows for precise measurement of our mitigation strategies' effectiveness.
The results of our comprehensive evaluation are presented in Fig. 3, where we analyze two critical metrics: the toxicity score and the generative perplexity of the samples. To assess the quality and coherence of the generated text, we measure perplexity using GPT-2 Large (Radford et al., 2019) as an independent evaluator.
We fine-tune the model using the Laistrib objective with Attll, following the procedure outlined in Alg. 3. To investigate
28

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
the impact of sampling density, conduct experiments with varying numbers of Monte Carlo samples N E (2, 4, 16) for estimating the importance weights, with results displayed in Fig. 3. For comparative analysis, include benchmark results from the pre-trained MDLM (Sahoo al., 2024) model using Best-of-N sampling with N E {4, 8), reported in (Singhal et al., 2025).
Our experimental results demonstrate several key findings. First, our approach exhibits superior scaling properties with respect to the number of Monte Carlo samples used for importance weight estimation. Second, our fine-tuning methodology achieves more effective toxicity mitigation compared to the pre-trained MDLM model, even when the latter employs Best- of-N sampling techniques. Notably, since our approach is based on fine-tuning rather than inference-time scaling, eliminates the need for multiple reward function evaluations during inference, resulting in reduced computational overhead and improved efficiency in practical applications.
G. TCSM Post-training with Preference Optimization
G.1. Detailed Algorithm
Problem Setting We introduce methodology for fine-tuning pre-trained diffusion models using pairwise preference data, denoted as ((q.x",xi)) In this formulation, q represents a query or instruction, while x and x represent the preferred (winning) and non-preferred (losing) responses, respectively.
The underlying preferences are assumed to emerge from latent reward model that is not directly observable. Among various approaches for modeling such preferences, we adopt the widely-recognized Bradley-Terry (BT) model (Bradley & Terry, 1952). This model provides an elegant framework for capturing human preference distributions. Specifically, the BT model expresses the probability of one response being preferred over another as:
(q, p'(xI" > x²(q)= exp(R((q,xx")) exp R (q, xx)
(29)
where R (2, x) represents the underlying reward function that quantifies the quality of response x given query q.
Building on this foundation, we define our target distribution emphasize preferred responses. This distribution can be formally expressed
Ptargot (xilq)= = (x"" (x) = (x is winner/q) = -Pi(xi Pp((yila) yrlg),
(30)
Y1
For practical implementation, we leverage a pre-trained diffusion model pij (q) as reference distribution, which serves as the starting point for our fine-tuning process.
Based on the TCSM with density ratio estimation approach in Sec. 5.1, we learn a new diffusion model pf relative to the pre-trained reference. The detailed algorithm is shown in Alg. 4, where we use BCE loss to estimate the density ratio as an example.
29

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Algorithm 4 Preference Optimization with TCSM using BCE loss
Require: Pre-trained diffusion model Pile
x Require: Preference dataset D ((c,x",,)) )
Require: Model parameters 0, learning rate 7), time distribution wt),, coefficient
1: for each training iteration do
2: t t~ w()
Sample diffusion time
3: (c.x",'') D
D Sample preference triplet Sample noisy sequence for preferred response >> Sample noisy sequence for non con-prfeered response D. Compute density ratios for preferred and non-preferred responses
4:
x ~ (11()
5:
x ~ Pipl (-12²)
6:
([]((e) - ex
7:
Pp] (x"
rij(() t (x (e)
8:
Apt) (x]
9:
> Compute loss
C - log (c)
10:
(c)
log (+)] (c)
11:
0 - 0 - VL
b Update model parameters
12: end for
G.2. Experimental Details and Results
To evaluate the effectiveness of preference optimization, we employed the IMDB-sentiment dataset (Maas al., 2011) as our primary evaluation benchmark, with the SiEBERT model (Hartmann al., 2023) serving as our reward function. For training data, we utilized carefully curated preference dataset constructed in prior work (Rafailov et al., 2023; Wang et al., 2023). As our foundation model, we used the pre-trained model from Sec. 4.1, which had been extensively trained on the OPENWEBTEXT dataset.
The fine-tuning process implemented our density ratio estimation framework, as detailed in Sec. 5.1, with Binary Cross- Entropy (BCE) loss serving as our optimization objective. We adopted parameterization strategy (i) from See. 5.1, which defines the density ratio as:
Plp |x] x:) Bp (of |x(xx xt)
-==0 (aflxt xx)==
Here, the coefficient ,9 plays a crucial role in balancing two competing objectives: maximizing preference reward optimiza- tion while maintaining fidelity to the original pre-trained model. The complete training procedure is outlined in Alg. 4.
Our training protocol consisted of full epochs with batch size of 256. We employed the Adam optimizer with a learning rate of 1 10-5 and weight decay of 1 10-5 To ensure stable training, we implemented linear learning rate warmup for the first 10% of training steps, with momentum parameters 31 0.9 and B3 0.95. The noise schedule remained consistent with that of the pre-trained model to maintain continuity in the diffusion process.
To thoroughly investigate the effects of preference optimization, we conducted experiments across a range of 3 values: 10.1, 0.5, 1, 5). Our evaluation focused on two key metrics: the mean reward achieved by the fine-tuned model and the entropy of generated samples. As shown in Fig. 2, we observed that models with stronger preference optimization (higher values) achieved both higher mean rewards and lower sample entropy. This suggests that our approach improves alignment with desired preferences but also leads to less diverse generation of preferred samples.
H. TCSM Post-training with AR - Diffusion Distillation
Problem setting In this case, we assume we have pre-trained autoregressive model p (xi) trained on the target distri-
bution Pi (x1), and we show that we can use TCSM to distill it to diffusion model pf (x,). Note that this deviates from the regular diffusion models setting, that we have the knowledge the target distribution P1 (x1)+ ss (x1), and we can use as teacher model. In this section, we set the target distribution be the AR teacher model distributoin pi (x1) = (x1).
30

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Algorithm 5 Top-K Estimation
1: procedure ccs_estimate(x:
teacher_moder L, V, tts) tcs)
2:
Xo: Input tokens; L: Sequence length; V: Vocabulary size; K: Top-X tokens select; tes: list
3: logits + teacher_model(xx.
IR V original_loo__prreo <- teacher_ baateee_mmod__o__proboolll
4: for / = 1 to L do
5: Get top-K tokens: top__oooeeeeeeeeeee tttkensss + (op
6: If x $ top_tokens, add it to sop_toeens
7:
Construct batch of new sequences Xy + [x top_ tokens, x
8:
Compute log probability of sequences log_prob from new_logits teacher_modele
9:
Compute log-density ratio: log density_ ratio log jog_probb- loo_lo___rrr__ prob
10: Append log-density ratio to list: tcs <- tcs + jes_density_ratee ratio
11: end for
12: return tcs
13:
end procedure
And akin to classical knowledge distillation, we are interested in how to distill the knowledge from the AR teacher model the diffusion student model.
TCSM objectives for distillation We show that our TCSM objectives naturally integrate the knowledge of the AR teacher model into the training objective.
We have
(x)(())Xx)) = 1/f (x) PT (xI)pq11(: (x1)
We can also use (x1) ) to estimate
pA aot.a xf Pej (xx |z] xf En; PA Piji (x+lvi. x
Pi Pro(ot xx)
Both score-based and distribution-hasee TCSM objectives can be used to distill the AR teacher model to the diffusion student model, we use the distribution- TCSM objective in our experiments and assume it is the default setting in following discussions.
Efficient estimation of distillation target To optimize the TCSM objective, we need compute the distillation target (x1). Naively, this requires (V 1) L + 1 likelihood evaluations of the teacher autoregressive model for each sequence E (x).... (x). Even though that the likelihood evaluation can be done in parallel for the autoregressive model, this procedure is still computationally prohibitive. To address this challenge, we introduce two approaches to efficiently estimate the target concrete score, Top-K estimation and First-order Taylor estimation.
Top-K K approximation Our empirical analysis reveals that distribution Pit (of [x] xz) are naturally sparse. As illustrated Fig. 6, tokens with high density ratios closely resemble the one-hot encoding of original tokens in the simplex space, but enriched with distributional information. This observation motivates approximating the score vector with only the top-K items, treating the rest as zero, for efficient computation. We leverage this property to propose efficient top- approximation that reduces computational complexity from O(VL) to O(KL) by considering only the K most probable tokens at each position. This approximation can be efficiently implemented using batched forward passes and proves effective even with < 128 for detailed implementation and the complete algorithm, we refer readers to Alg. 5 in the appendix.
First-order Taylor approximation We leverage the fact that autoregressive language models, despite operating on discrete tokens, differentiable functions that can be approximated using Taylor expansion. For sequences that differ by only one position, we can efficiently estimate the likelihood ratio using first-order Taylor approximation: log Pije Pijo x (x.) R log JogPiie (1j x + Vex, Pr (x//x)) (ey) ). This gradient- based estimation requires just one forward and backward pass through the teacher model; for detailed derivations and implementation, please refer to Alg. 7.
31

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
though
traveled
One-hot
America Africa
TCS
weekend
North
Token ID
to
Weed
this
month
South
week
came
We
Korea
went
last
traveled to
South
Carolina
last
summer
Figure 6: Visualization of the concrete score for sequence "I traveled to South Carolina last summer.". The x-axis represents the position in the sequence, and the y-axis represents the log- pro-prbbbbylity ratio. The red line represents the original token, and the blue lines represent the top-K tokens with the highest log- peg--bbbabiltyy ratios. The concrete score is highly sparse, with most of the probability mass concentrated on few tokens.
Experimental To validate our distillation approach, we conducted comprehensive experiments focusing on language mod- eling capabilities using the OPEN WEBBTEBBTE TEXT dataset. Our experimental setup involved two key components: a teacher model and a student model. For the teacher, we pre-trained transformer-based autoregressive model following the architectural configurations described in (Sahoo et al., 2024). As our student model, we employed an absorbing discrete diffusion model.
The training process utilized our Top-K estimation strategy with K 128, training the student model from scratch. To assess performance, tracked the validation negative log- ligeeiiol (NLL) loss on the OPENWEBTEXT dataset, which we visualize in Figure Fig. 4. The empirical results demonstrate two significant findings: First, our distillation approach substantially accelerates the student model's learning trajectory compared to standard training. Second, and perhaps more importantly, models trained with our distillation loss consistently achieve lower perplexity scores than baseline approaches throughout the entire training process, indicating improved model quality.
I. Connection to Continuous Target Score Matching
In this section, we elaborate on the relationship between the proposed Target Concrete Score Matching (TCSM) framework for discrete data and the established Target Score Matching (TSM) objective (Bortoli et al., 2024) used in continuous diffusion models. We first briefly review TSM in the context of language modeling via continuous diffusion and then demonstrate how TCSM can be viewed as its discrete analogue under certain approximations.
Continuous diffusion models for language often operate in a continuous embedding space. Let x1 [x]...., a1 be a discrete sequence from the vocabulary X' (1, V). Let E R be word embedding matrix, where is the embedding dimension. The one-hot vector for token & is e E (0, The embedding for token of is E E egt- The
forward noising process typically acts independently on these embeddings:
4: (2 = II gqp (z)1=1)= = II (( a c.
(31)
1=1
32

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Algorithm 6 Top-K with N-Gram Estimation
1: procedure tcs (cs_estimate(xi, teacher_maeeeeee moodel, ngram_model, L, V, K, tcs)
X1: Input tokens; L: Sequence length; V: Vocabulary size; K: Top-X K tokens select; tes: list 3: logits +- teacher_ [ IR V x original_ogg__pree <- baanbee_mooeellog___eee
2:
4: for / = 1 to L do
5: Get top-K tokens: top_toopeeee tokens + Top
6:
Get N-Gram score for all tokens: rrgram_scores +- ngram_ model ++1
X1 (
7:
Add another top-K tokens: cop_tokens < top_tokeme + TopK (n-gram scores, K)
8:
If x1 <1[1] & top_tokens, add it cop_tokens
9:
Construct batch of new sequences & + [xf', hop_tokens,xx
10:
Compute log probability of sequences loo_prob from new_ new_logits + Saacher_model(5
11:
Compute log-density ratio: log density_ratie ratio + log loo_probb orig. _og_prob
12:
Append log-density ratio to list: tcs <- tcs + bog_density_reee
13:
end for return
15:
end procedure
Algorithm 7 Concrete Score Estimation with first-order Taylor approximation
1: procedure ts__esimatt(teacher_mddee tokens, T)
2:
> tokens: Input tokens of shape (B, L); V: Vocabulary size; 7: Temperature
3:
f f one_hoottokens, V)
Convert one-hot vectors
4: Enable gradient computation for X1
5:
logits teacher_model((e
6:
prob + log (ogitts
7:
jog_prob + [x :::] prob:pro//:: -1,
8:
Compute gradient: grad_log_proot + Vx, log_
9:
D Compute log-density ratios
10:
log_prob prob_ratio +
- - (xi grad_log_prob)
Scale by temperature: jog_prob ratio log Bac__poob_rrtttteeee ratio//
12:
prob_ _o_rratoo exp(log_prob_rattio
13:
return prob_ratio
14:
end procedure
where (z)) forms the sequence of noisy embeddings R2dd and a; oz schedule parameters. The goal is to learn the score function Vz, log z (z,) of the marginal distribution (z,),) = 4: | [2, [x, (x,,
Target Score Matching (TSM) provides an objective when the score of the clean data distribution, Va, log (z)) (where 21 represents the clean embeddings and p1 is a density over them), known or can be estimated. The following identity connects the noisy score to the clean score:
Lemma L1 (Target Score Matching Identity, adapted from (Bortoli et al., 2024)). Let qen (z.lz) = W(zz; I define the forward process conditioned on clean continuous data 21, and let pı (2:) be differentiable distribution over Zn- Then, the score of the noisy marginal qe (z:)) = 4 1 (2, 2i)p1 (z, dzz is given by:
1
Vz. log (ogg((2))
E (en((e))).) [zz log (21)]
(32)
where (1)(2)(ze) is the posterior distribution.
Proof. The proof follows standard arguments, e.g., in Bortoli et (2024), adapted for the scaling factor 0g. Using the prop- erty Vz, log qe ( (21) = -a ae Vz, log d: ( (z)2z) and Bayes' rule dej (z, (2)) = gije ( (21), we take gradi- ents w.r.t. Zı: V,, a, log 9q11 (z.|z1) = Va, log 91p (z) (z]]2.) V., (z.) Combining these yields V., log Qup (2.|z.) (Vz, (ne 24) - Vz, log (21)). Finally, taking the expectation w.r.t. (n (z:): V., log qe (2.)
33

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
cs _ Eyj V log 91je] + ax q 22 log (2,)]. Since Beg V2 log log/1je) I Vx Pije (z) dzz = 0 (assuming boundary conditions), the identity holds.
(zil()))e,) (V. log (2)
91je)
Using Lemma 1.1, score network So(z;, t) can be trained by minimizing the TSM loss:
LTSM (0) = (1, Epr (annett so((z.))) Vn,,
Vxy log (21)
(33)
Alternatively, using the mean prediction parameterization (10(2),) ~ (2.2)))))) and Tweedie's formula 1 Vo. log (zz) + ze), the TSM objective becomes equivalent to minimizing (up to scaling by ai
( [21]
A az a//07))
STMM (0) = (0.1 (an mi) (oo(z. t) - 10- Ve, lz1) +
1
Zz
(34)
ax
Note: The exact form depends slightly on conventions; here we target scaled version of the clean score plus noise term. Let T(z1, Ze, t) = of vz, log (11) + 1 a.. 2, be the target for the mean predictor.
Now, let's connect this to the discrete TCSM ohjective. Consider the log-probability ratio (concrete score component) for the posterior distribution qıje (x1|z, in the continuous setting, where &1 differs from x1 only at position i (i.e., o = xi. and of = i for + i):
(x1(((//2)) log log (x]],x]
91 (x1) log 9:11 (zo) (x1) 9:11 (z, x1)
(35)
qi (x1)
The second term simplifies due to the product structure of uu/1:
log (2)12)()111
41 [2) (t{)
log 9: [z) 1
(36)
4:11 (z (x1)
|| a e2gl 112
2 a ezy 11?
(37)
207
20
02 (z E (e2) -e,))) (04))-a
at
(IE e2y 12 -E ezy 11²)
(38)
of
20,
Let's assume embeddings have similar norms, making the last term negligible, or absorb it into the definition.
(ix) For the first term, log [ we use first-order Taylor approximation in the continuous embedding space 2 E ...,..., E ex4] corresponding to x1. Let pı (Z1) be the density over these embeddings. Then:
Pi (zx,))
log
= log (zx.) ) + (Vz, log logpi(zz): ZR. bx,) - log P1 (zx,))
(39)
pi (Zx,)
= (Vz, log (Z) Zz, 2xi)
(40)
((V2, x, log (z) )): E (eaç -
(41)
where ()), denotes the gradient block corresponding to the i-th position embedding. Combining Eq. (38) (simplified) and Eq. (41), the target concrete score is approximately:
(x)((x + jizz)
1u (x1lzz) log
(42)
Qije (x1 zz)) ((Vz, log (2z));
04 z/ E (e, -0p]-ee;)) ezi))-
o?
(43)
Now, consider the model prediction po (xi 22), often parameterized via logits pa(2)))) such that po(at jlz.) boftma([[ug..dd The model's concrete score is:
Pp (x, [z) = [molas [molag = (lpo) e, - -zg)
(44)
34

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
The TCSM objective aims match Fpo to Fa The TSM objective (Eq. (34)) encourages uo(z(:)) 1 2z, (zi) + as zt. this holds, then from Eq. (44):
T
T (x1 (z)e, (TT C, o25)= =(ivv Ve, logpi(z))
+ 2 - - -
1
(45)
att
Comparing this to the target approximation in Eq. (43), we see they align (up scaling factors and potential embedding norm terms) if E I. When E + I, the alignment is approximate.
In summary, under the first-order Taylor approximation for the marginal discrete probability ratio and assuming word embeddings E behave similarly to an identity mapping (or have negligible impact on the inner products compared to the main terms), minimizing the TCSM objective, which matches discrete concrete scores, serves as an approximation to minimizing the continuous TSM objective. This provides a conceptual link between the two frameworks, highlighting how TCSM adapts score- score- matching principles to the discrete domain.
J. Experimental Configuration Summary Table
To enhance clarity and facilitate reproducibility, this section provides a comprehensive summary of the specific models, parameterizations, and training objectives used for each experimental result presented throughout the paper. Sec. J details the configuration for each key experiment, linking the reported results (identified by their table or figure number) to the underlying methodological choices, including the prior distribution (source distribution for diffusion), the structure of the denoising model p the proposal distribution h ((xx (x,) used within the loss computation (if applicable), and the specific TCSM training objective function employed.
K. Related Work
Generative modeling (Austin et al., 2021; Song et al., 2021; Ho et al., 2020; Vincent, 2011; Song & Ermon, 2019) has seen significant advances through diffusion models, initially developed for continuous data like images. Applying these principles effectively to discrete data, such as text or graphs, presents unique challenges due to the non-differentiable nature of discrete spaces and has spurred several distinct lines of research.
Score Matching and Continuous Diffusion Foundations The theoretical underpinning for many modern diffusion models is Score Matching (Hyvärinen et al., 2009). This method estimates parameters 0 for models p(x;0) q(x; 0) with intractable normalization constants by minimizing the difference between the model's score function Vx log g(x; 0) and the data score Vx (x)). A key insight by Hyvärinen et al. (2009) showed that this objective can be computed using only the model score and its derivatives on data samples, avoiding the need for the true data density or normalization constant. crucial practical development was Denoising Score Matching (DSM) (Vincent, 2011). which established an equivalence between score matching on noise-perturbed data and training specific denoising autoencoders (DAEs). DSM matches the model's score at noisy point to the score of the conditional denoising distribution, avoiding the second derivatives required by original score matching and making score estimation more tractable.
These principles were central to the development of diffusion models. Farly work framed diffusion via forward (noising) and reverse (denoising) Markov processes trained with a variational lower bound (VLB) (Sohl-Dickstein et al., 2015). Subsequently, score-based generative models (Song & Ermon, 2019) directly applied DSM by training a single Noise Conditional Score Network (NCSN) (x o)) to estimate scores Vx 9o: (x) across multiple noise levels {az). using annealed Langevin dynamies for sampling. Denoising Diffusion Probabilistic Models (DDPM) (Ho al., 2020) refined this, particularly for images, by parameterizing the reverse process predict the added noise and using a simplified VLB-derive objective shown to be equivalent to DSM over multiple noise scales. While highly successful, standard DSM can suffer from high variance at low noise levels. Target Score Matching (TSM) (Bortoli et al., 2024) addresses this by incorporating knowledge of the clean target score V log p(x) when available, leading to lower variance estimators in the low-noise regime.
Continuous Diffusion for Discrete Data One approach to handle discrete data involves operating within continuous embedding spaces, adapting standard continuous diffusion techniques. This allows leveraging powerful continuous models but requires mapping back to the discrete space. Diffusion-IM (Li et al., 2022) applied continuous diffusion to word
35

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
embeddings, enabling controllable text generation via gradient guidance during sampling. Plaid (Gulrajani & Hashimoto, 2023) focused on likelihood-basee hased training for text, jointly optimizing embeddings and model parameters using the VLB, categorical reparameterization, an output prior, a learned conditional likelihood p0z) and self-conditioning. CDCD (Dieleman et al., 2022) employed probability flow ODE on embeddings, using score interpolation to jointly train em- beddings and a denoising Transformer with a cross-entropy loss, along with time warping. Bit Diffusion (Chen et al., 2023) treated the binary representation of discrete data as continuous "analog bits," enhanced by self-conditioning and asymmetric time intervals. While effective, these methods rely on continuous approximations or embeddings, motivating research into models operating directly on discrete domains. Furthermore, many of these works explore non- con-autoregressive approaches enabling parallel generation (Bowman et al., 2016; Gu al., 2018: Li et al., 2022; Hoogeboom et al., 2021; Savinov et al., 2022; Che et al., 2017; Zhang et al., 2020; Yu et al., 2017; de Masson d' Autume et al., 2019; Deng et al., 2020), contrasting with sequential autoregressive models.
Discrete Diffusion Models A parallel line of research develops diffusion processes inherently designed for discrete state spaces, often using Markov chains. Building on early foundations (Sohl-Dickstein et al., 2015; Hoogeboom et al., 2021), D3PM (Austin et al., 2021) generalized discrete diffusion using various structured transition matrices (e.g.. uni- form, absorbing, Gaussian-like) and trained via hybrid VLB/cross-entropy loss. Campbell al. (2022) extended this to Continuous-Time Markov Chains (CTMCs), deriving a continuous-time ELBO and proposing efficient sampling methods like tau-leaping and predictor-corrector schemes, leveraging factorization for high-dimensional data.
Score-like Analogues and Masking Mechanisms for Discrete Diffusion Instead of direct Markov chain simulation, other works define score-like quantities for discrete diffusion. The concrete score, defined as the ratio of marginal proba- bilities pt (pt))) acts as a discrete analogue to the continuous score (Meng et al., 2022; Lou et al., 2024). SEDD (Lou et al., 2024) trained models using a score entropy objective (I-DSE) derived from this ratio, connecting it to the ELBO and using Tweedie r-leaping for sampling. Sun et al. (2023) developed categorical ratio matching within CTMC framework, learning singleton conditionals p. (a*4 x x(d) with tractable loss and an analytical reverse sampler. Building on this Ou et al. (2024) showed that for absorbing diffusion, the concrete score factorizes into a time-independent conditional and time-dependent scalar, simplifying the model (RADD) and yielding the Denoising Cross-Entropy (DCE) loss.
Masked (or absorbing) diffusion, which replaces tokens with a special [MASK] token during the forward process, has proven particularly effective. MDLM (Sahoo et al., 2024) introduced a substitution--aase parameterization (SUBS) and derived a simplified Rao-Blackwellized ELBO equivalent to weighted Masked Language Modeling (MLM) losses, en- abling generative training of encoder-only models. Shi et al. (2024) (MD4) further unified this framework, deriving a simple ELBO with SNR invariance properties similar to continuous diffusion and generalizing state-dependent masking schedules.
Further research has refined the parameterization and mechanisms of discrete diffusion. Reparameterized Discrete diffu- sion Models (RDM) (Zheng et al., 2023) identified an underlying route-and-denoise mechanism, simplifying the objective to cross-entropy on noisy tokens and enabling adaptive routing during sampling. Liu et al. (2024b) proposed Discrete Diffusion with Planned Denoising (DDPD), factorizing the reverse process into a planner (predicting corruption) and a denoiser, allowing adaptive sampling via the Gillespic algorithm guided by the planner.
Discrete Flow Matching offers another generalization pathway. Gat et al. (2024) defined probability paths interpolating discrete distributions and derived corresponding probability velocities, analogous to continuous flow matching, providing a unified sampling theory. (Campbell et al., 2024) formulated discrete flows using CTMCs, learning scores via cross- cross-entropy and enabling inference-time flexibility by adjusting the rate matrix family without retraining, also unifying multimodal generation. Discrete diffusion principles have also been applied to structured data, such as graphs in DiGress (Vignac et al., 2023), using specific noise transitions, auxiliary features, and classifier guidance.
Scaling and Adapting Pre-trained Models for Diffusion Language Modeling Significant recent effort has focused on scaling diffusion models for language generation, often by adapting large pre-trained autoregressive (AR) masked language models (MLMs). DiffusionBERT (He et al., 2023) integrated BERT into an absorbing-state diffusion framework, leveraging pre-trained weights and exploring novel noise schedules and time conditioning. Ye et al. (2023) adapted pre- trained MLMs (like XLM-R) for generative tasks by finetuning with an RDM objective, enabling issrruction-oolloinn capabilities. AR2Diff (Han et al., 2024) proposed converting pre-trained AR models to diffusson models by enabling bidirectional attention and continuing training with a diffusion objective. DiffulLaMa LAMA (Gong et al., 2024) presented
36

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
continual pre-training method to adapt AR models (like LLaMA) into time-embedding-fees diffusion models using attention mask annealing. LLaDA (Nie et al., 2025) developed large masked diffusion model trained with masking objective, adapting standard pre-training and SFT pipelines for this non-autoregreesivv paradigm. These works demonstrate the potential of leveraging existing large model architectures and weights to build capable diffusion language models.
Guidance and Control in Discrete Diffusion Controlling the generation process of discrete diffusion models is vital for their application. Several approaches modify the sampling procedure or the model itself. Nisonoff et al. (2024) introduced Discrete Guidance (DG), a principled framework for guidance in CTMC-based models, offering exact predictor guidance (PG), predictor-free guidance (PFG), and an efficient Ahyylo--pproximmted Guidance (TAG) variant by exploiting tractable normalization constants during inference. FK-steering (Singhal et al., 2025) provides general inference-time steering approach using Feynman-Kac interacting particle systems, applicable even with non-differentiable rewards via parallel simulation and resampling. An alternative strategy involves finetuning the model itself to incorporate guidance. Rector- Brooks et al. (2024) proposed Discrete Denoising Posterior Prediction (DDPP), framework for steering pre- treitrained Masked Diffusion Models (MDMs) according to reward function R(x1). DDPP reframes steering as learning an amor- tized sampler (via finetuning the MDM) for a target posterior distribution proportional to Pg (x1 R(x,). By exploiting the relationship between the target denoising posterior, the pre-trained model's posterior, and the reward, DDPP derives several simulation-free training objectives, offering scalable approach to bake reward- -based control into the model. Other meth- ods include informed corrector steps based confidence scores combined with architectural changes and novel training objectives for masked diffusion (Zhao et al., 2024b), and adaptations of standard classifier- free classifier-based guidance for discrete domains, sometimes coupled with improved ELBO formulations suitable for guidance (Schiff et al., 2024).
LLM Distillation Our work also relates to LLM distillation (Xu et al., 2024b), which focuses on transferring capabilities from large teacher models to smaller student models. Common techniques involve distribution matching, specialized loss functions (e.g., MiniLLM (Gu et al., 2024), DistiLLM (Ko et al., 2024)), using rationales (Hsich et al., 2023), or dynamic data selection (Liu et al., 2024a). While most existing methods distil knowledge between autoregressive models, our research explores knowledge transfer from powerful AR teachers to bidirectional diffusion students. This presents distinct challenges, particularly regarding the mismatch between the teacher's sequential generation process and the student's non- autoregressive, iterative refinement process, but potentially benefits from similar underlying principles aimed at effective knowledge transfer and mitigating distribution discrepancies.
37

---

Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Model Variant Name (Defining Section/Eq.)
Experiment (Table/Figure)
Prior (Source Dist.)
Denoising Model Parameteri-
Proposal bution (xx//xx))
distri-
Training Objective (Equation / Description)
zation Paje
Experiments on TEXT8 (Fig. lb)
TCSM Uniform Lacora (Sec. 4.2)
Hig. 1b
Uniform
Factorized:
Pielxilxx (2.))
Pije(Xi (x,)
Licare with Gen Kl. (Monte Carlo version:
Il's ((((()))))
Eq. (10))
TCSM Uniform Laistrib (Sec. 4.2)
Hig.
Uniform
Factorized (as ahove)
(1)((x1(x1))
Latserib with KI. (Cross- Entropy: Factorized ver- sion of Hq. (9)
TCSM Absorb Lacore (Sec. 4.2)
Fig.
Mask
Factorized (as above)
P1{((1|xx))
Licare with Gen KL. (Monte Carlo version: Eq. (10))
(Absorb-
ing)
TCSM Absorb Laistrib (Sec. 4.2)
Fig. lb
Mask
Factorized (as above)
Pije(X] [+)
Lasstrib with KL (Cross- Entropy: Factorized ver- sion Eq. (9)
(Absorb-
ing)
TCSM Absorb Laistrib (Sec. 5.1)
Fig. lb
Mask
Density egy ii): Pxj((x)(x))
Ratio
(Strat-
Mip Piç
m
Post-training phase:
(Absorb-
DRE objective using
ing)
Pije (xs) exp(ff((x (x))) (Ref = Pre-trained TCSM Ab-
Gen KL (Table 4, col- umn
sorb Latstrih)
Experiments OPEN (Table Fig. Ia, Fig.
TCSM Uniform Locore
Table
Uniform
Factorized (as above)
(1)((11
Lscore with Gen KL (Bq. (10))
(Sec. 4.2)
TCSM Uniform Leistrib
Table
Uniform
Factorized (as above)
P1((X1|xi))
Lassceto with KL (Factor- ized version of Eq. (9)
(Sec. 4.2)
TCSM Absorb Laistrib
Table
Mask
Factorized (as above)
P1p((11xx)
Laseriib with (Factor- ized version of Eq. (9))
(Sec. 4.2)
(Absorb-
ing)
TCSM Absorb Laistrib (Sec. 5.1)
Table
Mask
Densily Ratio (Strategy ii. as above) (Ref Pre-trained TCSM Absorb faistris)
Pile Fije
Post-training phase: DRE objective using Gen KL (Table 4. col- umn 3)
(Absorb-
ing)
TCSM-Ben (Sec. 4.2)
Hig.
Mask
Factorized (as ahove)
(11(x1
Latstrib with Kl. (Target Pije uses BERT approx. for
(Absorb-
ing)
TCSM-AR (Sec. 4.2)
Hig. la
Mask
Factorized (as above)
(1)((11x11)
Laistrio with KI. (Target
(Absorb-
Pite uses AR approx. for P1)
ing)
TCSM-Hollow (Sec. 4.2)
Hig. la
Mask
Factorized (as above)
P1)((x1xx,)
Lasstri with KI. (Target Pite uses Hollow approx.
(Absorb-
ing)
for P1)
TCSM (Sec. 5.4)
Distillation
Fig.
Mask
Factorized (Student Model)
(1)((1|x;))
Lasstris with KL (Target Pij uses AR Teacher via Top-K approx.)
(Absorb-
ing)
Density Ratio Estimation Bregman Comparison (Table 5)
TCSM BCE (Reimpl.)
Table
Mask
Density Ratio (Strategy ii)
PiR PiR
objective using BCE (Table 4, column 3)
(Sec. 5.1)
(Ahsorb-
ing)
TCSM LSIF (Sec. 5.1)
Table
Mask
Density Ratio (Strategy ii, as above)
DRE objective using
(Absorb-
LSII (Table column 3)
ing)
TCSM (Sec. 5.1)
Gen
KL
Table 5
Mask
Density Ratio (Strategy ii above)
pi = 1
DRE objective using Gen KL. (Table 4, col- umn 3)
(Absorb-
ing)
Post-training Fine-tuning Experiments
TCSM Reward Tuning
Fig. (Syn-
Uniform
Standard denoising model Ph (Factorized assumed)
PiP
Weighted KL. objec- tive for y with 1/tlll (Alg. 3, Line 7)
(Sec. 5.2)
thetic)
Table 8: Detailed summary of model configurations for experiments reported in the paper.
38


Reviews:
[{"review_id": "i3Y6GfydwQ", "reviewer": "Reviewer_vK3t", "summary": "Recent works have proposed various diffusion modeling frameworks for discrete data, this paper proposes target concrete score matching, a framework that unifies various discrete diffusion approaches, such as discrete flow matching, masked diffusion language modeling, etc. The unified framework allows for using various model parameterizations and inference/noising processes with either uniform or absorbing terminal distributions in a single framework. The paper also demonstrates post-training methods such as sampling reward tilted models using RL or preference fine-tuning, and distillation from auto-regressive models can be encompassed in the TCSM framework.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their positive review and valuable feedback. We greatly appreciate the opportunity to address the questions and provide additional clarifications about our work. In the following sections, we carefully respond to each point raised by the reviewer, quoting the relevant comments to ensure clear and direct responses. We have also included additional explanations that we believe will help further clarify our approach and results. Please don't hesitate to let us know if you have any follow-up questions or need further details on any aspect of our work.\n\n# Experimental Details Clarifications\n\n> the paper could benefit from a discussion of the experimental results.\n\nWe agree that more discussion of the experimental results would be helpful. We layout following additional details on the implementation details and experimental results.\n\n\n## Q1 Sampling\n> A discussion on sampling methods from masked diffusion models is missing\n\nWe agree that including explicit introduction of sampling methods used in this work would be helpful.\nTCSM's flexibility allows it to work with any diffusion model parameterization, compatible with standard sampling techniques like ancestral sampling and reverse CTMC simulation via Euler method.\n\nIn our experiments:\n* Masked diffusion models (Sec 5.2, Fig 3): Used ancestral sampling\n* Uniform diffusion models (Fig 5): Used reverse CTMC simulation with Euler method\n\nThese are standard approaches briefly noted in App A. We'll provide more details in the revision.\n\n\n## Q2 Fig 1 evaluation\nFig 1 shows how parametric models (Sec 4.2) enhance sample efficiency during training. We aimed to test if this approach accelerates convergence vs data-only baselines. To clearly show performance differences, we deliberately restricted all models to 26 billion tokens, creating a controlled comparison within this limited training budget.\nIn contrast, the results reported by Sahoo et al. (23.21 for MDLM, 24.10 for SEDD) baseline were obtained using full training data. Consequently, the PPL scores in our Fig 1 are not directly comparable to those specific results from the original papers.\nFor fair comparisons under identical conditions (with full training convergence), see Tab 3 and 6.\n\n\n\n# Further Clarifications\n\n## Q3 AR to Diffusion Distillation\nWe placed distillation in post-training to align with LLM community where distillation is typically a post-training technique.\n\nDistillation section highlights a key TCSM advantage: it's the only discrete diffusion framework enabling effective AR-to-diffusion distillation—valuable given today's many pre-trained LLMs. We developed novel techniques for this process (Top-K, first-order approximation) that enhance scalability beyond Sec 4.2.\n\n## Q4 Table 2 details\n\nWhile the key connections summarized in Table 2 are currently present in the manuscript, they are distributed across different sections. For instance, we establish that our simplified $\\ell_{distrib}$ objective (Eq. 9) corresponds to the standard cross-entropy loss employed by MD4/MDLM/DFM. Furthermore, the relationship with EDLM (2410.21357) is detailed in App E.3 (L. 1122), linking their objective (Eq. 10) to the BCE-based Bregman divergence shown in Table 2 (row BCE (ii)).\n\nTo improve readability, we will add a consolidated paragraph in the revised manuscript. This new paragraph will explicitly outline these connections and clearly explain the significance and utility of Table 2.\n\n## Q5 SEDD, SDDM comparison\nSEDD was included as a baseline in all language modeling experiments, with results shown in Tables 3, 4 and Fig 1.\n\nFor SDDM, we followed prior discrete diffusion studies by excluding it, as its focus is image generation. Comparable language modeling results and official code were unavailable, preventing direct comparison.\n\nWe selected DFM for Sec 2 due to its general framework covering various discrete diffusion approaches, including SEDD (score parameterization) and MD4/MDLM (mask source distribution). This makes DFM ideal for introducing core concepts before our specific contributions.\n\n## Extra references\n\nWe will make sure to include all the references pointed out by the reviewer.", "claims_and_evidence": "See below.", "methods_and_evaluation_criteria": "The paper enables studying various existing approaches, with different inference processes, model parameterization and objectives in a single framework.", "theoretical_claims": "The theoretical claims are justified and detailed and self-contained proofs are provided in the appendix.", "experimental_designs_or_analyses": "The experiments contain the relevant diffusion baselines, as well as experimenting with different objectives and choices in the TCSM framework. However, the paper could benefit from a discussion of the experimental results. For instance, for several baselines considered the inference process, objectives and model parameterization are similar (for instance, in table 3), therefore a discussion explaining the results or guidance on which component would benefit the paper.", "relation_to_broader_scientific_literature": "The authors consider several discrete space diffusion models and propose a unified framework for studying and making modeling choices.", "essential_references_not_discussed": "Sampling from discrete diffusion models is an active area of research as well as relevant for any unifying framework. Recent works have proposed various forms of sampling schemes, such as predictor correctors (zhao et al 2024), or gradient-based guidance (shi et al 2024). \n\nZhao, Yixiu, et al. \"Informed correctors for discrete diffusion models.\" arXiv preprint arXiv:2407.21243 (2024).\n\nShi, Jiaxin, et al. \"Simplified and generalized masked diffusion for discrete data.\" Advances in neural information processing systems 37 (2024): 103131-103167.", "other_strengths_and_weaknesses": "Strengths:\n1. a single framework to study multiple inference processes, model parameterizations, divergences is appealing since discrete diffusion models have shown promising performance. TCSM allows for making modeling choices without having to manually derive an objective or parameterization for each different choice. \n\nWeaknesses:\n1. A discussion on sampling methods from masked diffusion models, including correctors (Zhao et al 2024), re-masking, etc is missing. \n\n\nZhao, Yixiu, et al. \"Informed correctors for discrete diffusion models.\" arXiv preprint arXiv:2407.21243 (2024).\n\nShi, Jiaxin, et al. \"Simplified and generalized masked diffusion for discrete data.\" Advances in neural information processing systems 37 (2024): 103131-103167.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1IWVk5uHLB", "reviewer": "Reviewer_GgYJ", "summary": "This work presents a new paradigm for modeling discrete data, titled Target Concrete Score Matching (TCSM). Unlike recent works that match a denoising concrete score, starting from an objective inspired by discrete flow matching, the authors propose to model the concrete score of ‘clean’ (i.e., target) data. The framework of TCSM is claimed to admit other recent discrete diffusion / flow matching papers as special cases. After deriving tractable Monte Carlo formulations of the TCSM objective, the authors conduct experiments demonstrating the competitive language modeling capabilities of TCSM-trained models, as well as their amenability to post-training adaptations.\n\n## update after rebuttal\nDuring the rebuttal periods the authors addressed my big concern on whether the results represented \"fair comparisons\". I therefore increased my score to 4.\n\nI believe the authors should commit to adding the missing proof for Prop 4.1 and make the edits from the rebuttal period which are important for clarity and reproducibiliyt", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer's thorough, high-quality, and insightful feedback. We'll address all suggestions and correct typos in our revision.\n\nWe are particularly encouraged by the reviewer's positive assessment of our work as \"innovative\", \"compelling\", and \"representing a very big step forward.\" We also appreciate their willingness to increase the score once the concerns on experimental details are addressed. To directly address these concerns, we have organized our response to focus first on the experimental details.\n\nBelow we respond to each point (Q=Question, W=Weakness)\n\n# Experimental Details Concerns\n\n## [W3, Q1, Q2, Q3]\n* Sec 4.1 introduces TCSM using **only data samples** from training dataset, same as prior works (SEDD, MD4)\n* Sec 4.2 presents an alternative method using auxiliary parametric models (AR, BERT, Hollow) to estimate Target Concrete Score (TCS) via $p_1(x_1^i|\\mathbf{x}_1^{\\neq i})$\n\nThey describe orthogonal techniques within TCSM.\n\n## W3\nExperiments following Sec 4.1 use only **data-only** methodology, without auxiliary models from Sec 4.2.\n\n## Q1\nParametric models in Sec 4.2 are used **only during training** to estimate TCS. They're **not needed for inference/generation**. After training, standard sampling techniques are used. (Please refer to Q1 of vK3t for more details).\n\n## Q2\n\nOur procedure:\n1. Initialize parametric model from scratch and train on same dataset\n2. Use this trained model (frozen) to provide TCS estimates\n\n\"Pre-trained\" means auxiliary model is trained before being used in diffusion model training.\n\n## Q3\nYes, it requires additional forward passes during training. However:\n1. The overhead applies only during training, not inference, preserving generation speed\n2. We observed just a ~20% increase in training time\n3. As Fig 4 shows, these models significantly accelerate convergence, requiring fewer total training steps (and FLOPs) to reach target performance compared to data-only baselines\n\n\n## Q5\n* Pre-training: $h_{1|t}$ is true data posterior $p_{1|t}(x_1 | x_t)$\n* Post-training with reward: $h_{1|t}$ is pre-trained denoising model $p_{1|t}^{pre}$\n\n## Q8\nYes, we re-trained both baselines with same training budget as TCSM.\nPlease refer to Q2 of vK3t for more details.\n\n## Q10\n\"Reimpl.\" refers to our re-implementation of EDLM NCE. As TCSM with BCE in Eq.11 is equivalent to EDLM's NCE loss and their code isn't public, we implemented it ourselves (detailed in App E.3, L1105-L1110).\n\nThe MDLM number in Tab 6 comes from Table 2 in the EDLM paper. Thanks for noting this discrepancy. We'll update with the original number from Sahoo et al. 2024.\n\n## Q11\nLike SEDD Sec 5.2.3, we divide text into fixed-length, non-overlapping segments. AR models use sliding window approach.\n\n\n## W2 \nComparisons are fair. All TCSM models were trained from scratch on identical datasets and evaluated under the same conditions using the same validation set. TCSM was not initialized from any pre-trained model trained on any extra data.\n\nBaseline models were trained to convergence across multiple epochs, TCSM's gains come from our density ratio parameterization (see W5).\n\n## W4\nWe implemented AR baseline with 26B tokens, achieving 22.51 PPL. We'll add to Fig 1.\n\n## W5\nTab 6 gains come from our density ratio model $r_{1|t}^{\\theta}$, not extra training. This model captures token dependencies other approaches miss with conditional independence assumption.\n\n## W6\nPlease refer to Q3 in vK3t.\n\n## W7, W8\nAll experiments use linear probability paths. We built on MD4's codebase, maintaining identical architectures, sizes, training steps, and optimizers for fair comparison. Code will be released for reproducibility.\n\n\n# Other Concerns\n\n## Q4\nEq.4 and 5 are indeed specific instances of the general Eq.3, simplified by using the 1-Hamming neighborhood.\nThis adds no additional assumptions on the divergence operator, as the goal of matching model and TCS can use any divergence measure. The 1-Hamming structure only defines what scores we match, not how we measure their differences through the divergence.\n\n## Q6, Formatting issues 6\n'Multiset' in Def 3.1 refers to how we construct the concrete score matrix (L114, RHS) by duplicating the original sequence. This enables transition from score space to normalized probability distribution space (Sec 3, L185-L219, 'Target Concrete Score' paragraph). Our approach parallels concepts in prior work (Proposition 1, Eq. 9 in MD4).\n\n## Q7\nWe agree \"pseudo-likelihood\" in Prop 4.1 isn't used in exact sense. This term was chosen because minimizing $l_{pseudo}$ directly corresponds to maximizing standard pseudo-likelihood. We'll revise.\n\n\n## Q9\nAs in Table 5, when implementing Eq. 11, we treat samples from $p^{ref}$ as \"negative samples\" for training model $\\theta$. We don't backpropagate through sampling from $p^{ref}$.\n\n## W1\nPlease refer to Q4 of vK3t.\n\n## Extra References\nThank you for pointing out the extra references. We will add all suggested references.", "claims_and_evidence": "The claims are well supported by the experiments, however, as noted below I have some questions about the details of the experiments and believe more information should be provided in the manuscript.", "methods_and_evaluation_criteria": "Yes, benchmarks and tasks are well suited to the tasks at hand.", "theoretical_claims": "Proofs were checked. However, there is a proof missing in the manuscript. Currently the referenced proof for Proposition 4.1 is not found in the appendix. This should be remedied during the upcoming discussion period.\n\nAdditionally, I believe that the claim that TCSM unifies previous discrete diffusion and flow matching works under a single framework (i.e., Table 2) deserves more explicit derivation that demonstrates equivalence (could be added to the Appendix).", "experimental_designs_or_analyses": "My biggest questions / concerns are regarding the main NLP experiments. Specifically, I do not believe that the current manuscript contains enough information for reproducibility:\n-  The authors should provide details on how perplexity (or its lower bounds) were computed for TCSM methods.\n- More details for Figure 1 and Table 4 should be provided, i.e, model sizes and hyperparameters for training.\n    - More specifically, for Section 4.2, it is unclear whether the TCSM models used in Figure 1 are randomly initialized or come from pre-trained BERT/AR/Hollow transformer models. If the latter, the authors should more explicitly state the details for the pre-trained model, and moreover should clarify why the comparisons to the baselines are still “fair” if pre-trained models are used (i.e., TCSM would be receiving more training / FLOPs budget in that case). \n- All of the fine-tuning / post-training experiments should have more explicit details about the specific model used for initialization and the hyperparameter setup of the fine-tuning", "relation_to_broader_scientific_literature": "This work stands in contrast to the successful denoising score matching work of SEDD and is a generalization of other recent discrete diffusion and flow matching papers.", "essential_references_not_discussed": "Throughout the paper, I would recommend adding Ou et al., 2024 [1] to the reference lists that include MDLM and MD4, as Ou et al was concurrent with these works and derives a similar formulation.\n\nThe authors should cite works such as [2], [3], [4] for discussing the use of the taylor approx. in the AR parameterization/distillation\n\nAdditionally, references for the text8 [5] and OWT [6]  datasets should be added.\n\nFinally, it would be helpful to add references to the first sentence in Section 4.2.\n\n---\n\n[1] Ou, Jingyang, et al. \"Your absorbing discrete diffusion secretly models the conditional distributions of clean data.\" arXiv preprint arXiv:2406.03736 (2024).\n\n[2] Vignac, Clement, et al. \"Digress: Discrete denoising diffusion for graph generation.\" arXiv preprint arXiv:2209.14734 (2022).\n\n[3] Nisonoff, Hunter, et al. \"Unlocking guidance for discrete state-space diffusion and flow models.\" arXiv preprint arXiv:2406.01572 (2024).\n\n[4] Schiff, Yair, et al. \"Simple Guidance Mechanisms for Discrete Diffusion Models.\" arXiv preprint arXiv:2412.10193 (2024).\n\n[5] Matt Mahoney. Text8 dataset, 2011. URL http://mattmahoney.net/dc/textdata.\n\n[6] Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http:\n//Skylion007.github.io/OpenWebTextCorpus, 2019.", "other_strengths_and_weaknesses": "**Strengths**\n\nOverall, I think this is innovative work and find the new directions of post-training and various parameterizations compelling. Additionally, if the concerns / confusions I have about the experimental details are resolved, this work would represent a very big step forward towards closing the gap to the dominant AR approach for language modeling. My current score below reflects this and I would be more than happy to increase it if the concerns raised above and detailed below are addressed.\n\n**Weaknesses**\n\n1. The authors should provide more explicit derivations of equivalence between TCSM and the works listed in Table 2.\n2. The authors should clarify whether the comparison between TCSM and other models in Figure 1 and Tables 3 and 4 is “fair”. Was TCSM initialized from a pre-trained model here? If so, which one and additionally is the training budget comparable to baselines after accounting for the train of the initialization model.\n3. The presentation of an “experiments” section following Section 4.1 seems out of place, since my understanding is that the actual models used require the parameterization introduced in Section 4.2.\n4. An AR baseline should be added to Figure 1.\n5. Are the gains in Table 6 just a product of additional training? Or are the baselines indeed comparable to fine-tuned TCSM?\n6. The presentation of distillation in Section 5.4 as a post-training technique is somewhat confusing, considering this is the implementation of the AR parameterization of $p_1$ in Section 4.2, if I understand correctly. It seems the same methodology is being “pitched” for both pre and post training.\n7. More experimental details are required to ensure reproducible results. Specifically, for all experiments the specific probability path schedule (I am assuming linear is used) should be mentioned as well as other missing model size, training length, and hyperparameter details (many / all of these can go in the appendix).\n8. All the post-/fine-tuning experimental details should provide more information about the pre-trained model being used.", "other_comments_or_suggestions": "**Suggestions**\n1. It would be useful to have algorithms for training and generation with TCSM.\n2. Throughout the text, I think you can consider dropping the time subscripts when using the variable $y$. To me, this represents any alternative from the vocab and so the subscript is unnecessary.\n3. It would be helpful to make explicit what the “mild regularity assumptions” on the divergence are.\n4. Citations should be added to Tables 1 and 2.\n5. In Def 3.1, perhaps the notation for $\\mathcal{N}$ on Line 105-RHS should be changed to: $\\mathcal{N}(\\mathbf{x}) = \\{\\mathbf{x\\}_{n\\_1}, \\ldots, \\mathbf{x}\\_{n\\_{K\\_{\\mathbf{x}}}} \\}$.\n6. On Line 198 LHS, one can surmise what is meant by $[c]_{\\mathbf{y}_1}$ but this notation is not strictly well-defined in the text.\n7. Consider adding an explicit definition for the cross entropy notation $\\mathbb{H}$ on line 211 RHS.\n8. I know it is technically “concurrent” work according to ICLR guidelines, but it might be nice to add UDLM (Schiff et al., 2024) to Tables 2 and 4.\n9. The best values per column should be bolded in Table 3.\n\n**Typos / Formatting issues**\n1. Figure titles, legends, axes labels, and tickmarks are quite hard to read. Should be made bigger / more legible. Additionally in Figures 2 and 3, the dotted lines, the shading, and the meaning of the legend text (e.g. TCSM xx) are not explained anywhere.\n2. There are several places where $p_{1|t}^{\\theta}$ is replaced with $p_{t|1}^{\\theta}$, e.g.,  Line 78-RHS, Line84-RHS, Line89-RHS, Line 676…\n4. Several places have minor formatting issues after equations where the commas appear on a new line, e.g., after Equations (1) and (3)\n5. References in the main text to Eq (12) should instead refer to Eq (1).\n6. Line 134 LHS should say “ensures” instead of “ensuring”.\n7. On Lines 115-116 RHS I understand what is meant by describing the matrix representation of the concrete score, but I think this part of the sentence is a bit confusing “by replicating the original sequence $\\mathbf{x}$ $L$ times.” It’s not the values of $\\mathbf{x}$ that are actually used, it’s just the right shape? Is that what is trying to be conveyed here?\n8. Line 116 RHS is missing  \"$\\mid \\mathbf{x}_t$\" in the vector expression for each column $i$ of the concrete score.\n9. In the first two rows of Table 2, should “KL” instead be “GKL” or does the caption need to be updated?\n10. The numbering for Tables 3 and 4 should be switched. \n11. The actual parameterization of TCSM in Tables 3 and 4 should be indicated (i.e. BERT, vs. AR, vs Hollow).", "questions_for_authors": "1. How would one use the parameterizations in Section 4.2 for generation? For example, the pre-trained BERT model seems to require ground truth data to compute  $p_{1|t}(x_1^i\\mid \\mathbf{x}_1^{\\neq i})$? \n2. Are the parameterizations in Section 4.2 indeed “Pre-trained” (as indicated in the gray paragraph header in Line 256 RHS) or are they initialized from scratch?\n3. How expensive are the parameterizations in Section 4.2? Seems like it requires $L$ forward passes for each sentence when using BERT / Hollow models?\n4. Is it obvious that the divergence from Eq (3) can be decomposed into the component parts, as in Equations (4) and (5)? Aren’t there some assumptions being made about the divergence operator here that render these equivalent? If so, these should be explicitly stated. \n5. Why is it important to define $h(\\mathbf{x}_1 \\mid \\mathbf{x}_t)$? It seems that in all places, this is assumed to just be equal to $p(\\mathbf{x}_1 \\mid \\mathbf{x}_t)$.\n6. In Definition 3.1, why is it possible to have a “multi”set? Is this ever used?\n7. In Proposition 4.1, is it correct to use the term “pseudo-likelihood\" for $\\ell^{i}_{pseudo}$ since it also has the added $1 / (V\\cdot p)$ term?\n8. For Figure 1, why do the authors only report PPL after 26B tokens. I believe this is only ~10% of the training of the baseline models? Additionally, were SEDD and MDLM re-trained for this figure? If so, that should be clarified.\n9. In the Density Ration Parameterization of Section 5.1, wouldn’t this cause $\\theta$ to not be updated since the $p_{ref}$ in the parameterization of $r_{1|t}$ would cancel out with that in the first entry of the Bregman Divergence in Eq. 11?\n10. What does “Reimpl.” mean in the TCSM BCE line of Table 6? What is being reimplemented? Additionally, where is the MDLM number taken from? This is different than the one reported in Table 2 of Sahoo et al. 2024.\n11. Can the authors elaborate on what is meant by the differing methods of compute PPL noted in the Appendix (Liines 875-879)? Some notation/formulas explaining the differences would be useful here as well as an explicit mention of which baseline uses which methodology.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cV7l5ffE5I", "reviewer": "Reviewer_wm3g", "summary": "The paper introduces a novel training objective for discrete diffusion models, dubbed Target Concrete Score Matching (TCSM), which is based on the concrete score (Meng et al., 2022).\nSpecifically, two different objectives are proposed: One is based on some divergence between the target and predicted concrete score, whereas the other is based on some divergence between the target and predicted distribution (conditioned on a noisy version of the data). It is shown not only that the two objectives have the same global minimum, but also that if the global minimum is reached, then the target and model distribution are the same almost everywhere.\nBy choosing the forward KL-divergence as a specific instance of TCSM and under a factorized parameterization (where every token is modeled independently), the objective is claimed to reduce to the simple cross-entropy reconstruction objective, where the clean data is predicted from its noisy version.\nTo demonstrate the flexibility of TCSM, the proposed training objective is applied to pre-training a language model with or without some reference model, and also adapted for post-training tasks in the form of (offline) reward optimization, (offline) preference tuning, as well as knowledge distillation from some teacher language model.\nThe reported empirical results are on-par with or slightly ahead of prior work.\n\n## Update after rebuttal\nThank you to the authors for earnestly and extensively engaging with my questions and feedback. There were a lot of initial concerns, some of which were addressed by the authors' rebuttal and some of which turned into different concerns, some of which in turn have been addressed by the authors' followup response. In the following, I will summarize which of my concerns have and have not been addressed after the rebuttal:\n1. **Faster convergence of TCSM:** While this point was initially unclear, it seems that TCSM converges faster than baselines _initially_, but performs worse than the baselines (on masked diffusion) after full convergence. In light of this, claiming \"faster convergence\" is, IMO, unjustified as _convergence_ usually refers to the point in training where the model no longer improves, i.e. full convergence. In the absence of further evidence to the contrary, I believe that optimizing the diffusion ELBO directly is still superior for pre-training.\n2. **TCSM encompasses several existing methods:** This concern has been partially addressed. I now see how TCSM can be used to train various discrete diffusion models, including MDMs. However, claiming that, e.g., MDM is an instance of TCSM still requires a mathematical proof.\n3. **Likelihood computation:** TCSM relies on existing theory for likelihood bounds and does not provide a likelihood bound in and of itself. This confusion has been addressed.\n3. **Soundness of theoretical claims:** Most of my concerns have been adequately addressed. The intractability of $p\\_{1|t}$ remains a concern. This has now been reduced to the intractability of $p\\_t(x\\_t)$, which is still an issue if not conditioned on $x\\_1$. During training, we neither sample from nor approximate $p\\_t(x\\_t)$ but instead sample from $p\\_1(x\\_1)$ and then $p\\_t(x\\_t | x\\_1)$. This results in the same joint distribution, but would imply that the theory does not describe what we actually do in practice.\n4. **Data prediction:** The authors point towards DFM for this terminology, but the DFM targets are probability _flows_ rather than probability _distributions_. In the DFM sense, TCSM is neither noise nor data prediction, since we are dealing with a distribution over denoised tokens rather than a probability flow. I maintain that the usage of the term in this paper is unnecessary and confusing.\n\nOverall, given that many of the concerns on theoretical soundness have been addressed, I will raise my score from 1 (reject) to 2 (weak reject). Due to the initial extent of issues and given that TCSM for pre-training (and especially the claim of faster convergence) stands on weak evidence, I believe that major revisions in the form of improved clarity and framing are necessary. Unfortunately, this makes it hard for me to recommend an accepting decision, despite the novel and significant contributions.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough and insightful review. We address each point in detail below, quoting relevant comments, and will incorporate all suggestions into revision.\n\nDue to the paper's density, we aimed to balance presenting our method and providing essential background information. We omitted some standard practices for brevity, the concerns arising from these omissions can be clarified below with minor revisions, as they do not reflect flaws in our method or experiments.\n\n# Q5\n>Inconsistent Fig 1 baseline numbers\n\n>cited numbers are different from original paper. calls into question soundness of methodology or reported numbers.\n\nThe discrepancy comes from different experimental setups. Fig 1 shows models trained on 26B OpenWebText tokens, making scores incomparable to original papers. We mentioned this in Sec 4.2 (L301) and Fig 1's caption, but agree it needs more emphasis. See response to Q2 of vK3t for more details.\n\n# Q1\n>How to compute likelihood?\n\nWe use same ELBO methods as prior works:\n- Masked diffusion: MD4/MDLM approach (Eq.4 in [1])\n- Uniform diffusion: DFM method (Eq.7.32 in [3])\n\n>Are Eq(4,5) likelihood bounds?\n\n>Writing is overly general (is sampling t from arbitrary $\\omega(t)$ necessary?)\n\n>Masked diffusion models have ELBO as weighted loss based on noise schedule, the source of the scaling factor in TCSM isn't clear.\n\n$\\omega(t)$ provides necessary weighting for valid ELBO and scaling factors, following diffusion literature (Eq.7 in [5], [4 Sec2.1]). For masked diffusion, ELBO weighting is $\\lambda(t) = \\frac{d\\alpha_t/dt}{1-\\alpha_t}$, so $\\omega(t) \\propto \\lambda(t)$. We sample $t$ uniformly and apply $\\lambda(t)$, equivalent to $t \\sim \\omega(t)$.\nEq (4, 5) are likelihood bounds only when $\\omega(t)$ matches the ELBO weighting scheme.\n\n# Q2\nRefer to Q1 of vK3t.\n\n# Q3\nEq.10 comes from Eq.9 using conditional independence\n$p^{\\theta}_{1|t}(\\mathbf{x}_1 | \\mathbf{x}_t) = \\prod\\_{i=1}^{L} p^{\\theta}\\_{1|t} (x^i_1 | \\mathbf{x}_t)$.\nThus $p^{\\theta}\\_{1|t}(\\mathbf{x}\\_1 | \\mathbf{x}\\_1^{\\neq i}, \\mathbf{x}\\_t)= p^{\\theta}\\_{1|t}(\\mathbf{x}\\_1 | \\mathbf{x}\\_t)$, yielding Eq.10.\n\n# Q4\nIn pre-training, $h_{1|t} = p_{1|t}$ sampling from true data distribution, noted in Prop. 4.1 (L195) and L239.\n\n# Q6\nTable shows OWT val PPL from OWT-trained models. We'll add main text references.\n\n>Where is data-prediction (Eq.12) from?\n\n\"Data-pred\" follows DDPM conventions, used alongside \"noise-prediction\" or \"v-prediction\". It refers to predicting clean data $\\mathbf{x}_1$ from noisy $\\mathbf{x}_t$, versus predicting noise. See Eq. 109 in [4] for more.\n\n# Q7\nIn Sec 4.2, we discuss gradient variance to motivate using parametric models for faster convergence. Prior works (e.g., Fig 2 in [6]) shows high gradient variance can slow convergence in diffusion models. While we don't directly measure variance reduction, our experiments demonstrate TCSM's faster convergence, addressing this core issue. We'll clarify in the revision that sample efficiency is our direct objective and demonstrated benefit.\n\nUsing existing LLMs (e.g. LLaMa) as parametric models avoids pretraining costs. Though we trained from scratch on OWT for fair comparisons, results show potential for leveraging pretrained LLMs.\n\n# Q8\nSimplification comes from using $p_{1|t}$ as $h_{1|t}$ and forward KL. We'll add complete notation: $l^i\\_{\\text{distrib}} = - \\mathbb{E}\\_{p_{1|t}(x\\_1^i | \\mathbf{x}\\_t)} \\log p\\_{1|t}^{\\theta}(x^i\\_1| \\mathbf{x}\\_t) + C$.\n\n# Q9\n>RHS of Eq.19 repeated twice\n\nRHS terms differ by $\\mathbf{x}_1^{\\neq i}$ positions.\n\n>RHS of Eq.20 is identical to Eq.19\n\nThe proof shows equivalence between two concrete score views: Eq.19's LHS gives conditional score $p(x_1^i | \\mathbf{x}_1^{\\neq i}, \\mathbf{x}_t)$, while Eq.20's LHS gives $i$-th component of joint score $p(\\mathbf{x}_1 | \\mathbf{x}_t)$. They're equal as normalization constants cancel in ratio-based concrete score.  Though the RHS expressions appear identical, the key difference lies in their LHS interpretations\n\n# Q11\nSee response to Q5 of vK3t.\n\n# Q12\n>Prop 3 insight, Prop 4.1 Proof\n\nProp 3 enables practical TCSM objective estimation from data. Since $\\mathbf{c}\\_{p\\_{1|t}}$ is unknown, it reformulates the objective using KL and IS divergences, leading to the objective in Prop 4.1 used in experiments. It thus connects theory to practical estimation.\n\nWe'll include the proof of Prop 4.1 in appendix, which straightforwardly writes out the explicit forms of the KL and IS divergences.\n\n# Q13\n>Discussed divergence not used in exp\n\nGen. KL used in \"TCSM Absorb $L_{score}$\" (Table 4). Forward KL used in $\\ell_{\\text{distrib}}$ in \"TCSM Absorb $L_{distrib}$\" (Tables 3-4). Post-training uses f-divergence, giving objective (Eq. 11), tested in Tab 5-6 and App E.2.\n\n# Q14\n>Table 2 not elaborated\n\nPlease refer to Q4 of vK3t.\n\n# Q15\n>Samples\n\nWill be added in the revision.\n\n# Ref\n[1] 2406.04329\n[2] 2406.07524\n[3] 2412.06264\n[4] 2303.00848\n[5] 2011.13456\n[6] 2503.09573", "claims_and_evidence": "Section 4.2 claims that the proposed method helps mitigate “slow convergence, high variance gradients, and reduced sample efficiency compared to autoregressive models” by using a parametric model (a pre-trained language model, AR or non-AR). While using a parametric model indeed improves convergence compared to the TCSM baseline, no evidence on gradient variance is provided. Furthermore, the setup requires having access to a capable pre-trained model, which itself requires compute resources to train, thus offsetting any efficiency gains. I therefore cannot confidently conclude any of the above claims from the provided evidence.\n\nSection 4.1 (L268 ff.) claims that TCSM “encompasses several existing discrete diffusion methods, including MD4 (Shi et al., 2024), MDLM (Sahoo et al., 2024), and DFM (Gat et al., 2024)”, with different source distributions and divergence measure supposedly giving rise to various objectives from the literature (as per Table 2). However, this is stated without proof, and the claim is not at all obvious. For starters: Masked diffusion models (MD4 and MDLM) have an ELBO that takes the form of a weighted reconstruction loss, where the weights $\\frac{\\alpha_t’}{1 - \\alpha_t}$ are determined by the noise schedule $\\alpha_t$. While the proposed TCSM $l_\\mathrm{distrib}$ loss indeed seems to reduce to a reconstruction loss in certain cases, it is entirely unclear where the scaling factor would come from. Adding to the confusion is the fact that the equivalence is claimed for the $\\mathcal{L}_\\mathrm{distrib}$ loss, which is not actually an instance of TCSM (Eq. 3), but only has the same global minimum (as per Propositions 2).\n\nThe paper also claims an improvement in PPL compared to other discrete diffusion models (MDLM and SEDD), but the cited numbers are different from what is reported in the original paper. This calls into question the soundness of either the methodology or the reported numbers.", "methods_and_evaluation_criteria": "The paper relies heavily on likelihood as an evaluation criterion (BPC, PPL), and while the chosen datasets are consistent with prior work, there is no explanation on how one actually calculates the likelihood under the proposed model. While it is somewhat plausible that one _can_ calculate likelihood from the concrete score, it is not obvious (to me) how to do this, and the paper is lacking detail in this regard. If the intended claim is that the proposed $\\mathcal{L}\\_\\mathrm{score}$ and\n$\\mathcal{L}_\\mathrm{distrib}$ losses are likelihood bounds, this would have to be proven mathematically.", "theoretical_claims": "Proposition 4.1 refers to the appendix for proof, but I cannot find this proof anywhere. Therefore the claim is stated without proof.\n\nFor Eq. (9), it is claimed that the derivation is straightforward starting at Eq. (5), but I do not see how this simplification occurs. Specifically, while there are indeed some constant terms (which we cannot simply “drop” and claim _equality_), the expectation/sum over $p_{1|t}$ that is present in the KL-divergence but not Eq. (9) cannot (trivially) be ignored. Hence, and as far as I can tell, while Eq. (9) can still serve as a training objective, the claimed equality does not hold.\n\nThe simplified objectives arising from a factorized parameterization (Eq. 10 and L263) are also stated without proof, and it is non-obvious how they follow from the stated assumption.\n\nThe proof of Proposition 1 (App. B.1) assumes “mild regularity conditions”, which is a term that I am not familiar with, and I am unable to find any further elaboration in the paper. Clarifying these conditions would improve the mathematical rigor. Other than this, the proof seems correct.\n\nProof of Proposition 2:\n- RHS of Eq. (19) is repeated twice.\n- RHS of Eq. (20) is identical to Eq. (19). It seems to me like at least one of them should be different.\n- Consequently, I have a hard time coming to the claimed conclusion.\n\nProof of Proposition 3 seems fine after skimming.", "experimental_designs_or_analyses": "The paper relies on likelihood as a measure of model performance, which is fine, but the lack of qualitative examples somewhat casts doubt on the sample quality of the proposed model. I suggest providing at least some samples from each experiment (unconditional generation, IMDB sentiment tuning, toxicity tuning) in the appendix.", "relation_to_broader_scientific_literature": "The paper combines ideas from “Target Score Matching” (Bortoli et al., 2024) and “Concrete Score Matching” (Meng et al., 2022). While I am not deeply familiar with these two papers, it seems like a more detailed discussion on how the presented method is similar/different from and builds upon these two prior works would be appropriate, especially considering that the papers in question are only referenced in passing and are not mentioned in Section 2 (Preliminaries).", "essential_references_not_discussed": "It seems to me like the comparison to [1] and/or [2] would be appropriate, and arguably more pressing than comparing to discrete flow matching.\n\n- [1] Lou et al., 2023. https://arxiv.org/abs/2310.16834\n- [2] Sun et al., 2022. https://arxiv.org/abs/2211.16750", "other_strengths_and_weaknesses": "The paper provides an interesting and new perspective on discrete diffusion models and, if the claimed performance improvements are real, improves the state-of-the-art. This is the paper’s strength, which should be focused in future revisions.\n\nHowever, the mathematical rigor and the soundness of claimed results leaves a lot to be desired and makes it hard for me to recommend an accepting decision. I am happy to reevaluate my conclusion if these concerns can be addressed, but the rebuttal period constitutes a tight deadline given the extent of the issues. It may be deemed the better, less stressful option to take the necessary time to improve the writeup and promptly resubmit it at a future date.", "other_comments_or_suggestions": "On a more subjective note, I have found that the clarity of the writing has a lot of room for improvement. Things feel, at times, overly and unnecessarily general, in a way that obfuscates the contributions of the paper (e.g. do we really need to sample $t$ over an arbitrary distribution if this distribution in practice will always be $U(0, 1)$?). The theoretical results could also be put into perspective a bit better, leading the reader to the desired conclusions. For example: It is not clear to me why Proposition 3 gives new insight, considering that we have already proved equivalence of $l_\\mathrm{score}$ and $l_\\mathrm{distrib}$ in Proposition 2. Further, while Proposition 2 assumes some specific divergence measures, as far as I can tell, these are not the ones that are later used in experiments. Another example: Table 2 is not only stated without proof, but also never elaborated on, and it is left to the reader to “figure out” what it means and entails.\n\nTo summarize: I would suggest to streamline the theory by clearly stating all the conclusions we would like to take away, to lead the reader towards the desired conclusions as clearly and directly as possible, and to remove anything that does not contribute towards this goal.", "questions_for_authors": "- How does one compute likelihood under the proposed model? Are objectives (4) and (5) supposed to be likelihood bounds?\n- How does one generate samples with the proposed model?\n- How is Eq. (10) derived?\n- What is the proposal distribution $h(x_1 | x_t)$ during pretraining? This seems like an essential component of the training objective that is (as far as I can tell) never specified until Section 5.\n- How are baseline numbers for SEDD and MDLM obtained? The reported numbers (Figure 1) are different from what is claimed in each respective original paper: Sahoo et al. report a PPL of 23.21, not >30, for MDLM and 24.10, not >34, for SEDD. What is the reason for this discrepancy?\n- Table 6: What data is used to compute PPL? Also, the table is not referenced in the main text.\nWhere is the “data-prediction objective” (Eq. 12) from? I cannot find it in either of the two references (Campbell et al., 2024; Gat et al., 2024).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes target concrete score matching methods for discrete diffusion models. The study covers pre-training and post-training to demonstrate the advantage of target concrete score matching.

The proposed score estimation and training pipeline is the major contribution of the paper. This is well-recognized as novel and important contributions by all the reviewers. Concerns of the paper center around the clarity of the presentation and completeness of the results. For example, the formal justification of Proposition 4.1 is missing. Rough edits around equations, where comma appears in the beginning of a line. Faster convergence of target concrete score matching using a parametric model for $p_1$ seems to be not fully justified and involves unfair comparison.

Given these debates, the recommendation should be considered as a conditional acceptance. The positive recommendation owes to the complete pipeline provided in the paper and relatively sufficient empirical results. Nonetheless, the aforementioned issues must be resolved before the next revision.
Decision: Accept (poster)