Edit Flows: Variable Length Discrete Flow Matching with Sequence-Level Edit Operations
Authors: Marton Havasi
Brian Karrer
Itai Gat
Ricky T. Q. Chen
Venue: NeurIPS 2025 poster ()
Keywords: Discrete Flow Matching
Primary area: probabilistic_methods
TL;DR: A discrete flow model with native variable-length generation capabilities using edit operations and relying only on relative token positioning.

Abstract: Autoregressive generative models naturally generate variable-length sequences, while non-autoregressive models struggle, often imposing rigid, token-wise structures. We propose Edit Flows, a non-autoregressive model that overcomes these limitations by defining a discrete flow over sequences through edit operations---insertions, deletions, and substitutions. By modeling these operations within a Continuous-time Markov Chain over the sequence space, Edit Flows enable flexible, position-relative generation that aligns more closely with the structure of sequence data. Our training method leverages an expanded state space with auxiliary variables, making the learning process efficient and tractable. Empirical results show that Edit Flows outperforms both autoregressive and mask models on image captioning and significantly outperforms the mask construction in text and code generation.

Edit Flows: Variable Length Discrete Flow Matching with Sequence-Level Edit Operations
Marton Havasi FAIR at Meta
Brian Karrer FAIR at Meta
Itai Gat FAIR at Meta
Ricky Q. Chen FAIR at Meta
Abstract
Autoregressive generative models naturally generate variable-length sequences, while non-autoregressive models struggle, often imposing rigid, token-wise struc- tures. We propose Edit Flows, a non-autoregressive model that overcomes these limitations by defining discrete flow over sequences through edit operations- insertions, deletions, and substitutions. By modeling these operations within a Continuous-time Markov Chain over the sequence space, Edit Flows enable flex- ible, position-relative generation that aligns more closely with the structure of sequence data. Our training method leverages an expanded state space with auxil- iary variables, making the learning process efficient and tractable. Empirical results show that Edit Flows outperforms both autoregressive and mask models on image captioning and significantly outperforms the mask construction in text and code generation.
1 Introduction
Non-autoregresssive models have become the standard across high-dimensional modalities, thanks to their ability to produce coherent and globally consistent outputs. Recent advances include MovieGen (Polyak et al., 2025) for video, Audiobox (Vyas al., 2023) for audio, and Stable Diffusion (Esser et al., 2024) for images. This trend extends to discrete code and text generation well: recent diffusion-based models such as LLaDa (Nie et al., 2025), DREAM (Ye et al., 2025), and Mercury (Ermon al., 2025) show that fully parallel generation autoregressive baselines on certain open-ended language tasks. Despite these advances, current non-autoregressive models rely rigid, factorized representations with fixed token positions. They work by iteratively unmasking or replacing tokens the target sequence. Critically, they cannot add remove tokens: two fundamental operations for modeling sequential data.
match even surpass strong
In this paper, we propose Edit Flows, novel non-autoregressive framework that models generation as discrete flow over the space of sequences via edit innietionnn,sssiinnnn, deletions, and substitutions. We frame sequence generation as a stochastic process governed by Continuous-tim Markov Chain (CTMC) over full sequences, in contrast to the usual factorized representation with absolute token positions (Figure 1). The model learns to estimate the rate each possible edit operation conditioned the current sequence (Figure 2). This enables modeling based relative token positions and eliminates the need for masking or padding tokens during training or inference. Moreover, Edit Flows naturally accommodate variable-lengte sequences. In contrast to existing non-autoregressive models that generate tokens in fixed lengths or rely on hcuristic cemi-auutregrresivve sampling (Nie et al., 2025), Edit Flows produce longer shorter outputs adaptively, depending the context.
Despite the conceptual simplicity of modeling sequence transitions through edits, training such models is non-trivial. A direct optimization of full sequence-level stochastic processes typically demands costly computations. To address this, we introduce Flow Matching- (Lipman et al., 2024) training procedure that augments the state space with auxiliary variables that determine one possible chain edits that leads to the target sequence. By sampling these auxiliary variables in
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Inputs
Xyo
<806>
centence
avocadoo
the
caccence
strough
reyLeser
<BoG>
sentence
Outputts
Inc.
Ins.
Insart
rate
Ins
.808.
Sentence
thr ough
the
Dol
through
Ins
*yorrs
-808.
sentence
through
revtation
Sub
Subotitute
rate
thought
Ins
Xyooas
<800>
sentence
evolves
reviston
Del
Del:
Delete
rate
Del
Del
<006-
sentence
evolive
through
revision
Figure 1: Edit Flow sampling process. Starting with To containing random tokens or an empty sequence, the model applies edits at and reaches cohesive sentence at time 1.
Figure Edit Flow model inputs and out- puts. Given Tt, the model predicts the rate of each possible edit.
each training iteration (without exposing them to the model), obtain tractable training objective and the model automatically learns infer these auxiliary variables.
Empirically, Edit Flows show a strong and consistent improvement over fixed-length discrete flow and diffusion models (Campbell al., 2024b; Gat al., 2024; Shi et al., 2024) across several benchmarks, including image-to-tex generation at 280M parameter scale (MS-COCO, Image Captioning 3M), code generation at 1.3B parameter scale (HumanEval, MBPP), and open-ended text benchmarks 1.3B parameter scale (HellaSwage ARC, PIQA, OBQA, WinoGrande). On image-to-text generation, Edit Flows outperformed all baselines, including the autoregressive model, and code generation, has relative improvement of 138% over the mask model. We summarize our contributions:
We introduce Edit Flows, non-autoregressive generation framework expanding upon the operations-innertioons substitutions, and deletions.
Discrete Flow Matching recipe, with native support for variable-length sequences via edit
We construct sequence-level probability path, enabling CTMC-basce modeling directly over sequences of varying lengths, unlike prior work focused on loken-level transitions.
We demonstrate the effectiveness of Edit Flows on large-scale benchmarks in image caption- ing, open-ended text benchmarks, and code generation.
Preliminaries
2.1 Continuous-time Markov Chains
To form the basis of our discrete generative model (Campbell et al., 2024b; Gat et al., 2024; Hold- errieth et al., 2024; Shaul al., 2024), we make use of Continuous Markov Chains (CTMC) over a discrete space X. These are Markov processes that generate trajectories (XX))eeo,1) and is characterized by rate u, denoting the infinitesimal transition probabilities between states
P( = x|X = x.)) = 62, (x) hu, (x/xz)) + o(h)
(I)
where oh) satisfies lim o(h) = On. Sampling from CTMC can be done by iteratively applying the update formula (I). The rate uz (x(2;) denotes the infinitesimal probabilities transitioning from state to any other state time i, and for (1) to be a proper probability mass function, need both sides to sum to one. Hence, u, needs to satisfy
(x(x)) 2 0 for all r + Øt,
E. (x(((ar)) = O,
(2)
typically referred to as the rate conditions. Note this enforces (x(2)(11)) Eayoy, Ue (z(22))

---

We a rate 4 "generates" probability path if the time marginals of the associated CTMC are samples from Pt. i.e. X₁ Pt- Concretely, they should satisfy the Kolmogorov forward equation.
0 0t Pr(x) = Σ (r(x = Σ (e(xlu)p.()) (ul
flow
flow
That is, the change in probability of being in state is the total infinitesimal probability flowing into ar from other states minus the total infinitesimal probability flowing out of æ, determined by the rate.
2.2 Discrete Flow Matching
Discrete Flow Matching (DFM; Campbell et al. 2024b; Gat et al. 2024) is a conceptually simple framework for learning CTMC-based generative model to transport from source (e.g. noise) distribution p(x) to a target (e.g. data) distribution ((x over a discrete space x E X. For now, consider discrete space over sequences of fixed length N, so X TN where T {1..... M}
denotes a vocabulary of size M containing a discrete set of token values.
Discrete FM training relies on prescribing coupling distribution (1) that samples pairs (1) where the marginals are and q, i.e.,
Σz, n (Zo. 21) (x1))
Σx, ( 21) p(zu)-
(4)
The simplest case is of course the independent coupling ा (2o. x1) p(x00g((1)) Further, we would also prescribe a conditional CTMC characterized by conditional rate
Ut (2) To, .i) generating pe (x/220 s.t. Po [in, x1) = (x) pi (x x1) 6,,(2) (x) (5)
where å denotes Kronecker's s delta function. That is, the conditional probability path (r x1) interpolates between two points from the source and target. DFM then trains generative model that transports according to the marginal probability path pe (z), which interpolates between the source and target distributions.
(x) Exai,, Po(x) 21) 21) (20.
implying Po(x) p(z), (I(x) q(x).
(6)
can be shown that the marginal rate
u. (x((a)))) = 2e,, (1/ vo, 21)
generates the marginal probability path Pt (z), i.e. (x(2z)) characterizes CTMC that transports from the source p to the target data distribution q. In order to train model to approximate (7), prior works have used cross-entropy (Gat et al., 2024; Campbell et al., 2024b) and evidence lower bounds (Lou et al., 2024; Sahoo et al., 2024; Shi et al., 2024; Shaul et al., 2024) as training objectives, all of which are captured by the family of Bregman divergences (Holderrieth et al., 2024).
Token-wise mixture paths. The prescription of (4) and (5) is then left as a design choice. Most existing works have focused on the factorized token-wise conditional path (Gat et al., 2024)
(a 106, x1) = (1 (1-kz)ōag (x") +rrda (æ"), ue (ar 26. æ1)
Ri I-Ke
(dus (2*) duf (z")). (8)
where Ke is scheduler that satisfies Kip 0, K1 1. states that differ by one token, expressed concisely as
The multi multi-dimeesional case is to consider only
(r(2/ x1) II Pe Pe(x") =1), u.(rr zo, 21) E; 82, (x ")ur(x" [ 2b 2f),
where 8z. t) I[ ببة (z") is shorthand for denoting that all dimensions except are the same. That is, this rate is factorized in that it only describes loken-wise changes, though sampling can be done in parallel (1). This is a particular advantage of using continuous-time framework, requiring only a per-dimension parameterization of the model, at the cost of using an iterative procedure for sampling. It has been difficult to generalize beyond the token-wiss paths it quickly become intractable to prescribe a conditional CTMC (5) for training that has more general transitions over sequence space (Shaul al., 2024).
Mask construction. As noted by many existing works (Austin et al., 2021; Lou et al., 2024; Campbell et al., 2024b), the simplifying case of considering the source distribution to be mask
3

---

camp0
600-
through
revistes
throed
((0) (a)2s))))
[log uf
(2r) log (Del
[xx]]
Figure 3: Computing the loss starts with the two aligned sequences 2g and 21- Locations where =6 require insertion operation, locations where zj require deletion and locations where = A 2 require a substitution. 2g is sampled by applying a subset of the operations to 2g depending on the scheduler. Then, Fe is obtained by removing all tokens from 2z. The Monte-Carlo estimate of the loss contains the model output u (3((2)) in two terms: the negated sum of all the edit rates and the logarithms of the remaining edits between 2 and ø1.
distribution has significant theoretical and practical benefits. That is, setting po(x) 6m (zr), where m is a special mask token not found in the original vocabulary. Theoretically, this drastically simplifies the construction (Sahoo al., 2024; Shi et al., 2024) and practically has been shown scale (Nie et al., 2025; Ye et al., 2025; Ermon et al., 2025). The main benefits come from requiring only learning transitions between the mask token and the other tokens, with no transitions between tokens from the original vocabulary. However, this construction still has multiple downsides, as it does not make full use of the CTMC framework and equivalent to an any-order autoregressive model (Hoogeboom et al., 2022; Pannatier et al., 2024) though usually implemented with non-causal attention. As with all token-wise path constructions, the most glaring downside the lack of inherent support for variable- length generation. To handle variable length outside of the modeling framework, padding can be done during training but the excessive padding makes the model over-confident predicting padding tokens, an issue that currently relies on semi-autoregressive sampling to get around (Nie et al., 2025).
3
Edit Flows
3.1
Edit Flows: a continuous-time Markov chain using edit operations
We design new CTMC-based generative model through the Discrete Flow Matching framework using edit operations to enable variable length generation, while encompassing existing constructions as special cases. Let T be as defined previously to be a vocabulary of size M. Then our state space is
defined as the set of all possible sequences up to some maximum length N, i.e., X' = U T
We will now describe the Edit Flow model which is a CTMC that operates directly sequences, and discuss tractable training using generalization of the DFM recipe later in Section 3.2. Specifically, parameterize the rate of CTMC ul. For two sequences x, Hy E X, of (x((); is allowed non-zero only if x and x, differ by one edit operation. An edit operation is one of either insertion, deletion, substitution, which model. Specifically, given a sequence z with variable length n (z)), we define the edit operations that be performed x concretely as follows.
the space of
to transition between sequences in our generative
Let ins(x, i, a), x E X' E fl...., n(x)), .a a E T, be the sequence resulting from inserting the token value to the right side of position of the sequence resulting in
ins(r, i, a) (x+ "...,'', a, ++1 x())),
(10)
Let del(a, i), X, i (1.....,(([)}, be the sequence resulting from deleting the i-th token from the sequence F, resulting in del(x, = (x ........ 1-1 s+1
n(x)).
(11)
Let sub(a, i, a). x E X ( (()),, }e E T, be the sequence resulting from substituting the token value a into position of the sequence H, resulting in
sub(z, i, (x) ..... x 1-1
1... p(z)).
(12)

---

Generated tokens:
def is (s-meen:: int) bool:
a"" "CCheck if a number is prime or not.
in <2 return False
A small white dog wearing black hat top head.
for range(2, n):
if % i = return False
return True
Figure 4: Edit Flow generation examples with X, 0 (i.e. insert-only model). The tokens are color coded to denote the timestep that they were generated in. Left: Coding model conditioned on the function signature. Right: Image captioning model conditioned on the image.
These edit operations define the support of the rate uf (-(are). Figure 1 shows example of CTMC transitioning through sequences using edit operations. Since insertions, deletions, and substitutions result in sequences that mutually exclusive, can parameterize each separately.
uy (ins(xx a)(=)) X!, ((aa))))))
for i fl....,,ll n(x)
(13)
" (del(z,, ((2,,)))zz ()) (x)
for i E (l....,n(z))
(14)
d (sub(rr, a)) (x (z) (als)
for i E [(,.).,,(([))}
(15)
With this parameterization, the Ae, 2 O are the total rates of inserting, deleting, or substituting any token position and determines the chances of each operation occurring; (a)(())) and Qū, Quu (alx) are the (normalized) distributions over token values if an insertion or substitution occurs at position i Equations (13)-(15) ensure rates non-negative and the summation to satisfy (2) is more tractable:
u (xzlZx) = Enr (a, -
-E In( ( (n(a)) (xx))
(16)
Figure shows the model outputs corresponding to (13)(155).
Special cases. The framework of Edit Flows actually generalizes many existing constructions, as one can restrict the rates to recover existing discrete generative models. For instance, the token-wise probability paths (8) are substitution-only, i.e. Xim Addl O, with the mask construction having an additional constraint X (z) 0 if m. As such, the token-wise CTMCs are incapable of increasing or decreasing sequence length. An autoregressive model can also be recovered by only allowing insertions to occur at the rightmost location, i.e., all rates are zero except X (a)) As such, the model is incapable of making corrections to the existing sequence other than inserting new tokens in prescribed order. It can be seen that Edit Flows is simple yet natural generalization of these existing discrete generative modeling constructions.
3.2 Training Edit Flows
Since Edit Flows generalizes beyond the token- wise paths that have been previously explored, it cannot easily make use of existing cross-entroee entropy or evidence lower bound objectives for training. as these are difficult or intractable to derive. The main difficulty in deriving a conditional rate (5) that lies X' the need to account for all possible transitions that can transport from one sequence to another, such multiple possible insertions that transition to the equivalent sequence. Instead, we propose an extension of the DFM training recipe to include an auxiliary Markov process, and in doing resulting in allowing Bregman divergences for training Edit Flows.
Discrete Flow Matching with auxiliary Markov processes. Suppose we wish to train CTMC that lies in space X' and it follows the marginals of a CTMC that lies in an augmented space (a, 2) X Z with a probability path (z, 2). We show that it is possible recover the CTMC that transports directly in X. automatically inferring the auxiliary process in 2. This is concisely formalized in the following Theorem 3.1. Further details and proofs are provided in Appendix B. We note that in contrast to the original Flow Matching derivation (Lipman et al., 2024), this result shows that we can marginalize over time-dependen processes, not just time-independent variables. Finally, this result is more generally applicable than just training Edit Flows; we showcase another application
5

---

of Theorem 3.1 in Appendix C.I to train with localized propagation rates which incentivizes localized edits, going beyond existing independent probability paths.
Theorem 3.1 (Flow Matching with Auxiliary Processes). Let us uz 2/21, 26) be a rate over the augmented space of X' x 2 that generates pr(z, z), then
us (x(()) Σx Bps(asl2a)) ue (12,,
generates
Pc(a)) - Pc(,,22)) (17)
and furthermore, for any Bregman divergence Do(a, - <((a) - -bb)- - (a (a- b. (b)) defined by a convex function 6, we have that
d
E Pe.. Do Eg uel 2e(,, do
do (x) Do (u (-o() u((((1x))) (18)
Training with an auxiliary alignment process. As previously mentioned, it is difficult to directly construct conditional rate (5) for Edit Flows, even if given points zo and æ1, as there can be multiple sets of edit operations that transitions from to 21. Instead, we can consider an augmented space where simpler construction exists. In particular, we will define an auxiliary process using alignments.
Given two sequences To and I1, an alignment can be used to define a precise set of edit operations that transform Ti to Ti. In general, there are many possible alignments for every pair of sequences. For example, below are illustrations of three example alignments between the words 'kitten' and "smitten" (the most optimal, sub-optimal padding-to-the-rre to-the-right strategy, and the least optimal):
KEITTEN 11
KITTEN EETEENEEEE EEEEEEEE 14J LIL
KIT BATTENT EREEEEER
SMITTEN
3EEEE
E 33 e 8 S
The special token e is blank token that is not added to the vocabulary, i.e., it is not part of the input or output of the model. Instead, we will only use it to define an auxiliary process that will provide a training signal for Edit Flows via Theorem 3.1. As can be seen, given an alignment, we can recover edit operations as tuples (a - b) with o, E T {s}. interpreted as an insertion if a = E, a deletion if b E, substitution if a + and b E.
Formally, let us define the space of aligned sequences as Z (Tu {E)) Furthermore, we define the function fim-blanks - + X' as the operation of stripping away all the tokens. Note that since this is many-to-one function, this implies X < |z]. Following the DFM recipe, we would need to prescribe a coupling and conditional CTMC that transports from point to point. Given samples from the source 2i ~ p(x) and target 211 ~ q(ær) in X, we can directly construct aligned sequences 2o and 21 in 2, e.g., by randomly padding the sequences, or by solving for the optimal alignment that corresponds to the minimal edit distance. This defines a coupling π (20, 21) over the auxiliary variables satisfying the correct marginal distributions
p(z) = E Ez, (2u, (zn)
q(z)= E., (20, 21 21)0, f
(19)
Then, given 2o, 21 n,, we define a conditional probability path over the augmented space of X
Z
(1 2 Imo, 20, 2 21) Pa (s. 2/20, =1) = Pe(2) =0, 21 21)0 (z) (1)
(20)
where Pe(2120, 21) is a token- wise mixture probability path (8). A conditional rate that transports along the augmented probability path then given by (see Lemma B.2)
u (x. 2| 2/ 2t, 20, 21) = fan (z) E
Rs I-ne
(6=;(xt) - (=(
(21)
Note that this rate only transports between sequences Xt + x that differ by one edit operation, perfectly mapping to Edit Flow's transitions (13)-(15). Applying Theorem 3.1, the marginal rate that transports from p(x) to ((x) can be expressed as
(x|x,) = Σ. (po (2r, 2|2,, 2t, 20. 21),
(22)
which we learn using Bregman divergence as the training loss (see Appendix B. simplifying to
N
ht Kt [xi
(())= = E
Σ (x(())))) -
Σ
1
log (,,
(23)
(10.21))
(xpe. 21)
1=1
where I(z,, 21) = fim-blanks (z) 22
1-1
,,, 21
i+1
..... 2
N which directly corresponds to of

---

the edit operations in (13)-(15). This loss can be interpreted as minimizing all the output rates of the model, while having weighted cross-entropy over edit operations that bring Ty closer to 21-
Interestingly, even when trained with the least optimal alignment, which deletes all tokens from zo and inserts all tokens in æ1,, the trained model has a preference towards minimizing the number of edits during its generation process (see Appendix F). learning a non-trivial coupling between zo and 21. This is analogous to the kinetic energy minimization that is observed for Flow Matching in continuous space (Shaul et al., 2023).
3.3 Algorithms and advanced techniques for Edit Flows
In this section, we provide details on the sampling procedure and advanced techniques that make use of the Edit Flows framework. We only provide a summary of each technique here, focusing on the resulting algorithmic procedures and high-level intuition; complete details are in Appendix C.
Sampling. Sampling from the model requires transporting a source sample Xo to time t = 1, simulating the CTMC defined with the learned rate uf. Following previous works (Campbell et al., 2022: Gat et al., 2024), we leverage the first-order approximation in (1). Sampling thus iterates: with current state X, and step size h, independently determine whether each insertion, deletion and substitution, occurs with probability (X,,, then perform all edit operations simultancously.
Classifier-free guidance. We considered a few approaches to add classifier ffreeeer--reeeeeo guidance (CFG; Ho and Salimans 2022) to Edit Flows. The scheme that we found to be the most reliable, and which we use throughout all experiments, is apply CFG independently to À and Q.
Sharpening Q. We also explored ad-hoc adjustments to the Q distributions, such as temperature, top-p and top-k sampling, generally intended to sharpen the distribution over the most likely values.
Reverse rates. We can also formulate and learn a CTMC that transports from @ to p. We call this to t = 0. Combining the forward and reverse rates allows us to introduce a stationary component that corrects the samples but does not modify the distribution of the samples, introducing extra inference- -iiee computation for the ability to self-correct during sampling. When applied in practice, we take step forwards in time with u. to ++ h(1 + ax) for a; 0 followed by step in reverse time with - (1++a)+a.)) back to t + h.
a reverse rate ti as we apply
reverse time, from
Localized edit operations. The default rates that we use for the alignments 2, have been factorized per token (21), resulting independent edit operations. While this allows the use of conditional rates from prior work (8), this could be problematic for Edit Flows as when the sequence length becomes large, noisy sequences 2, will consist of noi-beiig tokens. Instead, we propose a non-factorized locality-based construction in which if an edit operation has occurred, it incites nearby edit operations to occur, thereby encouraging locally consistent subsequences in Tr. We construct this by creating a novel auxiliary CTMC that locally propagates the occurrence of edit operations in 2 space, and applying Theorem 3.1 to easily obtain tractable training objective. All details can be found in Appendix C. We find localized Edit Flow models to be especially more performant at generating long sequences, leading to a 48% increase in Pass@1 I on code generation.
4 Related work
Discrete diffusion and flows for language modeling. Generative models based on iterative refine- ment such as diffusion (Sohl-Dickstein et al., 2015; Ho et al., 2020) and flow models (Lipman et al., 2024) have seen their fair share of discrete adaptations. Both aim learn CTMC-based generative model but approach the construction differently. Discrete diffusion models typically start with a corruption process which is then reversed (Austin al., 2021; Lou et al., 2024). Discrete flow models, in contrast, aim to transport between two distributions with an interpolating scheme (Campbell et al., 2024b; Gat et al., 2024). With the DFM framework, Shaul et al. (2024) also proposed new ways of constructing general discrete token- wise paths. However, despite the large design space, none have been able to reliably surpass the simple mask construction, which has been the core focus of many recent works (Sahoo et al., 2024; Shi et al., 2024; Ou et al., 2024; Zheng et al., 2024), motivated by the success of masked language modeling (Devlin et al., 2019; Ghazvininejad et al., 2019; Yang et al., 2019; Chang et al., 2022). In particular, the mask construction has shown to perform well at scale, though it currently still shy of autoregressive models on code generation tasks and requires heuristic or semi semi-autoregressive sampling schemes (Nic et al., 2025; Ye et al., 2025; Ermon et al., 2025). In
7

---

stark contrast, we explored in the opposite direction, making full use of the CTMC- construction instead of simplifying it. This allowed us to generalize the existing DFM construction enable variable-length generation and construct a model using position-relative edits as generative process.
Non-autoregressive variable length generation. When the generative modeling framework does not inherendy allow variable length generation, such as many non-autoregressive approaches, the stereotypical method of handling it is to utilize a separate length prediction model (e.g. Lee al. 2018). More integrated approaches have considered edit operations, though many of the existing con- structions are heuristic- beuse and do not show that they properly sample from the target distribution. Levenshtein Transformer (Gu et al., 2019b) and DiffusER (Reid et al., 2022) are edit- alit-based sequence generation models. They consider sequential expert policy that performs a series of edits each step, and the model is trained through imitation learning. Unlike Edit Flows, DiffusER uses a causal masked model (Aghajanyan et al., 2022) to fill in insertions and substitutions autoregressively and trained to match a discrete-time corruption process that is sequentially simulated. Chan et al. (2020) considers sequence alignments using only deletion operations and leverages marginalization over latent alignments. Gu et al. (2019a) and Stern et al. (2019) propose insertion-only models that sequen- tially predict what and where to insert tokens. The most similar work to ours is perhaps Campbell et al. (2024a), who proposed modeling inserts in a jump diffusion framework, relying on generator theory and evidence lower bounds for training. However, extending this direct derivation approach to more than a singular insertion, and to introduce deletions and substitutions, is very challenging and arguably intractable; an issue that we got around by making simple use of Theorem 3.1.
Relative positions for language modeling. There is growing trend to incorporate only relative positional information into neural network architectures (Liutkus et al., 2021; Press et al., 2021; Peebles and Xie, 2023; Su et al., 2024; Ding et al., 2024). However, on the methods side, there has not yet been a shift due to non-autoregressive models mainly using a token-wise construction. As such, every token generated must also account for the exact position (e.g., exact number of neighboring mask tokens) when deciding on token value. Edit Flows one of the first models to use only relative and localized operations in the method construction, sample generation time, and in the architecture. Beyond the capability of variable length generation, enabling the use of position-relative generation may be key advancement and could be the underlying reason that allows Edit Flows to outperform methods based on absolute positioning.
Iterative editing models. Several prior works on constrained generation employ iterative editing or sampling procedures. Welleck et al. decouple an existing language generator from learned iterative corrector that refines its outputs, whereas Edit Flows uses a single model to begin from random or null sequence and directly generate outputs through a sequence of discrete token edits. Miao et al. (2019) employ Metropolis-Hasting sampling over insertion/éleeiin/eppaacccc to satisfy lexical constraints; by contrast, Edit Flows deterministically takes a fixed number of flow- fow-matching steps from noise to data without an acceptance criterion. Qin et al. (2022) propose COLD decoding, an mncrgy-based approach that iteratively refines whole sequences via Langevin dynamics under constraints, whereas Edit Flows incrementally edits tokens rather than resampling full sequences. Finally, Sha (2020) formulates exxically-constraine optimization problem using differentiable fluency objective to guide edits, but Edit Flows requires no external objective or backpropagation at test time. These contrasts underscore that Edit Flows integrates generation and editing within a single flow-matching model, rather than relying on separate generation and correction modules or auxiliary sampling schemes.
operations
generation as a gradient- -uidded
Experiments
We experimentally validate the performance of Edit Flows on multiple text generation tasks, including image-to-text generation using 280M models, text and code generation benchmarks with 1.3B models.
Baselines. We primarily compare against a state-of-the-art Autoregressive model (Vaswani et al., 2017: Touvron et al., 2023) with standard left-to-right generation, and Mask DFM (Gat et al., 2024) which is the most relevant and best performing non-autoregressive framework currently for text generation, equivalent to discrete mask diffusion models.
Models. We test two variants of our models with different p(Xo). For the default Edit Flow we use = óg so that the flow generates using combination of insertions and deletions, with the forward and reverse rates, respectively. A variant Uniform X, Edit Flow use Xo xx........ x100) where X² Pemp, with Pemp being the (marginalized) empirical dis-
p=
8

---

MS COCO
Image Captioning 3M
Method
METEOR
CIDEr
SPICE
ROUGE-L
CIDEr
SPICE
VI.P (Zhou et al., 2020)
28.4
117.7
21.3
24.3
77.5
16.5
ClipCap (Mokady et al, 2021)
27.1
108.3
20.1
26.7
87.2
18.5
Llama3 Autoregressive
25.7
95.5
19.6
25.2
85.8
17.8
Mask DFM
25.3
95.6
19.2
27.4
96.2
20.3
Edit Flow (Ours)
27.4
108.1
21.1
29.0
101.9
21.7
Localized Edit Flow (Ours)
27.4
105.1
22.1
28.3
99.7
20.8
Table 1: Image captioning benchmarks using Llama3 280M models. Thesse works used pretrained models that were trained larger amount of data and cannot be directly compared; they are shown for reference only. Colors show the best and second best among each metric.
Method
HellaSwag
ARC-E
ARC-C
PIQA
OBQA
WinoGrande
Llama3 Autoregressive
49.5
71.0
36.3
76.0
30.4
62.1
Mask DFM
38.3
55.4
27.8
65.3
22.6
52.3
Edit Flow (Ours)
49.0
63.1
33.0
68.8
28.6
53.6
Table 2: Zero-shot text benchmarks using Llama3 1.3B parameter models trained on DCLM-baseline 1.0 (Li et al., 2024).
tribution of the tokens in the training set. When constructing the alignment between 2g and 21, 50 of the initial tokens deleted and the other 50 are substituted, with the remaining tokens inserted. Finally, a Localized Edit Flow that makes use of a localized propagation process Appendix C.I, which encourages localized edits during generation.
Architecture and hyperparameters. We use 280M and 1.3B parameter variants of the Llama architecture (Grattafiori et al., 2024; Touvron et al., 2023) for all of our models and baselines. The maximum sequence length during training is set to 1024 tokens for all models. The Autogressive baseline uses causal attention, while the Mask DFM and Edit Flow models use full self-attention, including an additional token encoding the value al., 2024) to handle batches of variable lengths, allowing to not require special padding tokens and significantly increasing token efficiency during training. our experiments, Edit Flows are able ingest x more training data per iteration while using the compute and memory Mask DFM. We train all models and baselines using the same compute budget for fair comparison. We use a cubic scheduler K 1 for Edit Flows and Mask DFM, which found perform better than the linear scheduler as also observed by Gat et al. (2024). Further hyperparameter details are in Appendix E.
For Edit Flow, we use FlexAttention (Dong
HumanEval
HumanEval+
MBPP
Method
Pass @
Pass 10
Pass@1
Pass@
Pass @
Pass @ I0
Autoregressive (Gat 2024)
14.3
21.3
17.0
34.3
Llama3 Autoregressivef
17.0
34.7
14.0
28.6
25.6
45.4
Mask DFM (Car
6.7
13.4
6.7
20.6
Mask DFM (Oncle I Lengh) (Gat 2004)
11.6
18.3
13.1
Non-AR
Mask DFM*
9.1
17.6
7.9
13.4
6.2
25.0
Uniform X, + Edit Flow (Ours)
24.3
9.7
19.5
9.4
33.4
Edit Flow (Ours)
12.8
24.3
10.4
20.7
10.0
36.4
Localized Edit Flow (Ours)
14.0
22.6
18.9
34.0
Table 3: Code generation benchmarks using Llamaa 1.3B parameter models trained on the CodeLlama (Roziere al., 2023) datamix. *Superscript denotes our own implementation. We highlight the best non-autoregressive models, where colors show the best and second best among each metric.

---

Image captioning. We train on the task of image to text generation, using image captioning datasets for training and validation. Specifically, we train from scratch on the MS COCO dataset (Lin et al. 2014; CC-BY 4.0) and an image captioning dataset containing 3M image-caption pairs. Results are shown in Table I, where we also provide prior works as references that used large pretrained models. By training on the larger Image Captioning 3M dataset, our models can match the performance of these references. We see that for generation of short sequences such as captions, non-autoregressive models can be better than autoregressive models. Furthermore, we see sizeable improvement in performance from using our Edit Flow models. We attribute this improvement to the native capabilities of handling variable lengths. We see that the Localized Edit Flow performs on par but does not outperform the default Edit Flow, which is expected for short length generation. Examples of the generation process are shown in Figure 5.
Text benchmarks. For text benchmarks, we trained our models using the DCLM baseline 1.0 (Li et al. 2024; CC-BY 4.0) dataset. We show the results for common text benchmarks in Table 2. Following (Nie et al., 2025), we perform CFG during evaluation, which has multiple ways to be extended when applied to general CTMC processes. The Edit Flow model is significantly better than the Mask DFM model, but it is slightly behind Autoregressive.
Code benchmarks. For the code generation benchmarks, we used the CodeLlama datamix (Roziere et al., 2023). Results are shown in Table As additional baselines, we compare against the results reported by Gat et al. (2024), which includes an oracle where the ground truth length is provided to the model. Interestingly, we see that Edit Flows can outperform even the model with oracle length provided. We note that on such large scale data sets, the lengths of the sequence seen during training are not very informative and we need to crop sequences to a maximum length anyhow (see Figures 9,10); however, the ability of Edit Flows to generate and process using only relative positions still gives Edit Flow a superior edge. Furthermore, our Edit Flow models are competitive with the Autoregressive model reported by Gat et al. (2024), though it still falls short compared to our own implementation. An interesting result is that the Localized Edit Flow model significantly outperforms the other non-autoregressive models on MBPP, which is known to require generating long sequences of code, with a relative improvement of 48% at Pass@ I over the non-localized Edit Flow and a 138% relative improvement over Mask DFM.
Limitations
We identify two key limitations in our empirical results.
The pre-training configuration we employed favors autoregressive models. During autoregressive training, the model exposed all possible conditioning contexts within the input sequence. In contrast, we selected a random subset of the input to serve as conditioning. If this randomly chosen subset does not closely align with the evaluation scenario, the resulting learning signal may be diminished.
The text benchmarks used in study focus on likelihood estimation rather than text generation. Since non-autoregressive models lack a closed-form expression for likelihood, we relied on the best available alternative- noisy estimate of the ELBO. Although this provides reasonable approximation, these benchmarks do not directly assess the quality of the generated text produced by the models.
7 Conclusion
Edit Flows operate using position-relative edit operations and naturally support variable-length generation. By modeling sequence generation as CTMC, our approach captures expressive sequence- level transition dynamics without relying on rigid, factorized processes. Empirically, Edit Flows show consistent improvement over the mask construction across range of large scale benchmarks. In our initial results, they surpass autoregressive models in image captioning and but fall slightly behind them in text-benchmarks and code generation. However, many training pipelines and benchmarks are designed for autoregressive models, and we believe that further efforts can significantly boost performance.
10

---

References
Armen Aghajanyan, Bernie Huang. Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, al. Cm3: A causal masked multimodal model of the internet. arXiv preprint arXiv:2201.0753
2022.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34: 1 798 1-17993, 2021.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266-28279, 2022.
Andrew Campbell, William Harvey, Christian Weilbach, Valentin De Bortoli, Thomas Rainforth, and Arnaud Doucet. Trans-dimensional generative modeling via jump diffusion models. Advances in Neural Information Processing Systems, 36, 2024a.
Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications protein co-design. arXiv preprint wwXiv:2402.04997 2024b.
William Chan, Chitwan Saharia, Geoffrey Hinton, Mohammad Norouzi, and Navdeep Jaitly. Imputer: Sequence modelling via imputation and dynamic programming. In International Conference on Machine Learning, pages 1413. PMLR, 2020.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1 11315-10 11325, 2022.
Jacob Devlin, Ming-Wa Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pages 4171-4186, 2019.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending IIm context window beyond 2 million tokens. arXiv preprint arXiv:2402. 13753, 2024.
Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A programming model for generating optimized attention kernels. arXiv preprint arXiv:4412.0549% 2024.
Stefano Ermon, Aditya Grover, Volodymyr Kuleshov, and other Inception Labs employees. Introduc- ing mercury, 2025. URL https: inceptionlale wosptts..nnab......mmmmooo ai ai/introducing iaaiing-merurry.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari. Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-furst international conference on machine learning, 2024.
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. Advances in Neural Information Processing Systems, 37: 133345-133385, 2024.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel
decoding of conditional masked language models. arXiv preprint 194.094904 2019.
Daniel T Gillespie. A general method for numerically simulating the stochastic time evolution of
coupled chemical reactions. Journal of computational physics, 22(4):403-434 1976.
Daniel Gillespie. Exact stochastic simulation of coupled chemical reactions. The journal of physical chemistry, 81(25):2340-2361, 1977.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv preprint arXiv:2407.22 1783, 2024.
Jiatao Gu, Qi Liu, and Kyunghyun Cho. Insertion-based decoding with automatically inferred generation order. Transactions of the Association for Computational Linguistics, 7:661-676. 2019a.
11

---

Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. Advances in neural informa- tion processing systems, 32, 2019b.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 125988. 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851 2020.
Peter Holderrieth, Marton Havasi. Jason Yim, Neta Shaul, Itai Gat, Tommi Jaakkola, Brian Karrer, Ricky T. Q. Chen, and Yaron Lipman. Generator matching: Generative modeling with arbitrary markov processes. arXiv preprint urXiv:2410.20587 2024.
Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole. Rianne van den Berg. and Tim Salimans. Autoregressive diffusion models. In International Conference on Learning Representations, 2022. URL htw.h: httt:::://operevieww met/forrum
Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic no--attoreeeeeeessssse neural sequence modeling by iterative refinement. arXiv preprint urXiv: 2018.
Jeffrey Li, Alex Fang. Georgios Smymiss, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg. Rui Xin. Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Yu Hsich, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang. Khyathi Chandu. Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby. Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-lin In search of the next generation of training sets for language models, 2024.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision- ECCV 2014: 13th European conference, zurich, Swifzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky T. Q. Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code. arXiv preprint arXiv:2412.00624. 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26306, 2024.
Antoine Liutkus, Ondřej Cıfka, Shih- Wu, Umut Simsekli, Yi- Yi-Hsuan Yang. and Gael Richard. Relative positional encoding for transformers with linear complexity. In International Conference on Machine Learning. pages 7067-7079, PMLR, 2021.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Proceedings of the 41st International Conference on Machine Learning.
pages 32819-32844 2024.
Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. Cgmh: Constrained sentence generation by metropolli-hhastings sampling. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6834-6842, 2019.
Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv
preprint 1 1099344, 2021.
Shen Nie, Fenggi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, JUN ZHOU, Yankai Lin, RinRong Wen, and Chongxuan Li. Large language diffusion models. In ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, 2025.
Hunter Nisonoff, Junhao Xiong, Stephan Allenspach, and Jennifer Listgarten. Unlocking guidance for discrete state-space diffusion and flow models. arXiv preprint arXiv:2406.00552,, 2024.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint ar arXiv:2406.037.0 2024.
12

---

Arnaud Pannatier, Evann Courdier, and François Fleuret. o-gpts: A new approach to autoregressive models. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 143-159. Springer, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Ma. Ching Yao Chuang, David Yan, Dhruv Choudhary. Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang. Kiran Jagadeesh, Kunpeng Li, Luxin Zhang. Mannat Singh, Mary Williamson, Matt Le, Matthew Yu, Mitesh Kumar Singh, Peizhao Zhang, Peter Vajda, Quentin Duval, Rohit Girdhar, Roshan Sumbaly, Sai Saketh Ramb- hatla, Sam Tsai, Samaneh Azadi, Samyak Datta, Sanyuan Chen, Sean Bell, Sharadh Ramaswamy, Shelly Sheynin, Siddharth Bhattacharya, Simran Motwani, Tao Xu, Tianhe Li, Tingbo Hou, Wei- Ning Hsu, Xi Yin, Xiaoliang Dai, Yaniv Taigman, Yaqiao Luo, Yen-Cheng Liu, Yi-Chiao Wu, Yue Zhao, Yuval Kirstain, Zecheng He, Zijian He, Albert Pumarola, Ali Thabet, Artsiom Sanakoyeu, Arun Mallya, Baishan Guo, Boris Araya, Breena Kerr, Carleigh Wood, Ce Liu, Cen Peng. Dimitry Vengertsev, Edgar Schonfeld, Elliot Blanchard, Felix Juefei-Xu, Fraylie Nord, Jeff Liang, John Hoffmman, Jonas Kohler, Kaolin Fire. Karthik Sivakumar, Lawrence Chen, Licheng Yu, Luya Gao, Markos Georgopoulos, Rashel Moritz, Sara K. Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du Movie gen: A cast of media foundation models, 2025. URL /tttpi:///axv...oo hwwgg/abs//11 13720.
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108. 112008.11499 2021.
Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy-basee
constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems, 35:9538-9551, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763. PmL.R, 2021.
Jakob Gulddahl Rasmussen. Lecture notes: Temporal point processes and the conditional intensity function. arXiv preprint arXiv: 2018
Machel Reid, Vincent J Hellendoorn, and Graham Neubig. Diffuser: Discrete diffusion via edit-based reconstruction. arXiv preprint arXiv:2210. 16886, 2022.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:13011 101113311100000 2024.
Lei Sha. Gradient-guide guided unsupervised lexically constrained text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2020.
Neta Shaul, Ricky TQ Chen, Maximilian Nickel, Matthew Le, and Yaron Lipman. On kinetic optimal probability paths for generative models. In International Conference on Machine Learning, pages 30883-30901 PMLR, 2023.
Neta Shaul, Itai Gat, Marton Havasi, Daniel Severo, Anuroop Sriram, Peter Holderrieth, Brian Karrer, Yaron Lipman, and Ricky T. Q. Chen. Flow matching with general discrete paths: A kinetic-optimal perspective. arXiv preprint arXiv:2412.03487 2024.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37: 103131-103167, 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning, pages 2256-2265, pmlr, 2015.
13

---

Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion transformer: Flexible sequence generation via insertion operations. International Conference on Machine Learning. pages 5976-5985. PMLR, 2019.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing. 568: 127063, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:2307.09288 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Iilia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Apoorv Vyas, Bowen Shi, Matthew Le. Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, Jeff Wang. Ivan Cruz. Bapi Akula, Akinniyi Akinyemi, Brian Ellis, Rashel Moritz, Yael Yungster, Alice Rakotoarison, Liang Tan, Chris Summers, Carleigh Wood, Joshua Lane, Mary Williamson, and Wei-Ning Hsu, Audiobox: Unified audio generation with natural language prompts, 2023. URL htttp:://akkaiii http://aaaii..o..a/////aa org/abs/2311.. 15821.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. In The Eleventh International Conference on Learning Representations.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Jiacheng Ye. Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang. Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025. URL https: / /twwnlkkkpppp//p/ gi thub. httpooo///2///////////// dremm.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yo Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409. 02908, 2024.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 13041-13049, 2020.
14

---

A
Qualitative Examples
Input Image
Edit Flow caption generation process
dog
dog-
dog television.
dog television.
doges an animal television.
A doges of an animal television.
A brown and doges of an animal a television.
A brown and dog watches of an animal a television.
A brown and white dog watches of an on a television.
A brown and white dog watches image of animal on television.
black
black hat
aring black hat dogaring black hat head. dogaring black hat on head.
A dogaring black hat on top head.
A dog wearing black hat on top head.
A white dog wearing black hat top of head.
A white dog wearing a black hat on top of head.
A small white dog wearing black hat on top of head.
tree
tree a a tree a of tree a close of on tree close up of birds on a tree branch close up of birds on a tree branch pot
close up of birds tree branch with pot
over
over.
lies over
lies over a street.
Anlies over a street.
Anlies over a street cars.
Anlies street with cars.
An flies over street with cars.
An air flies over street with cars.
An airplane flies over street with cars.
An airplane flies over a street with cars.
Figure 5: Example input images and the stochastic sequential generation of captions from an Edit
Flows model.
15

---

Generated tokens:
def truncate_mit truneate_nuuber(nnnnne nunber(number:: float) -> float:
""" Given a positive floating point number, it can be decomposed into
and integer part (largest integer smaller than given number) and decimals (leftover part always smaller than 1).
Return the decimal part of the number. >>> truncate (3.5
0.5
return number- int(number 0.0)
from typing import List, Tuple
def sum product(numbers: List[int]) -> Tuple[int, int]:
For a given list of integers, return tuple consisting of sum and product of all the integers in list. Empty sum should be equal to 0 and empty product should be equal to 1.
>>> sum product([)
(0, I)
(10, 24)
0
product
not numbers:
return sum, product
for index, number (numerattemumbers)::
sum += index
product *= number
return sum, product
def sum product empty(numbers: Lis((int] >> Tuple[int, int]:
>00 product([)
(0, 1)
sum_ www_suummmeeessssss product([1,
(10, 24)
name main !:
print( summ product) print(sum, fato(ssu__prouuuute prroduct_empt(((6)
def' string sequence(n: int) sir:
Return string containing space-delimited numbers starting from 0 upto inclusive. sequencee))))
'0'
>>> sequencse)))))
2112355
numbers []
add numbers to sequence
numbers += [str(value) value rango(n+1)) +1]
space delimited
return str(" join(numbers))
Figure 6: Edit Flow generation examples with X, / (i.e. insert-only model) without divergence- free component. 100 sampling steps. The function signature and the docstring serve as prompts.
16

---

Initial State (Step 0):
def is prime (n: int) bool: if n <= 1: return True
for in range (2, n): if % 0:
return True return False
Intermediate State (Step 150):
def is s9_prime(ns (n: int) -> bool: if <= 1: return False
for in range n): if. % n == 0:
return False
True
6
Final State (Step 300):
def is _s_prime (n: int) -> bool: if <= 1: return False
for in range (2, n):
if n % == 0:
return False return True
Figure 7: Edit Flow error correction example for correcting the is is_prime function. The initial implementation is incorrect, because the three return statement are negated. The model starts from an incorrect implementation, makes 117 edits over 300 steps, and reaches the correct final state. Note that intermediate states may contain extra tokens that later deleted.
B Theorems and proofs
Theorem 3.1 (Flow Matching with Auxiliary Processes). Let uz 2/2,, 2)) a rate over the augmented space of X Z that generates Pe(x, 2), then
( Σ. (1, 2x)
generates
Pr(x) - E. (17)
and furthermore, for any Bregman divergence Dø(a, b) = ó(a) - o(b) - (a - b, dbo(b)) defined by convex function 6, we have that
d E. Eenarpo Do E. 2/1 2x), uf (1(())))) = do Brrnpi Do (ue(-lox), (-(o))) (18) do
Proof. For the first part of the theorem (17), since ue( z|w,, 2.) generates pe(x,, 2), they satisfy the Kolmogorov forward equation
Pr (z,
ΣΣ 2
pe 2/
17

---

then we can show (z) and pt (z) also satisfy the Kolmogorov forward equation
① on p((x)
Σ
0
Pc(a,
ΣEΣ
Pe 24 21), 21)
0t
Σ
ue 2/ 21)
(e 2x) (x()
pe (x()
21
Σ (el
Additionally, (2, 2 2 2x) satisfies the rate conditions by assumption. Assume (r > 0. Then
Ex ue(x mx) Ex, EE, Ex ut (z, 2|zr, (p) z() > 0)) p((e)))) O. Further, u. (x)(2))) 2
pe(2x)
when x A xy and pa (xx) > 0 because un (x, a|2a,, =) 2 0 when (xr + (xx,, ze) and (r =4) > 0.
Terms with pr (aee =x) 0 do not contribute in the sum. So u, (1rl2r)) satisfies the rate conditions.
For the second part of the theorem (18), note that
d
ΣΣ ue 2 2/12, 22) du
ouu (z|x1))
(e 21)
Pe (3e )u (2, =|1 2,) du
(l ()))
(e(
1
2
= Σ Pt (x(
olu (z|z1))
du
d (z(2)) du
Exppp(() Σ
s(u bluf (+|=))
then we directly prove the result
E (e, Do
Σ ue 2,,,, zx), (1((11z))
d0
d Exnee [0 vel, 2/2,, -
- E. ul. 26) - (1(xe) a ((
d0
d E Exneemp((a))) (+(o)) - Ex E. u 2 1,, 2
+ E. u ( dhe (x)+))
d0
d Ex (x) (o(u((=2)) )(2((122)) E. (
Ex ul (x| du (1
d0
d
d0 Be) (lu.(1Izz)) ((u(()lae)) (1.(z));)) ul (c|oz)) du sl (x (x<(z)))))
d De ((()))r))
d0
To apply theorem 1. we require a rate us uz(2, 2|w4, 2t) in the augmented space of 2 that generates Pc(x,22 2). The following lemma can simplify this construction.
Lemma B.1 (Rates that generate pt (x) p(x))))p(=)). Let pe(x,,2 2) be a distribution over augmented space of X x Z where (x(2)2) = p(x|2) is time-independent. Let ue uz(2(2)) be a rate over Z that generates (z). Then
(1 2 (T 21) = (1 - -8, ()))((())) (x. (2)
(24)
is rate over augmented space of X x Z that generates px(x,2)
18

---

Proof. We first check rate conditions (2) for u1 2|,,,
21). When (x) o (æ,, zr) and Pe( 2x) >
uz(2, 2 2| 2x) = (1 6z. (2))) 2
because Pe( > Then
Σ u1 2| 21)
= Σ (1 - 82, (3) + de, (z)) (2))6,)))
2,3
2,3
Σ (e(212e) uc + (z)l2r)
Σ (x(())
where the last equality uses that (z(2)2)) is rate over Z and again Pe( 24) > 0 = Pr( > 0.
Now show ue ue(z, 2|2x, ze) also satisfies the Kolmogorov forward equation (3) for pc 2) which proves the result
Σ
(, 2/ 24 (xt,,
Σ
(1-8xx 82, (2))p(())))) + Ss, (x)6 (2) (z) 20) p (1t
Deite
p(alz)
(n<=l=)p (=<22)))(((z) ( 2) (=(=)))p((()
a
= =((a]) oe P((2)
((,
When the relationship between given auxiliary is not only time-independent, but also deterministic, this Lemma B.1 leads to the following Lemma stated inline in the main text
Lemma B.2 (Rates that generate pe pe(x, 2) 55(²) (z)) Let pc(x, = (x))(=))p(()) be distribution over augmented space of X Z where po(2(2) (x)=)(x) is time- iine-independent and deterministic. Let ux(2/z)) be a rate over 2 that generates (z))). Then
u1(2, 2 F1 21) = f((z) (1),)())((()))) (2(2g)
(25)
is rate over augmented space of X Z that generates pe (xx,)) 2).
Proof. From Lemma B. the rate in equation (24) generates this pr(z, 2) using uz(2)2). Because only this rate when pe (Zt, 2)) > O, this rate will always be evaluated at ae f(z) giving ur z, [zx), xx) (1 82, (()))) =)) + (2) (-5 + ((()(())) u1(2(21)
+ 5( (x)6 (z)ur(z,
B.1 A Bregman divergence as the training loss for Edit Flows
Given velocities u,(:, =lae, =e) and ug (-(oz) that satisfy the rate conditions, we define
slue (-(2))) = Σ ue (x(x)x)) log (x)p)))))
(26)
The Bregman divergence corresponding to this Φ is:
D(((()) ((f(-Iz))) - o(g(-(=z)))
Σ (f((l)))) + Ing((/a)))
atxi
Σ (-((o)a))) - f()(ae) g(0((22)) + s(alze)) 232
(27)
f(x(21)
19

---

Therefore the training loss for Edit Flows with this øi can be written
(()) = (2o. (o0. Do
Σ 211,, ul (-1222) (-
E (ap
(z 2/ Ze, 20, 21) + Σ ue (x)(2)))
ult (z) (01)
E saarghor
ue(2, 2/20, Ze, 20, 21 log Ue (1 2 2z, 2o, 21)
E, (200)))
ul (2(20/11)) + +E Σ
(2,22//01 2e, 20, 21) log ut (2/x²)
+ const.
2
(28)
C Advanced techniques for Edit Flows
Sampling. Sampling from the model requires transporting source sample Xo ~ to time 1, simulating the CTMC defined with the learned rate u/. Exact simulation (Gillespie, 1976, 1977) is intractable as it requires integration of uf. With the Edit Flow parameterization (13)-(15), the exact probability an edit operation characterized by the rate Aτ,5 occurring within an interval (t, 8 + h) is
1 1( de. (X.)dd (X.).
(29)
Following previous works (Camphell et al., 2022; Gat et al., 2024), we leverage the first-order approximation. Sampling thus iterates the following procedure: with current state X, and step size h, independently determine the probability of each insertion, deletion and substitution, then perform all edit operations simultaneously.
For each position i, sample whether to insert with probability (X,, and whether to delete or substitute with probability (Xx, + (X,)). Since deletions and substitu- tions the same position are exclusive, if either occurs, select deletion with probability Adel (X.)/ (X.) + X (X))), otherwise substitution.
2. If insertion substitution at i, sample the new token value from Q (-1X.).
3. t < t ++ h
Classifier-free guidance. We considered three approaches to add classifier freeee guidance to Edit
Flows. Classifier-fre guidance (CFG) considers training model with and without conditioning c and combining those two models at sampling time using weighting hyperparameter w.
Our first approach is weighted rate CFG which follows Nisonoff et al. 2024 and uses (for x + xg and within one edit operation)
u (x c) - (aaal)))--uutaa e) )e, c)..(a with ( (a,(( c) & (alx,) "0, c)
(30)
(2, Σ. " (a,
(31)
where Ar,s and Qt, are for specific edit operation taking it + x.
Our second fixed rate CFG which uses ūz(x] c) A Ae(are ())r(alz.)))
Our third approach is naive rate CFG which uses T,(2)),,)) c) & Xe, (alze, c) where (e( =
(x,
Note that these CFG methods only differ in how the modified \ is constructed, impacting the probability of an edit operation. For all of our benchmarks, the naive rate CFG consistently performed the best, with fixed rate CFG very close in performance; however, the weighted rate CFG was consistently worse than either options. When CFG is applied in conjunction with reverse rates, we applied CFG to both the forward and reverse rates.
20

---

Reverse rates. A CTMC Markov process can also be defined via reverse time simulation from t 1of 0 using rates Ut
P(X-A x/ X2 2x) 6z, (x) hūg (2r(2a)) o(h)
(32)
where o(h) satisfies - h except that the transition is from to t h instead of to + h. Like (1), in order for (32) to define a valid probability distribution, reverse rates üt must obey the rate conditions in (2).
((h) 0. This equation is identical to forward-time simulation (1)
rate Un "generates" a probability path Pt. if the time marginals of the associated reverse-time simulation are samples from Pt, ie., Xt. Pt. Concretely, they should satisfy the Kolmogorov forward equation reverse (i.e., with minus sign)
① p(()) Σ (m(
Σ (.(clu)p:()) y
Σ (x()))))aa))) u
(33)
reverse flow into
reverse flow out of
We construct ūt that generates pt (and in fact is CTMC with the same joint distribution) from Ue that generates Pr via the following procedure. Assume generates Pt. For x & x'. consider that the probability flux from r in forward time towards x' equals the probability flux from x' to æ in reverse time as follows
Ty (x(p))))px(())
(34)
reverse Blx iitto x Inserting into the Kolmogorov forward equation satisfied by un
( ufr
(x) Σ ua(aly)pe()) - Σ (1(l)))p(())
(,lx))p)
Σ (<(=)))p<())
Σ (x(l))))(v).
(35)
so ūt generates Pt-
Now consider ur + i. which satisfies (2) and probabbilty-preserving such that En, (x|xa))
Te (xx) = O. If we perform forward simulation with this rate using (1) starting from Pt (x) and sampling z', we maintain that x' ~ (z).. This allows corrector steps that can correct errors in the marginal distribution via repeatedly applying such a step without updating time.
We also have that (1+ a-u, at,, for a 2 generates pt in forward time. This combination rate can be simulated via stepping forward from 2; to fi--((1aa) using it and then backwards to Ze-h using --)A(10)- To see this is equivalent for small h, let = T2+A(1+0) and consider the distribution of -e-h after the combination of these steps
Σ (Zeth)) + hoxu+ + o(h)) (1) h(1 + a)ue(vl + o(h))
= (2r4)) h (aū + a) + o(h). (36)
C.1 Localized propagation paths
Edit Flows leverage an underlying conditional probability path Pr(2]20, 21) and associated rates (z(2,),, so far given by the factorized token-wise mixture. Let us further generalize this probability path and associated rate to non-factorized, applying auxiliary variables again. We first re-express this probability path through auxiliary boolean variable E (false, true)
(= 21) = Σ pe(mmaa z0, 21 (+ (z|m, 20. 21),
(37)
where Pr(2/m 20, 21)
II
1 [-my (-myo 0gg(2) + [mm] [mm))02g (2*),
(38)
1=1
21

---

t=0
C
t=1
Figure 8: Illustration of the localized generation path. Tokens that are neighboring to existing tokens have a much higher likelihood of getting inserted next. Tokens that are not next already inserted tokens (e.g. token I) have small, but non-zero likelihood of getting inserted in the current timestep.
where [j] is the indicator funetion and returns one if the input is true, zero otherwise. That is, m indicates whether 2" is equal zó z1. the case of Pe 21) being be a factorized distribution, this would recover the factorized probability path (8).
Pe(m| 20; 21) II [m co: 21),
Pe(mm) 20: 21) 1 [-m (1 (6)) + 11 [mi]ie
(39)
This helps ensure that the conditional rates can be constructed easily. However, this could be prob- lematic for Edit Flows when the sequence length becomes large, noisy sequences F, will consist of non-neighboring tokens. Instead, we will propose a non-factorized locality-based construction in which if m true, incites nearby values (m3-1 and m m/+1) 1) to transition their value true, thereby encouraging nearby neighbors to have similar values.
Let consider an extended space of boolean variables denoted by M E {true, false) NxN and consider N independent CTMC processes, starting at all values being false. For each row M', we create a process where a first switches to true according to time-dependent rate A indep and this then propagates to neighboring values according propagation rate Aprop, This can be concisely expressed as the following CTMC process for each M
We uM(M MM)
indep 8ij [M VM. VM propp)
(I Ap -M; (M+))),
M false,
(40)
where dip = 1 if j and d.j = otherwise. Breaking this down, X indep is an independent rate for switching M to true regardless of the value of M, at other positions-it we only have this independent part, then this formulation will be equivalent to the factorized case-and doop is the rate for the off-diagonals Mij if a neighbor is true, responsible for propagating along local neighborhoods-foo simplicity, this part time independent. We then map from this extended space to the space of m by the mapping:
m = M.
M V ... M.N.J
(41)
That is, m² true if any value in the column of M; is true.
Augmented rate. We now have rate u, u,(M] (M]M,, 20. 21) that generates Pt (M]20, 21) and can apply Lemma B.2 twice determine rate (e(2, m, M|2,, m,, M,, 20. 21) that generates Pi (z, m, M| 21). The target summed rate we need for training a localized path model (where we consider z as observed and (m, M) as auxiliary) is for + 2t
Σ m.M
uz(2, m, M|24, m., Me. 20, 21)
(42)
032 (x))
Simlep + Ee I -M
if = and 7, differ only in j-th token otherwise
(43)
To utilize specifically for localized edit flows, we extend our rates again to generate Pe 2, M|20, 21) and the rate needed for training localized edit flows, prior to the sum over
22

---

additional auxiliary 2, is simply (42) multiplied by ō fum (z) Following the same steps as before, the edit flow training loss using localized rates is:
N
C(0, A) -1,,
ul (xsl(()))
#+] 1,1 log u (21(2,, 1), xx)
(20.2))
(44)
where ur(2s, 21,8) E; |M
Jim-blanks (2) , i-1 _+11
12 1
and
A
indep
Parameterization. For indep we can reuse the same form from the factorized case defined by scheduler Ki,
indop
R4 Kig
Ki O,
(45)
which allows us ensure that m] = true for all i and whose integral can be obtained easily. For APOO, we choose an appropriate constant, the value of which corresponds to the expected number of propagations within unit interval of time.
Sampling. In order to allow efficient training, we need to sample (m, M.) for a given f without simulating the CTMC (40). The construction of (40) is designed explicitly to allow efficient sampling. Since the CTMC processes are independent for each M', we can simulate them independently. Furthermore, for every Mi- the source of the propagation use of the following 2-step sampling algorithm given t:
only be from M Thus,
make
1. For each i, independently sample the time 1 [0, that each M would switch to true based on the independent rate X indep If t I t, then M;" is set to true.
2. For each i such that t s t. sample the number of neighbors the left and right that switched to true due to propagation with rate Arru from M during the time interval [t]
Afterwards,
set m M v M v... M
Step of this sampling algorithm requires determining the time of the switch 6* This is equivalent to finding the occurrence time of an inhomogeneous Poisson process with intensity function indop, This can be done via the inverse method (Rasmussen, 2018) follows.
1. Sample u ~ Uni 1)
2. Compute t" s.t. u = exp(- - for X
indep dt) dtd
For the parameterization in (45), we can analytically derive this.
-" (u)
where ~~ Unif(()
(46)
Step 2 of the sampling algorithm consists of determining how many neighbors get propagated from each source a within certain time interval [4) Since neighbors the same side only get propagated sequentially, this is equivalent to determining the number of occurrences from homogencous Poisson process with intensity drrae The formula for this is
N; Pois(:: A (pp),t,), where At, =t
(47)
We would sample variables i.i.d. N and N for the number of neighbors propagated to the left and to the right of ' respectively. The logic for M. can be concisely expressed
s = M, [(U < <())
(Ú ( ()
(48)
All computations within each step of the sampling algorithm can be completely parallelized, resulting in fast sampling of m
23

---

HumanEval pass@10
20
HumanEval+ @
19
18
10
17
0.00
0.25
0.50
0.75
1.00
1.25
10
102
10a
CFG scale
Sampling steps
Figure 11: Effect of CFG scale sampling time on the code generation benchmark using the 1.3B parameter Edit Flow model.
Figure 12: Effect of the number of sampling steps on the code generation benchmark using the 1.3B parameter Edit Flow model.
Training data analysis
DCLM Baseline 1.0
CodeLlama
Beginning sequence
Beginning sequence
Portion traliinng dala
0.4
Continuation of cropped sequence
Continuation of crapped sequence
0.3
ortaainnns
0.2
0.2
0.1
0.0
0.0
1-255
256-511 512-767 Sequence length
768 768-1023
1024
1-255
256-511 512-767 768-1023 Sequence length
1024
Figure 9: 54% of the training data consists of sequences length 1024 and 57% of these are self contained sequences (meaning that they start with a <BOS> token and have < 1024 tokens in total).
Figure 10: 50% of the training data consists of sequences of length < 1024 and 72% of these are self contained sequences (meaning that they start with a <BOS> token and have < 1024 tokens in total).
E Further experimental details
Training: All models were trained 500,000 steps with batch size of 4096 distributed across 16 x 8 H100 GPUs, which resulted in 2T tokens used for the Autoregressive and Mask DFM models. Since the Edit models do not use compute for tokens that are missing from the sequence, they are considerably more compute efficient. They were able ingest 6T tokens during the same 500,000 training steps.
Architecture:
Table shows the details of the architecture and optimizer used in experiments.
Conditioning: A beginning of each sequence in the training set is designated to be conditioning. The portion of the sequence used conditioning is randomly chosen to be d where U 1). For 10% of the sequences, we drop the conditioning to allow for unconditional prediction and CFG scaling inference time.
Image conditioning: To condition our model on an image input, we follow Liu al. (2024) and early fusion approach of appending image embeddings prompts our sequence models.

---

Hyperparameter
280M configuration
1.3B configuration
Vocabulary size
32k
32k
Model dimension
1024
2048
Conditioning dimension
32
64
Number of layers
12
16
Number of heads
16
32
Number of KV heads
8
8
Feed-forward dimension
1740
3072
hidden dimension
6963
12288
Training steps
500k
500k
Batch size
4096
4096
Optimizer
AdamW
Adam W
Learning rate
3e-4
3e-4
Beta
0.9
0.9
Beta
0.95
0.95
Warmup steps
2000
2000
Learning rate schedule
cosine
cosine
Table 4: Details of the Llama3 architecture and optimizer used in our experiments. Conditioning dimension is used in the text and code experiments: it denotes the dimensionality of the embedding carrying the binary signal whether a given token is part of the conditioning or not.
Sampler lbpprrprrameerr
Astongroseive
Mask DFM
Edit Flex
Uniform X,, + Edit Flow
Pasei
Passii
Paesti
Prsse 10
Pass
6ass6:10
Pase 10
Sumpling sieps
10030
Clussilies guidance
Temperature
0.0
1.0
08
0.8t
1,01 f)
0.8t 1.00L
Divergence free component
- 1/5
10 1/A3
00 - eya
150e4
10
0000222111 - 1y5.ss
10,0e (100))011-))))))))0 -tj
Topp
00
0.7
0.5
0.3
0,7
0.9
Top
Revens
Revene tempenttre
0.5
0.
Resenstopp top
0.8
08
Table 5: Sampling parameters used in the code experiments. The parameters were tuned by running random search (N=200 runs for pass@I and N=20 runs for pass @ 10) on the HumanEval benchmark. The HumanEval results were then re-computed using a new random seed to avoid evaluation set leakage.
We use frozen CLIP embeddings (Radford al., 2021) for computing image embeddings and then map it to the same dimension as the sequence model with a I-layer MLP projector.
Sampling: For the pass @ and @ 10 benchmarks, tuned the sampling parameters (tempera- ture, top_p. sampling steps, CFG, divergence-free component) for each model separately with the goal of maximizing performance. Figures 11 and 12 show the impact of CFG scale and the number of sampling steps on generation quality. Table shows the sampling parameters used for evaluation in the code benchmarks.
Mask DFM: The Mask DFM bascline is trained using the ELBO objective (Shaul et al., 2024) in the image captioning experiments and using the cross-entropy objective (Gat et al., 2024) in the code and text experiments. Training data that does not meet the sequence length 1024 used by the model is padded using padding token. This padding token, if generated by the model, is removed at inference time.
Text benchmarks: Table 6 shows the CFG scales tuned for the text benchmarks.
25

---

Method
HellaSwag
ARC-E
ARC-C
PIQA
OBQA
WinoGrande
Mask DFM
0.0
0.5
0.0
0.5
0.0
0.0
Edit Flow
1.0
0.5
0.5
0.5
1.0
0.5
Table 6: CFG scales used in the text benchmarks. We only tuned CFG scale: we swept the values 0.0,
0.5, 1.0, 2.0, 5.0 and 10.0 every benchmark and report the best results.
F
Model preference for minimal edits
Similar to continuous flow matching, the generated coupling p1 (Z1 may differ from the coupling used during training, denoted as ( [z0). The model learns coupling that involves fewer edits than the average observed training. To illustrate this, we applied edit flows to toy dataset that includes only insert and delete operations, with substitutions. The distributions of π x and π (m) are both uniform over strings of length containing only the characters A and (as shown in Figure 14). The probability path is defined such that every character in .o gets deleted and every character in gets inserted (least optimal alignment). The coupling at training time is uniform.
However, the model does not retain the uniform coupling from training. Figure 14 demonstrates that prioritizes To, 21 pairings that require the fewest edits. For example, Zi -AAA is 20x more likely to generate
AAAA (requiring no edits) than 21 = BBBB (requiring 4 insertions and 4 deletions). Generally, the cells with the highest values of P1 (z1 |2b) correspond to pairings that require only few edits, while the lowest values correspond to pairings that require many edits.
AAAA
409 GES
KiM
0.175
AAEA
AADB
ABAA
0.325
ABDA
5:4
A888
1/6
BAAA
0.075
BABA
308
BABB
528
0.000
BBAA
0.025
BBBA
A0
ouo Air
8888
028
AMM-
AAIA
AABB
ABAA-
ABBA
BAAA-
BAAB
BABA-
BA88
BBM
BBAB
8BBA-
AAAA
AA-8-
AABA
AA88
ABMA
ABMB
AUIAA-
KBE8
BAM
BAAD
6404
BBM
8888
Figure 14: Comparison of the training time coupling (π (zo)) with the coupling learned by the
edit flow (f (a) (zo)). The model prioritizes pairings that require few edits.
26

---

der pet_a(ide: lint(int)) > (1tet [int]tnt]]:
nun_substitution cleenoncios) torget..nnsssue www.sosstttttiin nunddelstions target_aus, deee_.o..elleierr target_nuw, wwrppe.ww..sssttsstte.... wwb.stttttttonss www.substitutions
x.0 x.0 ra-randon. ( highavocab. size, 2scenBue, decatiora nUR www.ssattttiosss.tlllllloo. sub. id +
2
[epe:lon___1d] (lantide) mun, mibessisstttio
[Subb.d]] -ww.sobbticcttooss
stuffle(s)
List[inc]
List[int]
aca_index
3_0_ineex
for token in if token 1ptt
T_B. DEs____s__E_____ooooo
2. (ec.annccossooon.
1.8_irdex +o
epsilon_0.0
append
append index])
frdex ida irdex
rature 2.8,
def (1_0 list[int), 1.1: lsst(iet), kappa: flost) >> list[int]:
reture
token_a if rp..radoo undforw() kapps else token.l for toker_e; tokes_] 2ip 2_1)
Training loop
training_sspples
(xarple,
7L8 [hos_1s] -1.
[hhs__s]]
float np..rindos. uniooo).
kappa: fleat
Using linear schedule
scapps: flast 1.8
2.1: list[int]] jet_ 2.1, keppa]
1:4: list(int] renove sps:ion(2_5) +.t_tersor: forch, Tensor torch toddet)
pass
irsert_landda insert.q, delete_.arbde, sobstituro_larbdb substitute.e rodelc x_t_tensor,
caleilate lose
losx_tere_1: Jarch.Teneen torch. sse(
insert_lanbda derete_ siaetitute_labbd
Tess_term_2: torch, Tensor a torch, tessor (8.8, devitendevice)
Intei int -1 Corresoonding indea in 1.t. for token 1, tosen in ziplz i, 1.:
If token.t epsiiin, 8. and token.t Ie epsilen,1.id: srt_irdex
epasion, token.1 epaslen___.dd:
(eappa kappa])
insert lonbdaCx.
elfr tokerat I- opsilon_e_id
eps:lon.1.id:
Extra taken muxt coleted
loss tere. less,cerm. 2.- (ckappa kappe]) torch, Log(
aist
token.t I- epsiion. 8. and token_1 is cosilet-l-fe
taken_t taken_]
Incorrect must aubatituted wabeittre_lamaattt...oo
kapps)) Sessstatt_aD_t_indeee
Tess: terch Tessor Toss_tere_l Tossutern_2 ostimizer grad() less. (ad
optiatzer, (tep
Figure 13: Simplified training code for Edit Flows. The helper functions get_z 2 and get_z_ noisy and target token sequences, while the training loop computes the loss and updates the model parameters. For brevity, we did not include features such as batching, conditioning on a random portion of the sequence and scaling the model outputs by the rate.
f_____t generate
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our theoretical and empirical results support the itemized claims made in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We highlight the limitations of our approach dedicated section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
28

---

Justification:
Our theoretical results are rigorously stated and proved in the supplementary
material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The experimental results section combined with the experimental details and code in the supplementary material give sufficient details to reproduce our experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
29

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: All the data used is already open-source. Regarding the source-code, are not able to publish it at this time due to our organization's 'ss policy. We hope overcome the administrative challenges and publish our code in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co public/ htte://uddeeeeeeeuiuue
ss ssionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips.
coiiiiissooooiiyyyoc for more details. Btlc//guide///oeeiiiiee
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provide detailed explanations in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We support our claims with a range of experimental results, but the error bars are not commonly reported for these benchmarks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
30

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide these details in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:///eeriipseeeee http://ppp....clllctctcccccooooa ines?
Answer:(Yer Yes]
Justification:
We read and comply with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work does not have novel socictal impact beyond that of already existing generative models.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
31

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our data open source and we do not plan on releasing our models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: This work does not re-use code models from third parties. Data is properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
32

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or human experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing or human experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
33

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This research does not involve LLMs any important, original, or non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
for Please refer our LLM policy http:///eerrpiscccooeeeeeeeeeeee what should or should not be described.
34


Reviews:
[{"review_id": "2tybLCsz5h", "reviewer": "Reviewer_arpi", "summary": "Edit Flows proposes a discrete flow matching approach to sequence modeling using edit operations which allows for variable length processing with the flow matching framework, rather than the standard \"blockwise with padding\" approach generally taken. This increases training efficiency, and leads to numerous downstream benefits, as well as operating in a highly interpretable space of actions. Applied to image captioning, text generation, and code generation, Edit Flows shows competitive performance to relevant baselines, aligning with the intuitions and mathematical derivations, which justify the approach as well as numerous variants of interest.", "questions": "Will the authors release code, or pseudo-code for the key methods of this work, in order to facilitate reproduction in open source?\n\nIn Table 6, what were the CFG values tuned on? Train, train and validation, or validation only?\n\nOn figure 7, do the authors have any comparison of parallelization versus sampling steps to fit a given computational budget, similar to \"Large Language Monkeys: Scaling Inference Compute with Repeated Sampling\"? For example, is it better to do pass @64, with smaller steps, or pass @10 with more steps?\n\nSimilar to the above, reverse rates were discussed in the paper, but I did not see any concrete study of result quality over compute with reverse rate enabled, versus more parallelism or increased number of steps - do the authors have any results on this?\n\nOne concern I have on this approach is vocabulary scaling beyond 32k. Though this method should scale moderately well, do the authors have any empirical or theoretical insights into how the compute cost grows as the vocabulary increases, compared to (say) standard Transformer models? Ideally this would be similar to Transformer scaling, and allow for larger scale experiments with sufficient large scale compute to train up to the 20-70B range, with larger vocabulary sizes to match common models in that range.\n\nDo the authors have any qualitative examples of injecting invalid text (false facts, code vulnerability, typos, etc) during the sampling chain, and seeing if the model returns to edit and remove these? The space of human-in-the-sampling-chain seems possible given the model setup, depending on this behavior. Similarly, given it should be possible to mark a subsequence \"not for editing\" - does having some subsequence marked fixed change the following sampling behavior?\n\nAny intuition on CFG on rates versus CFG on Bregman divergences and when to prefer one or the other?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\nEdit Flows thoroughly introduces the background, key motivation, and driving theory behind the approach. It covers many relevant background papers, and a number of variations on Edit Flow methods for ablating key mechanisms. Figure 8 is very nice, and would make a compelling video visualization. Overall, the paper was an enjoyable read, and had an excellent treatment of the material, background, and various application areas, along with a detailed appendix.\n\nWeaknesses:\nIt would be nice to see some larger and more recent image captioning models, to compare best-on-best performance (though this may need to be independent of model size). As it stands, the qualitative examples in the appendix are compelling.\n\nThe \"Autoregressive\" comparison in the tables is a bit too generic, providing a relevant citation of which model, if it was self-trained or from another paper, and so on. The citations in section 5 indicate it is a Transformer, but what flavor? Being detailed in the tables will help contextualize the results. Reusing the same superscript, to mean different things (\"not comparable\" in one table, and \"our own implementation\" in another) is also something I would change.\n\nGenerally, as always, more comparisons would help, and especially pre-trained comparisons would be useful given the recent uptick in publicly available mask-based generators. There are a huge amount of comparisons that could be used in Table 1 or Table 2 even if some need qualification (different training set, different model size, and so on), and some qualitative study of failure cases could be really useful/\n\nThe efficiency argument in appendix D is a bit of a drawback in disguise - what do the baselines look like with equivalent token budgets (even if it might take 3x more steps, and more compute)? This may not be feasible, but some discussion about this point is worthwhile given the importance of training, overtraining, and matching token budgets in general. Alternatively, reporting Edit Flow performance at 2T tokens could highlight the same. Being more efficient in terms of tokens-per-step is definitely a benefit, but being careful in the competitive comparison is important.\n\nThe mathematical treatment here is dense, in line with other work in this area, but some condensing of the mathematical bits to a relevant section, wrapped by higher level descriptions of the key ideas *behind* the formulations, could help readers who are not familiar with the flow matching literature and notations get to the meat of the contribution.", "was_revised": true, "final_justification": "The updated rebuttal and discussion really highlighted the strengths of the method and expanded the impact of the work. I would rate this paper \"strong accept\" in general, without hesitation given the updates. Accounting for scaling limitations, the rest is very thorough and well done - the response to every reviewer has greatly enhanced the understanding and clarity of the work. If I saw this paper in *any* major conference, or even as a spotlight talk, it would make sense to me given the additional focus on results, scaling, and integration into some larger experimentation of interest to many researchers.", "rebuttal": "Thank you very much for your thorough and insightful review of our submission. We appreciate your detailed feedback and the time you took to engage with our work. Below, we address each of your questions and concerns in turn.\n\n> It would be nice to see some larger and more recent image captioning models, to compare best-on-best performance (though this may need to be independent of model size). As it stands, the qualitative examples in the appendix are compelling.\n\nWe note that the primary value of all of our experiments is to analyze the ability of the models on their own when trained from scratch, without additional data or pretrained weights. This allows us to examine model performance differences without confounding variables such as varying training data. Most papers that report image captioning use pretrained weights (e.g. VLP and ClipCap which we included for reference). \n\nHowever, to satisfy our and the reviewer's curiosity, we have conducted an additional experiment where we initialize the weights using a Llama 3 1B model, as a means to test how well Edit Flows can adopt the weights of a pretrained AR model trained on internet-scale data. This increased our CIDEr score on MSCOCO from 108 to 124, which now outperforms both VLP and ClipCap (a 1.5B GPT-2 model). Going further than this would require additional ideas from the literature, such as data synthesis methods like recaptioning, which are orthogonal to the modeling framework and is out of scope for us.\n\n> The \"Autoregressive\" comparison in the tables is a bit too generic, providing a relevant citation of which model, if it was self-trained or from another paper, and so on. The citations in section 5 indicate it is a Transformer, but what flavor? Being detailed in the tables will help contextualize the results. \n\nThank you for pointing this out. We use the 280M and 1.3B parameter variants of the Llama3 architecture based on the official Llama3 repository. We updated the tables in the manuscript to denote this clearly. Further architecture and training details are included in appendix D.\n\n> Reusing the same superscript, to mean different things (\"not comparable\" in one table, and \"our own implementation\" in another) is also something I would change.\n\nThank you for raising this issue, as we did not notice this. We updated the superscripts to be consistent across all tables.\n\n> There are a huge amount of comparisons that could be used in Table 1 or Table 2 even if some need qualification (different training set, different model size, and so on), and some qualitative study of failure cases could be really useful/\n\nWe understand the reviewer's desire to scale up, as we would like to as well. However, we emphasize that the value of our experiments is in having a concrete apples-to-apples comparison on the same data and compute scale. This is crucial for understanding design choices, and it is very difficult to understand the tradeoffs of different modeling approaches when there are confounding variables such as data and archiecture differences. We will investigate scaling laws in a separate work, which will provide clearer insights for comparing across different model sizes than a tabular comparison.\n\nFor a qualitative comparison, we generated outputs from the Mask DFM and Edit Flow models on the code benchmarks and we are includin them in the appendix. Similar to prior work, we find that when mask models are trained for variable length generation using <PAD> tokens, their predictions over-emphasize the <PAD> tokens, and thus confidence-based unmasking fails as it prioritizes sampling <PAD> tokens first. In contrast, Edit Flow models never see padding tokens, though we find that regular sampling and confidence-based sampling perform on par.\n\n> The efficiency argument in appendix D is a bit of a drawback in disguise - what do the baselines look like with equivalent token budgets (even if it might take 3x more steps, and more compute)? This may not be feasible, but some discussion about this point is worthwhile given the importance of training, overtraining, and matching token budgets in general. \n\nWe evaluated the models in the equal compute setting because we believe this is the most fair comparison. Had we shown a more modest improvement while using 3x less compute, we would be understating the benefits of our approach.\n\nTo understand the effect of token efficiency better, we performed additional experiments on the code benchmarks. When we consume the same number of data sequences as the Mask DFM, our model achieves 12.1% on HumanEval pass@1. When we use the same compute budget as the Mask DFM, our model achieves 12.8% on HumanEval pass@1. Both are better than the Mask DFM which is at 9.1%.\n\n\n> Will the authors release code, or pseudo-code for the key methods of this work, in order to facilitate reproduction in open source?\n\nWe are adding a section to the appendix containing a simple, self-contained implementation of Edit Flows.\n\n> In Table 6, what were the CFG values tuned on? Train, train and validation, or validation only?\n\nThe benchmarks do not have train-validation splits. We tested CFG values 0.0, 0.5, 1.0, 2.0, 5.0, and 10.0 on each benchmark and report the best results. No other hyperparameters were tuned.\n\n> On figure 7, do the authors have any comparison of parallelization versus sampling steps to fit a given computational budget, similar to \"Large Language Monkeys: Scaling Inference Compute with Repeated Sampling\"? For example, is it better to do pass @64, with smaller steps, or pass @10 with more steps?\n\nThank you for bringing this work to our attention. The technique is certainly applicable, and it can be used to optimize the sample quality at a given compute cost. We did not investigate sampling efficiency in this submission, but we believe this is a worthwhile idea to improve inference cost and we hope to improve on this aspect in future work.\n\n> Similar to the above, reverse rates were discussed in the paper, but I did not see any concrete study of result quality over compute with reverse rate enabled, versus more parallelism or increased number of steps - do the authors have any results on this?\n\nFirst note that learning the reverse rate does not cost extra compute as it is learned simultaneously with the forward rate. The reverse rate is an extra output head with the same loss but time-inverted, incurring no additional compute cost during training.\n\nWith the reverse rates disabled during sampling results in HumanEval+ pass@1, we observe a 2.2% drop for Edit Flow and a 1.9% drop for Uniform $X_0$ + Edit Flow.\n\n> One concern I have on this approach is vocabulary scaling beyond 32k.\n\nScaling is analogous to that of standard Transformer models. Using a larger vocabulary size results in larger output and embedding layers, but does not affect the intermediate transformer blocks. \n\n> Do the authors have any qualitative examples of injecting invalid text (false facts, code vulnerability, typos, etc) during the sampling chain, and seeing if the model returns to edit and remove these? The space of human-in-the-sampling-chain seems possible given the model setup, depending on this behavior. Similarly, given it should be possible to mark a subsequence \"not for editing\" - does having some subsequence marked fixed change the following sampling behavior?\n\nWe are adding an example to the appendix to showcase the model’s correction capability. Edit Flows are able to fix an incorrect implementation of the `is_prime` function (where the three return statement are negated). We start with the string\n```\ndef is_prime(n: int) -> bool:\n    if n <= 1:\n        return True\n    for i in range(2, n):\n        if i % n == 0:\n            return True\n    return False\n```\nOver the course of 300 steps, the model makes 117 edits and reaches the final state\n```\ndef is_prime(n: int) -> bool:\n    if n <= 1:\n        return False\n    for i in range(2, n):\n        if n % i == 0:\n            return False\n    return True\n``` \nNote that the model does not make the minimal number of edits. For example, the state at step 150 contains a few extra random tokens which are deleted later in the sampling process:\n```\ndef is_prime(n: int) -> bool:\n    if n <= 1:\n        return False\n    for i in range(2, n):\n        if. % n == 0:\n            return False\n\n\n True\n\n\n,\n6\n```\nNote that these are sampled from the model while disabling edit operations on the function signature. Likewise, we can also disable editing in any subsequence at sampling time, but our model does not take such a mask as input and would not change its behavior apart from seeing different noisy sequences. As shown above, however, it is fine to prompt the model with invalid or wrong inputs and the model can still \"denoise\" the inputs.\n\n> Any intuition on CFG on rates versus CFG on Bregman divergences and when to prefer one or the other?\n\nIn general, it is unclear how CFG affects the likelihood of the generated samples. This is the case for continuous-space diffusion models and is also the case for Edit flows. Only for the mask construction, it can be shown that CFG is equivalent to sampling from a similarly modified likelihood --- which corresponds to both CFG on rates and CFG on Bregman divergences (ELBO) as they are equivalent. Out of the two options for Edit Flows, the CFG on the Bregman divergence loss / ELBO is slightly more interpretable.\n\nList of manuscript changes in response to the feedback:\n1. Results table names the specific autoregressive model (Llama3) and the superscript notation is now consistent.\n2. Added a qualitative example to the appendix to showcase how Edit Flows can fix an incorrect implementation of the `is_prime` function.\n3. Added qualitative examples to showcase the Mask DFM and Edit Flow generation processes."}, {"review_id": "iDykQdRdmE", "reviewer": "Reviewer_C6NG", "summary": "The paper develops a discrete flow framework with edit operations for the markov chain transitions instead of the masking (insertion/replacement). It constructs the corresponding theoretical framework, develops a high-performing practical implementation and designs a set of improved training and inference techniques (related to corruption strategies, classifier-free guidance, etc.). It evaluates the method on VQA, text and code generation and shows competitive performance with autoregressive models and better performance compared to masked diffusion methods.", "questions": "- Why do you think the model works worse for code generation? \n- Does the model generalize well to the autoregressive generation order? DiffuCoder has recently shown that discrete text diffusion models somewhat diverge to the autoregressive order anyway. I was curious if something like that happens for your model and if not, how well would it generate autoregressively?\n- Autoregressive models can generalize to longer in the sliding window fashion with overlaps. How well would the proposed model do the same?\n- I suspect that δxt(x¬i) is a bit non-strictly defined in its current form since there are many inputs $x$ that would have a density of \"infinity\", so instead it should be defined as a mixture of delta distributions (a delta distribution per each \"correct\" input $x$).\n- Did you perform any quantitative exploration of self-correction? There is many possible inference schedules for self-correction (i.e., how many steps to do, at which time-steps to do it, etc.). Have you run any evaluations for it?\n- L300: \"In our experiments, Edit Flows are able ingest 3× more training data per iteration while using the same compute and memory as Mask DFM.\". Do you use 3x larger batch size or 3x longer (on average) sequence length to train Edit Flows?", "limitations": "There is a reasonable discussion of the limitations in the experiments section.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The overall framework is novel and feels very natural for text generation tasks — i.e. much more natural than existing masked formulations that feel far-fetched. It's cool to see the out-of-the-box self-correction mechanism.\n- The work develops a set of improved techniques for both training and inference, which makes it readily competitive with modern masked/autoregressive approaches\n- The theoretical framework is rigorous and i didn't spot any problems with it (I have not gone through the proofs, but closely read through the exposition)\n- The empirical results are thorough and convincing\n\nWeaknesses:\n- I'm not sure that it's fair to use 3x time more tokens to train Edit Flows compared to the baselines. What is the performance at 2T tokens?\n- Appendix section \"B.1 Localized propagation path\" seems to be crucial for good performance, but it's 2 full pages of technical text. I spent ~10 minutes trying to understand the high level of what's going on, but couldn't do that. I understand that it's some \"local edits\", but it would be good to add a paragraph describing the high level idea for those who wants to understand the intuition, but do not need to know the gritty details.", "was_revised": true, "final_justification": "I've read the authors' response and believe that it has resolved my concerns. I believe it's a very interesting paper and decided to increase my score.\n\nI also went through the fellow negative review of czyg and believe that their raised concerns are nonsensical.", "rebuttal": "Thank you for your thoughtful and detailed review of our paper. We appreciate your insights and the opportunity to address your questions and concerns. Below, we provide responses to each of your points:\n\n> I'm not sure that it's fair to use 3x time more tokens to train Edit Flows compared to the baselines. What is the performance at 2T tokens?\n\nWe were unclear in discussing this point. The compute budget of Edit Flows matches that of the baseline autoregressive model and Mask DFM in our experiments. Edit flows use roughly 3x less compute per data sequence than Mask DFM due to not needing `<MASK>` tokens.\n\nA simple explanation for the 3x efficiency gain is, for the $\\kappa_t = t^3$ scheduler, we approximately remove 2/3 of the tokens per sequence. The Mask DFM model replaces removed tokens with a special `<MASK>` token, whereas Edit Flow actually deletes the tokens. This allows Edit Flows to use 3x less compute during training.\n\nTo understand the effect of token efficiency better, we performed additional experiments. When we consume the same number of data sequences as the Mask DFM, our model achieves 12.1% on HumanEval pass@1. When we use the same compute budget as the Mask DFM, our model achieves 12.8% on HumanEval pass@1. Both are better than the Mask DFM which is at 9.1%.\n\n> Appendix section \"B.1 Localized propagation path\" seems to be crucial for good performance, but it's 2 full pages of technical text.\n\nThe standard path will delete tokens independently, leading to e.g.\n\n$X_1$ = \"The quick brown fox jumps over the lazy dog\"\n\n$X_t$ = \"The `<DEL>` brown fox `<DEL>` over `<DEL>` lazy `<DEL>`\"\n\nThe localized path is used to incentivize retaining subsequences, and results in e.g\n\n$X_t$ = \"`<DEL>` quick brown fox `<DEL>` `<DEL>` `<DEL>` lazy dog\"\n\nWe are adding a figure showing how the token removal process is affected by a localized path in the appendix, which can provide high-level insights. We believe that the reason localized paths work better is because it is more aligned with the model's sampling process. During sampling, the model tends to generate well-formed subsequences since it may be easier to predict nearby tokens than faraway tokens (a frequent example is generating a token that completes half a word for multi-token words).\n\n> Why do you think the model works worse for code generation?\n\nOur experiments show that Edit Flows outperform the mask construction, the current most popular non-AR model, in code generation. In our experiments, we do find that autoregressive models outperform Flow models on code benchmarks at this scale at equal compute cost. We theorize that Autoregressive models have the advantage that, due to their causal nature, they learn all possible left-conditioned sequences, whereas non-autoregressive models do not always condition on tokens that correspond to the prompt at inference time. Furthermore, we only used pretraining data, which contains large documents and a lot of sequences are larger than 1024 sequence length which we had to crop. We note that [1] showed recently that diffusion models are able to close this gap with further training and even outperform autoregressive models in data-limited settings. Though their data sets are much smaller than our setup, we believe future improvements in data efficiency can significant close this gap.\n\n> Does the model generalize well to the autoregressive generation order?\n\nUnlike mask models, which are equivalent to any-order models, Edit Flows cannot be post-hoc queried to generate in an \"autoregressive\" order. The Edit Flow model predicts a bag-of-tokens that includes all missing tokens, but does not predict the order in which they should be inserted. We also tried using high-confidence sampling for Edit Flow models; it is not autoregressive and it performed on par with regular sampling. We are including additional qualitative examples to showcase the generation process on code examples.\n\n> Autoregressive models can generalize to longer in the sliding window fashion with overlaps. How well would the proposed model do the same?\n\nWe used sequences of size 1024 for training both autoregressive and non-autoregressive models. For MBPP, had to resort to a similar method, because the prompt + generation length can be longer than 1024. In order to evaluate on MBPP, if a prompt was longer than 768 tokens, we only used the last 768 tokens of the prompt as conditioning. Conversely, we did not attempt evaluating the model on longer than 1024 sequence length, though we imagine it would not perform well compared to shortening the prompt.\n\n> I suspect that δxt(x¬i) is a bit non-strictly defined in its current form since there are many inputs  that would have a density of \"infinity\", so instead it should be defined as a mixture of delta distributions (a delta distribution per each \"correct\" input )\n\nWe used $\\delta$ as shorthand for the Kronecker delta function which is an indicator function (it is either 1 or 0) rather than as a Dirac-delta distribution. We thank you for pointing this out and we will make this definition clear.\n\n> Did you perform any quantitative exploration of self-correction? There is many possible inference schedules for self-correction (i.e., how many steps to do, at which time-steps to do it, etc.). Have you run any evaluations for it?\n\nWe swept the hyperparameters for the divergence-free component (which uses both the forward and reverse rates) and found that it improved the results by +2.2% for the EditFlow model and by +1.9% for the Uniform $X_0$ + Edit Flow model on the HumanEval+ pass@1 benchmark.\n\nAdditionally, we are updating the appendix with qualitative examples that showcase how Edit Flows are able to identify and fix bugs in an incorrect implementation.\n\n> L300: \"In our experiments, Edit Flows are able ingest 3× more training data per iteration while using the same compute and memory as Mask DFM.\". Do you use 3x larger batch size or 3x longer (on average) sequence length to train Edit Flows?\n\nThe compute budget, the number of iterations and sequence length matched in all models. Edit Flows were able to train with 3x more data sequences per iteration because of their compute efficiency. See our detailed explanation and additional ablation experiment in Question 1.\n\n> Minor formatting concern: Figure 3 text is too small.\n\nThank you for pointing this out. We have adjusted this figure to increase the font size and make the figure more legible.\n\nWe hope these responses address your concerns and provide clarity on the points raised. Thank you again for your valuable feedback.\n\n[1] Prabhudesai, Mihir, et al. \"Diffusion Beats Autoregressive in Data-Constrained Settings.\" arXiv preprint arXiv:2507.15857 (2025).\n\nList of manuscript changes in response to the feedback:\n1. Expanded the discussion on the reason why Edit Flows use 3x less compute per token during training.\n2. Included a figure to explain the sampling process of localized path model.\n3. Clarified the notation of $\\delta$.\n4. Included qualitative examples from the code generation model to showcase the generation process.\n5. Added a qualitative example to the appendix to showcase how Edit Flows can fix an incorrect implementation of the `is_prime` function."}, {"review_id": "PLMJH10uZQ", "reviewer": "Reviewer_LxY2", "summary": "The  authors introduces Edit Flows, a non-autoregressive generative framework that models sequence generation via edit operations (insertions, deletions, substitutions) within a Continuous-Time Markov Chain (CTMC). Unlike prior non-autoregressive models that rely on fixed-length token-wise transitions, Edit Flows naturally support variable-length generation by defining a discrete flow over sequences. The authors propose a tractable training method using auxiliary alignment processes and demonstrate strong empirical performance across image captioning, code generation, and open-ended text tasks.", "questions": "1.  Will the authors provide a minimal implementation free of proprietary dependencies?\n\n2. Why does Edit Flows require 5K–10K sampling steps for code generation? Are there potential optimizations to reduce this computational cost without sacrificing output quality?\n\n3. Why does Edit Flows still lag behind autoregressive models in text/code tasks? Is this a fundamental limitation of the approach, or could it be addressed with improved training or data?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The proposed method demonstrates strong novelty, and a solid theoretical foundation is provided.\n\n- Extensive experimental evaluations are conducted on code generation and image captioning tasks, with promising results.\n\nWeaknesses\n- The performance of the proposed method still falls short of autoregressive approaches in the image captioning task.\n\n- The method appears to require sampling 5,000–10,000 steps for code generation, which may raise latency concerns.", "was_revised": true, "final_justification": "The concerns have been addressed. Thus, I keep my rating.", "rebuttal": "Thank you for your detailed review and for highlighting both the strengths and areas for improvement in our paper. We appreciate the opportunity to address your questions and concerns. Here are our responses:\n\n> The performance of the proposed method still falls short of autoregressive approaches in the image captioning task.\n\nWe note that the primary value of all of our experiments is to analyze the ability of the models on their own when trained from scratch, without additional data or pretrained weights. This allows us to analyze model performance differences without confounding variables such as varying training data. For instance, most papers that report image captioning use pretrained weights (e.g. VLP and ClipCap which we included for reference). When doing this apples-to-apples comparison, we outperform the autoregressive model. \n\nHowever, to satisfy our and the reviewer's curiosity, we have conducted an additional experiment where we initialize the weights using a Llama 3 1B model, as a means to test how well Edit Flows can adopt the weights of a pretrained AR model trained on internet-scale data. This increased our CIDEr score on MSCOCO from 108 to 124, which now outperforms both VLP and ClipCap (a 1.5B GPT-2 model). Going further than this would require additional ideas from the literature, such as data synthesis methods like recaptioning, which are orthogonal to the modeling framework and is out of scope for us.\n\n> Will the authors provide a minimal implementation free of proprietary dependencies?\n\nWe are adding a code example for Edit Flows in the appendix. This example is designed to be minimal and free of proprietary dependencies, allowing for easy replication and further exploration by the community.\n\n> Why does Edit Flows require 5K–10K sampling steps for code generation? Are there potential optimizations to reduce this computational cost without sacrificing output quality?\n\nWe decoupled the investigations into performance and sampling efficiency. In this work, we only optimized for generation quality. \n\nOn the other hand, our sampling code is very minimalistic and designed for easy batching. A more sophisticated implementation would likely bound the number of model evaluations by the total number of edit operations per generated sequence. We believe that there is a large scope for improving generation speed. Most of the 5,000–10,000 steps don’t change the input at all (and we informally found that only approximately 10-20% of the steps have an edit operation performed on the state). We hope to reduce the significant inference-time computational cost of Edit Flows in future work.\n\n> Why does Edit Flows still lag behind autoregressive models in text/code tasks? Is this a fundamental limitation of the approach, or could it be addressed with improved training or data?\n\nOur experiments show that Edit Flows outperform the mask construction, the current most popular non-AR model, in code generation. In our experiments, we do find that autoregressive models outperform Flow models on code benchmarks at this scale at equal compute cost. We theorize that Autoregressive models have the advantage that, due to their causal nature, they learn all possible left-conditioned sequences, whereas non-autoregressive models do not always condition on tokens that correspond to the prompt at inference time. Furthermore, we only used pretraining data, which contains large documents and a lot of sequences are larger than 1024 sequence length which we had to crop. We note that [1] showed recently that diffusion models are able to close this gap with further training and even outperform autoregressive models in data-limited settings. Though their data sets are much smaller than our setup, we believe future improvements in data efficiency can significant close this gap.\n\nWe hope these responses address your concerns and provide clarity on the points raised. Thank you again for your valuable feedback.\n\n[1] Prabhudesai, Mihir, et al. \"Diffusion Beats Autoregressive in Data-Constrained Settings.\" arXiv preprint arXiv:2507.15857 (2025).\n\nList of manuscript changes in response to the feedback:\n1. Added a reference implementation of Edit Flows to the appendix."}, {"review_id": "AkzHgxFFZl", "reviewer": "Reviewer_czyg", "summary": "Problem: Non-autoregressive models struggle with variable-length sequence generation, unlike autoregressive models.\nProposed solution: Introduce Edit Flows, a non-autoregressive model using edit operations (insertions, deletions, substitutions). This allows position-relative, non-rigid generation, better matching real sequence structure. For training this, they use an alignment between start and end states to form data that they then train on. They also use auxiliary variables (epsilon) to denote empty words after deletion or before insertion.", "questions": "I think The real details actually start from line 170. Up to this point, it's all formalisms about continuous processes and how they may connect to discrete time. \n\n\nTraining \n=======\nHere is my understanding (and let me know if I am understanding it correctly). To train this model to map x_0 to x_1, you want to create a path that connects the starting point to the finish line. And for doing this, you do some manual alignment. For example, as your example shows, you use the path characterized by edit distance between two sequences. \n\nIf the above is correct, you should probably discuss this work that has a similar data generation pipeline and contrast against it: https://arxiv.org/abs/2211.00053\n\nOne thing that I don't fully understand: do you think this edit-distance-based data augmentation can be done for all tasks? \n\n\nExperiments ;\n=========\nI am not convinced your baselines are actually strong. For example, for OBQA and ARC-C (Table 2), the numbers are barely above random (25%). Meanwhile, we know that the scores on these datasets have been much higher as far as we know, even with. 1B model. \n\nOne other issue: why do these tasks require sequence editing? To be honest, I don't have any concrete suggestions. But I do wonder if there are more appropriate choices. \n\n\n\nOther works \n===========\nMy suggestion is to move the related work to the end of your paper to not create a break between formalism and experiments. \n\nOther missing citations: \nhttps://arxiv.org/pdf/1811.10996\nhttps://arxiv.org/abs/2202.11705 \nhttps://aclanthology.org/2020.emnlp-main.701.pdf", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- I am convinced by the motivation: we need to go beyond auto-regressive (one-directional) generation/sampling. \n- However, I am not convinced whether the proposed framework/algorithm is adding anything to our existing knowledge. If you distill the idea, it all comes down to curating training data for text editing and training on it; it's a pretty simple framework that is wrapped inside an esoteric formalism. \n- I am also not convinced whether the proposed framework is adding anything to our existing empirical toolkit. The experimental setup is weak, to my understanding.", "was_revised": true, "final_justification": "First, I want to acknowledge the theoretical contribution of this paper (as I have in my review). \nI remain unconvinced by the empirical results.", "rebuttal": "We thank the reviewer for their review and for going through the paper despite difficulty in understanding our mathematical notation, which we found very unfortunate but acknowledge the feedback. However, we believe that what we proposed are significant modeling improvements beyond simple data manipulation, and we hope that the following statements may help clarify our contributions:\n\n  - Unlike the works that the reviewer brought up [1,3,4], we do not provide extra knowledge to the model (such a scoring function for intermediate sentences, or an ordered sequence of target edit operations). Our models only see the data sequences, and either (i) deletes all tokens or (ii) performs completely random edit operations to transform to a noise distribution. In either case, we simply map from a noise distribution (the null sequence or a sequence of random tokens) to the data sequence, and there is no external knowledge given to the model.\n\n  - The alignments that the reviewer refers to as data processing (line 170) is actually only for training tractability. We use this to align the model's outputs (which has a shorter length due to deletion of tokens) back to the original sequence (which contains all tokens). In practice, these alignments are sampled randomly (e.g., from randomly deleting tokens independently) and do not provide the model with additional information.\n\n  - The insert-only Edit Flow model sees the *exact* same data and noise sequences as the mask diffusion baseline (denoted Mask DFM). The core difference is only whether we actually delete a token (Edit Flow) or replace it with a `<MASK>` token (Mask DFM). As such, any improvements we see are due to the differences in modeling paradigms, not data processing.\n\n  - The complexity in the  formalism is in order to model the stopping criterion \"when to stop editing\". Unlike the works that the reviewer brought up [2,3], we do not use a non-terminating MCMC algorithm but model a terminating process. As such, the model automatically *learns when edits should be done* and stops when no more edits are needed. These are the rates denoted as $\\lambda$, which have not been used by most prior works. Prior methods used different stopping criterion because they were restricted models (e.g., AR uses EOS token but are limited to left-to-right generation, mask diffusion stops when all tokens are unmasked but are limited to fixed sequence lengths).\n\nWe hope the above statements help clarify our contributions. We understand that the continuous-time Markov chain formalism is new to the community and had hoped to introduce and extend it as gently as possible.\n\nBelow, we answer the specific questions.\n\n> I am not convinced whether the proposed framework/algorithm is adding anything to our existing knowledge. If you distill the idea, it all comes down to curating training data for text editing and training on it; it's a pretty simple framework that is wrapped inside an esoteric formalism.\n\nFirstly, note that our proposed model has the ability to generate variable length sequences in a non-autoregressive fashion, which is not possible with the Mask DFM construction.\n\nSecondly, we note that introducing edit operations in the generation process (see line 140) introduces interesting behaviors. The Edit Flow model predicts a set of tokens at to be inserted at each location; however, the model doesn't need to be optimal and can predict any one of the tokens within this set since the generation process can insert the other tokens at a later time. This allows the model to generate whatever token it is most confident about and then self-correct (e.g. fix the grammar) later on. We are including additional qualitative samples in the appendix to showcase this generative behavior.\n\n> I am also not convinced whether the proposed framework is adding anything to our existing empirical toolkit.\n\nOur experiments are aimed at an apples-to-apples comparison between modeling frameworks. These ablation studies are vital in understanding the tradeoffs between different frameworks. Even at the 1B scale, the text and code models already take weeks to train, making understanding their design choices crucial prior to scaling up.\n\n> [paraphrased] Confirmation of understanding: The training involves creating a path connecting start to finish using manual alignment, e.g., edit distance between sequences. Is this correct?\n\nWe first note that we are mapping from a completely noise distribution (either null sequence or completely random tokens) to a data sequence. The alignments are completely random and are not user-provided manually-tuned edit sequences. For instance, they do not correspond to the optimal alignment using edit distance as it wouldn't make sense with one sequence having random tokens. We believe this may be a source of misunderstanding, as some prior works have trained on curated sequences of edits; *our paper is not about curating sequences of edit operations*.\n\n> [...] path characterized by edit distance between two sequences\n\nTo satisfy our curiosity, we tested the reviewer's suggestion of using edit distance to produce an optimal alignment between a model-generated sequence and the data sequence (this new experiment does use additional knowledge from a pretrained model but does not impose any priorities on the edits). Surprisingly, we found that this did not improve but hurt performance. We believe it is because the model does not see diverse enough sequences during training.\n\n> If the above is correct, you should probably discuss this work that has a similar data generation pipeline and contrast against it: [1]\n\nThank you for bringing our attention to this work and the other works you mentioned; we will update the related work section. In the referenced work, one model proposes complete sentences and another model scores the sentences. In contrast, Edit Flows do not require a separate base-generator and a corrector model; a single model is used to generate the samples. Our generated samples are also not necessarily complete or coherent sentences until the final generation step. We added more qualitative examples in the appendix to illustrate this.\n\n> One thing that I don't fully understand: do you think this edit-distance-based data augmentation can be done for all tasks?\n\nSince we do not introduce additional knowledge, the framework works for any generative sequence modeling task. \n\n> I am not convinced your baselines are actually strong. For example, for OBQA and ARC-C (Table 2), the numbers are barely above random (25%). Meanwhile, we know that the scores on these datasets have been much higher as far as we know, even with. 1B model.\n\nThank you for bringing this to our attention. If the reviewer is aware of a 1B model trained on the DCLM-baselines-1.0 dataset only, please let us know so we can cross reference. All of our reported results are zero-shot, whereas many prior works used few-shot evaluation or performed model finetuning. The goal of our experiments is a systematic study of the different modeling paradigms, aimed at improving our understanding.\n\nThanks to the reviewer's comments, we investigated our DCLM results and we found that our models slightly underperform on these benchmarks because we were training them only on a subset of DCLM. We are correcting this and we will update the results. We emphasize that the experiment still serves its objective as an ablation study. All models were trained on the same data using the same resources, therefore we do not expect the conclusions to change.\n\n> One other issue: why do these tasks require sequence editing? To be honest, I don't have any concrete suggestions. But I do wonder if there are more appropriate choices.\n\nWe incorporate edit operations mainly to handle variable length generation with non-autoregressive models, which are required for most text generation tasks such as captioning and code generation.\n\n> Other missing citations: [...]\n\nThank you for bringing these works to our attention. We will include them in our related works section; a summary of their differences:\n[2] is similar to Edit Flows in the sense that the model makes predictions in the space of edits. A key difference, however, is that Edit Flows sample from the target distribution in a fixed number of steps in the Flow Matching framework, as opposed to [2] that generates samples from a stationary distribution until acceptance.\n[3] and Edit Flows both iteratively improve on the sentence until the task is complete, however Edit Flows make predictions in the space of token edits, whereas [3] predicts the next sequence in the generation process as a whole.\n[4] makes predictions in the space of edits, however, unlike Edit Flows, [4] requires a differentiable objective function that scores language fluency and semantic meaning. Edit Flows only require a single model and does not use backpropagation at test time.\n\nPlease let us know if you have additional questions. We are happy to further clarify our contributions, and hope to rectify any misunderstanding.\n\n[1] Welleck, Sean, et al. \"Generating Sequences by Learning to Self-Correct.\" The Eleventh International Conference on Learning Representations.\n\n[2] Miao, Ning, et al. \"Cgmh: Constrained sentence generation by metropolis-hastings sampling.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.\n\n[3] Qin, Lianhui, et al. \"Cold decoding: Energy-based constrained text generation with langevin dynamics.\" Advances in Neural Information Processing Systems 35 (2022): 9538-9551.\n\n[4] Sha, Lei. \"Gradient-guided unsupervised lexically constrained text generation.\" Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.\n\nList of manuscript changes in response to the feedback:\n1. Expanded the related works section to include the four missing references.\n2. Added qualitative examples to showcase the Edit Flow generation process."}]

Author rebuttal:

Meta-review: This paper introduces Edit Flows, a discrete flow matching framework for non-autoregressive sequence generation that uses edit operations (insertions, deletions, substitutions) rather than masking. The work addresses the limitation of fixed-length generation in existing non-autoregressive models by enabling variable-length sequence generation through a continuous-time Markov chain formulation. Strengths include strong theoretical foundations with rigorous mathematical framework, novel modeling approach that naturally handles variable-length sequences, comprehensive experimental evaluation across image captioning, text generation, and code generation tasks, and consistent improvements over masked diffusion baselines. The paper demonstrates 3x computational efficiency during training compared to masked approaches and includes interpretable generation processes with self-correction capabilities. However, weaknesses include performance still lagging behind autoregressive models on text/code tasks, high inference cost requiring 5K-10K sampling steps, limited scalability evaluation (models only trained up to 1B parameters), and some empirical results showing modest improvements with relatively low absolute performance on certain benchmarks.

The rebuttal period generated substantial discussion, particularly around empirical validation concerns. Reviewer arpi initially raised questions about baseline comparisons and efficiency claims but was satisfied by the authors' detailed responses, including additional experiments with Llama-3 initialization and clarifications about computational efficiency, ultimately increasing their score to "Strong Accept." Reviewer C6NG asked about computational fairness and theoretical details, and after receiving comprehensive explanations about the 3x efficiency gain and localized propagation paths, also moved to "Strong Accept." Reviewer LxY2 maintained their "Borderline Accept" after authors addressed concerns about implementation availability and sampling efficiency. However, reviewer czyg remained unconvinced despite extensive exchanges, raising persistent concerns about statistical significance of results, appropriateness of confidence interval calculations, and fundamental questions about the method's practical value. The reviewer argued that confidence intervals were incorrectly computed by treating multiple runs as independent trials rather than using bootstrap resampling, and questioned whether modest improvements on benchmarks with relatively low absolute scores constitute meaningful progress. 

Despite czyg's remaining concerns about empirical validation which I resonate with, the strong theoretical contributions, novel approach to variable-length generation, and positive reception from three other reviewers support acceptance, as the work advances our understanding of non-autoregressive generation methods and provides a valuable foundation for future research.
Decision: Accept (poster)