Flow of Spans: Generalizing Language Models to Dynamic Span-Vocabulary via GFlowNets
Authors: Bo Xue
Yunchong Song
Fanghao Shao
Xuekai Zhu
Lin Chen
Luoyi Fu
Xinbing Wang
Zhouhan Lin
Venue: ICLR 2026 Poster (2026)
Keywords: generative flow networks
language models
Primary area: foundation or frontier models, including LLMs

Abstract: Standard autoregressive language models generate text token-by-token from a fixed vocabulary, inducing a *tree-structured state space* when viewing token sampling as an action, which limits flexibility and expressiveness. Recent work introduces dynamic vocabulary by sampling retrieved text spans but overlooks that the same sentence can be composed of spans of varying lengths, lacking explicit modeling of the *directed acyclic graph (DAG) state space*. This leads to restricted exploration of compositional paths and is biased toward the chosen path. Generative Flow Networks (GFlowNets) are powerful for efficient exploring and generalizing over state spaces, particularly those with a DAG structure. However, prior GFlowNets-based language models operate at the token level and remain confined to tree-structured spaces, limiting their potential. In this work, we propose **F**low **o**f **S**pan**S** (**FOSS**), a principled GFlowNets framework for span generation. FoSS constructs a dynamic span vocabulary by segmenting the retrieved text flexibly, ensuring a DAG-structured state space, which allows GFlowNets to explore diverse compositional paths and improve generalization. With specialized reward models, FoSS generates diverse, high-quality text. Empirically, FoSS improves MAUVE scores by up to 12.5\% over Transformer on text generation and achieves 3.5\% gains on knowledge-intensive tasks, consistently outperforming state-of-the-art methods. Scaling experiments further demonstrate FoSS benefits from larger models, more data, and richer retrieval corpora, retaining its advantage over strong baselines.

Published as a conference paper at ICLR 2026
FLOW OF SPANS: GENERALIZING LANGUAGE MODELS TO DYNAMIC SPAN- VOCABULRYY VIA GFLOWNETS
Bo Xue Yunchong Song?;
Fanghao Shao', Xuekai Zhu', Lin Chen' Luoyi F Xinbing Wang", Zhouhan Lin
'LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University "Shangaai Artificial Intelligence Laboratory
SShanghai Jiao Tong University 'Shanghai Innovation Institute
sappho x8 jtu edu. en, lin, zhouhan@gmaili com
songyunchongg@jjlbb org. cn,
ABSTRACT
Standard autoregressive language models generate text token-by-tokee from a fixed vocabulary, inducing tree-structured state space when viewing token sampling as an action, which limits flexibility and expressiveness. Recent work introduces dynamic vocabulary by sampling retrieved text spans but overlooks that the same sentence can be composed of spans varying lengths, lacking explicit modeling of the directed acyclic graph (DAG) state space. This leads to restricted exploration of compositional paths and is biased toward the chosen path. Generative Flow Net- state
works (GFlowNets) powerful for efficient exploring and generalizing spaces, particularly those with a DAG structure. However, prior GFlowNets Nets-baedd language models operate at the token level and remain confined to tree-structured spaces, limiting their potential. In this work, we propose Flow of SpanS (FoSS), a principled GFlowNets framework for span generation. FoSS constructs dy- namic span vocabulary by segmenting the retrieved text flexibly, ensuring a DAG- structured state space, which allows GFlowNets to explore diverse compositional paths and improve generalization. With specialized reward models, FoSS generates diverse, high-quality text. Empirically, FoSS improves MAUVE scores by up 12.5% over Transformer on text generation and achieves 3.5% gains on knowledge- intensive tasks, consistently outperforming sate-offtheearr methods. Scaling exper- iments further demonstrate FoSS benefits from larger models, more data, and richer retrieval corpora, retaining its advantage over strong baselines. Code available at https://githuu com http://hhh...cccoaaooooocoott/oa http://appppooaaaa ooolow of -f---anss
INTRODUCTION
Standard autoregressive language models generate token-by-token from fixed, finite vocabu- lary (Brown et al.] 2020 Radford et al. 2019; [Sennrich et al. 2016). However, this approach has inherent limitations. The fixed vocabulary restricts the granularity of generation, and the resulting tree-structured state space, as illustrated in Figure where each state has unique prefix predecessor, limits the model's ability to efficiently explore alternative generation paths. such a structure, each sample can only follow single path, whereas directed acyclic graph (DAG) structure allows simultancous exploration of multiple paths, covering a broader region of the state space (see the green area in the DAG and tree state space in Figure Ip. To address these limitations, recent work has introduced dynamic vocabularies, and variable-length generation units, enabling models to construct
context-aware, multi-word vocabularies (Cao al. 2024 Martins al., 2022 Li al. 2024 Lan
et al. 2023). However, these methods overlook the inherent DAG nature of sentence composition, where the same sentence (e.g.. " "ABCDEFGH") can be formed through different span combinations, such as + or "D"""""""""""""""""""""""""""""" "H" Consequently, existing ap-
"Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2026
ABCDEFH
ABCDE
PREFIX:
ABCDE
ABCD
ABCDEF
ABC
Tronsformer
State
Action
ABCDEG
ABCE
ABDEG
Transition
ABD
ABCDEFGZ
ABDE
ABDEP
Policy Network
REGAL
ABCD
ABCDEFGH
Reword Models
cb
EFGH
6H
Credit
Docs
Vecabulary
Spon Encoder
Corpes
Given the prefix, standard language model generates token-by-token and forms free state space, trained with next-token prediction (NTP) loss; while in FoSS, since a sentence can be composed of spans in multiple ways, we construct DAG state space and optimize it with GFlowNets.
Figure 1:
proaches do not explicitly model the DAG-structured state space, relying instead training data incidentally reflect such structures and lacking theoretical guarantees for coverage or generalization.
Generative Flow Networks (GFlowNets) are promising method for solving the aforementioned problems. GFlowNets are class of generative model that samples with probability proportional the reward, known for their excellent performance and theoretical guarantees in exploring and generalizing over state spaces, especially those with DAG structure (Shen et al. 2023 Krichel al. 2024] Atanackovic Bengio 2024). There has been limited work on applying Nets to language models (Hu et 2024] Yu et al. 2025 Lee et al., 2025), and these efforts primarily treat GFlowNets as trainable alternative sampling strategy for language models, without constructing DAG-structured state space. This reduces the GFlowNets framework to degraded learning paradigm that limits the exploration of state transitions and is unable to fully leverage the potential of GFlowNets (Li et al. 2023 Shen et al., 2023 Malkin et al. 2022 Bengio al. 2023 In this work, we introduce the GelowNets framework to span language models for the first time, explicitly constructing DAG-structured state space that allows GFlowNets fully unleash its power in the context of span generation. At the same time, unlike previous scenarios such as reasoning, where clear answers are available (Yu et al. 2025), explore the design of reward models within the GFlowNets framework to enhance the performance of general text generation.
We propose FoSS, span language model within principled GFlowNets framework. FoSS introduces a flexible text segmentation algorithm to construct the DAAG-structured state space. Then, we explicitly model the DAG space with GFlowNets, enabling efficient exploration and generalization over the state space. Benefiting from the specialized reward models, FoSS can be efficiently trained while maintaining both the quality and diversity of text generation. Our extensive evaluation demonstrates that FoSS significantly outperforms existing methods across various tasks. In text generation, FoSS achieves a 5.51% MAUVE score improvement over the state-of-the-an while attaining consistently higher scores in GPT-4-based evaluations, which aligned with human preferences (Zheng al.) 2023). In the domain adaptation task, FoSS outperforms fully fine-tuned models without any training, showing robust generalization ability. Comprehensive ablation studies show that without the DAG-state space design or GFlowNets training, the model's performance drops sharply, validating our design as indispensable component. Furthermore, scaling experiments confirm that FoSS effectively leverages larger model capacities, increased training data, and richer retrieval corpora, revealing the tremendous potential of FoSS when allocating a larger computational budget.
PRELIMINARIES
GFlowNets designed leam stochastic policies that sample from unnormalized distribution over complex objects through a sequence of discrete actions within a Markov Decision Process
(MDP) framework (Pan et al. 2024 Jain al. 2022 Shen al. 2023). Let the state space be S
2

---

Published as a conference paper at ICLR 2026
Algorithm 1: FoSS Training
Input:
Training corpus D. Forward policy PF(-1:0), Reward function R(-) (Eq Learning rate 7J, Bernoulli mixing probability π
Output: Fine-tuned parameters
Pre-processing:
Apply DAG-Inducing Span Segmentation algorithm to D to obtain D
Initialize prefix-traeectry buffer B t- 6, epoch counter
while not converged do
e
// Mini batch implicitly handled
foreach segmented document d E DD' do
Truncate to prefix and residual d Build vocabulary V conditioned on d!
e then
construct trajectory using spans from d' c action sequence
- BU [(T, R(T)))
else
if (ernoulli (T) then
// Online sampling
r - sample trajectory from Pp(-13 from 8g =
B < BU [T, R(T)))
else
// offline sampling
sample trajectory from ((c) and compute Pp(-13V,0)
0 0 VoL((; (Fq
return +- 0
and the set of edges be E C for directed acyclic graph (DAG) G E), where vertices represent states and edges represent transitions between states. The process begins from an initial state so E S, with no incoming edges, and terminates terminal state X S, where X the set of terminal states.
Sn x) consists of sequence of states, with transitions (s; + $;+1) E E corresponding to actions taken at each step. The objective of GFlowNets learn forward policy Pr (s (s;), which defines the probability of transitioning from one state to the next. The probability of trajectory 7 is the product of the transition probabilities along the sequence: PF(T)
Each complete trajectory
(5o 81
I
The terminal distribution PJ (x:) represents the marginal distribution over terminal states, defined as: PT (x) E Pr(r), where the sum is taken over all trajectories that terminate in terminal state The objective of GFlowNets is to learn the forward policy such that the terminal distribution PE (x) is proportional to reward function R(x).. i.c., PTT (x) x R(x), where R(x) is an unnormalized density over terminal states (Bengio et al. 2023). To achieve this, GFlowNets associate flow with each state in the graph. Concretely, state-flow function F S - R_ is defined that assigns non-negative value to each state, representing the total flow passing through it. This function enables various training objectives for GFlowNets. In addition to the forward policy, GFlowNets incorporate a backward policy PB(s; (5441), which sequentially deconstructs compositional objects, transforming the intractable matching of marginal distributions over terminal states into tractable one of matching distributions over complete trajectories (Gritsaev et al. 2025 Jang et al. 2024).
3 SPAN-GENERATION WITH GFLOWNETS
this section, we introduce our novel span-based text generation approach utilizing GFlowNets. Prior language modeling approaches typically operate with word-level vocabularies where each element referred to as a token. These static vocabularies impose constraints on generation granularity. In framework, vocabulary elements can be either single words multi word phrases, which we collectively refer to as spans. We first formalize the span- -eneratinn problem within MDP framework, establishing comprehensive formulation that enables flexible span sampling. Then,
3

---

Published as a conference paper at ICLR 2026
present learning objective and training policy that effectively navigates the DAG-structured state space. Furthermore, we detail the key components that parameterize our model: the policy network architecture that generates spans from dynamic vocabulary, and our reward function formulation that balances text fluency and alignment with human-written text.
3.1 MARKOV DECISION PROCESS FORMULATION FOR GFLOWNETS LEARNING
To formalize span- basdd text generation within the GFlowNets framework, we establish a complete MDP formulation with the following components: State: Each state s; S is represented as string = CONCAT 42), where is the prefix text, i; denotes the text generated so far consisting of previously sampled spans, and CONCAT represents string concatenation. Action: We denote by candidate action from the action space, and by a; the actual action sampled. An action involves selecting span to be appended to the current state 8;. Transition: Given a state 8; and an action a, the transition function deterministically yields the next state 8;41 by appending the selected span to St. Reward: The reward function R + R+ assigns a non-negative value to each terminal state E X C S, where employ specialized reward models to evaluate the quality and coherence of the generated text.
Since actions involve spans of variable length, multiple distinct trajectories lead the same state, ensuring the underlying graph G is a DAG. For example, in Figure 1 the state "ABCD" can be reached via two different trajectories: (AB ABCD) and (AB ABC ABCD). This contrasts with conventional token-by-token autoregressive language models, where each state can only be reached by single unique trajectory, forming tree (Lee al. 2025 Hu et al., 2024 Yu al.) 2025). As shown in Figure 1, token-by-tor by-token autoregressive language models generate single words sequentially with accumulative prefixes. This structure restricts each non-initial state have exactly one parent node. For example, the state "ABCD" can only be reached from state "ABC", creating rigid tree structure. This makes the backward policy deterministic with (s ( degenerated case, GFlowNet, reduces to discrete-action soft Q-learning (Malkin et al. 2022). limiting the number of possible transitions and impeding exploration of the expression space Li et al. 2023).
In such
We instantiate the forward policy Pp with span language model parameterized by 0, denoted by PsLM(:;0), GFlowNets in this setup learn stochastic policies for sampling terminal states by taking actions drawn from the action space. Given unnormalized reward function R defined over terminal states, the GFlowNets navigates through the state space following the forward policy Pr(s:+1|s;; 0). Concretely, for sequence generation, the states 8; CONCAT(c, t;) represent the partial sequence thus far, and action corresponds to a span sampled from the span language model: a, PSLM(al (a) s5; 0). Since the action a, directly determines the next state through span concatenation, write: Pp Ss+1 0)
PSLM PaLM(a, s4; 0), where $141 CONCAT(s;, a.). This continues until a terminal action is sampled, yielding a terminal state sn, E X that consists of the prefix, all generated spans, and the termination symbol T.
3.2 LEARNING OBJECTIVE AND TRAINING POLICY
We adopt the subtrajectory balance learning objective for training GelowNets, which, compared to flow matching (Bengio et al., 2021). detailed balance (Bengio et al. 2023) and trajectory bal- ance (Malkin et al. (2022), enables more efficient exploration of DAG-structured state space while providing improved stability, particularly for longer sequence generation (Madan et al. (202)). Given a forward policy PF. a backward policy PB, and state flow function F. the objective over partial trajectory (sp - 8e) is defined as:
LSWTB((T) log F(sh) I Pr(sc4ilss) 2
F(se) I PB(s
When GFlowNets converge, for valid state 8e that contains a complete sentence, we define s as the terminal state reached after sampling the termination symbol from S.. We then have R(s?) F(se) (P[T] | se) for such states (Deleu et al. 2022 Pan et al. 2023 Hu et al. 2024). Using this, we derive the final learning objective by summing over all partial valid trajectories within
complete trajectory (so -> 81 - Sn)) with equal weight:
4

---

Published as a conference paper at ICLR 2026
T Pr P1 2
log R(s,) R(s} I (B( 118) (p
[(T)=
E (s 8y) OSiciSn
(2)
where (sa (8t, 8j) is an indicator function that equals 1 if both 8; and 8j are complete sentences, and O otherwise. This selective summation focuses the learning signal on transitions between meaningful, complete sentences, leading more efficient and stable training.
To address the challenge of exploring the combinatorially large sampling space, we propose a hybrid online-offliee training approach leveraging readily available training data. While replay buffers enhance GFlowNets training (Jain et al.) 2022), these online off-policy methods often suffer from high variance and can become trapped in suboptimal trajectories (Fedus et al., 2020; Vemgal et al.). Drawing inspiration from offline pretraining strategies (Pandey et al. 2025), our approach constructs high-reward trajectories from training data, providing more effective exploration initialization. Specif- ically, we compose the mini-bated during training using trajectories from three sources: (1) the policy PF, (2) a reward-prioritizes replay buffer containing past high-reward trajectories, and (3) trajectories from the training set. This hybrid approach is crucial for FoSS to effectively navigate the combinatorially large DAG-structured state space. Meanwhile, to ensure that the trajectories derived from the training naturally induce a DAG, employ DAG-Inducing span segmentation algorithm that applies a controlled stochastic early-stopping mechanism to standard forward maxi- mum matching. This mechanism results in multiple distinct segmentation trajectories per training set sentence, where different span paths share common sub-sequences, explicitly modeling the DAG structure. For more details, please refer Appendix E To further mitigate variance in sparse reward regimes and stabilize training, we follow eHu al. 2024) and initially fine-tune the policy network on readily available training set data before standard GFlowNets training, equipping it with the capacity to sample high-reward states.
3.3 POLICY NETWORK
We instantiate the forward policy Pr using span language model that processes the current state 8 = CONCAT(c, t)) with transformer-based prefix encoder, mapping the input to a contextual vector h; Rdd via causal attention (Vaswani et al., 2017). For the action space, a E TUUUUT (T), where T represents phrases from external corpora and V is the fixed word- level vocabulary. A span encoder computes context-dependent representations for each potential action a, yielding embedding V. E R2_ For phrases from T. we compute vector representations for all candidate phrases from the supporting documents, ensuring the internal state space forms DAG. We employ a bidirectional Transformer (Devlin et al. 2019) to generate contextualized representations, then transform and concatenate start and end position embeddings through MLPs to form the complete phrase representation. For word-level actions from V/ and the terminal action T, we utilize standard token embeddings. Our architecture draws inspiration from recent advances in dynamic vocabulary language models, particularly CoG (Lan al.) 2023), the forward policy distribution over actions is formulated as: PSLM (a) ss;; 0) xx exp (h; v.). For the (s we employed an uniform distribution over all possible suffixes of $141 in the dynamic vocabulary. Details on dynamic vocabulary construction and policy network are provided in Appendices C and
consider
3.4 REWARD FUNCTION
Our reward function for span-based text generation balances fluency and alignment with the distri- bution of hwwan-wwriiii wwrtten text through two components: a language model (LM) and a preference model (PM). The LM component encourages grammatical correctness and contextual relevance (Ide) et al. 2024 Fu et al., 2024). However, high-likelihood regions of language model distributions often correspond to repetitive, generic patterns but fail to represent the diversity and quality of human text (Holtzman et al. Welleck et al. 2020). To mitigate this, our PM component discriminates be- outputs, guiding generation toward more human-like
tween human-written text and expressions. Specifically, the LM component provides a natural measure of textual quality through PLM(s, /c). which represents the likelihood of the complete generated sequence given the prefix. For the PM component, our approach employs a Bradley- Terry (Bradley & Terry/ 1952) log-likelihool loss with a score-centering regularizer (Eisenstein et al. 2023), which trains the model to assign
5

---

Published as a conference paper at ICLR 2026
Table 1: MAUVE scores on open-ended text generation across in-domain, out-of-domain, and scaling settings. Results are reported for both greedy decoding and nucleus sampling.
In Domain
Out of Domain
Scaling Data Store
Method
Greedy
Nuclcus
Greedy
Nucleus
Greedy
Nucleus
Transformer w/o FT Transformer w/ FT
18.64
22.37
20.32
25.21
19.87
23.43
19.87
23.43
23.00
26.85
20.21
21.31
KNN-LM LNN-LM Khandelwal et al. 2020
19.92
22.50
23.31
24.75
23.21
23.39
RETRO al. 2022
21.19
22.86
18.70
20.35
19.75
22.87
LoG et al. 2023)
26.01
26.14
21.31
28.14
24.68
26.97
GFlowNettsFI Huu al. 2024
26.58
29.61
26.49
28.62
GDV Liu al.
2024
25.69
24.34
26.35
24.80
FoSS
30.78
31.65
27.84
32.17
27.88
33.79
higher scores human- human-written references from the training set than to sequences generated by the policy network. The sigmoid- -oaooidddtrssffomeedd score (sm(),, serves the learned reward signal.
Combining these complementary objectives, we formulate reward function for terminal state sn as:
R(s,) = exp (a log (s,(cc))) + (1 - a.) log (s,) E (0,
where a controls the trade-off between language model fluency and preference model alignment. Both models operate at the word-level, requiring tokenization of 8a (ONCAAT(c,L, (.) into its constituent tokens [W1, W2,...; WNN,, where the first M tokens correspond to the prefix e and the remaining tokens represent the generated text in total sequence of N tokens. The language model likelihood then be computed as: pT (c) = I PLM (w: [W); ...y W;-1).
For the preference model, compute (Pm( )m)) (ffM (W1, ....... WN)) )), where fPM represents the discriminator function of the preference model that maps the token sequence to a scalar logit, and a()) denotes the sigmoid function. A detailed description of the initialization processes for both the LM and PM components is provided in the Appendix C.3
EXPERIMENTS
For our experimental evaluation, compare approach against base model and state-of-the-arr methods: Transformer (Vaswani et al. 2017). the de facto standard architecture for neural language modeling: KNN-LM (Khandelwal al. 2020 extends pre-trained language model by interpolating its predictions with a k-nearest neighbors retrieval, using similar context examples from a datastore to refine next token predictions; RETRO (Borgeaud et al. 2022) integrates frozen BERT (Devlin) et al. 2019) retriever with a chunked cross-attention mechanism, allowing the model to condition on retrieved document chunks predict the next token; CoG (Lan et al. (023) uses phrase encoder to index the training corpus and prefix encoder for the current context, retrieving similar phrases based prefix for sentence continuation, which inspired the architecture of our policy network: GFlowNett-FFT (Hu et al., 2024) leverages GFlowNets to fine-tune standard Transformers under the token-by-token autoregressive paradigm, resulting in a tree-structured graph G; GDV (Generation with Dynamic Vocabulary) (Liu et al. 2024) builds upon similar retrieval framework CoG, but introduces a novel dynamic vocabulary loss during training to encourage generation beyond the static vocabulary.
To ensure a fair comparison, the prefix encoders the Transformer, KNN-LM, CoG, GFlowNets-FT. GDV, and FoSS are all initialized from pre-trained GPT-2 (Radford et al.) 2019). For the phrase encoder in both CoG and FoSS, we fine-tune the pre-trained BERT-base SOOT-bbse-carae casee model (Devlin et al. 2019). In GFlowNets-FT the reward model employs both the LM and PM components from FoSS. For more implementation details, please refer to the Appendix We report baseline results from their original publications where possible.

---

Published as a conference paper at ICLR 2026
Table 2: Pairwise comparison of FoSS against baselines using GPT-4 as evaluator for text quality assessment.
In Domain
Out of Domain
GPT4 Preference
GPT4 Preference
FoSS vs.
FoSS vs.
Better Neutral Worse
Better Neutral Worse
Transformer
53%
19%
28%
Transformer
56%
23%
21%
kNN-LM
67%
15%
18%
kNN-LM
75%
13%
12%
CoG
42%
31%
27%
CoG
74%
6%
20%
GFlowNets-FT
55%
29%
16%
GFlowNets-FTT
49%
24%
27%
Following previous work (Su al. 2022; Liu al. 2024 et al. 2024), we evaluate models using MAUVE (Pillutla et al. 2021> I and Diversity (Welleck et al. 2020) metrics. MAUVE measures the divergence between model-geeeraaee
and human- written text distributions using a quantized embed- ding space and Diversity quantifies the non-repetitive nature of generated content through n-gram statistics. Further details on the evaluation metrics are provided in Appendix [ For decoding, we use greedy search, which selects the pighest-probability span each step, and nucleus sampling, which samples the next span from the normalized probability distribution with top-p set to 0.95 (Holtzman) a..).
4.1 IN DOMAIN EVALUATION
For in-domain evaluation, models are trained on the training set of the WikiText-103 (Merity et al. 2016) dataset and evaluated on its test set. The WikiText-103 dataset is a large-scale corpus com- prising 1,801,350 training samples, 3,760 development samples, and 4,358 test samples derived from Wikipedia articles. which serves as a standard benchmark for assessing language modeling performance (Su et al. 2022
Li al. 2024).
Table presents the performance comparison between the baselines and our proposed FoSS on the WikiTeext1033 test set. Our FoSS achieves MAUVE score absolute improvements of 5.51 and 8.22 over CoG and fine-tuned Transformer with nucleus sampling, respectively. This improvement can be attributed GFlowNet's ability model DAG-structured internal state space and generalize better. Meanwhile, FoSS outperforms GFlow.eet--F by 2.04. highlighting the advantage of our DAG-structured state space over the tree-structured approach used GFlowNets-P which enables more comprehensive exploration of state transitions. Notably, despite the established tendency of greedy search to produce degeneration problems (Welleck et al.] 2020), FoSS with greedy search outperforms both the fine-tuned Transformer and the CoGi with nucleus sampling by 7.35 and 4.64 in MAUVE, respectively.
To further evaluate the quality of generated text, we employ GPT-4 to assess the continuations generated by the models for each test prefix. Prior work has shown that powerful LLMs, such as GPT-4, capable of matching human preferences well, achieving over 80% agreement, which is comparable to the agreement between humans (Zheng et al., 2023). Our evaluation focuses on key aspects of text quality: fluency, coherence, informativeness, and grammatical correctness. Further details can be found in the Appendix G As shown in Table the GPT-4 evaluation results reveal that
our FoSS model consistently outperforms all baselines across all quality dimensions.
4.2 OUT OF DOMAIN EVALUATION
In the domain adaptation setting, the models trained on the WikiText- 103 dataset are tested on different domain. Following previous work(Liu et al., 2024 Cao et al. 2024), we use the English portion of Law-MT (Koehn & Knowles 2017). which is English-Germa translation dataset in the legal domain containing 389,292 training samples with 2,000 samples each for development and testing purposes. The memory of kNN-LAN -LM, RETRO, CoG, GDV and FoSS are constructed from the training set of Law-MT. Transformer w/o is fine-tuned on the WikiText103 training set, and Transformer w/ ft is additionally trained the Law-MT training set.
7

---

Published as a conference paper at ICLR 2026
34
12
Tresiheman
32
H4: Traniforent
MAN sc00e
-0
MAUNEE 4mme
MAUNE
1.0
28
20
25
25
24
0.001
03,03
0.01
0.03
0.1
0.3
1.0
0.47
0.54
3.60
3.75
7.5
15
100M
20004
4.035
500M
3600M
The pwwportttoonoooommmm En-WWld dettaee
The propetton (27loooNete training date (S.)
Sondel
Figure Generation quality of Figure 3: Generation quality of Figure Generation quality FoSS with different sizes of the FoSS with different sizes of of- of FoSS with different sizes of span index.
fline data trained.
models.
As shown in Table 1 FoSS consistently outperforms all baseline methods, including the Law-MT fine-tuned Transformer. Specifically, FoSS achieves MAUVE score improvements of 6.96 and 4.03 over the fine-tuned Transformer and CoG, respectively. These results demonstrate that FoSS effectively adapts new domains by simply updating the source text collection without requiring domain-specific training. The superior performance can be attributed to the generalization capabilities of GFlowNets, which enable more effective knowledge transfer across domains.
Additionally, conducted GPT-4 evaluation under this setting, which follows similar procedure as outlined Section 4.1 As presented Table FoSS-generated continuations were preferred in 56% and 74% of cases when compared to the fine-tuned Transformer and CoG, respectively. These human-aligme aligned evaluation results demonstrate the effectiveness FoSS in cross-domain generation scenarios without requiring domain-speciie parameter updates.
4.3 SCALING EVALUATION
In this section, we evaluate the scaling properties of FoSS across three dimensions: memory size, training data volume, and model scale. For memory scaling, we constructed retrieval datastores for KNN-LM, RETRO, CoG, and FoSS from the En-Wik corpus and evaluated on the WikiText- 103 test set. set. The Transformer w/o FT was trained solely on WikiText-103, while Transformer FT was additionally trained on the En-Wiki.
As shown Table I FoSS with the En- Wiki memory consistently outperforms other strong baselines, including FoSS with the smaller WikiText- 103 memory. This demonstrates the plug-and-play capability of FoSS, which can leverage larger memory resources for performance gains without requiring additional training. To further investigate the impact of memory size generation quality, we created subsets of En- En-Wiki at varying scales to serve as the retrieval datastore. FigureZiilusstn illusstaats that as increase the FoSS memory size, text generation quality consistently improves. For comparative purposes, we use Transformer w/o FT as the baseline, since the fine-tuned Transformer exhibited performance degradation when trained on the additional En-Wiki data.
To assess the impact of training data volume on FoSS performance, we sampled subsets of the WikiText-103 training set at different scales and trained both models accordingly, evaluating them on the test set. As illustrated in Figure 3 FoSS outperforms fully-trained CoG and Transformer models even when trained on an extremely limited data fraction (0.47%). Furthermore, FoSS demonstrates robust scaling behavior, with performance consistently improving as training data volume increases. This finding highlights the sample efficiency of our approach and suggests that FoSS effectively leverages the compositional nature of the DAG-structured state space to generalize from limited examples, while maintaining strong scaling properties with additional training data.
Finally, to evaluate model scaling effects, we initialized the prefix encoder with progressively larger versions of GPT-2: GPT-2 (base), GPT-2-Medium. GPT-2-Large. and GPT-2-XL. All models were trained on WikiText- 103 and evaluated its test Figure 4 shows that text generation quality consistently improves as we scale up the model size, confirming that FoSS effectively leverages increased model capacity.
8

---

Published as a conference paper at ICLR 2026
Table Accuracy of Different Methods on Knowledge-Intensiv Tasks
Method
TruthfulQA OpenhookQA ARC-Challenge MedMCQA Med-USMLE
Transformer (w/o FT)
28.02
22.67
23.82
26.70
24.89
Transformer (w FT)
28.76
22.71
24.00
26.95
24.15
CoG
29.38
24.29
24.34
27.55
25.06
FoSS
30.45
26.20
24.63
27.44
25.27
4.4 DOWNSTREAM EVALUATION
Consistent with prior research (Sanh al. 2021 Brown et al. 2020), we adopt classification- with-options approach to evaluate model performance. For assessment, we employ five diverse knowledge-intesd -itteddgg--tteevveeeeeee intensive datasets: TruthfulQA (Lin et al. 2022 for evaluating factual aceuracy, Open- BookQA [Mihaylov et al.) 2018) and ARC-Challenge (Clark et al. (018)) for testing scientific reasoning at different complexity levels, and MedMCQA (Pal al. 2022 and Med-USMLE (Jin et al. 2021) for assessing specialized medical knowledge.
In this framework, the model is presented with set of candidate options, and the likelihood of each option being the correct answer is computed. The option with the highest probability selected as the model's prediction. We then report the accuracy of the model's predictions. To calculate the likelihood of a given text, we approximate the likelihood by summing over all possible generation 2024).
paths using dynamic programming (Cao
Table 3 presents the accuracy comparisons between the baselines and our proposed FoSS on five question-answeriigg datasets. The Transformer w/ FT baseline was fine-tuned on the WikiText-103 dataset. Our results show that the proposed FoSS consistently outperforms the haselines across most datasets. Specifically, compared to the standard Transformer FT, our model improves accuracy on the TruthfulQA and OpenBookQA datasets, increasing from 28.76% to 30.45% and from 22.71% 71% to 26. 26,20% respectively. While CoG shows competitive performance on MedMCQA, our method still achieves the best overall performance across the majority of the benchmarks.
4.5 ABLATION STUDY
Table 4: Ablation study on key components of FoSS: DAG structure, preference model (PM), and language model (LM) rewards.
In Domain
Out of Domain
Scaling Data Store
Method
Mauve
Diversity
Mauve
Diversity
Mauve
Diversity
FoSS w/o DAG
29.61
65.72
28.62
81.44
FoSS w/o PM
28.25
89.91
30.49
92.53
29.81
91.42
FoSS w/o LM
29.09
92.77
29.79
94,72
29.53
92.46
FOSS
31.65
92.48
32.17
93.00
33.79
92.51
To validate the effectiveness of modeling G as general DAG rather than tree, we conducted an experiment where we removed all phrases from the vocabulary during training, ensuring that G' forms tree structure. As shown in Table 4 the results demonstrate that FoSS significantly outperforms the tree-structured variant across evaluation settings. Specifically, the DAG-based model achieves 9.65% relative improvement in MAUVE score and 27. 27.46% increase in Diversity score across different settings. This indicates that the DAG structure allows for more diverse and flexible generation paths, enabling better exploration of the generation space and mitigating the limitations posed by tree-structured generation, which restricts the ability to explore diverse phrase combinations.
To assess the impact of different reward components, we conducted ablation experiments by separately removing the LM and PM from Equation 3 As demonstrated in Table 4 removing either component

---

Published as a conference paper at ICLR 2026
leads to decreased MAUVE scores across all settings, indicating that both components essential and complementary. It is worth noting that the PM-only variant (FoSS w/o LM) achieves the highest Diversity scores in both in-domain and out-of-domain settings, suggesting that PM inherently favors text with greater diversity. This aligns with PM's objective to differentiate human-written text, which typically exhibits richer diversity from model-generated texts (Zhang al. 2025)
5 CONCLUSION
In this work, we construct dynamic span-vocabulary for text generation, framing the state space as DAG. We then model the span generation within the GFlowNets framework, unleashing the potential of GFlowNets in language models within the DAG-structured space. Our proposed FoSS excels in text generation, domain adaptation, and knowledge-intensiig tasks, consistently outperforming strong baseline models. The scaling experiment further demonstrated that FoSS consistently improves in performance with increases in model parameters, training data, and retrieval data, validating its scalability and potential with greater computational budgets.
ACKNOWLEDGMENTS
This work is sponsored by the National Natural Science Foundation of China (NSFC) grant (No.62525209, 62576211).
REFERENCES
Lazar Atanackovic and Emmanuel Bengio. Investigating generalization behaviours of generative flow networks. In ICML 2024 Workshop on Structured Probabilistic Inference {{&} Generative Modeling, 2024.
Emmanuel Bengio. Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381-27394 2021.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. Journal of Machine Learning Research, 24(210):1-55, 2023.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmaann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 2206-2240. PMLR, 2022.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345 1952.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot leamners. Advances in neural information processing systems, 33:1877-1901, 2020.
Bowen Cao, Deng Cai, Leyang Cui, Xuxin Cheng, Wei Bi. Yuexian Zou, and Shuming Shi. Retrieval is accurate generation. The Twelfth International Conference on Learning Representations, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. 2018.
Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks. In Uncertainty in Artificial Intelligence, pp. 518-528. PMLR, 2022.
10

---

Published as a conference paper at ICLR 2026
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Ming-Wei
Razvan-Gabriel Dumitru, Minglai Yang, Vikas Yadav, and Mihai Surdeanu. Copyspec: Accel- erating Ilms with speculative copy-and-paste without compromising quality, arXiv preprini arXiv:2502.08923, 2025.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, al. Helping herd- ing? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint wrXiv:2312.09244, 2023.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International conference on machine learning, pp. 3061-3071. PMLR, 2020.
Jinlan Fu, See Kiong Ng. Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 6556 6576, 2024.
Giorgio Giannone, Neil Tenenholtz, James Hall, Nicolo Fusi, and David Alvarez-Melis, Enhancing language models for technical domains with dynamic token injection. NeurlPS 2023 Generative Al and Biology (GenBio) Workshop.
Timofei Gritsaev, Nikita Morozov, Sergey Samsonov, and Danii Tiapkin. Optimizing backward policies in gflownets via trajectory likelihood maximization. In The Thirleenth International Conference on Learning Representations, 2025.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. International Conference on Learning Representations.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Edward J Hu, Nikolay Malkin, Moksh Jain, Katie E Everett, Alexandros Graikos, and Yoshua Bengio. Gflownet-em for learning compositional latent variable models. In International Conference on Machine Learning, pp. 13528-1155 13549.. PMLR, 2023.
Edward J Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference large language models. In The Twelfih International Conference Learning Representations, 2024.
Yusuke Ide, Yuto Nishida, Justin Vasselli, Miyu Oba, Yusuke Sakai, Hidetaka Kamigaito, and Taro Watanabe. How to make the most of Ilms' grammatical knowledge for acceptability judgments. arXiv preprint (:X.x:2408.096399 2024.
Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garvia, Jarrid BrotorrBrroos, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference Machine Learning, pp. 9786-9801. PMLR, 2022.
Hyosoon Jang. Yunhui Jang, Minsu Kim, Jinkyoo Park, and Sungsoo Ahn. Pessimistic backward policy for gflownets. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang. and Peter Szolovits. What disease does this patient have? large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scall similarity search with gpus. JEEE Transactions Big Data, 7(3):535-5477 2019.
11

---

Published as a conference paper at ICLR 2026
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pp. 6769-6768887 2020.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference Learning Representations, 2020.
Hyconah Kim, Minsu Kim, Tacyoung Yun, Sanghyeok Choi, Emmanuel Bengio, Alex Hernández- García, and Jinkyoo Park. Improved off-policy reinforcement learning in biological sequence design. In NeurlPS 2024 Workshop Al for New Drug Modalities.
Philipp Koehn and Rebecca Knowles. Six challenges for neural machine translation. Proceedings of the First Workshop Neural Machine Translation, pp. 28-39-8999 2017.
Michal Koziarski, Andrei Rekesh, Dmytro Shevchuk, Almer van der Sloot, Piotr Gaiński, Yoshua Bengio, Chenghao Liu, Mike Tyers, and Robert Batey. Rgfn: Synthesizable molecular generation using gflownets. Advances in Neural Information Processing Systems, 37:46908-46955, 2024.
Anas Krichel, Nikolay Malkin, Salem Lahlou, and Yoshua Bengio. On generalization for generative flow networks. arXiv preprint arXiv:2407.03105, 2024.
Tian Lan, Deng Cai, Yan Wang, Heyan Huang, and Xian-Ling Mao. Copy is all you need. In The Eleventh International Conference on Learning Representations, 2023.
Seanie Lee, Minsu Kim, Lynn Cherif, David Dobre, Juho Lee, Sung Ju Hwang, Kenji Kawaguchi, Gauthier Gidel, Yoshua Bengio, Nikolay Malkin, et al. Learning diverse attacks on large language models for robust red- red-teaming and safety tuning. In The Thirteenth International Conference Learning Representations, 2025.
Minghan Li, Xilun Chen, Ari Holtzman, Beidi Chen, Jimmy Lin, Scott Yih, and Victoria Lin. Nearest neighbor speculative decoding for I1m generation and attribution. Advances in Neural Information Processing Systems, 37:80987-81015 2024.
Sida Li, Ioana Marinescu, and Sebastian Musslick. Gfn-sr: Symbolic regression with generative flow networks. In NeurIPS 2023 Al for Science Workshop, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3214-3252, 2022.
Yanting Liu, Tao Ji, Changzhi Sun, Yuanbin Wu, and Xiaoling Wang. Generation with dynamic vocabulary. In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 18931-18948, 2024.
Stephen Zhewen Lu, Ziqing Lu, Ehsan Hajiramezanali, Tommaso Biancalani, Yoshua Bengio, Gabriele Scalia, and Michal Koziarski. Cell morphology-guide small molecule generation with gflownets. In ICML 2024 Workshop on Structured Probabilistic Inference {(& Generative Modeling, 2024.
Pouya M Ghari, Alex Tseng, Gokcen Eraslan, Romain Lopez, Tommaso Biancalani, Gabriele Scalia, and Ehsan Hajiramezanali. Gflownet assisted biological sequence editing. Advances in Neural Information Processing Systems, 111:10064111066665 2024.
Kanika Madan, Jarrid Rector-Brooks. Maksym Korablyov, Emmanuel Bengio, Moksh Jain, An- drei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability. In International Conference on Machine Learning, pp. 23467-23483. PMLR, 2023.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. Advances in Neural Information Processing Systems, 35: 5955-5967, 2022.
12

---

Published as a conference paper at ICLR 2026
Pedro Henrique Martins, Zita Marinho, and André FT Martins. Chunk- nearest neighbor machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4228 0228-4245, 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 1609.00609907843 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct clectricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 2381-2391, 2018.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pp. 248-260. PMLR, 2022.
Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of gilownets with local credit and incomplete trajectories. In International Conference on Machine Learning, pp. 26878 26890. PMLR, 2023.
Ling Pan, Moksh Jain, Kanika Madan, and Yoshua Bengio. Pre-training and fine-tuning generative flow networks. In The Twelfth International Conference on Learning Representations, 2024.
Mohit Pandey, Gopeshh Subbaraj, Artem Cherkasov, Martin Ester, and Emmanuel Bengio. Pretraining generative flow networks with inexpensive rewards for molecular graph generation. In Forty-second International Conference on Machine Learning, 2025.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems,
2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprini arXiv:21 10.08007 2021.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1715-1725, 2016.
Max W Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving gflownet training. In International conference on machine learning, pp. 30956-30977 PMLR, 2023.
Tony Shen, Seonghwan Seo, Grayson Lee, Mohit Pandey, Jason R Smith, Artem Cherkasov, Woo Youn Kim, and Martin Ester. Tacogfn: Target-conditioned gflownet for structure- based drug design. Transactions Machine Learning Research.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. Advances in Neural Information Processing Systems, 35: 11548-21561, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Nikhil Murali Vemgal, Flaine Lau, and Doina Precup. An empirical study of the effectiveness of using replay buffer mode discovery in gflownets. In ICMI. 2023 Workshop on Structured Probabilistic Inference { Generative Modeling.
Peiyi Wang. Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, al. Large language models not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9440-9450, 2024.
13

---

Published as a conference paper at ICLR 2026
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. International Conference on Learning Representations, 2020.
Fangxu Yu, Lai Jiang, Haoqiang Kang, Shibo Hao, and Lianhui Qin. Flow of reasoning: Training Ilms for divergent reasoning with minimal examples. In Forty-second International Conference on Machine Learning, 2025.
Yiming Zhang, Harshita Diddee, Susan Holm, Hanchen Liu, Xinyue Liu, Vinay Samuel, Barry Wang. and Daphne Ippolito. Noveltybench: Evaluating language models for humanlike diversity, 2025.
URL
wtt.//gggg////////// 05228
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
14

---

Published as a conference paper at ICLR 2026
A RELATED WORK
A.1 MODELS BEYOND FIXED VOCABULARY
Recent advancements in language modeling have shifted towards dynamic vocabularies and flexible text spans to address the limitations of traditional token-by-token autoregressive generation. Retrieval- based methods have gained significant attention, with approaches like CoG-2 (Cao et al., (2024) and CoG (Lan et al.) 2023) leveraging external datastores to dynamically retrieve and assemble contextually relevant phrases. This allows for a flexible vocabulary that enhances both semantic richness and fluency in generated text. Similarly, chunk-based methods, such as the kNN machine translation model by (Martins et al. 2022), retrieve variable-length token sequences, improving translation coherence and fluency. Speculative decoding techniques, like CopySpec (Dumitru et al., 2025), further optimize this idea by reusing repeated token sequences, which accelerates inference and reduces redundancy. Dommann-specifii enhancements, such as those proposed by (Giannone et ) dynamically inject specialist tokens into the model's vocabulary, enabling more effective handling of domain-specific knowledge without the need for extensive fine-tuning. In the realm of semi- parametric methods, NEST (Li et al., 2024) introduces text span retrieval from a corpus, allowing for more diverse and semantically coherent text generation. Additionally, (Liu et al.) (2024) propose dynamic vocabulary approach that incorporates arbitrary text spans as fundamental generation units, demonstrating improvements in both generation quality and efficiency across various tasks. Collectively, these approaches represent a paradigm shift from static token generation more flexible, context-aware methods, enabling language models to produce semantically richer and more diverse content.
However, while methods such as (Lan et al., 2023), (Cao et al. 2024), and (Liu et al. 2024) implicitly induce a DAG-structured internal state space (each state can be reached via two segmentation paths), this structural property remains unrecognized and unexploited. Consequently, existing approaches exhibit biases towards specific segmentation patterns. Moreover, thoroughly exploring and generalizing over the DAG-structured state space is inherently nontrivial, it requires exponentially growing data to sufficiently cover the combinatorial explosion of compositional paths. To address this challenge, we identify GFlowNets principled solution, enabling effective generalization and exploration in DAG-structured spaces (Shen et al. 2023 Krichel et al. 2024 Atanackovic & Bengio 2024
A.2 GENERATIVE FLOW NETWORKS
Generative Flow Networks (GFlowNets) offer framework for training stochastic policies to sample compositional objects with probabilities proportional to reward function (Bengio et al.) 2023). This unique property makes them particularly effective in applications that require generating diverse yet high-reward samples, such as biological sequence design (Jain et al.] 2022; M Ghari et al.
2024; Kim et al.) and molecule generation (Koziarski et al.) 2024 Lu et al. 2024). Recently, GFlowNets have also been successfully extended to various inference tasks, including Bayesian structure learning (Shen et al.) and variational inference with discrete latent variables (Hu al.) 2023).
In the context of language modeling, (Hu et al. (024) introduced a method to amortize intractable inference in large language models using GFlowNets. demonstrating its potential in guiding language model behavior. Subsequently. (Yu et al. 2025) utilized GFlowNets to generate diverse reasoning paths, enhancing the exploration of solutions in complex problem-solving scenarios. Additionally, recent studies have explored the use of GFlowNets for automated red-teaminn -eamieaming of large language models. For instance, (Lee et al.) 2025) proposed two-stage approach combining GelowNets fine-tuning and MI.E smoothing to generate diverse adversarial prompts, mitigating mode collapse methods.
issues typical in reinforcement learning-
The original motivation for the GFlow machinery, and its primary novelty, are in MDPs with many trajectories per state (Shen et al., 2023 Bengio et al. 202I). However, existing GFlowNets approaches integrated with LLMs remain constrained by token-level generation. Consequently, they inherently form tree-structured state spaces, which reduces the GFlowNets framework to a degraded learning paradigm that limits the exploration of state transition. In contrast, our method constructs general DAG by employing text spans as actions, allowing for multiple generation paths to the same state, thus fully leveraging the exploration and generalization potential of GelowNets.
15

---

Published as a conference paper at ICLR 2026
B DATASETS AND METRICS DETAILS
B.I DATASETS DETAILS
The open-ended text generation experiments in this paper include three benchmarks: (1) Wixi 103, large-scale corpus comprising 1,801,350 training samples, 3,760 development samples, and 4.358 test samples derived from Wikipedia articles, containing over 100 million words and widely used for evaluating universal language modeling performance; (2) the English portion of Law-MT, an English-German translation dataset in the legal domain containing 389,292 training samples with 2,000 samples each for development and testing purposes; and (3) En- an extensive corpus utilized for enlarged phrase indexing experiments, containing over 4,848,348 long English Wikipedia documents with more than 3 billion words, substantially larger than WikiText-103.
Given that Wiki serves the retrieval datastore in the scaling experiments, we examined the overlap between En- Wiki and the WikiText- 103 test set. Empirically, only 8.5 x 10 of En Wiki documents appear in the WikiText-103 test set. When the retriever queries En Wiki using prefixes from the WikiText- 103 test samples and retrieves 1,024 documents per prefix, only 8 test samples have any retrieved document containing text identical the corresponding test sample.
For knowledge-intensive evaluation tasks, we utilize several established benchmarks: (1) Open- BookQA (Mihaylov al. 2018), which evaluates understanding of scientific concepts through 5,957 multple-chooce elementary science questions (4,957 train, 500 development, 500 test) and probes 1,326 core scientific facts requiring broad commonsense knowledge we utilize its test split comprising 500 questions; (2) ARC- ARC-Challenge (Clark et al. 2018), containing 7,787 grade-school level, multiple-choice science questions across diverse topics, with its Challenge Set specifically including questions answered incorrectly by retrieval-basee and word co-occurrence algorithms- we use the test split of this Challenge Set, consisting of 1,172 questions; (3) TruthfulQA (Lin et al. 20022) benchmark designed to evaluate the truthfulness of language models, consisting of 817 multiple- choice questions across 38 categories including health, law, finance, and politics; (4) MedMCQA (Pal let al. 2022), , comprising 194,000 multiple-choice questions covering 21 medical subjects and 2,400 healthcare topics-we we use its validation split consisting of 4,183 questions; and (5) Med-USMLE (Jin) al. 2021), which comprises 12,723 multiple-choice questions with four options each, sourced from the United States Medical Licensing Examination- utilize its test split containing 1,273 questions.
B.2 METRICS DETAILS
MAUVE (Pillutla et al., 2021) measures the divergence between model-generated and human- written text distributions using a quantized embedding space. By computing the divergence between discrete distributions derived from neural embeddings, MAUVE effectively captures both quality and diversity aspects of generated text, correlating strongly with human judgments while remaining computationally efficient.
Diversity quantifies the non- non-repetitive nature of generated content through n-gram statistics. Formally defined as (1 Rep-")) )), where Rep-n represents the percentage of duplicate n-grams, this metric penalizes redundant patterns while rewarding informative and varied text. Higher Diversity scores indicate more information-rich generations with fewer repetitive structures.
C IMPLEMENTATION DETAILS
C.1 TRAINING SETUP AND HYPERPARAMETERS
The training of our proposed model was conducted on eight NVIDIA A800 GPUs, each equipped with 80GB memory. We employed the AdamW optimizer with learning rate of lc-7 and a linear learning rate schedule. To manage memory constraints, we implemented gradient accumulation with a step size of 32. For the first epoch, training relied exclusively on offline training set data. Starting from the second epoch, each step, we sampled trajectories with probability of 0.2 from the replay buffer (offline) and with a probability 0.8 from muline-generated trajectories. For the language model, we followed the configuration in (Hu al. 2024), applying LoRA (Hu al. 2022)
16

---

Published as a conference paper at ICLR 2026
for finettuning GPT-2 and BERT with the following hyperparameters: rank 64, scaling factor a = 16, and LoRA dropout rate of 0.1.
C.2 DYNAMIC VOCABULARY CONSTRUCTION
To enhance the efficiency of FoSS, we pre-encoded all documents in the source text collection offline. However, retrieving from such extensive phrase collection poses significant engineering challenges. To address this, we adopted coarse-to- pipeline, consistent with the approach in CoG. Specifically, during training, the dynamic vocabulary of FoSS comprises two components: (I) a token-level vocabulary and (2) phrases derived from the batch of training documents. augmented by the top-k documents (with k 128) that share similar topics with the prefix of the training documents. For the second component, we selected all substrings of length 2-8 tokens from the documents, ensuring that the policy network can sample any sequence in exponentially scalable manner. Notably, experiments revealed that relying solely on the batch of training documents, which were obtained through the span segmentation algorithm, resulted in poor-quality online-sampled sentences that adversely affected training stability. Additionally, during the FoSS training, we needed to segment prefixes from training samples before trajectory sampling. our experiments. we first used NLTK's sentence tokenizer to split training samples into sentences, then progressively concatenated these sentences to the prefix until length approached 32 tokens, which matches the prefix length used during testing. Due to computational resource constraints, we sampled 15% of the data for GFlowNets training. During inference, we first employed a document retriever to identify the top-k related documents for a given prefix, with k 1024. Subsequently, the corresponding phrase representations were collected for generation. In this work, we utilized a widely adopted semantic matching model, DPR (Karpukhin et al. 2020), combined with the FAISS (Johnson et al.) 2019) vector search toolkit, as the document retriever to recall documents with topics similar to the prefix.
C.3 REWARD MODEL TRAINING
For both the LM and PM components of the reward function, we utilized GPT-2 for full parameter fine-tuning. In the case of LM fine-tuning, we employed causal language modeling loss on the training dataset. For the PM fine-tune tuning, we employ Gradley-Terry objective that encourages the model to assign higher score to the preferred continuation than to its non non-preferred counterpart. We train on a dataset DPm = {(xt ; )), where x+ is the human-written references from the training set and x= the continuations produced by the initial policy. Let (PM( denote the scalar-valued score predicted by the PM. Our training objective is:
CPM -B(ut (((ut,-)~DDppp [logor o (f)M(x=))))))))))))))))))))))))) fPM f +A +X (()+,a)))DDM [(fem(xt)) + fem(x }}
where af) is the sigmoid function, m represents an optional margin hyperparameter and À controls the strength of the centering regularization. Here, we employ the score-centering regularizer (Eisenstein) ct al., 2023), adding an auxiliary loss that minimizes the squared sum of the scores. This encourages the model to produce mean-zero outputs.
D POLICY NETWORK DETAILS
Our policy network architecture comprises two primary components: prefix encoder and a span encoder. The prefix encoder processes the current state as a sequence of tokens, where previously sampled spans are tokenized. This sequence is then encoded using standard Transformer architecture with causal attention mechanisms. The representation of the prefix is derived from the final layer's hidden state corresponding to the last token in the sequence.
For the span encoder, we compute vector representations for all candidate phrases from the supporting documents (constituting T). Specifically, we extract all possible substrings of length 2-8 tokens from the supporting documents to form T, ensuring that the internal state space forms a DAG. To encode these phrases, we employ deep bidirectional Transformer (Devlin et al. 2019) to generate contextualized token representations for each supporting document. For a phrase spanning from position s e, apply separate MI.Ps transform token representations into start and end
17

---

Published as a conference paper at ICLR 2026
embeddings of dimension 2 each, then concatenate the start embedding position with the end embedding at position e to form the complete phrase representation. For single words in V and the terminal action T, utilize standard token embeddings.
E SPAN SEGMENTATION ALGORITHM
In this work, we introduce a DAG-Inducing Probabilistic Span Segmentation algorithm that generates stochastically varied segmentation trajectories for each training document, specifically designed to facilitate the offline construction DAGs over sequences. Our span segmentation algorithm draws inspiration from CoG. Building upon the standard forward maximum matching strategy, our method integrates a probabilistic early-stopping mechanism that allows phrase extraction to terminate at controlled random points, governed by set of thresholds p {po- 0. Pr 0 P: 1, Vi = 0, o.....n...
Concretely, given document, we tokenize it into sequence of tokens and iteratively scan from left to right. At each step, we attempt to find the longest token span that appears in either the document itself or its top-k retrieved nearest neighbors. a candidate span is found and its length falls within pre-specified bounds, it is selected as a phrase with probability Pr, where p- E P. If selected, we commit this candidate span as a phrase in the segmentation and reset our span search begin the next token. If not selected, we continue the segmentation process by extending the current span with one additional token, repeating this procedure until either find selectable span, reach the maximum span length limit, or the extended span no longer appears in the retrieved documents. The details of our proposed DAG-Inducing Probabilistic Span Segmentation algorithm can be found in Algorithm 2 This probabilistic termination mechanism results in multiple distinct segmentation trajectories per document, where different span paths share common sub-sequences. Crucially, by generating overlapping and diverging segmentations offline, ensure that the resulting training trajectories data naturally induce a DAG structure.
F EFFICIENCY AND DIVERSITY ANALYSIS
We compare the average time cost of different methods for completing the generation on the test set following the speed testing setup of Lan et al. (003).. The results are reported in Table As can be seen, FoSS achieves comparable inference efficiency with the standard Transformer baseline. This is because the copied phrases typically consist of multiple tokens. As result, FoSS requires fewer decoding steps to generate text of the same length. Unlike FoSS, which uses a coarse-to-fing search pipeline, kNN-LM performs large-scale vector search at every decoding step. This leads to significantly higher inference latency compared to other methods.
Regarding training efficiency, FoSS incurs additional computational costs compared to offline training methods due its online sampling mechanism, which is inherent to RI--based approaches. The most comparable baseline in terms of training paradigm is GFlowNets-FT (Hu et af.) (0244). On GPUs, FoSS requires 10.5 hours per epoch, compared to 9.25 hours for GFlowNets-FT. This modest overhead stems from the expanded vocabulary needed to explore the DAG-structured state space, which enables more comprehensive exploration of compositional paths.
Meanwhile, we also compare the diversity of the methods. It can be observed that FoSS shows improvements in diversity compared CoG under both generation strategies. with the diversity score increasing from 43.03 to 43.50, and from 89.07 to 92.48, respectively. This enhancement can be attributed the fundamental training objective of GelowNets, which learns a policy that samples from a distribution proportional to the reward function (P (z) R(x)) rather than simply maximizing the reward, which encourages exploration of diverse generation trajectories and enhances the diversity of the generated text. While the diversity of our method under nucleus sampling is slightly lower than KNN-LM, this can be attributed to differences in the inference strategy. kNN-LM conducts large-scale vector search at every decoding step, whereas FoSS performs single retrieval step during the generation process, which affects the richness of the retrieved content and subsequently impacts the overall diversity of the generated text.
18

---

Published as a conference paper at ICLR 2026
Algorithm 2: DAG- Inducing Probabilistic Phrase Segmentation
Data:
Document set: D {dz. {dz}, 1 where d, denotes the i-th document. K denotes the number of retrieved documents. N denotes the number of documents in the training set. Pre-defined maximum and minimum phrase lengths: Lmaz and Lmii- Phrase segmentation thresholds: p {po = 0, P........12 Pn. S pt <1,, Vi 0, n1..
Result: DAG-inducing segmented document set:
D' (1 {(p), (dy, pos,))
)>>!!!!!! )r }r=0)==1 iN
|14:11;"
where pi denotes the rrthh phrase in the r-th segmentation of d, appearing in document dj. Preprocess: Split each document into token-level sequences using an off-the-shelf tokenizer. Define empty result set D' = {..
for i +-++1 1 to N do for pr E do
cursor f- 0
PhraseCollection <- 0 cache, + (} labeltast False
while cursor < ||di|le do
if Linin (en(ccche
Imaz then
random () 2 p, then
label rest +- SearchPhrase (cachep)
else
labelnow- rest +- False, {
else
if len(cache > Liaaz then cache, 5
if labeltast = True and label False then cursor f- cursor -1 currasCollectionaaeenddcccc, rest) cache, {}
else
if labellast False and labelnow False then Crrraseoollcctioaaapeeeeeee cache, 0
None)
cursor f- cursor +1
labeltast label
D' (oppennc((rasseoClleetin)
Table 5: Comparison of Diversity and Latency Among Methods for In Domain Generation
Greedy
Nucleus
Method
Diversityf
Latency(s)
Diversityf
Latency(s)
Transformer
22.37
1.32
93.22
1.48
KNN-LM
22.13
10.36
95,80
10.42
RETRO
21.19
4.39
91.19
4.51
CoG
43.03
1.29
89.07
1.54
FoSS
43.50
1.29
92.48
1.51
19

---

Published as a conference paper at ICLR 2026
Case Prefix
The production selected two adjacent properties on the Warner backlot's 'Blondie Street' for the Burnham and Fitts' homes. The crew
Generation of FoSS
continued filming at the same location over several years, aiming to maintain continuity for scenes set at the school. The producers noted that filming would need several years to complete and that it was casier to edit the film and prepare it for the same location used for other scenes in the film.
Generation of Transformer w/ FT
moved there in 1950, using three yards of undeveloped land Hawthorne Street between 9 and 10 Westheimer Street ( the site of the National Training Center at that time) ), district section of South Main Street between 14th and 15th Streets, and the neighborhood North College Street opposite Latimer streets.
Figure 5: Case Study. The blue part represents directly sampling phrase.
FoSS
x
CoG
Transformer
32
MAUVE score
30
28
26
24
124M
355M
774M Model Size
1.5B
Figure 6: Generation quality FoSS with different sizes of models.
G GPT4 EVALUATION
Due to GPT-4's sensitivity to the order of candidate sentences (Wang et al.) 2024), we adopt the method described in (Wang et al. 2024 Liu al., 2024). computing the final result by averaging outcomes from pairs of evaluations with reversed presentation order. Figure 7 shows the detailed prompt used for GPT-4, which we adopt from (Liu et al., 2024). Specifically, for each triplet (prefix, generation_1, generation.2), generation 1), in order to mitigate any impact that the order of the two generations might have on GPT-4's evaluation.
include another corresponding triplet (prefix, generation 22,
20

---

Published as a conference paper at ICLR 2026
System:
You area helpful and precise assistant for checking the quality of the text.
User: [Prefix]
[Prefix of the Test Sample]
[The Start of Assistant 1's Generation]
[Generation_]]
[The Assistant 1's Generation]
(The Start of Assistant 2's Generation]
[Generation_2 2]
(The End of Assistant 2's Generation]
[System]
We would like to request your feedback on the performance of two Al assistants in response the user prefix displayed ahove. Please rate the fluency, coherence, informativeness, and grammar. Each assistant receives overall indicates better overall performance. Please first provide comprehensive explanation your evaluation, avoiding any potential bias and ensuring the order in which the responses were presented does not affect your judgment. Then, output two lines indicating the scores for Assistant and 2, respectively.
scale of 1 10, where higher score
Your response should be in the following **json** format: "Json
EEvalattion evidence": "your evaluation explanation here",
Sccoee of the Assistant 1": "score", "Score the Assistant "score"
Figure The GPT-4 evaluation template with three slot (Prefix of the Test Sample), [Generation_]]] [Generation.2]
H CASE STUDY
Figure presents a comparative example between FoSS and fine-tuned GPT2 model. Given an identical film production prefix, FoSS generates continuation that maintains thematic coherence, discussing filming logistics, continuity considerations, and production planning. The text flows naturally from the established context about Warner's backlot filming. In contrast, the GPT2 model produces text that, while grammatically correct, diverges semantically with abrupt shifts to specific street locations and unrelated elements like "National Training Center" that lack connection to the film production theme.
I
LIMITATIONS
One limitation of FoSS is its design to perform single retrieval operation per prefix, in contrast to token-level retrieval strategies (e.g., kNN-LM). While this approach yields substantial gains in infer- ence latency (as detailed in Table the comparatively lower generation diversity observed against such baselines an anticipated trade-off. Future work could explore augmenting retrieval frequency potentially enhance diversity and overall performance; however, this direction is orthogonal the primary methodological contributions of this study and is thus reserved for subsequent investigation.
J THE USE OF LLMS
We used LLMs for minor language polishing to improve clarity and readability. Additionally, we used GPT-4 evaluate the quality of gener-eeee texts, as described the Experiments section.
21


Reviews:
[{"review_id": "PLMxDQ5tO2", "reviewer": "Reviewer_JUcH", "summary": "The paper introduces FoSS, a span-based language modeling framework that builds an explicit DAG over segmentations and learns a GFlowNet policy to traverse it. Concretely, span actions (length 2–8 tokens) are drawn from a dynamic, retrieval-augmented vocabulary; training uses subtrajectory balance with a hybrid online/offline regimen, a uniform backward policy over suffixes, and a composite reward combining a fine-tuned LM likelihood and a preference model. On open-ended generation (WikiText-103), FoSS improves MAUVE over strong baselines, is preferred by GPT-4 on multiple criteria, and shows gains on several knowledge-intensive multiple-choice tasks.", "questions": "* How many training tokens and GPU-hours are used for FoSS vs. each baseline?\n* Why do spans have lengths of 2–8 tokens? What motivated this design choice?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Casting span generation as an explicit DAG and optimizing it with a GFlowNet is a clean way to expose multiple compositional paths, addressing a known bias of tree-structured token decoders. I find the core idea both interesting and promising.\n\n- The experiments are comprehensive and the results are highly positive. FoSS improves MAUVE both in-domain and out-of-domain (Table 1), shows positive GPT-4 preferences (Table 2), and is competitive on QA tasks (Table 3). The ablation study showing a drop in performance without the DAG structure supports the central claim that “many-paths-per-state” matters (Table 4).", "weaknesses": "- Section 4.3 (“Scaling behavior”) is somewhat misleading. In Figures 2 and 3, the x-axis is rendered with equal spacing while the labels are logarithmically spaced (0.001 → 1.0 with tripling and 0.47 → 15 with doubling). Please use a logarithmic x-axis and plot against numeric proportions to avoid misrepresenting the trends. Figure 4 should use the number of parameters as the x-axis instead of model names. The sizes for GPT-2 Small, Medium, Large, and XL are 124M, 355M, 774M, and 1.5B, respectively. Once corrected, the diminishing returns with model scaling appear quite strong, so I am not sure this method would truly generalize to larger models.\n- The En-Wiki datastore (used for WikiText-103 evaluation) likely contains near-duplicates of test articles. Even if baselines share the same memory, absolute improvements can be inflated. Please filter out any pages overlapping with the WikiText-103 test set, or at least check and report overlap statistics.\n- Although you cite a paper claiming that strong LLM-based evaluations mostly align with human judgments, other work suggests this alignment can be dataset-dependent (https://aclanthology.org/2025.acl-short.20/). The reliability of LLM-based evaluation remains debatable. A small human study (e.g., 100 pairs) would strengthen the conclusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HOAXtINDly", "reviewer": "Reviewer_4EKG", "summary": "The paper describes an approach to generating text, Flow of SpanS (FoSS), that involves using multi-token spans of existing text.  This situates it with some other recent work that also uses multi-token spans like e.g. CoG retrieving and assembling contextually relevant phrases; it differs in that it recognizes that this corresponds to a DAG-structured internal state space, which then permits the application of GFlowNets as a way of working with this structure.  This also situates it with respect to other work that uses GFlowNets, but this other work typically is applied just to single token-level generation.  Evaluations are carried out with respect to several appropriate text generation baselines under the MAUVE and Diversity metrics, showing strong performance by FoSS.  There are also evaluations on downstream tasks and ablation studies, as well as explorations of scaling.", "questions": "No questions.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* Overall, I think this is a great paper.  The idea is a very neat one, the sort that comes from recognizing the structure of data and realizing what this implies for methods to apply.\n\n* The experiments are quite comprehensive, and show the effectiveness of FoSS, with e.g. quite large improvements in MAUVE scores (Table 1) and strong preferences for FoSS text (Table 2).  Beyond the values of the metrics, the case study in Fig 5 in the appendix gives a feel for how much better the text generated by FoSS can be.  I could see this approach being widely used.  (The latency analysis of App F is also encouraging as to the practicality of using the method.)\n\n* There are several insights throughout the paper as well, as for example in the ablation discussion on the role of the PM component in Eqn (3).", "weaknesses": "Nothing at all major.\n\n* There are a few places where some additional remarks on setup could help.  For instance, there’s no discussion of greedy vs nucleus in terms of setup, what they might tell us, etc; the numbers are just presented in Table 1.  Table captions in general could be a bit more informative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1WX6mzTZ53", "reviewer": "Reviewer_WyH2", "summary": "This paper proposes a novel span-based text generation framework using GFlowNets. The vocabulary is extended to include multi-word phrases, and the decoding process is reformulated as a Directed Acyclic Graph (DAG) problem. Extensive analyses and experiments are conducted to demonstrate the effectiveness of the proposed approach.", "questions": "N.A.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper connects well with relevant literature. Each component of the proposed FoSS model is discussed in relation to previous studies, making the distinctions from existing work clear.\n\n2.While the method combines ideas from GFlowNets and dynamic span vocabularies (neither originally introduced by the authors), it tackles non-trivial challenges in integrating these concepts. The authors address these challenges in a thoughtful and technically sound manner.\n\n3.The paper provides detailed explanations of each component and implementation aspect, ensuring reproducibility and transparency.\n\n4.The experimental section is extensive, covering in-domain, out-of-domain, and downstream evaluations that collectively validate the proposed approach.", "weaknesses": "1.Lack of comparison with more recent and important baselines. As mentioned by the authors in the related work, two recent and important studies on dynamic vocabularies [1][2] were not included in the main experimental comparisons.\n\n2.For scalability, the largest model evaluated in the scalability experiments is GPT2-XL (~1.5B parameters). It would be valuable to test the approach on larger and more recent models such as LLaMA 3 or Qwen 3. If direct training is infeasible due to resource constraints, a cross-model comparison (e.g., FoSS on GPT2-XL vs. vanilla LLaMA 3/Qwen 3) could still provide useful insights.\n\n3.Although inference-time latency is reported, the paper lacks discussion of the training overhead compared to other baselines.\n\n[1] RETRIEVAL IS ACCURATE GENERATION, ICLR 2024\n\n[2] Nearest Neighbor Speculative Decoding for LLMGeneration and Attribution, NeurIPS 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "No5aET3tAv", "reviewer": "Reviewer_z7qh", "summary": "This paper proposes FoSS (Flow of SpanS), a span-based language generation framework that models generation as trajectories on a DAG (instead of the usual token-level tree) and trains a span-generating policy with Generative Flow Networks (GFlowNets). The authors introduce a DAG-inducing probabilistic span segmentation to produce multiple segmentation trajectories per sentence, construct a dynamic span vocabulary from retrieved documents, and instantiate a span policy (prefix encoder + span encoder) trained with a subtrajectory-balance GFlowNet objective. A combined reward (fine-tuned LM likelihood + a preference model) steers generation toward fluent, human-like outputs. Empirically, FoSS reports substantial MAUVE gains over strong baselines (CoG, kNN-LM, RETRO, Transformer variants) and wins in GPT-4 preference comparisons; ablations show the DAG design and reward components are important.", "questions": "1. Segmentation sensitivity: How sensitive are results to the choice of span length bounds (2–8 tokens) and the probability schedule P used in DAG-inducing segmentation? Can you provide an ablation or plot showing MAUVE / Diversity vs. max span length and vs. different stochastic early-stop rates?\n\n2. Reward hyperparameters & robustness: How was α (tradeoff between LM and PM) selected? Do generated distributions sharply change for small α perturbations? Have you observed reward-hacking behaviors (e.g., overly short/high-likelihood degenerate outputs) and how were they mitigated? Please report any experiments on robustness of PM (overfitting to training references, ensemble PMs, or rejection sampling). \n\n3. Computational and latency accounting: Could you provide concrete training GPU-hour numbers and inference latency breakdowns (per prefix average, top-k retrieval, span scoring) compared to each baseline (Transformer, CoG, kNN-LM)? Table 5 is mentioned but not visible in my copy — please expand on it.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Clear methodological novelty — DAG + GFlowNets for spans.\nModeling span selection as actions yields a DAG where the same final string can be composed via multiple span paths; using GFlowNets to learn a forward policy which samples terminal sequences with probability proportional to reward is a principled fit for such DAG-structured compositional spaces. This is a clean conceptual advance over token-level GFlowNet applications. \n\n\n\n2. Practical span segmentation and dynamic vocabulary.\nThe DAG-inducing probabilistic span segmentation (probabilistic early stopping on forward maximum matching) creates multiple offline trajectories per document and makes the DAG explicit for training. The dynamic vocabulary construction (token vocab + substrings from retrieved documents, 2–8 tokens) is a sensible engineering choice to make the action space manageable while expressive.\n\n3. Thoughtful hybrid training recipe.\nThe hybrid online/offline batching (policy samples + reward-prioritized replay + training-derived trajectories) and pre-fine-tuning of policy components to stabilize sparse-reward GFlowNet training are appropriate design choices for the large combinatorial space. Algorithm 1 clearly describes the training loop. \n\n\n4. Extensive empirical evaluation.\nThe paper reports (i) MAUVE and Diversity on WikiText-103 and Law-MT, (ii) GPT-4 preference evaluations, (iii) downstream knowledge tasks (TruthfulQA, OpenBookQA, ARC, MedMCQA, Med-USMLE), (iv) scaling experiments (memory, training data fraction, model size), and (v) ablations showing the DAG and reward components matter. The breadth of evaluation supports the central claims.", "weaknesses": "1. Reward design — potential for reward hacking / calibration issues.\nThe reward R = exp(α log p_LM + (1−α) log p_PM) combines LM likelihood and a learned preference model. Preference models can be brittle and susceptible to distributional shift / reward hacking; the paper briefly mentions score-centering regularizer but lacks a systematic analysis of sensitivity to α, PM training data composition, or adversarial failures. More diagnostics would strengthen confidence. \n\n2. Computational cost / scalability concerns not fully quantified.\nTraining uses eight A800 GPUs with gradient accumulation and LoRA; the span index and retrieval (up to k=1024 at inference) plus GFlowNet trajectory sampling could be expensive. The latency table (Table 5) is referenced, but details are not shown in the snippet; a clearer cost vs. baseline comparison (GPU hours, wall-time, memory) is needed to judge practicality at scale. \n\n3. Single retrieval per prefix — diversity tradeoff.\nThe authors acknowledge FoSS does only one retrieval per prefix (vs. token-level retrieval in kNN-LM), which reduces latency but may limit diversity; while FoSS improves MAUVE/diversity over many baselines, kNN-LM sometimes shows higher diversity under nucleus sampling. The paper states this as a limitation but does not explore hybrid retrieval-frequency regimes. \n\n4. Ablation could be deeper.\nAblations remove DAG (only tokens), LM, or PM, but the paper doesn’t show results for: (a) different max span lengths, (b) varying the retrieval k, (c) different segmentation probability schedules P (the stochastic early-stop thresholds), or (d) alternatives to the uniform backward policy (they chose uniform suffix PB). These choices could materially affect performance and stability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)