Amortizing intractable inference in large language models
Authors: Edward J Hu
Moksh Jain
Eric Elmoznino
Younesse Kaddar
Guillaume Lajoie
Yoshua Bengio
Nikolay Malkin
Venue: ICLR 2024 oral (2024)
Keywords: large language models
LLMs
Bayesian inference
chain-of-thought reasoning
latent variable models
generative flow networks
GFlowNets
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We fine-tune LLMs to sample from intractable posteriors for tasks such as infilling, chain-of-thought reasoning, and tool-augmented inference.

Abstract: Autoregressive large language models (LLMs) compress knowledge from their training data through next-token conditional distributions. This limits tractable querying of this knowledge to start-to-end autoregressive sampling. However, many tasks of interest---including sequence continuation, infilling, and other forms of constrained generation---involve sampling from intractable posterior distributions. We address this limitation by using amortized Bayesian inference to sample from these intractable posteriors. Such amortization is algorithmically achieved by fine-tuning LLMs via diversity-seeking reinforcement learning algorithms: generative flow networks (GFlowNets). We empirically demonstrate that this distribution-matching paradigm of LLM fine-tuning can serve as an effective alternative to maximum-likelihood training and reward-maximizing policy optimization. As an important application, we interpret chain-of-thought reasoning as a latent variable modeling problem and demonstrate that our approach enables data-efficient adaptation of LLMs to tasks that require multi-step rationalization and tool use.

Published as a conference paper at ICLR 2024
AMORTIZING INTRACTABLE INFERENCE IN LARGE LANGUAGE MODELS
Edward J. Hu", Moksh Jain', Eric Elmoznino Mila Quebec Al Institute, Université de Montréal Becward.hu,/e moksh. jain, eric. elmoznino,
Younesse Kaddar
University of Oxford
younesse kaddargench. Kounesse.kaddaareecchi. acc
Guillaume Lajoie', Yoshua Bengio',
Nikolay Malkin
Mila Quebec Al Institute, Université de Montréal
guilluuieee lajoie, yoshua bengio, nikolay wal.aa...lkkiiii www.maaaaiiimimmmmm queee
ABSTRACT
Autoregressive large language models (LLMs) compress knowledge from their training data through next-token conditional distributions. This limits tractable querying of this knowledge to start-lo-end autoregressive sampling. However, many tasks of interest-includin sequence continuation, infilling, and other forms of constrained generation-involve involle sampling from intractable posterior distributions. We address this limitation by using amortized Bayesian inference to sample from these intractable posteriors. Such amortization is algorithmically achieved by fine- tuning LLMs via diversity-sceking reinforcement learning algorithms: generative flow networks (GFlowNets). We empirically demonstrate that this distribution- matching paradigm of LLM fine-tuning can serve as an effective alternative to Mekimum-llkeelhenn training and wawarddmaximiziig policy optimization. As an important application, we interpret chain-of tftouuggttthhtttt reasoning as a latent variable modeling problem and demonstrate that our approach enables data-efficient adapta- tion of L.MMs tasks that require multi-step rationalization and tool use. Code: https: / / gi thub. com/ GFN GgNorg/gEn. /gin- Im- tuning.
INTRODUCTION
Autoregressive large language models (LLMs) trained on general-domain data are vast stores of world knowledge (Petroni et al., 2019). They are typically optimized by predicting token given its preceding context; therefore, tractable inference over this knowledge is limited to sampling conditioned on a prefix. Many useful tasks, such as infilling (7hu et al., 2019; Liu et al., 2019), generating text conditioned on length or lexical constraints (Hokamp & Liu, 2017; Hu et al., 2019), and finding the most likely sequence continuation, involve intractable inference LLMs.
Such tasks are related to the problem of reasoning, which has been framed as one of probabilistic inference (Gershman & Goodman, 2014). Correspondingly, the linguistic expression of reasoning can be seen as inference over language. For example, we can interpret chain-of -f-thuuhttt reasoning (Wei et al., 2022; Kojima al., 2022), paradigm of reasoning in language models, as a problem of intractable posterior inference. Given a dnestiin-answer pair (X, Y), we are interested in finding latent chains of thought token sequences Z that contribute the most to the conditional likelihood
p( X) = Σ PLM(ZY X) =
Σ PLM(Y (2))LM(2 X),
where PIM denotes the likelihood assigned to a sequence by a language model and apposition of variables (e.g., XZY) denotes the concatenation of the token sequences.
While past work has relied on prompting and in-context learning to produce Z's that lead to the correct Y, treating Z as hidden variable in latent variable model (LVM) renders thain-of-thoughe reasoning a Bayesian inference problem (Fig. 1). For this LVM, the distribution we must sample Pim(XZY) Σz P) Such sampling is intractable: while it is easy
from is the posterior (Z (Z X) to evaluate PLM( (XZY), the conditional distributions needed to sample Z from PLM(Z (Z) X, Y) one token time are not easy compute.
* Equul contribution. o0 Work done during internship Mila. 'CIFAR AI Chair. CCIAAR Senior Fellow.

---

Published as a conference paper at ICLR 2024
She caught and ate a mouse. / She meowed until she was fed. /
The cat was hungry.
Now the cal is sleepy, not hungry.
GFlowNet (as fine-tuned LM)
The review expresses a personal opinion.
eval(3 4) 7.eval (7- 8)
9amw (.)X)
reward R(Z.X.Y)
"A deeply moving storyline."
Label: Subjective
8
Answer: -1
Three problems of reasoning in language- - sentence infilling, chain-of-thought reasoning, and problem-solving with external tool use can all he seen as instances of the latent variable model at the top left, where an input (X) generates the output (Y) via latent variable (Z). Right: We fine-tune an LLM to sample from the Bayesian posterior over Z, conditioning on X and optionally Y. If conditioned on Y, the trained policy can be used to sample diverse latent sequences (e.g.. for infilling, $4.2). If not conditioned on Y, the policy can sample Z. and thus predict Y. for inputs X not seen during training (e.g., for classification and multi-step reasoning, $4.3, 4.4). As shown in $4.4, modeling the full diversity of the posterior aids generalization.
Figure 1: Left:
A standard method to sample approximately from intractable posterior distributions is Markoy chain Monte Carlo (MCMC), but it is difficult to craft good proposal distributions for multi-modal distributions over language data (Miao et al., 2019; Zhang et al., 2020a; Lew et al., 2023), and inference on new input may be prohibitively slow. Alternatively, one can turn to reinforcement learning (RL) approaches such as proximal policy optimization (PPO; Schulman et al., 2017), where the language model is treated as policy to be fine-tuned. However, these do not aim to model the full diversity of the distribution; instead, learned policies settle around a small number of modes. In both cases, issues with this mode collapse are exacerbated when the target distribution is misspecified, leading to the undesirable behavior of overoptimized samplers (Gao et al., 2023).
Amortized probabilistic inference - that is, training a model to approximate a distribution of interest - provides a principled, efficient, and potentially scalable way to draw samples from the distribution (Beal, 2003). One way to implement amortized inference for high-dimensional discrete data such as text is using generative flow networks (GFlowNets. Bengio et al., 2021), which are diversity-seeking reinforcement learning algorithms that train policies to sample objects (such as a token sequence Z.) with probability proportional to given reward function, such as the joint PLM(XXY)
In this work, we present method that initializes the GFlowNet policy with a pretrained LLM and continues to train it with a reward objective that can be evaluated with the same LLM. The result is a different type of fine-tuning (FT) procedure for text generation that has a number of advantages, including improved sample diversity. data efficiency, and out-of- distrribbttinn generaliza- tion. GFlowNet fine-tuning makes the language model sample from the target distribution, enabling amortized inference in a number of applications (Fig. 1).
Leveraging this approach, empirically demonstrate the possibilities and benefits of learning to sample from intractable distributions over text continuations, latent reasoning chains, and tool use sequences using GFlowNet fine-tuning. Notably, the diversity of samples from the models trained with GFlowNet fine-tuning is beneficial in Bayesian model averaging settings, such as when aggregating answers questions obtained via multiple reasoning chains. For example, using pretrained language model with 6B parameters, our method shows an absolute improvement of 10.9% over supervised fine-tuning subjectivity classification with only 10 labeled examples ($4.3) and outperforms supervised fine-tuning and PPO by 63% on integer arithmetic with 50 demonstrations, with notable improvements in -itto---iitrrittionn generalization ($4.4). Moreover, the benefits of amortized inference allow us to efficiently sample from the fine-tuned model at scale. Our contributions include:
(1) A general algorithm for amortized sampling from intractable LLM posteriors. (2) A probabilistic approach fine-tuning L.MSs to perform chain-of-thought reasoning.
(3) Empirical results on sequence continuation, natural language reasoning, integer arithmetic with tool use, and story infilling.
2 MOTIVATING EXAMPLE: GENERATING RANDOM NUMBERS WITH LLMS
We consider a simple task that highlights the limitations of seward-maximizing reinforcement learning (RL) methods in fine-tuning L.Mss. For readers unfamiliar with RL, we refer to Sutton & Barto (2018) and include a glossary in SA.1 to define key terms used throughout this paper. The task involves generating random numbers from given distribution when prompted 'The following is random integer drawn uniformly between 0 and 100: :. This task is minimal instantiation of the problem we study in the paper: sample from target distribution given unnormalized density.
2

---

Published as a conference paper at ICLR 2024
16
10
Probability mass
10
10
10
10
100
100
20
Generated number
Generated number
Generated number
(a) Base model 50.5% 5% of samples are valid numbers.
(b) PPO fine-tuning 95.8% of samples are valid numbers.
(c) GFlowNet fine-tuning 100% of samples are valid numbers.
Figure 2: Empirical distributions of 512,000 integers from to 100 generated by GPT-J fine-tuned with PPO (reward-maximizing: h) and GFlowNet fine-tuning diistilbation-maccinge c). Note the logarithmic y-scale.
Although the target distribution is tractable, making the task seemingly straightforward, it serves as a useful illustration of the behaviors of different fine-tuning methods.
Renda al. (2023) found that pretrained LLMs perform quite poorly on this task: the distribution of numbers generated with the above prompt will be far from uniform (Fig. 2a shows an example using an instruction fine-tuned GPT-J 6B (Wang & Komatsuzaki, 2021)!). There may be many reasons for this, among them the effects of instruction fine-tuning and the LLM's 's possible bias towards numbers that are more frequent in the training data (e.g.. numbers starting with '1' more frequent due to the properties of many natural data- lata-generating processes (Benford, 1938)).
While
RL can teach the model to generate valid numbers (by penalizing outputs that are not numbers from 100), it would not resolve the distribution skew introduced during pretraining. Indeed, rewarding all valid integers equally leads to expected gradient of zero for policy gradient methods. Fig. 2b shows that while most samples are valid numbers after PPO training, the distribution remains highly skewed.
Instead, we can take a principled approach by training the LLM to match the target distribution with a GelowNet learning objective. Such an objective directly optimizes the likelihood of the model gen- erating number to be proportional the reward for that number, which is the number's (potentially unnormalized) probability under the target distribution. When the policy is initialized as the pretrained LLM, the resulting distribution after GelowNet fine-tuning is shown in Fig. 2c. Quantitatively, the KI. divergence from the sampling distribution to the target (uniform) distribution decreases from 3.37 for the original LLM (on the support [0, 100]) to 9.75 10 5 for the GFlowNet-fine -enetee----e model.
This example illustrates a general point: GFlowNet objectives provide principled and flexible approach to fine-tuning LLMs to match target distribution where goward-maxiizza RL fails to. On this simple task, this distribution matching could also be achieved through supervised fine-tuning however, this would require access to samples from the target distribution, which are unavailable in general (though not in this simple example). In the following sections, we further illustrate this point in non- vonviaivial problems involving intractable inference, reasoning with latent variables, and tool use.
FINE-TUNING LLMS TO SAMPLE FROM INTRACTABLE DISTRIBUTIONS
We first describe how intractable inference emerges from interesting applications of LLMs, one of which is chain-of-thour thought reasoning seen through the lens of latent variable models, where the posterior distribution over the latent variable is intractable. We then discuss how GFlowNet ohjecties can be used to train amortized samplers to perform such intractable inference.
PROBLEM: INTRACTABLE INFERENCE IN LARGE LANGUAGE MODELS
Autoregressive language models decompose the distribution over sequences of tokens as product of ordered conditionals: p(WI:N) = p(w1) (ww) w1)... pw.. I wi:N-1). While this decomposition
makes left-tt-right sampling from the distribution tractable, sampling from other conditional distribu-
We the Dnstruct-GPT-J model availahle co

---

Published as a conference paper at ICLR 2024
Table 1: Objects in language posterior inference. Given a pretrained 'teacher' LM PLM we train Gel 4GFN to sample the posterior p(Z X. Amortization and generalization are achieved by making X. and
optionally Y, an imput 4GFN-
Object
Meaning
Example (infilling)
Example (subjectivity classification)
X
cuuse / condition / question mechanism / reasoning chain effect/ / answer conditional prior
The cat was hungry.
A deeply moving storyline.
Z
She ate a mouse.
This review expresses personal ferling feelings.
Y
Now the cat sleepy, not humgry.
Answer: Subjective
P(Z X)
PLM(Z X)
P( (X2))
likelihood of effect given cause and mechanism
PIM X2)
p(Z,Y (X)
conditional joint, reward for Z
PIM(ZY (X)
p(Z X.Y)
posterior (intractable!)
approximated and umortized by GFlowNet qaFx (Z X(,Y))
| XX)
posterior predictive/ /
approximated as Σz quax X) XZ).
Bayesian model average
sampled as Z ~ g X). Pix XZ)
tions is intractable. Various problems of language modeling can be viewed as sampling from such intractable conditionals in the distribution over sequences of LLM; we give two such examples and related terminologies in Table 1. Some tasks we study in $4 are instances of these examples.
Tempered and contrastive sampling. In many applications (e.g., translation, summarization, dialogue systems), one wishes to sample from a low-temperature distribution over sequences Z conditioned on a prefix X. i.e., q(Z | X) x PLM ( for some temperature T < 1, as high- likelihood samples are more likely to be fluent or accurate continuations of X (Tillmann & Ney, 2003). The limit of T - 0 gives a distribution that is peaky on the most likely continuation. However, sampling from q. or finding its mode, is intractable, and it is common to resort to approximations, such tempering the tokemwise conditional distributions or using beam search search for mode. A related problem is sampling a continuation with a correction for its unconditional likelihood, O, where applications again resort to
egg. gZ X) PIM(XZ) ( with B < O and approximating the next-token conditionals of by tempering (Malkin er al., 2022b; Li et al., 2023).
Infilling and reverse generation. Infilling is the task of sampling sequence of tokens conditioned on both its prior and subsequent context, which can be understood as sampling from the distribution (Z X, Y) x PLM(XZY) where X and Y are fixed. Reverse generation is special case, where X an empty sequence. Besides being meaningful task in its own right (Liu et al., 2019; Zhu et al., 2019; Donahue et al., 2020; Susanto et al., 2020; Lu et al., 2022a), infilling and reverse generation are key components of newly emerging methods of L.M prompting, such as when LLMS are tasked with optimizing their own instruction sequences or reasoning steps (Zhou et al., 2023; Sordoni et al., 2023; Xu et al., 2023). Current applications achieve this by resorting to aand-engineered instructions and inverted prompts.
Constrained generation. Sampling of text with constraints and penalties for example, those on the presence or the absence of certain words or on the score of an auxiliary classifier evaluated on the text can be understood as sampling from distribution 9 (Z) æ PLM (Z)c()c (Z), where is an externally specified constraint. Current approaches to the problem use tokenwise approximations (Liu et al., 2021), various cubblem-specffic beam search and local search techniques (e.g.. Schmaltz et al., 2016; Hokamp & Liu, 2017; Hu et al., 2019; Sha, 2020; Lu et al., 2022b) or classifier-guided conditional generation approaches (e.g., Yang & Klein, 2021; Meng et al., 2022).
3.2 REASONING THROOGH LATENT VARIABLES
Chain-of-thought reasoning (Wei et al., 2022: Kojima al., 2022) helps LLMs solve complex problems by producing a reasoning chain before giving the final answer. LLMs pretrained on general domain data can Iearn to produce useful chains of thoughts given demonstrations, which usually handcrafted or generated by prompting the LM. Interestingly, although the capacity for chain-of- thought reasoning only emerges in large language models, knowledge can also be extracted from smaller language models when they are carefully fine-tuned (Schick & Schütze, 2021).
Motivated by this, we connect chain-of-thoupe reasoning to the general problem of inference in latent variable models illustrated in Fig. 1. Here, reasoning can be seen as posterior inference: sampling from the posterior distribution over a string of tokens Z conditioned on a prefix X and a suffix Y, given an autoregressive language model PLM. The posterior is defined
PLM (Z X,Y) = (XZY
Σz (XM(Y & PXM(X
(2)
Our goal is to train models to sample Z from this posterior distribution. Intuitively, this allows us to sample likely reasoning chains that lead to the desired outcome Y. Although we take Z to be string
4

---

Published as a conference paper at ICLR 2024
of tokens, the same formalism and the GFlowNet objectives apply to other structured latent objects. such as trees or sets of natural language statements, as long as one has access to a likelihood model p( XZ). While not investigated in this work, these generalizations could be important for formal reasoning and multi-step chains of inference. See, e.g., Yao et al. (2023); Hao et al. (2023); Besta et al. (2024) for approaches to reasoning in language using tree- or list-structured state spaces.
A latent variable model of this form is useful when the marginal distribution PLM(Y X) is harder to model than PLM(Z X) and PLM (Y) XZ), i.e., a difficult inference is broken down into a chain of easier ones. By training model to match the Bayesian posterior PLM X, Y), we can learn to sample latent reasoning chains that increase the likelihood of producing Y from X via the sampled Z.
However, we can also fine- thne the language model PI Z XY) itself to maximize the likelihood of data pairs (X.Y) under the LVM. While it generally intractable directly maximize the data likelihood PI.(X.Y) Y) Σz Pi (XZY) because of the summation over Z, the (variational) expectation- maximization (EM) algorithm (Dempster et al., 1977: Beal, 2003; Koller Friedman, 2009) can be used for this purpose. In the expectation step (E-step), we draw samples from the posterior over the latent variable (L X, Y), which could come from an amortized sampler of Z. In the maximization step (M-step), we maximize the log-likelihooe sampled latent variables ( log PLM (XZY) with respect to the parameters of the language model PLM- This combination of amortized inference (learning to sample the chain of thought) and supervised fine-tuning (optimizing the language model with the "supervision" involving Z sampled from the amortized posterior) will be illustrated in one of our experiments (54.3, Table 3).
ceeetationmmaimizzatooon
of the joint probability of the
3.3
AMORTIZED INFERENCE WITH GELOWNET OBJECTIVES
For inference in the latent variable model, we leverage the probabilistic framework of generative flow networks (GFlowNets; Bengio et al.. 2021: 2023). Using notation from Malkin et al. (2022a), we briefly introduce relevant GFlowNet concepts pertaining to autoregressive sequence generation. Here, GelowNets learn policies to sample sequences Z Zizz .......... Zn T E Z (where denotes stop symbol) from distribution over the space of sequences Z. given an unnormalized density (reward) R: Z - R=o. The generative process is the same as in autoregressive language models: generation begins with an empty string, and at the i-th step a token z; is sampled from a policy (GFN(Z) t; Z1=i-1), which is then appended to the sequence. This process continues until stop symbol T is generated.
The marginal likelihood 4PP (Z) of sampling a terminal state Z = Zi:nn T is given by IT;- 4GFN I 21:-1 (ai-1))GFN((( [T] z), where 21:0 is understood to be the empty string. The goal of GFlowNet training is to fit a parametric policy 9GFN (: :; such that TPN (Z) x R(Z), i.e., the likelihood of generating a complete sequence is proportional to its reward.
Learning objective. We use modified version of the subtrajectory balance (SubTB; Madan et al., 2023) objective to account for trajectories being terminable at all states (Deleu et al., 2022). The objective for sequence Z. Z1:h.T T is
R(zia 11 4GFN 11*-1)9GEN(T Zz:))
2
L(Z:0)= =
Σ 0si<isn
log
R(ZigT))GFNN 21:))
For sequence generation tasks, the SubTB objective is equivalent to the path consistency objec- tive (Nachum al., 2017) in max-entropy RL (Haarnoja et al., 2017), which has been previously used in the context of text generation (Guo et al., 2021). See Appendix A.2 for further discussion.
Training policy. As the objective in Eq. 3 can be minimized to 0 for all trajectories simultancously given enough model capacity, we can use trajectories sampled from any full-support distribution (training policy) to perform gradient descent sampling from is combinatorially large, it is important to have a training policy that can efficiently explore Z. To this end, we compose the mini-batch during training using trajectories from three sources: (1) the policy qGEN, (2) a tempered version of the current policy qGFN and (3) replay buffer storing past trajectories. Replay buffers have been shown to be quite effective in improving GFlowNet training (Jain et al., 2022: Deleu et al., 2022; Shen et al., 2023).
L(T; 0) with respect to 0. As the space
Parametrization, amortization, and generalization. To sample the latent sequence Z from the posterior defined in Eq. 2, we parametrize the GFlowNet policy as an autoregressive language model that samples the latent Z one token at a time from left to right. By setting the reward R(Z)= PLM (XZ) x PZ X, Y), we learn sampler for the posterior at convergence.
5

---

Published as a conference paper at ICLR 2024
As illustrated in Fig. depending on the task, we can condition the GFlowNet policy on either X or Y.Y. Y. In cases such as reasoning ($3.2), where there is only a single correct Y for each X and we interested in predicting Y for unseen X at test time, we can simply condition on X. In this case, the GFlowNet policy is simply a language model that generates Z as a continuation of X. To be precise, we initialize 4GFN copy of PLM that is conditioned on the prefix X, and then fine-tune" it with GelowNet objective. With this view, sampling Z. is an inverse problem: we need to infer Z. given a (conditional) prior (Z X) and an observation Y under likelihood model PLM Y XZ).
Allowing the GelowNet policy to explicitly take X as input amortizes the sampling procedure and allows generalization to unseen X. In this sense, the GFlowNet is a Bayesian model (akin to a LM cascade (Dohan et al., 2022) or deep language network (Sordoni et al., 2023)), in which Z are conditionally sampled 'parameters' that transform X into Y. To predict the Y for an unseen X, one performs Bayesian model averaging by drawing samples of Z from 9GFN [Z] X) followed by sampling from (Y XZ).
In tasks such as infilling ($4.2), however. the mapping from X to Y is one-to one-to-many and Y is available at test-time. Here, we are interested in Z itself, rather than using it as an intermediate variable en route to generating Y. The GFlowNet policy thus has to be conditioned on both X and Y. To achieve this, the policy is conditioned on a prompt that contains both X and Y (for example, see Appendix C).
4 EMPIRICAL RESULTS
We first validate GFlowNet fine-tuning text generation, where we seek to find likely sentence continuation given prompt ($4.1) 1) or fill in a missing sentence in a story ($4.2). Then, we study reasoning tasks that benefit from chain-of-thougle reasoning ($4.3) and external tool use ($4.4).
4.1
SENTENCE CONTINUATION
Diverse heam
Task description. A natural application for au- toregressive language models is that of sequence continuation: given a prompt, the model should generate high-likelln lieeliool completion. In ap- plications such as creative writing, we would
Muximum sentence log- likeliiklhoodd
-10
(SX compute)
GFlowNet fine-tuning
Divense beam
-15
Sampling w/ T
T=9.95
like the continuations to be semantically diverse while still having high likelihood under the language model. To demonstrate the benefits of GFlowNet fine-tuning, we consider the task of sampling the next sentence following a prompt.
-20
-25
Groody
Nuclous
Sampling
-30
Sampling autoregressively from the LM until
" token is reached is unlikely to produce samples that have a high likelihood because the distribution over sentences has a fat tail. Exist- ing approaches to generate sequence continua- tions include beam search and its variations (Vi- jayakumar et al., 2018; Shao et al., 2017), top-k sampling (Fan et al., 2018), nucleus sampling (Holtzman et al., 2019), tempered autoregressive sampling, and fine-tuning using importance sampling (Shih et al., 2023), among others. While useful, most of these methods are ultimately hand-crafted heuristics that leave room for improvement. Furthermore, some of these methods (e.g., beam search) involve a computationally expensive search procedure, compared to single pass of a learned in- ference model that amortizes over prompts. Our GFlowNet policy autoregressively samples the sequence until period is sampled, indicating the end of the sentence. Given prompts X, the LM is fine-tuned to generate the continuations Z from the tempered posterior by being trained with reward R(2) PLM + When 7 = 1, the GelowNet will trivially sample proportional to PLM without any fine-tuning, so we consider 0 < T < 1 to focus the likely continuations.
0.675
0,700
0.725
0.750
0.775
0,800
Sentenge diversity
Figure 3: Maximum log-likelihood and diversity of con- tinuations sampled for fixed prompts. GFlowNet fine- tuning (*) samples higher log-likelihood sentences while maintaining more sample diversity than the baselines (+ and -), even when they are given 5x the compute.
We consider dataset of prompts from WebTex (Gokaslan et al., 2019) with a 1.5B param GPT-2 XL (Radford et al., 2019) as the base model. We draw 8 samples from the fine-tuned model conditioned on fixed prompt, consider the and report the average over the dataset of prompts. To measure the semantic diversity of the
sample under the LM,
We use LoRA (Hu et al., 2022) instead of full fine-tuning for hardware efficiency in all experiments.

---

Published as a conference paper at ICLR 2024
samples, we compute the mean pairwise cosine distance between the embeddings (from a pretrained encoder (Reimers & Gurevych, 2019)) of the generated samples and average it over the dataset. We compare to basclines that are commonly used for producing continuations from LMs at inference time (beam search, diverse beam scarch, nucleus sampling, autoregressive sampling, tempered autoregressive sampling, and greedy generation).
Results. Quantitative results are reported in Fig. and empirical samples are shown in Appendix B. At lower temperatures, our method excels in generating high-likelihood sentences, outperforming the leading baseline, diverse beam search. If we increase the number of beams (and therefore compute) to 5x the number of samples produced by the GFlowNet, our performance remains comparable. Nevertheless, even in this scenario, the GelowNet's generated samples exhibit notably higher diversity compared to diverse beam search and are on par with the best diversity- scoriin benchmarks.
4.2 INFILLING STORIES
Task description. Next, we consider the story infilling task, a special case of the general infilling problem ($3.1), where given the beginning X and end Y of story, the goal is to generate the middle of the story Z (Zhu et al., 2019). This is challenging for a language model sampled left to right since continuations Z conditioned only on X might be incompatible with the ending Y. We use the ROCStories corpus (Mostafazadeh et al., 2016), a dataset of short stories containing exactly sentences each. Given the first 3 sentences and the last sentence, the goal is to generate the fourth sentence, which often involves a turning point in the story and is thus challenging to fill in.
As we expect the base model to contain the required knowledge, for this task we use a GPT-2 Large model (Radford et al., 2019) fine-tuned on the entire ROCStories training set as the base model. For evaluating the approach, we consider 900 samples from the dataset as training data to learn qGEN (Z) X. Y) and evaluate the similarity of the generated infills on a dataset of 100 unseen stories. Along with the GFlow
model, we also consider two baselines; prompting the model to infill the story and supervised fine-tuning on the same data. Further details are in Appendix C.
To measure the similarity of the gen- erated infills with the reference infills available in the dataset, we compute BERTScore (Zhang et al., 2020b), with DeBERTa (He et al., 2021) whicc is correlated with human judgments-
Results.
Table Evaluation of the generated infills.
Method
BEKTScore
BLEU-4
GLBU4
GFT4Eval
Prompting
0.081 = 0.009
13405
3.1
24
Supervised fine-tuning
0.094 - 0.007
16
0.
GelowNer tuiing
0.184 0.004
2.1002
4.20.7
along with BLEU-4 (Papineni et al., 2002) and GLEU-4 (better suited for sentences; Wu et al., 2016)
metrics. Additionally, we also evaluate each method using GPT-4 as a judge. From our results summarized in Table 2, we observe that the infills generated by the model with GFlowNet fine-tuning are closer to the reference infills the dataset than the baselines. By sampling from PLM (Z)X, Y), the GFlowNet is able to account for the ending while generating the infill, resulting in infills that link the beginning and the end of the story coherently. For further analysis and details see Appendix C.
4.3 SUBJECTIVITY CLASSIFICATION
Task description. SUBJ (Pang & Lee, 2004) is a binary classification dataset for natural lan- guage understanding. It is a collection of movie reviews in which each review is labeled as objec- tive, meaning that it references facts about the movie, or subjective, meaning that it expresses an opinion of the reviewer (see Table D.I for ex- amples). Given an unlabeled review, the model must predict whether is objective or subjective. While supervised fine-tuning on the full dataset can achieve high test accuracy, we are interested in the low-data regime where we only have tens of labeled examples. We use the same instruction -unedd GPT-J 6B variant as in $2 for this experiment. Without any demonstrations, the model struggles with this task using the prompt in Table D.2 and achieves only 51.7% zero-shot accuracy.
Table 3: Test accuracy (%) on SUBJ using an instruct-
mne-tuned GPT-J 6B. Method Zero-shot prompting
Test accuracy (S)
51.7
Training samples
10
20
50
Few-shot prompting
6.2
6.48
65.8 a 10.5
Supervised fine-tuning
64.3
6.81
89.7 1.4
Gilloweer finc-tuning
71.4
81.1
8.2
+ Supervised fine-tuning
75.2 1.8
78.7
7 1.6
89.9 + 0.2
This task is hard likely because it requires a latent reasoning step. A review could be considered objective because it analyzes the plot or facts about the movie, or it could be subjective because expresses a personal opinion makes a judgment. We denote the review X, the predicted subjectivity Y, and the latent reason Z. Then, we GFlowNett-ine-
the LLM qGEN (Z) | X), initialized with the
7

---

Published as a conference paper at ICLR 2024
base model PLM match the Bayesian posterior over rationales in Eq. 2. At test time, qGFN X) generates 10 latent rationales (Z's) for an unseen X. The LLM PLM then autoregressively samples from PLM(Y XZ) to produce 10 answers, the majority vote of which becomes the final prediction.
This posterior inference corresponds to the E-step in the EM algorithm, where the posterior PLM (Z X, is defined in Eq. 2. Further, as described in $3.2, we can take an M-step by updating PLM to maximize log PLM XZ) over collection of Z's sampled from the amortized posterior 9GEN. This is equivalent to applying supervised fine-tuning after GFlowNet fine-tuning,
Results. We present few-shot prompting and supervised fine-tuning with LoRA baselines. In few-shot prompting, we prepend 0, 10, 20, or 50 training examples to each test example using the prompt shown in Table D.2. We randomly shuffle few-shot demonstrations and report the mean and variance in Table In supervised fine-tuning, we directly maximize log PIM
X) over the same
10, 20, or 50 (X, Y) pairs. The variance is over model initialization and batch order. All entries except zero-shot prompting are aggregated over random seeds. See Appendix D for experiment details. GFlowNet fine-tuning consistently outperforms supervised fine-tuning in the low-data regime, as shown in Table 3. In some cases. performing supervised fine-tuning on top, which corresponds to running one step of the EM algorithm, further improves the performance.
4.4 SOLVING ARITHMETIC PROBLEMS STEP BY STEP
Task description. Arithmetic reasoning is fitting benchmark to evaluate reasoning abilities of large language models as it requires multi- step reasoning and correctness is easy to evalu- ate (Cobbe et al., 2021). While the distribution of pretraining and fine-tuning data (Magister et al., 2023; Lee et al., 2023; Luo et al.. 2023) and prompting choices (Imani et al.. 2023) play a critical role in their arithmetic abilities, LLMs are susceptible to poor generalization by learn- ing 'shortcuts' to reasoning (Dziri et al., 2023). We consider a simple integer arithmetic task (Fig. 1), with general pretrained base model, rather than a one pretrained on mathematical tasks (Jelassi et al., 2023). To avoid the pitfalls of symbolic calculations with language models, we adopt the tool use setting (Schick et al., 2023), where the model is equipped with calculator that can perform parts of the computation, implemented as in Cobbe et al. (2021): when the model outputs '=' the expression preceding it is evaluated and appended to the sequence. To prevent the model from "cheating", we limit the calculator to evaluate only two-term expressions. Consequently, reasoning here involves learning to plan using tool with limited capabilities (Hao al., 2023).
Table Test accuracy (%) on an integer arithmetic task with addition and subtraction using a GPT-J 6B model. Training data only include samples with 3 or 4 operands.
Number Operands
In-distribution
O0D
Method
3
5
k-shot CoT
0
10.2
6.4
3,2
117
0.2
== 5
20.4 10.4
0.6
6.6
k
26.5
15.2
8.9
20
35.5
11
10.5
Supervised fine tuning
1.3
19.6+
12,8 5.7
PPO GFlowNet fine-tuning
30.6
13.7
5.6
95.2 1.3
7.9
40.7
9.1
For training, we use synthetic dataset of arithmetic expressions, limited to addition and subtraction. Following Zelikman et al. (2022), we use small set of 50 demonstrations (X, Z, Y) to seed the replay buffer in addition to 1000 examples (X, Y). We use the same instruction-tune -unne GPT-J as in $4.3 as the base model. Further details are in Appendix E. We report the accuracy on two types of examples: (1) unseen in- diddstriibuton expressions (3 or 4 operands) and (2) longer (5 operands). As baselines, we consider zero-shot chain-off sfiin--f-thuhhh thought prompting, k-shot prompting, supervised fine-tuning on the tool use sequences, and fine-tuning with PPO (Schulman et al.. 2017). For all methods, we enable tool use and limit the model to generate only numbers and operators.
distriibutt expressions
Results. From the results summarized in Table 4, the base model performs poorly even with chain- of-thought prompts. Including examples in context improves the performance considerably, with monotonic improvements as the number of examples increases. Supervised fine-tuning improves the performance significantly on the in-distribution examples, but the model still struggles to generalize on the dittof-distribution examples. Fine-tuning with PPO results also yields poor performance, caused in part by the poor calibration of the base reward model, i.e. it cannot distinguish good rationales from bad ones. Even though the sequences generated with PPO (illustrated in Appendix E) have high rewards, they are spurious and do not even define valid calls to the tool.
Such overoptimization to a misspecified reward is a widely noted issue in LLMs trained with RL (Gao et al., 2023). On the other hand, by matching the entire distribution, Gel fine-tuning avoids collapsing to a single mode of the reward, thereby being robust to the misspecification of
8

---

Published as a conference paper at ICLR 2024
the reward (Eysenbach & Levine, 2022) and achieving significantly better performance on in and dist----iitiitttionn examples. See Appendix E for additional results and analysis.
5 FURTHER RELATED WORK
Sampling from intractable marginals. Beyond the approximations mentioned in $3.1, sampling from intractable posterior distributions given by pretrained models for tasks such as infilling and constrained generation has been object of study. Miao et al. (2019); Zhang al. (2020a) use MCMC for these problems, Malkin et al. (2021) used a seriable-neigborrhee modes, and sequential Monte Carlo approach was recently proposed by Lew et al. (2023). Others have studied the problem with masked language models, using them to perform variants of Gibbs sampling (Wang& & Cho, 2019; Goyal et al., 2022: Yamakoshi et al., 2022) and recovering marginal distributions over small sets of tokens (Torroba Hennigen & Kim, 2023).
ascent for finding
GFlowNets. GelowNets (Bengio et al., 2021) were originally proposed to learn policies for sam- pling discrete compositional objects from an unnormalized reward distribution, motivated by the need to sample diverse high-reward objects in scientific discovery (Jain et al., 2023), in particular, for biological sequence generation (Jain al., 2022). The interpretation of GFlowNets as variational inference algorithms (Malkin et al., 2023; Zimmermann et al., 2023) makes them appropriate for sampling Bayesian posterior distributions over structured objects (e.g. Deleu et al., 2022; 2023; van Krieken et al., 2023; Hu et al., 2023).
Chain-of- of-thought reasoning in LLMs. In recent work on classification and completion with language models, the latent reasoning chain Z, in the notation $3.1, is called a 'chain of thought' (Wei et al., 2022). The chain of thought is typically generated by conditioning the language model on X with the use of specialized demonstrations or prompts (Kojima et al., 2022), with no guarantee of sampling the posterior accurately. Related to our Bayesian formulation, Wang et al. (2023b) noted that appropriately aggregating the conclusions Y from several latent chains improves predictive performance. In Xu et al. (2023); Zhou et al. (2022), a posterior over latent token sequences is sampled using MCMC, while Zelikman et al. (2022) propose fine-tuning on successful (high-reward. in our language) chains of thought, which achieves reward maximization but gives no guarantee of diversity. In concurrent work, Phan et al. (2023) use MCMC sample chains-of-thought in problems with binary feedback. We expect these methods to generalize poorly to difficult exploration problems, while GelowNet fine-tuning takes advantage of generalizable structure in the posterior and has goal of sampling the full posterior over latent reasoning chains.
CONCLUSION
The knowledge compressed in LLMs is crucial for tasks such as infilling and constrained generation, but querying this knowledge involves sampling from intractable posterior distributions. We propose to use GFlowNet objectives to train LLMs to sample from such posterior distributions. Empirical results show that GFlowNet fine-tuning finds a better fidelity-diversity trade-off for text generation and also improves sample efficiency and generalization on downstream tasks compared to maximum- policy optimization. As an amortized inference algorithm,
likelihood training or reward-maximize our method converts computation into better test-time performance without additional data.
Future work should investigate transfer and generalization across tasks, in particular, building a 'universal reasoner' as model 9Z X) shared between X from different tasks, as was recently considered by Wang et (2023a). One should investigate the benefit of using a better knowledge model, e.g., more capable base L.M,, as starting point for GFlowNet fine-tuning. The ability to draw multiple samples from a GelowNet can also used to quantify epistemic uncertainty. Finally, we adopt the GFlowNet formalisms with the perspective of generalizing to latent variables Z with richer generative process than left-to-right sampling. We hope that the GFlowNet paradigm will enable more flexible reasoning with LLMs in the future: extending probabilistic programming with language variables (Beurer-Kellner et al., 2023), using structured chains of thought (Yao et al., 2023; Besta et al., 2024), and extending to program synthesis and planning with world models.
Linitations. Due resource constraints, experiments use models up to 6B parameters, but expect the conclusions to hold for larger models. In fact, our method can potentially benefit larger models more: it is harder to optimize larger model with maximizing objectives data, As with any on-policy method, exploration, especially in problems with more complex latents, remains an open problem. Furthermore, our method improves inference but not the knowledge in the LM. Issues such as hallucination or miscalibration, which are closely related to the knowledge representation, are thus not addressed.
small amount of

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
While we foresee no immediate negative societal consequences of our work, we hope that future researchers who build upon it will, as we have, bear mind the potential of LLMs and in particular of human-like reasoning in LLMs to be used both for good and for harm.
Research areas in safe and explainable AI that can benefit from GFlowNet funiing include (1) interpretability of LLMs' reasoning processes and (2) fine-tuning with human feedback or an external reward, where diverse sampling can help prevent 'reward hacking' and overfitting to misspecified target.
REPRODUCIBILITY
We discuss the details of the proposed algorithms in $3.3 and provide all the implementation details and hyperparameters for the experiments in the main paper and appendix. Code for our experiments is available at https: http:::/ggiiii http//ggttttttt/ee/ com/ NOrg/ / g-n- Im-tuning
ACKNOWLEDGMENT
The authors are grateful to Bonaventure Dossou and Salem Lahlou for their help in the early stages of this project. We also thank Robert Hawkins, Arian Hosseini, Zhen Wang, and Anirudh Goyal for valuable discussions and suggestions of related work.
GL acknowledges funding from CIFAR, Samsung, and Canada Research Chair in Neural Computa- tion and Interfacing.
YB acknowledges funding from CIFAR, NSERC, IBM, Intel, Genentech, and Samsung.
The research was enabled in part by computational resources provided by the Digital Research Alliance of Canada pettps:/alllaanecaan ca), Mila (https: //mm//mm.a./////////////////aall quebec), and NVIDIA.
REFERENCES
Matthew J. Beal. Variational algorithms for approximate Bayesian inference, 2003. URL https: //cse buffflo.............. edu/ www.ltaattttaaaaleeeeaa /tbttal////llaa/eaa http:///baaaaaaaaaeeeeeeeeeeeeee
pwpprs /baal003beee000 pdf.
Frank Benford. The law of anomalous numbers. Proceedings of the American Philosophical Society,
78(4):551 572, 1938. ISSN 0003049X. URL http: //wp:://wwww../ jstor. wwtportallleeeeo 984480988
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Neural Information Processing Systems (NeurIPS), 2021.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward Hu, Mo Tiwari, and Emmanuel Bengio. GFlowNet foundations. Journal of Machine Learning Research, (24):1-76, 2023.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. Association for the Advancement of Artificial Intelligence (AAAI), 2024.
Luca Beurer-Kellnng Marc Fischer, and Martin Vechev. Prompting programming: A query language for large language models. Proceedings of the ACM on Programming Languages, 7, jun 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14.14. 2021.
Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks. Uncertainty in Artificial Intelligence (UAI), 2022.
Tristan Deleu, Mizu Nishikawa- Nismkyawa-Toomey, Jithendaraa Subramanian, Nikolay Malkin, Laurent Charlin, and Yoshua Bengio. Joint Bayesian inference of graphical structure and parameters with a single generative flow network. Neural Information Processing Systems (NeurIPS), 2023.
10

---

Published as a conference paper at ICLR 2024
A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society B, 39(1):1-38, 1977.
David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes. Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl- Dick-iicst,ii,, Kevin Murphy, and Charles Sutton. Language model cascades. arXiv preprint
2034. 2022.
Chris Donahue, Mina Lee, and Percy Liang. Enabling language models to fill in the blanks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2492-2501, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v11222 1/2020. acl-main.225. URL https::/aclaattaiaet ha///aaaaaantoooyyy... org/ 2tgg/2222 2020. acl mall-mman.... 225.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D Hwang, al. Faith and fate: Limits of transformers on compositionallty. Neural Information Processing Systems (NeurIPS), 2023.
Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. International Conference on Learning Representations (ICLR), 2022.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889-898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v 1/P18- 1082. URL ht https: //caaattoooooooooooeeooaaaooo org/P18 1082.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. International Conference on Machine Learning (ICML), 2023.
Samuel Gershman and Noah D. Goodman. Amortized inference in probabilistic reasoning. Cognitive Science, 36, 2014.
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwehtext corpus. http: 1/5kylion007 git..b..hh....gg io/ http///eeebeeetoopp....... 2019.
Kartik Goyal, Chris Dyer, and Taylor Berg- Berg-Kirkpatriek. Exposing the implicit energy networks behind masked language models via Metropolis- International Conference on Learning Representations (ICLR), 2022.
Han Guo, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. Efficient (soft) Q-learning for text generation with limited good data. arXiv preprint arXiv:2 2021.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. International Conference Machine Learning (ICML). 2017.
Shibo Hao, Yi Gu, Haodi Ma. Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8154 8173, Singapore, December 2023. Association for Computational Linguis- URL https: aaw.athoootttyyyyyccccccc org/2023.
tics. doi: 10. 22.18653/v1/222 wtt.2222...mmmmaaiii.... emnlp-main.500 507.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enchanced BERT with disentangled attention. International Conference on Learning Representations (ICLR), 2021.
Chris Hokamp and Qun Liu. Lexically constrained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1535-1546, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 118866111111--1111 ATg/P17-1144
URL https: / ac anthol logy.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. International Conference on Learning Representations (ICLR), 2019.
11

---

Published as a conference paper at ICLR 2024
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low- Low-rank adaptation of large language models. International Conference on Learning Representations (ICLR), 2022.
Edward J Hu, Nikolay Malkin, Moksh Jain, Katie Everett, Alexandros Graikos, and Yoshua Bengio. GFlowNet-EM for learning compositional latent variable models. International Conference on Machine Learning (ICML), 2023.
J. Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia. Tongfei Chen, Matt Post, and Benjamin Van Durme. Improved lexically constrained decoding for translation and monolingual rewriting. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 839-850, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10.18653/v 1/N13//1//111190 URL https: hww.aaaaaaantooooaaaoooo org/Nig 1-NN-1110 1090.
Shima Imani, Liang Du, and Harsh Shrivastava. MathPrompter: Mathematical reasoning us- ing large language models. In Proceedings of the 6 Anmual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pp. 37-42, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 1 acl-industry.4. URL https: / aaaaaaaaantthooooo org/ 2023. aww--innnnuurr..
Moksh Jain, Emmanuel Bengio, Alex Hernandez-Gariia, Jarrid Rector-Brooks Bonaventure F.P. Dossou, Chanakya Ekbote, Jie Fu, Tianyu Zhang, Micheal Kilgour, Dinghuai Zhang, Lena Simine, Payel Das, and Yoshua Bengio. Biological sequence design with GFlowNets. International Conference on Machine Learning (ICML), 2022.
Moksh Jain, Tristan Deleu, Jason Hartford, Cheng-Hao Liu, Alex Hernandez-Garcia, and Yoshua Bengio. GFlowNets for Al-driven scientific discovery. Digital Discovery, 50))5557-577,000
Samy Jelassi, Stéphane d' Ascoli, Carles Comingo-Enrich, Yuhuai Wu, Yuanzhi Li, and François Charton. Length generalization in arithmetic transformers. arXiv preprint arXiv:2306.1544 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Language models are zero-shot reasoners. Neural Information Processing Systems (NeurIPS), 2022.
Daphne Koller and Nir Friedman. Probubilistic graphical models: principles and techniques. MIT press, 2009.
Nayoung Lee, Kartik Sreenivasan, Jason D Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small transformers. arXiv preprint arXiv:2307.03381, 2023.
Alexander K. Lew, Tan Zhi-Xen ShaaX Gabriel Grand, and Vikash K. Mansinghka. Sequential Monte Carlo steering of large language models using probabilistic programs. arXiv preprint wxXiv2306.000881 2023.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as op- timization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12286-12312, Toronto, Canada, July 2023. Asso- ciation for Computational Linguistics. doi: 10. (0.18653/v1/2DD) h2t01//222222222...666... acl-long..67. acl- 687.
URL https:
//aclanthology wwwechology..rg/200202 org/202333322.
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta. Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pp. 6691-6706 Online, August 2021. Association for Computational Linguistics. doi: 18.18653/ v1/2021 1al.022.aal-loog.55233 URL https:tpp... /at/ancaaanttttt hology org/202///00 2021002222220 acl- aaw.0.a.........
522.
Dayiheng Liu, Jie Fu, Pengfei Liu, and Jiancheng Lv. TIGS: An inference algorithm for text infilling with gradient search. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pp. 4146-4156, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18865/11/119. 1406. URL wwwp::tttp//omm aac anthology.ooreo 9- 1406.
12

---

Published as a conference paper at ICLR 2024
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG evaluation using GPT-4 with better human alignment. arXiv preprint arXiv:2303. 2023.
Sidi Lu, Tao Meng, and Nanyun Peng. Insnet: An efficient, flexible, and performant insertion-based text generation model. Neural Information Processing Systems (NeurIPS). 2022a.
Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, Noah A. Smith, and Yejin Choi. Neu- roLogic a*esque decoding: Constrained text generation with lookahead heuristics. In Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Humma Language Technologies, pp. 780-799, Seattle, United States, July 2022b. Association for Computational Linguistics. doi: 10. 1653/v waw.0222naal-mmann.a. URL org/ 2022. naacl- 57.
https: http///aaaaaathhhhaa
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-Guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. WizardMath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint 2985: 2023.
Kanika Madan, Jarrid Rector-Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, An- drei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning GFlowNets from partial episodes for improved convergence and stability. International Conference on Machine Learning (ICML), 2023.
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1773-1781, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 111118661122233adlelc 151. URL ht /ata/acaaanntooooooe
Nikolay Malkin, Sameera Lanka, Pranav Goel, and Nebojsa Jojic. Studying word order through iterative shuffling. In Proceedings of the 202/ Conference Empirical Methods in Natural Language Processing, pp. 10351-10366, Online and Punta Cana, Dominican Republic, November Associatio for Computational Linguisties. doi: 10. 10.11865/112221..eml Saillp-mmii.8099 URL https: /atc:://allllaa http:/::///caaathooogyy/o org/ wttg//2111..../// emalp-ma inniaiii.... 809.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in GFlowNets. Neural Information Processing Systems (NeurIPS), 2022a.
Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. Coherence boosting: When your pretrained language model is not paying enough attention. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8214-8236, Dublin, Ireland, May 2022b. Association for Computational Linguistics. doi: 10. 1/:53///1/0222accl-ooo ht https: / /aaccaaann aallnnhollogy org/ 2022. acl -onn.loon........ 565.
URL
Nikolay Malkin, Salem Lahlou, Tristan Deleu, Xu Ji, Edward Hu, Katie Everett, Dinghuai Zhang and Yoshua Bengio. GFlowNets and variational inference. International Conference on Learning Representations (ICLR), 2023.
Tao Meng, Sidi Lu, Nanyun Peng, and Kai-Wei Chang. Controllable text generation with neurally- decomposed oracle. Neural Information Processing Systems (NeurIPS), 2022.
Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. CGMH: Constrained sentence generation by Metropolis- Haattiol sampling. Association for the Advancement of Artificial Intelligence (AAAI), 2019.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Van- derwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper under- standing of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 839-849, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10. 10.18653/v1/N16- 10988. URL https: /aall antho www.laaoooooooogglooolooo org/ 6 10908
13

---

Published as a conference paper at ICLR 2024
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Neural Information Processing Systems (NIPS), 2017.
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summa- rization based minimum cuts. In Proceedings of the 42nd Annual Meeting of the Associa- sion foo Computational Linguistics (ACI--04), pp- 271-278, Barcelona, Spain, July 2004. doi: 10.3115/121855. 111199555111999999 1218990. URL htttps://accaatthheeete
.rgg//04-003...
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for auto- matic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/1073083.0733333 URL (aazg//02-1044
https: /tt///clannttttt
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joins Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 2463-2477 Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/w1/D19- 1250. URL
Du Phan, Matthew Douglas Hoffman, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, Rif A Saurous, al. Training cfain-of-thought via latent-variable inference. Neural Information Processing Systems (NeurIPS), 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Nils Reimers and Iryna Gurevych. Sentence-BEiE Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-
IJCNLP), pp. 3982 3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/D19 1410. URL http:: hatpat///ollaaoooooy.... wrrg org/Dis-1410
Alex Renda, Aspen K. Hopkins, and Michael Carbin. Can LLMs generate random numbers? evaluating LLM sampling in controlled domains, 2023. URL
http:/pppppl..slllllppppppppppla csail.mit.
edu/renda/ / 1m sampl haapppling-paappp papppp.pp.....
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2339-2352, Online, June 2021, Association for Computational Linguistics. doi: 10. 10.18653/v1/2022 -aa.lllman....mi.... 185. URL https: ac www.clllnntoooooo..ooooooooooooe 2ogg/2222 maacl-mat memmmain... 185.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Neural Information Processing Systems (NeurIPS), 2023.
Allen Schmaltz, Alexander M. Rush, and Stuart Shieber. Word ordering without syntax. In Pro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2319-2324, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 1.118653/1/DDI--11
URL https::///// /aac//aaaa www.cannnollooyooooooooooooooeoe org/Dle 6-11661111 1255.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Lei Sha. Gradient-guided unsupervised lexically constrained text generation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 8692- 8703, Online, November 2020. Association for Computational Linguistics. doi: 10. 00.18633/v1/0020 emnlp-main.70%. 701. URL https : anthology wtrgg/220000 emn 1p-ma n. 701.
14

---

Published as a conference paper at ICLR 2024
Yuanlong Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. Generating high-quality and informative conversation responses with sequence-to te-suennce models. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 2210-2219, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 1LL18653/1/DD1-1222 URL httt:://acaaatthhoooo httphh/owwooo.oooooooooooo
Max W Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving gflownet training. International Conference on Machine Learning (ICML), 2023.
Andy Shih, Dorsa Sadigh, and Stefano Ermon. Long horizon temperature scaling. International Conference on Machine Learning (ICML), 2023.
Alessandro Sordoni, Xingdi Yuan, Maro-Alexandre Côté, Matheus Pereira, Adam Trischler, Ziang Xiao, Arian Hosseini, Friederike Niedtner, and Nicolas Le Roux. Joint prompt optimization of stacked LLMs using variational inference. Neural Information Processing Systems (NeurIPS), 2023.
Raymond Hendy Susanto, Shamil Chollampatt, and Liling Tan. Lexically constrained neural ma- chine translation with Levenshtein transformer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3536-3543, Online, July 2020. Asso- ciation for Computational Linguistics. doi: 10. 10.18653/v1/020... maa.aa/....mmmmm..33.33.333 URL https: //aclanthology. wttp////..2...ooooo/.. acl-main.....mmmmmm 325.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018 URL http:://nncompleeeedeeeee net.book/eeeoooooooeooooooooooo http://oookktteeboo---2222 html.
Christoph Tillmann and Hermann Ney. Word Reordering and Dynamic Programming Beam Search Algorithm for Statistical Machine Translation. Computational Linguistics, 29(1):77133, 03 2003. ISSN 0891-2017. doi:
URL https: /tttt::://oo.oog//1111 org//0.....0.... 1162/
089120103321337453
Lucas Torroba Hennigen and Yoon Kim. Deriving language models from masked language models. In Proceedings of the st Annual Meeting of the Association for Computational Linguistics (Volume
2: Short Papers). pp. 1149-1159, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. http://ww1//23.llllsshh 99.
URL https: / http:////aaaaahot//aa//e org/
2023.
Emile van Krieken, Thiviyan Thanapalasingam, Jakub Tomezak, Frank van Harmelen, and Annette ten Teije. A-NeSI: A scalable approximate method for probabilistic neurosymbolic inference. Neural Information Processing Systems (NeurIPS), 2023.
Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju. Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural sequence models. Association for the Advancement of Artificial Intelligence (AAAT), 2018.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. https:/ / thub.
com/huggingg waaefface//rrr 2020.
Alex Wang and Kyunghyun Cho. BERT has mouth, and it must speak: BERT Markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation. pp. 30-36. Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. (0.18653//1/W11-22304 URL https: http:///aaaaaatot/ao//// org/ /119 9- 2204
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 billion parameter autoregressive language model. https: / //gggttthhhh com/kingoflolleoocococoooooooooe wwweeteranssttteee transformer- jax, May 2021.
Peiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao, Tianyu Liu, and Zhifang Sui. Making large language models better reasoners with alignment. arXiv preprint arXivv2309.02. 144, 2023a.
15

---

Published as a conference paper at ICLR 2024
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang. Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. International Conference Learning Representations (ICLR), 2023b.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, prompting elicits reasoning in large language models. Neural
and Denny Zhou. Information Processing Systems (NeurIPS). 2022.
Sean Welleck, Ilia Kulikov, Jaedeok Kim, Richard Yuanzhe Pang, and Kyunghyun Cho. Consistency of a recurrent language model with respect to incomplete decoding. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5353-5568, Online, November 2020. Association for Computational Linguisties. doi: 10. 10.186//1//2220
emolp-main.4453 448. URL hwwp::tttps... : a wawoooloocooooooooooooooooooooo org/2020 emn 1 ma i n. 448.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang. Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint 160i 001000881444..0 2016.
Weijia Xu, Andrzej Banburski-Fahey and Nebojsa Jojic. Reprompting: Automated chain-of-though prompt inference through Gibbs sampling. arXiv preprint arXiv:2305.0099) 009993, 2023.
Takateru Yamakoshi, Thomas Griffiths, and Robert Hawkins. Probing BERT's s priors with serial reproduction chains. In Findings of the Association for Computational Linguisties: ACL 2022, pp. 3977-3992, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10.18653/ v1/2022. findin2.innnns--ccl ac1.314. URL https: aww.aaatnnnntoyyyy.accc. org/ foro222findinng-aee 314.
Kevin Yang and Dan Klein. FUDGE: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technologies, pp. 35 1-335,,,, Online, June 2021. Association for Computational Linguistics. doi: 10. 18653/v 1/2021 maaccl-main.276 URL
https: / /tttss:/accaaniie http://aaaaatttooo........ .rge 2021. naacl ma in.inn.. 276.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Neural Information Processing Systems (NeurIPS), 2023.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. STaR: Bootstrapping reasoning with reasoning. Neural Information Processing Systems (NeurIPS), 2022.
Maosen Zhang, Nan Jiang, Lei Li, and Yexiang Xue. Language generation via combinatorial constraint satisfaction: A tree search enhanced Monte-Carlo approach. In Findings of the Asso- ciation for Computational Linguisties: EMNLP 2020, pp. 1286-1298, Online, November 2020a. Association for Computational Linguistics. doi: 10.18653/v 1/2020. http:///////.....ngnnggeeeee findingg-emmll.11. URL pettps://aclantien woganholog........o org/ 2020. f ndi nni ngs- emm 1 pp 115.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. BERTScore: Evaluating text generation with BERT. International Conference on Learning Representations (ICLR), 2020b.
Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi. Teaching algorithmic reasoning via in-context learning. arXiv preprini arXiv:221 1.09066, 2022.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. International Conference on Learning Representations (ICLR), 2023.
Wanrong Zhu, Zhiting Hu, and Eric Xing. Text infilling. arXiv preprint arXiv:1901.00153. 2019.
Heiko Zimmermann, Fredrik Lindsten, Jan-Willem van de Meent, and Christian A. Naesseth. A variational perspective on generative flow networks. Transactions on Machine Learning Research (TMLR), 2023.
16

---

Published as a conference paper at ICLR 2024
A ADDITIONAL BACKGROUND
A.1 GLOSSARY OF RL FOR LLMS
We provide definitions for key terms used throughout the paper, with focus on their relevance to our setting of fine-tuning large language models (LLMs).
Reinforcement learning. Reinforcement learning (RL) is branch of machine learning concerned with how agents should take actions in an environment to maximize cumulative rewards. In our context, RL is used to fine-tune the reciiiinnmmkknng process of LLMs to improve their performance on specific tasks. For a more comprehensive overview, we refer readers to Sutton & Barto (2018).
Policy. In RL, a policy is a strategy that defines the behavior of an agent by mapping states of the environment to actions. In our setting, policy dictates how the language model generates text sequences based on the current context and learned parameters.
Reward. A reward is a signal that evaluates the quality of an action taken by an agent in a particular state. In the fine-tuning of LLMs, rewards are used to guide the model towards generating more desirable text, such more accurate predictions or more coherent continuations. More specifically, in the context of GelswNets, the reward corresponds to the unnormalized posterior probability, and the GFlowNet aims to match it by learning policy that generates samples proportional to their reward.
Matching a distribution. Matching a distribution involves training model approximate target probability distribution. In our work, this concept is applied to fine-tune LLMss so that their generated text matches the desired characteristics, such adhering to particular style content constraint.
Policy gradient methods. Policy gradient methods (such as PPO) are subset of RL. algorithms that optimize policy by computing gradients of the expected reward with respect to the policy parameters. In the context of LLMs, these methods are used fine-tune the model's parameters to increase the likelihood of generating high-reward text sequences.
A.2 LEARNING OBJECTIVE
We use the subtrajectory balance learning objective for training GFlowNets (Madan et al., 2023). In the notation of that paper, with forward policy Pr, backward policy PB and state flow function F, the objective over a partial trajectory
Sn is defined as follows
L (2:0) = log F(sm) M (F(See)
F(sn) 1 P (3(S)(5H)))
In the case of autoregressive generation of sequence of tokens in fixed order (left-right). the generative process is a tree, so there is only a single path to each state, and each state has a single parent. Thus, PB(s|s') trivially. Additionally, since each state is valid terminable state, can incorporate the modification to account for this from Delcu et al. (2022). Specifically, note that at convergence F(s,) = R(s,))/F(( separately, reducing additional complexity. The only learned object is now the forward sampling policy PF, which we refer to as 9GFN. Summing this over all partial trajectories a trajectory with equal weight (1 1), we get the final learning objective in Eq. (3).
have R(s,)) ) PF
sn). Using this, we can simply substitute sn) in Eq. (4). This allows us to avoid parameterizing a flow function
B SENTENCE CONTINUATION
Additional details. We choose sentences as the level of gramularity for our sequence continuation task because they are natural unit of generation in language. Moreso than individual words, sentences are analogous to whole thoughts, reasoning steps, etc. because the compositional rules of syntax operate at the level of sentences. Whereas the meaning of word is often ambiguous and context-dependent, the meaning of a sentence tends to be more self-contained.
A naive solution to the task is to simply sample autoregressively from the LM until a "." token is reached, marking the end of the sentence. In practice, however, this is unlikely produce samples that have high likelihood because the distribution over sentences has a long tail: a vast number of sentences have small but non-zero probability, and collectively account for substantial portion of the total probability mass. Instead, it would be desirable to sample from a low-temperature distribution over sentences, that the distribution becomes more sparse and highly probable sentences
17

---

Published as a conference paper at ICLR 2024
sampled more often. However, this in itself is difficult distribution to sample from, and it is different from simply sampling autoregressively from the LM with a lower temperature applied to the distributions over the next words. For instance, as the temperature approaches 0, the desired distribution over sentences should converge to an argmax and produce the single most likely next sentence. However. if we try to apply the temperature to the distribution over the next words and sample autoregressively, then as the temperature approaches O, the resulting policy will greedily construct a sentence by sequentially picking the most likely next word, which is unlikely to produce the highest probability sentence.
Our GelowNet sampling policy parametrizes a distribution PGEN (w using the same archi- tecture the LM PLM(WWA) [W1r), and at the start of training, it is initialized with the same weights. The initial state of the GFlowNet consists of a prompt 11: that conditions generation (i.e., the text whose next sentence we trying sample). Each subsequent state Wi:k+i is extension of this text and is obtained by sampling autoregressively from PGPN. This process terminates when a period "." is sampled, indicating the end of the next sentence. If predefined maximum sentence length is reached, we force a "." action to manually terminate the generation.
The GFlowNet policy is trained to sample sentences in proportion to their tempered probability under the original LM given the prompt. Denote the length of the prompt by k. the length of the sampled sentence by m, and the temperature by T. Then, the reward is:
P W = (n PLM (IPM(WwalWwaH-))) +
m
R(W1:k+m) def
(5)
The intuition behind this reward is that we are assuming that the LM can reliably assign high likelihood to high-probability sentences, that we wish to preferentially sample over low-probability ones. If we were to set 7 = 1, then the solution for the GFlowNet would be sample proportionally to PLM (which it is initialized to). The preferential sampling of high-probability sentences is therefore obtained by setting 0 T
To run our experiment, we obtained a dataset of 1000 prompts from OpenWebTerx (Gokaslan et al., 2019) that were each 1-3 sentences long, 50 of which were used for validation. Our LM consisted of a pretrained 1.5B parameter GPT2-XL (Radford et al., 2019), and our GFlowNet was a copy of this model that was fine-tuned in a lower-dimensional space of 80M parameters using LoRA (Hu et al., 2022).
Additional results. We show examples of empirical next-sentence samples generated by our model in Table B.1 compared to baselines. The model was trained using reward temperature of 0.875, which achieves good balance between log- ligelhoodoo and diversity.
C INFILLING STORIES
Additional details. See Table C.1 for training examples from the subset of the ROC Stories dataset used for the task. To condition the model on X and Y, as well as for the prompting baseline, we use the following prompt:
"Beginning: {x)in End: {n)nn Middle:
An assumption we make in this paper is that the base language model already contains the knowledge required for the task, and the goal is to perform inference over this knowledge. However, for this task, none of the pretrained base models were good at assigning high likelihoods to plausible stories. Thus, we fine-tuned GPT-2 Large model (Radford et al., 2019) the entirety of the stories dataset and used this fine-tuned model as the reward model. This was done with full fine-tuning using the trl library (von Werra et al., 2020). We trained for 20 epochs with batch size of 64 and 32 gradient accumulation steps and a learning rate of 0.0005.
We detail the hyperparameters used for training GFlowNets in our experiments in Table C.2. During training, we sample (X, Y) from the dataset and then sample (batch size) Zs for every (X,Y), before using PIM (ZY) as the reward. We use the replay buffer described in $3.3 and seed it with rationales from the dataset. We linearly anneal the reward temperature, the temperature of the behavior policy during training, and the learning rate during warmup. For supervised fine-tuning, we use a batch size of 256 and train for 10 epochs with a learning rate of 0.0001 with rll and LoRA. At test time, we sample 1024 infills for each example in the test set from all the models at temperature 0.9, and average over 10 such draws.
18

---

Published as a conference paper at ICLR 2024
Table B.I: Empirical examples for sequence continuation. Sentences generated from GelowNet fine-tuning tend to be more reasonable than autoregressively generated samples from the LM with temperature 1.0 and tend to be more diverse than samples generated from diverse beam search. Input Prompt:
The matching campaign starts on Friday and will continue through midnight on Monday.
Sampling w/ T = 1.0:
(1) Participate with a fairly low S1 donation so we can motivate more volunteers on Sunday afternoon.
(2) However, the information regarding Cutler's suspicious death
may not become widely known until early in the week.
Diverse beam search:
(1) If you are interested in participating in the matching campaign. you can do so by clicking here. (2) There no limit the number of times you can enter.
GFlowNet fine-tuning:
(1) If you are interested in signing up you can do so here.
(2) Please share.
Input Prompt:
want hockey to become a huge thing in Houston.
Sampling w/ T = 1.0:
(1) We'll be here in Texas from late November till end March.
(2) To that end, I've been following the Houston Aeros (Houston Dynamo's 's minor-league affiliate) and their AHL affiliate (the Texas Stars).
Diverse beam search:
(I) That's why I'm here.
(2) That's why I'm doing this.
GFlowNet fine-tuning:
(1) This is something I've always wanted.
(2) When I was teenager in middle school, went to the Ice Arena in University of Houston and loved it.
Input Prompt:
That incident got a lot of attention in part because was captured on video. Israel said he recorded what happened at the synagogue, and made it public, to document it and leave no doubt about what transpired.
Sampling w/ T 1.0:
(1) He blogged about it here as well. (2) Isralli TV stations broadcast the video before aired in Israel, as the country's 's rule requires.
Diverse beam search:
(1) However, there is no evidence that he did so.
(2) However, there no video of what happened inside the synagogue.
GFlowNet fine-tuning:
(1) He is on tape doing all of it.
(2) It's a message countries can use to deter future attacks.
Input Prompt:
The Rolling Stones Concert has since been released solely by ABKCO Records. The band would remain incensed with Klein for decades for that act. Klein died in 2009.
Sampling w/ T = 1.0:
(1) Actually art has become as obscene as investment banking.
(2) Some believe he shot himself in the chest.
Diverse beam search:
(1) The Rolling Stones, however, would not be deterred.
(2) The Rolling Stones would go on to release their own version of
The Rolling Stones in Concert.
GFlowNet fine-tuning:
(1) He received a lifetime achievement award from the Jazz Times.
(2) Sometimes it seems like we are destined to repeat our own mistakes.
19

---

Published as a conference paper at ICLR 2024
Table C.I: Examples of training samples for the stories infilling task.
Beginning (X)
Middle (Z)
End (Y)
going a Halloween party. looked through my clothes but could not find costume. I up my old clothes and constructed a costume.
put my costume and went to the party.
My friends loved my cos- tume.
Allen thought he was very talented poet. He attended college to study cre- ative writing. In college, he met a boy named Carl.
Carl told him that he
Because of this, Allen swore off poetry forever.
wasn't 't very good.
Table C.2: Hyperparameters for the story infilling task.
LoRA rank
64
LoRA scaling factor
LoRA dropout
0.1
Batch size
64
Gradient accumulation steps
16
Learning rate
0.0001
Optimizer
Adam W
Pr temperature max
2
Pr temperature min
0.5
Reward temperature start
1.1
Reward temperature end
0.85
Reward temperature horizon
100
Buffer capacity
25
Number of training steps
1000
Evaluation temperature
0.8
Maximum length
25
Minimum length
5
20

---

Published as a conference paper at ICLR 2024
Analysis. Table C.3, C.4, C.S, C.6, C.7, C.8 illustrate some examples of the infills generated with the prompting baseline, supervised fine-tuned and GFlowNet fine-tuned models on the test examples.
For evaluating the infills we generate rating for the stories with infills from each of the methods based on the coherence of the story. We average the rating over 10 infills sampled from each method and average it over all the 100 test examples. The prompt used for evaluation the following, adapted from Liu et al. (2023). Stories with infills generated by the GFlowNet fine-tuned model on average receive much higher rating than the baselines. The average rating for the reference infills is 4.3 which should be viewed as an upper bound, as the stories may potentially be present in the GPT-4 training data.
You will be given a short story.
Your task is to rate the story on one metric.
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
Evaluation Criteria:
Ccherence (1-5) the collective quality of all sentences. The story should be well-structured and well-organized, The story should not just be a heap of related information, but should build from sentence to a coherent narrative. The story should also be realistic and all the sentences when put together should make sense.
Evaluation Steps:
1. Read the story carefully. Check if the story is coherent and all the sentences follow logical order.
2. Assign score for coherence on scale of 1 to 5, where is the lowest and
the highest based on the Evaluation Criteria. If there are grammatical errors or logical inconsistenciess
simply assign lower score and do not elaborate on the reasoning.
Example:
Randy had recently separated from his wife.
He felt very lonely and sad all the time.
considered trying to patch things up with Vera, his ex-wife. He decided to get a puppy instead. His decision made him happy and he no longer felt sad.
Coherence:
Randy had recently separated from his wife. He felt very lonely and sad all the time.
considered trying to patch things up with Vera, his ex-wife. Eventually Randy missed Vera very much and developedWhile making memories with
His decision made him happy and he no longer felt sad.
Coheerence
21

---

Published as a conference paper at ICLR 2024
story:
((Story))
Evaluation Form (scores ONLY)
Coherence:
D SUBJECTIVITY CLASSIFICATION
Additional details. See Table D.I for some training examples in the SUBJ dataset. We use the prompts in Table D.2 for all baselines.
We run GFlowNet tiee-tunigg for 1000 steps with a linear warmup over 200 steps, a fixed learning rate of 0.0005, and batch size of 512 samples; Table D.3 for all the hyperparameters used. Each batch consists of 8 queries (X's). randomly drawn with replacement. We then sample 64 rationales (Z's) for every X, before using PLM ZY| X) the reward. We also use the replay buffer described in $3.3 and seed it with potential rationales generated from PLM For supervised fine-tuning, both on its own and on top of GelowNet fine-tuning, we use batch size of 256 with 8 queries, randomly drawn with replacement. We train for 100 steps with a linear warm-up over 20 steps and a constant learning rate. We sweep the learning rate in [0.001, 0.0001] and report the best performance. The reward temperature is annealed from 1.2 down to over the first 150 steps of training.
Additional results. To encourage exploration in the space of Z, we inverse-prompt the model to generate potential chains of thoughts and store them in replay buffer at the beginning of training. The replay buffer is a priority queue indexed by the reward. Under-performing chains of thoughts evicted as we collect more trajectories. We ablate the effect of seeding the replay buffer with inverse-prompted chains of thoughts and aggregating multiple chains at test time in Table D.4.
E INTEGER ARITHMETIC
Additional details. See Table E.1 for examples from synthetically generated training data for the integer arithmetic task.
As with the infilling task, for the integer arithmetic task, the pretrained base models we tested were imperfect knowledge models, i.e. incorrect rationales are sometimes assigned very high likelihood. To assign the correct rationales higher rewards, we prepend some demonstrations with hand-crafted rationales to the query when computing the reward, i.e. PLM( (LLM(XY][(XX) (X;Zx)) experiments. This improves the calibration to some extent. Note that these (X;, Zi, Y,) are taken from the dataset and used only in the reward. The GFlowNet policy here is conditioned only on X, iee., (
where
3 in our
We detail the hyperparameters used for training GelowNets in our experiments in Table E.2. During training, we sample (X, from the dataset and then sample (batch size) Zs for every (X,Y), before using PLM (XZY](XX whenever a "=" is generated we extract the preceding expression and evaluate the last two terms using eval in Python. We use the replay buffer described in $3.3 and seed it with rationales from the dataset PLM- We linearly anneal the reward temperature, and the learning rate during warmup. aggregate the response over multiple samples drawn from
the reward. During the generation of Z, as mentioned in $4.4,
During evaluation, for all methods, the model at some fixed temperature. For the zero-shot bascline, we observe the best performance with the "Let us think step by step." prompt appended the end of the question. For supervised fine-tuning and PPO, we use the implementation from rrl. For supervised fine-tuning we use batch size of 128 with 8 gradient accumulation steps and train for 10 epochs with learning rate of 0.0001 using LoRA. For PPO we use a minibatch size of 64 with 16 gradient accumulation steps, a learning rate of 0.0001, 4 epochs (on the minibatch), clip range of 0.2, and adaptive KL coefficient initialized at 0.2.
Effect of the number of rationales. As mentioned in $4.4, we seed the buffer with rationales from dataset. Table E.3 summarizes the results of ablation on the number of examples used seed the buffer. We observe, as expected, that the performance generally improves as we the number of examples used to seed the buffer grows. When buffers are used to seed the buffer, the performance is quite poor. We hypothesize that without the good rationales used to seed the buffer, the exploration problem of searching for good rationales becomes very challenging.
22

---

Published as a conference paper at ICLR 2024
Table C.3: Examples of infills generated by prompting baseline along with reference infills for the stories infilling task.
Beginning (X)
Middle (Z)
End (Y)
He decided to get puppy instead
He now happily had new wife who really liked.
Randy had recently separated from his wife. He felt very lonely and sad all the time. He considered trying to patch things up with Vera, his ex-wife.
His wife missed him, but for the wrong reasons. Now
His decision made him happy and he no longer felt sad.
He finally found someone he could be happy with. It was
He began spending more time with his new girlffriend and friends.
Randy remained single and mostly satis- fied with that. He found his
When Randy was happy he went to the aquarium, place He started going out more with display of male affection.
He bought Vera a moonstone and gave it to her as
He and Vera got back together, even though they had stayed He introduced Paige to his new employ- ees and they were great. Randy happily married Vera and they have healthy relationship He found new lady with lot of the same interests
He didn't want to wait for another beer to get cold He went to sleep and came awake with an angry stomach.
Ken put bottle of beer in the freezer. He heard a popping noise. He looked in the freezer and saw the bottle had burst.
Ken had another beer as a treat. After the third beer
He drank warm beer instead.
He stored the beer in the freezer and for- got about it.
He walked around the bar a bit. He thinks he'll
Weeks later, Ken preferred his cold beer. However, he
He poured a fresh bottle with a sigh of relief. It
He went to the bar for cold beer. No one knew
Ken hung up the phone on his friend. He didn't
He bought some veggies from the gro- cery store. It was
Ken watched the movie Goalkeeper. He was glad the cold
He laid down for a rest. His belly was full!
He laid down for the night and finally sleep again. The
23

---

Published as a conference paper at ICLR 2024
Table C.4: Additional examples of infills generated by the prompting baseline along with reference infills for the stories infilling task.
Beginning (X)
Middle (Z)
End (Y)
The room did not have pencil
Jasmine made her pencil and done her work. She looked Her mom brought her pencil. Jasmine wrote her homework
Jasmine had homework to do. She did not have pencil. She looked around the room for one,
Jasmine decided not to do her homework.
Jasmine took her pencil out of the room. She now
Jasmine pretended not to have done her homework. She got
Her brother finally brought pencil, They could finally start! Her brother brought a pencil and solved her problem. Jasmine Her brother brought a pencil and solved her problem. Jasmine Her friend offered her pencil. Jasmine used the pencil
Her teacher saw her looking her phone. Then, Jas
Her mom came in and handed her the extra pencil! Jas Her mom brought her pencil. Jasmine then got to Jasmine sent her friend money do it. She was
Jane arrived at work an hour late Despite arriving late, Jane greeted warmly by her coworkers Jane learned that you never can predict the future. It was Jane learned not to rush into anything. Jane made a good Many of her peers were not pleased with her. Jane got
Jane had recently gotten a new job. She was nervous about her first day of work. On the first day of work, Jane overslept.
Jane did make good impression at her new job.
Jane ended up quitting her new job early. She was happy More time with the wrong people gave her anxiety. She was Jane learned that you do not mess up at job well
Jane realized she needed to find new work contacts.* 5. At
After her first day of work, Jane was too tired to
Secretly, Jane hated her new job. It made her
Jane realized she worked at mid-sized company with long
Jane's 's boss realized she did not show to work on
24

---

Published as a conference paper at ICLR 2024
Table C.S:
Examples of infills generated by the supervised fine-tuned model along with reference infills for the stories infilling task.
Beginning (X)
Middle (Z)
End (Y)
He decided get a puppy instead
He started enjoying his time with Vera even more. Now.
Randy had recently separated from his wife. He felt very lonely and sad all the time. He considered trying patch things up with Vera, his ex-wife.
Eventually Randy missed Vera very much and developedWhill making mem- ories with
His decision made him happy and he no longer felt sad.
He finally decided he had enough pain for no good reason.
He began spending more time with his new girlfriend. Now he For the first time since the divorce he started to make new Soon Randy and Vera were rekindling their romance. They
He started seeing someone new day. Now, he feels
He signed up for class in Same Sex Loving Uniqueness For the first time in months, Randy and Vera celebrated a
He introduced Paige to his new employ- ees. They are now good After some contemplating, he decided to patch things up with Vera He now looks for another partner to joy his life with.
He didn't want to wait for another beer to get cold
The popping noise was a freezer needle getting caught. It fell
Ken put bottle of beer in the freezer. He heard popping noise. He looked in the freezer and saw the bottle had burst.
Ken had a cold for thirty minutes. After the popping noise He stored the beer in the freezer. The pouring sound was
He drank warm beer instead.
He didn't know why didn't stop the ringing.
Weeks later, Ken knew the reason for the freezing. It
Later Ken learned the gas is out gas and will end
He left the freezer for the beer. It stayed in there
Before knew it, the popping noise was gone. He
The popping noise that Ken heard indi- cated it should have freezer shut
The popping noise became louder and more frequent. Now Ken's He froze the beer and drank it from a mason jar He went the store buy another. Now, he
25

---

Published as a conference paper at ICLR 2024
Table C.6: Additional examples of infills generated by the supervised fine-tuned model along with reference infills for the stories infilling task.
Beginning (X)
Middle (Z)
End (Y)
The room did not have pencil
Jasmine decided to talk her mom about it. They
Her dad now has teach her math! Pick- ing up
Jasmine had homework to do. She did not have a pencil. She looked around the room for one,
Jasmine decided not to do her homework.
Jasmine took her math test later in the day. She
Jasmine got A her assignment. She continued to
Her brother finally brought pencil, They could finally start! Her brother brought a pencil and solved her problem. Jasmine Jasmine used her pink and brown high- lighter instead.
Jasmine went to school. She was able to finish her
Jasmine made her pencil. At school, everyone was
Her brotherly earrings helped her stay more. While
Jasmine made her pencil. She ended up getting good
Jasmine sent her sister to do it. Now it was
Jane arrived at work an hour late After arriving home, Jane realized her apartment was not ready.
After work, Jane returned home and ate lousy sandwich.
Jane had recently gotten a new job. She was nervous about her first day of work. On the first day of work, Jane overslept.
Jane did make good impression at her new job.
After her first day of work, Jane learned that mistakes happen
Due to her bad first day, Jane was fired from her
Ending: Jane ended up late for work. Although she
On her 2nd day of work, Jane came home late On the second day of work, Jane arrived late and unprepared By the end of the day, Jane was tired and weary
After her first day of work, Jane's boss recommended her Secretly, Jane hated her new job. Se- cretly.
After work, Jane felt worn out and tired. The end
By the end of the day Jane felt confident and like she
26

---

Published as a conference paper at ICLR 2024
Table C.7:
Examples infills generated by the GFlowNet fine-tuned model along with reference infills stories infilling task.
Beginning (X)
Middle (Z)
End (Y)
He decided to get a puppy instead
He went to friend's house to talk. He took Vera the park for picnic. He finally decided to call Vera on the phone.
Randy had recently separated from his wife. He felt very lonely and sad all the time. He considered trying patch things up with Vera, his ex-wife.
His decision made him happy and he no longer felt sad.
He asked Vera to meet him and they did. He decided get dog for his family.
He went to Vera's 's house. He told Vera that he was ready to love again.
He bought Vera nice gift. He had two dogs and felt like he missed Vera.
He had good couple of years.
He wanted get Vera and he eventually did.
He wanted to sit in
alone.
He didn't want to wait for another beer to get cold
He decided to stop drinking beer. Ken suddenly smelled a bottle of beer.
Ken put bottle of beer in the freezer. He heard a popping noise. He looked in the freezer and saw the bottle had burst.
He thought it was something He walked around the freezer to see the bottle.
He drank a warm beer instead.
He tried to clean up the freezer. He was angry when he had to drink it. He stepped down from the freezer He made a big joke that night. He looked in a jug and see was, out. He then looked for beer the freezer. He tried to enjoy the rest. He watered the bottle,
27

---

Published as a conference paper at ICLR 2024
Table C.8: Additional examples of infills generated by the GFlowNet fine-tuned model along with reference infills for the stories infilling task.
Beginning (X)
Middle (Z)
End (Y)
The room did not have pencil
She was in panic.
She had to place the pencil in her pocket.
She had several her classmates.
Jasmine had homework to do. She did not have pencil. She looked around the room for one.
She looked for a neat pencil.
Jasmine decided not to do her homework.
She thought she looked everywhere. Her brother walked in and borrowed her pencil.
She only had a few minutes to do it. She never had her pencil. She saw a pen in her closet. She searched everywhere for pencil. She went to her room.
She didn't find one.
Jane arrived at work an hour late She lost her job after the first day of work.
She was egged on by her boss. She decided over hyp her first day. She arrived late and missed her first day of work.
Jane had recently gotten new job. She was nervous about her first day of work. On the first day of work, Jane overslept.
Jane did not make good impression her new job.
She was nervous about going to work. She ended up getting good job. Jane waited almost day to get to work. Her first day was very difficult. She made a couple of phone calls. She arrived at the office and was fired.
She was all out coffee. Jane was surprised she did not make good impression.
Table D.1: Two training examples from the SUBJ dataset.
Text (X)
Label
another story follows the relationship between stepfather neeson ) and his young stepson. hoffman 's performance authentic to the core his being.
objective subjective
28

---

Published as a conference paper at ICLR 2024
Table D.2: Prompts used for subjectivity classification.
Few-shot learning Supervised fine-tuning
GFlowNet fine-tuning
Classify this movie review as objective or sub- jective: "[X]" This review is [Y].
Classify this movie review as objective or sub- jective: "[X]" This review is [Z], so it is (Y).
Table D.3: Hyperparameters for GFlowNet fine- tin-ttning on subjectivity classification.
LoRA rank
256
LoRA scaling factor
16
LoRA dropout
0.
Batch size
16
Gradient accumulation steps
32
Learning rate
0.0005
Optimizer
AdamW W
PF temperature max
2
PF temperature min
0.5
Reward temperature start
1.2
Reward temperature
Reward temperature horizon
150
Buffer capacity
50
Number of steps
100
Number of samples
Maximum length
5
Minimum length
Analysis. In Table E.4 we show some examples generated by PPO and GFlowNet fine-tuned models. We observe that PPO generates sequences with high rewards under the base model. These sequences, however, are not valid expressions, and, in fact, do not even call the tool. Instead, the model learns to simply repeat the expression. Repetitions being assigned high likelihood an issue that has also been noted by Welleck et al. (2020). On the other hand, GFlowNets are able to generate the correct rationale to evaluate the expression.
There are still cases where the GFlowNet fails to produce the correct reasoning chain. We illustrate some of these examples in Table E.5. The errors fall mainly into categories: 1) missing operands in longer OOD problems 2) incorrect operand being copied 3) incorrect operator being copied. The potential source for such errors is that the reward model assigns equally high rewards to rationales with these minor mistakes and thus the model generates them with likelihood. These errors can potentially be reduced by using better and potentially larger reward models.
Table D.4: Ablation studies on subjectivity classification using GPT-J 6.8B.
Method
Test accuracy (%)
Training samples
10
20
50
GFlowNet fine-tuning
71.4 1.8
81.1
87.7
(-) Seed buffer
64.7 8.9
68.7 1.7
7..
(-) Seed buffer (-) Aggregating
63.9 7.1
2.4
3.3
29

---

Published as a conference paper at ICLR 2024
Table D.5: Top sample rationales for the SUBJ test set using the instruct-fine-tunt fine-tuned GPT-J 6B as both the reward model and the base model for the GFlowNet, which is trained on 50 labeled examples.
True label
Rationale
Frequency
This review is
about a factual event describing factual event based on a factual statement about a historical event about a movie review based on facts about a factual statement
11.45%
10.23%
9.47%
Objective
8.77%
7.41%
5,64%
4.61%
This review
about a movie review about the movie experience about the movie describing movie review about the movie review based on a fictional story describing the movie experience
26.21%
26.17%
7.60%
Subjective
3.44%
2.31%
2.30%
2.01% %
Table E.1: Examples from the integer arithmetic dataset. Note that the results of the two-term expressions evaluated the calculator.
Question (X)
Rationale (7.
Answer (Y)
Question: 6-0-4 4 4-8 = Answer:
6-0=,6-44
2- -
The answer is -6.
Question: 9 4--4---0 8 Answer:
9+4
The is 5.
Table E.2: Hyperparameters for the Integer Arithmetic Task
LoRA rank
64
LoRA scaling factor
16
LoRA dropout
Batch size
16
Gradient accumulation steps
32
Learning rate
0.0001
Optimizer
AdamW
PF temperature max
2
Pf temperature min
0.5
Reward temperature start
1.1
Reward temperature end
0.5
Reward Temperature horizon
150
Buffer capacity
50
Number of training steps
1000
Evaluation temperature
0.1
Number of samples
10
Maximum length
20
Minimum length
5
Table E.3: Ablation study to understand effect the number rationales used to huffer on the test.
accuracy (%) for the integer arithmetic task.
Number of operands
In-distribution
OOD
Number of seed rationales
3
4
5
22.6
5.8
3.4
10
58.6
52.3
20.2
25
48.8
56.8
24.6
50
95.2
75.4
40.7
30

---

Published as a conference paper at ICLR 2024
Table E.4: Samples generated by PPO fine-tuned and GFlowNet fine-tuned models.
Question (X)
Generated rationale (Z)
log R
Question:
+
?
1-9-8
-13.17
Answer:
1-9 = -8
-27.75
Question:
8 + 7
7 ?
8+7+2+7
-2.39
Answer:
8+7= 15, 15 + 2 17,
-11.72
Question: 7-5+8-0-6-
7-5+
-1.22
Answer:
7-
10, 10 10,
-
-7.99
=4
Table E.S: Examples of errrr made by the GFlowNet- fine- tuned model.
Question (X)
Generated rationale (Z)
Question: 7 Answer:
7+ 10 9
1
135 12 16,
Question: 8 Answer:
4-
045-5,5--6211116---77777
Question: Answer:
31


Reviews:
[{"review_id": "7T1yZ54H62", "reviewer": "Reviewer_ZdVU", "summary": "Many applications of LLMs like text infilling and constrained generation requires probabilistic inference that is intractable for LLMs. E.g., for the task of text infilling we need to be able to compute the conditional probability p(text | prefix, suffix). The paper proposes to use tackle this problem by fine-tuning LMs with GFlowNet. Specifically, the author proposes to fine-tune LMs to approximate the desired conditional distribution, e.g., p(text | prefix, suffix) by mat. This paper conducted empirical evaluations on various benchmarks including text infilling and numerical reasoning.", "questions": "See above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Empirical results on synthetic arithmetic reasoning benchmarks seem to be very strong.", "weaknesses": "Overall the paper is hard to follow: the authors provide little background on reinforcement learning and GFlowNet training. In particular, the authors use many terminologies without/before defining them clearly, examples include “policy”, “reward”, “matching” a target distribution, “rewarding all valid integers equally leads to an expected gradient of zero for policy gradient methods.”\n\nIn section 2, by looking at the problem of using LLMs to generate random numbers between 0 - 100, the authors try to motivate the use of GFlowNet instead of PPO training. PPO training does not resolve the distribution skew because the reward function only considers whether the number lies between 0 - 100. One correct way to do it could be asking the LLM to generate a **sequence** of numbers sampled from 0 - 100 uniformly and assign a positive reward only if the frequency of the numbers are close to uniform. A major part of the introduction focuses on intractable posterior inference/conditional probabilities and the fact that Section 2 mentions nothing about them makes it hard to follow.\n\nIn section 3 the authors introduced some related problems in NLP that could potentially be solved by GFlowNet and in section 3.3 on page 5 that the authors finally describes GFlowNet and their training objective. What is the original subtrajectory balance objective? How do you modify it? What is the semantics of your objective function? Answer to these questions can help distinguish GFlowNet from other approaches from the methodology perspective.\n\nBesides, some important related works of the field are missing from Section 3:\n-for temperature scaling: \n[1] leverages importance sampling to fine-tune LM p(x) such that it approximates the desired distribution p(x)^{1/T}. Their approach suffer from various problems such that the variance of loss is high due to the exponent 1/T. Given that the authors study this empirically, does the GFlowNet objective also suffer from this issue? If so, how is it resolved?\n\n-for text infilling:\n[2] and [3] both studies the problem of text infilling where [2] adopted a fine-tuning based approach. [6] and [7] tackles this problem by training insertion-based language models.\n\n-for constrained generation:\n\n`Current approaches to the problem use tokenwise approximations (Liu et al., 2021) or various problem-specific beam search and local search techniques`\n\nOther than search-based approaches, frameworks like FUDGE [4] and NADO [5] trains auxiliary models (classifiers) and combine it with LMs to approximate the desired conditional distribution. \n\nTo summarize, GFlowNet seems to be a very very general framework that allows you to fine-tune an LM to approximate any distribution that is proportional to an arbitrary reward function r(x). Despite the experiment results showing advantages against vanilla baselines, the authors did not make a strong argument showing why GFlowNet would work better on these downstream tasks against existing approaches, including the ones mentioned above.\n\nThe main argument may be stronger/clearer if the authors focus more on the chain-of-thought reasoning part other than trying to provide a generic solution to all intractable inference for LMs.\n\n\n\n[1] Shih, Andy, Dorsa Sadigh, and Stefano Ermon. \"Long Horizon Temperature Scaling.\" arXiv preprint arXiv:2302.03686 (2023).\n\n[2] Donahue, Chris, Mina Lee, and Percy Liang. \"Enabling Language Models to Fill in the Blanks.\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.\n\n[3] Zhu, Wanrong, Zhiting Hu, and Eric Xing. \"Text infilling.\" arXiv preprint arXiv:1901.00158 (2019).\n\n[4] Yang, Kevin, and Dan Klein. \"FUDGE: Controlled Text Generation With Future Discriminators.\" Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.\n\n[5] Meng, Tao, et al. \"Controllable text generation with neurally-decomposed oracle.\" Advances in Neural Information Processing Systems 35 (2022): 28125-28139.\n\n[6] Lu, Sidi, Tao Meng, and Nanyun Peng. \"Insnet: An efficient, flexible, and performant insertion-based text generation model.\" Advances in Neural Information Processing Systems 35 (2022): 7011-7023.\n\n[7] Susanto, R. H., Chollampatt, S., and Tan, L. Lexically constrained neural machine translation with levenshtein transformer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H7Xo9kzIBm", "reviewer": "Reviewer_H3x9", "summary": "This paper presents a new technique for fine-tuning LLMs, to perform amortized inference in probabilistic models (also defined using LLMs). This enables tuning LLMs for more interesting objectives than traditional RL or supervised fine-tuning techniques. The authors present several examples of such objectives: optimizing chain-of-thought reasoning so that it more often leads to the correct answer, optimizing for useful tool use, infilling plausible middles of stories with beginnings and ends, and whole-sentence temperature sampling for higher-quality sentence completion. In each, the proposed method is shown to outperform baselines.", "questions": "# Questions\n\n* Around how long (in wall-clock time) does it take to LoRA fine-tune a GFlowNet on your tasks, e.g. for a 6B-parameter model?\n\n* In Table 3, how were Test Accuracy numbers in the final row (\"+ Supervised Fine-Tuning\") generated? Were 10 samples of Z taken from the fine-tuned model, and aggregated via voting? Or are the Z samples still generated from q_{GFN} but now completed with Y drawn from the fine-tuned LM? Or is there no longer a voting procedure?\n\n* In principle, for tasks like story infilling, supervised fine-tuning (SFT) should be optimizing the same architecture as the GFN for an objective that has the same optimum (i.e., SFT is also a distribution-matching objective, where the optimum is the intractable posterior). Qualitatively, how do the samples from the SFT baseline look? (I think it would be nice to add them to Table B3 if possible!) If they are noticeably worse than the GFN samples, what would you attribute that to? Also: for SFT and for the \"just prompt the model to infill\" baseline, do you start with the base language model, or the reward language model that you fine-tuned with stories?\n\n* What is \"reward temperature horizon\"? What are the P_F min and max temperatures? (I saw that reward temperature was annealed during training, but did not see a reference to annealing the QFN's own temperature.)\n\n* You write on p23 that the reward model could often not distinguish between good and bad rationales. Does this mean that, given a prompt (e.g.) Z=\"..., 1 + 4 = 5. The answer is:\", the reward model assigns roughly equal probability to (a) the known correct answer Y from the training data (e.g., Y=14) and (b) the most-recently computed number (in this case, 5)? That's somewhat surprising to me!\n\n* For sentence continuation, do you see interesting pathologies at lower reward-model temperatures (e.g., bias toward very short completions, or very repetitive completions)?\n\n# Minor Comments\n\n* There are a couple points that I found confusing when first reading the paper, even though they are clarified later. \n\n  (1) The clause \"finding the most likely sequence continuation\" in the first paragraph was confusing. \"Finding likely sequence continuations\" is precisely what LLMs are trained to do, and would not seem to require intractable inference; I considered briefly that you might mean finding the literal maximum-probability sequence, but that also seemed wrong because I was expecting a list of posterior sampling tasks, not optimization tasks. Later I realized that you meant long-range (i.e., not per-token) reduced-temperature sampling, but this wasn't obvious from the intro.\n\n  (2) At multiple points you discuss chain-of-thought reasoning as an instance of intractable inference, with the formula P(Z | X, Y). But at test time, in chain-of-thought reasoning tasks, we do not *see* the answer Y, so it's not really a posterior sampling task. (If I give you only a single instance of a 'problem' in the chain-of-thought reasoning task, there is no clear MCMC target distribution you could specify over \"good chains of thought\" without already having access to the final answer Y. This is in contrast to the other tasks, like long-range temperature sampling, infilling, etc. where the reward $R$ can be evaluated at test time.)  After reading the whole paper, I have a clearer understanding of what it is you're doing in these (very neat) chain-of-thought examples. But their inclusion at the beginning of the paper, without sufficient explanation of how they work, makes it trickier to understand the proposed framework. Even later in the paper, there are two separate resolutions to the question of \"what to do without Y\" -- one is to use the (X, Y) pairs you have in order to generate Z's for fine-tuning (the \"EM\" idea), and the other is to train the GFN without access to Y, which could perhaps be interpreted as training it to do posterior inference conditioned on the event that the final answer is correct, rather than on a particular final answer.\n\n* The idea that fine-tuning to do better chain-of-thought reasoning might be viewed as a kind of EM was previously proposed by Dohan et al. (although not implemented).\n\n* At the bottom of p1, one of your citations is to a method that uses SMC, not MCMC, for which the notion of \"mixing between modes\" is not quite appropriate.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This is a very strong paper. Some of its key strengths are:\n\n* There is a very nice, pedagogical discussion of why it may be desirable to sample intractable distributions, which provides great motivation for the proposed approach. The random-number example in Section 2 also nicely illustrates the limitations of reinforcement learning.\n\n* Several researchers have proposed using \"online\" (i.e., test-time) inference methods to sample LLM posteriors, but those methods are only appropriate in settings where the increased cost of exploring multiple samples at test time is not prohibitive. This paper's technique enables offline training, and produces a network that can generate approximate posterior samples directly at test time. What's more, even in settings where test-time inference *is* feasible, Monte Carlo algorithms could use the amortized networks introduced by this paper as proposals, to rapidly speed convergence in cases the amortized networks handle well, and more gracefully handle cases where the amortized networks do not generalize.\n\n* Unlike some formulations of amortized inference, which require exact posterior samples to fine-tune on (e.g., https://arxiv.org/abs/1610.05735), this paper requires only the ability to evaluate the unnormalized posterior (the reward $R$). \n\n* The experiments suggest that this technique is applicable to a compellingly broad range of tasks. The experiments showing that the technique can help train LLMs to perform better reasoning over latent variables (the thoughts in chain-of-thought, or the tool invocations in tool-use applications) are particularly nice.\n\n* The writing is clear (if somewhat terse) throughout.", "weaknesses": "Overall, I really like the paper, but I do think there are a few places it could be improved:\n\n1. **Limited discussion of the training objective and its relationship to possible alternatives.** The training objective is introduced very briefly and without much intuition. I realize that there is an extensive literature on training GFlowNets and there is not space to go into full detail here. But are there reasons that this objective (among many other GFlowNet objectives) was particularly well-suited to the language modeling case? Why GFlowNets at all instead of e.g. reweighted wake sleep (a common method for amortizing intractable posterior inference)? How sensitive is performance to the distribution you use to generate training trajectories? How important is the replay buffer, and how is it populated? In fairness, I am not sure how many of these questions need to be addressed in a short conference paper.\n\n2. **Limited discussion of the limitations of the proposed technique.** Ultimately, the training method given here is a mostly-on-policy reinforcement learning method. A key challenge for such methods is exploration -- finding high-reward samples to reinforce. I would have appreciated more discussion of the sorts of posterior inference tasks that are and aren't likely solvable with the proposed techniques (at least without further innovations), possibly along with potential mitigations for these weaknesses.\n\n3. **Metrics for infilling.** I had reservations about some of the metrics used to evaluate the proposed approach, in particular for the story infilling task. It is unclear that measuring similarity to a single reference sentence is very meaningful--especially since a purported strength of the method is sampling the full posterior. It would be nice if (randomly selected) qualitative examples were presented for all baselines. It may also be worth considering an automated evaluation of the coherence of the resulting story (e.g., by asking GPT-4 to rate coherence). Despite the many (valid) critiques of such LLM-powered evaluations, I do think they are at least a better fit for creative coherent generation tasks like this one than metrics like BLEU.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qLYtirtCmU", "reviewer": "Reviewer_Vqdw", "summary": "This paper investigates the challenge of sampling latent variables from a posterior distribution in large language models (LLMs), where the latent variables might take the form of prompts, reasoning chains, etc. However, sampling from a posterior distribution is typically intractable. To address it, the paper proposes to use generative flow networks (GFlowNets), which sample a composite object (a sequence of tokens) via a sequence of constructive steps, with a probability proportional to a reward function (the product of the likelihood and the prior, leading to the joint distribution). This is different from MLE-based fine-tuning and reward-maximization-based fine-tuning, which tend to make the learned distribution more concentrated on one or few modes, potentially leading to incorrect outputs. In contrast, Bayesian inference aims to learn a distribution that encompasses all possible outputs, thus promoting diversity and preventing from overfitting to a wrong target. The authors used a modifed version of the SubTB training objective for fine-tuning and their experimental results demonstrate the effectiveness of GFlowNets-based fine-tuning in improving text generation and reasoning tasks.", "questions": "## Amortized inference with GFlowNet Objectives\n\n- GFlowNets start with an empty string and add one token at a time in a left-to-right manner. Depending on different tasks, $Z$ should be generated conditional on $X$ or $X, Y$? Here, $X$ or $X, Y$ is omitted?\n\n### Learning objective\n\n- Besides that 1) the ability to avoid to estimate the flow function $F$; 2) SubTB can have a better bias-variance trade-off in GFlowNet training, are there any other benefits to use a modified version? Also, did you try the conventional SubTB objective with the flow function considered?\n\n- Did you consider the hyper-parameter $\\lambda^{j-i}$ over incomplete trajectories with variable lengths $0 \\leq i < j \\leq n+1$, like SubTB($\\lambda$)?\n\n- Given that the generation order is fixed (i.e., left-to-right), it results in $P_{B} = 1$. For readers who might be unfamiliar with GFlowNets, it might be helpful to include an explanation or mention $P_B=1$ somewhere in the paper to ensure accessibility for all readers?\n\n### Parameterization, amortization, and generalization\n\n- $R(Z) = p_{LM}(XZY) \\propto p_{LM}(Z | X, Y)$ --> should be $p_{LM}(Z | X, Y) \\propto R(Z) = p_{LM}(XZY)$?\n\n\n## Empirical results\n\n- How to understand GFlowNet fine-tuning and supervised fine-tuning solely? The former is to train the LM with Eq.3, while the later is to train the LM by maximizing $\\log p_{LM}(XZY)$ with $Z$. Supervised fine-tuning corresponds to the variational EM - first update GFlowNet polices and then LM pamemters with $Z$? Thus, supervised fine-tuning should already include GFlowNet fine-tuning?\n\n- In Table 3, GFlowNet fine-tuning + supervised fine-tuning was considered. Then why not to consider it as well in Table 2 & 4?\n\n### 4.1 Sentence continuation - task description\n\n- $R(Z) = p_{LM}(Z | X)^{\\frac{1}{T}}$ --> should be $R(Z) = p_{LM}(XZ)^{\\frac{1}{T}}$?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Motivation:\n\nOne limitation of existing fine-tuning techniques, i.e., MLE and reward maximization - the learned distribution will end up focusing around one or very few outputs, due to the nature of maximization. If the wrong one was picked up, the consequence could be catastrophic. This is where the Bayesian posterior comes in - it can contain all the information over the potential outputs. However, sampling from a posterior distribution is typically intractable. GFlowNets have recently been shown to approximate a complicated multimodal distribution well. To this end, GFlowNets are used to sample composite latent variables via a sequence of steps, with a probability proportional to $p_{LM}(XZY)$ or $p_{LM}(XZ)$.\n\n\n### Originality:\n\nThe proposed GFlowNet fine-tuning builds on GFlowNets and Bayesian posteriors. The authors utilize GFlowNets as an amortized inference machine, to sample composite latent variables from an intractable posterior distribution in LLMs. This is different from MLE-based and reward-maximization-based fine-tuning techniques. The resulting GFlowNet fine-tuning shows good performance on various tasks. The originality is good.\n\n\n### Clarity:\n\nThe paper is well-organized.", "weaknesses": "Please see the following questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qpeZLq7zrA", "reviewer": "Reviewer_7WZA", "summary": "LLMs are good at auto-regressive sampling, as that is how they are defined and trained.  However, there are many types of inference queries can make LLMs more useful by allowing involved reasoning.  There are multiple examples where this involves sampling from intractable posterior distributions.  \n\nGFlownets (GFN) provide a way to fine-tune LLMs for more specific inference queries as it can tune the generative distribution to a non-next-token reward function.  This is particularly relevant as many current reasoning methods (e.g., chain of thoughts), can instead be thought of as alternative inference queries in a probabilistic model.  \n\nTo motivate the method, the paper illustrates that GFN, in contrast to PPO, can match a posterior distribution over random numbers, while PPO can only ensure that all the samples are valid, without ensuring that the distribution can match.  Supervised fine-turning also works here, but not when there are no samples available to match with, which is not always the case.  \n\nThe paper then goes over a variety of interesting intractable distributions that can now be sampled from with GFNs, allowing tasks like non-local low temperature sampling, infilling and constrained generation.  Note that each of these tasks requires separate and inference specific fine-tuning.  So while GFNs can sample high quality and diverse low temperature sentences by fine tuning for a given temperature, diverse beam search requires 5x more compute at inference time, but also doesn’t require retraining.\n\nThe paper also shows how to use variational EM to optimize the chain of thought reasoning to get the correct reasoning without providing additional training data for the reasoning.\n\nThe paper then demonstrates the empirical benefits of their approach in 4 experiments, low temperature sampling, infilling, subjectivity classification and solving arithmetic problems, in each case demonstrating their superior performance to good baselines.", "questions": "What do the reference in-fills look like for the other distributions?  It only shows the GFlowNet examples in Table B.3.\n\nWhat’s the speed of learning/convergence relative to, e.g. supervised fine tuning?  Is the inference unstable/need many restarts, etc?  How about compared to PPO training?\n\nDid the authors fine-tune separately for every temperature, or was this done in the style of an amortized sampler where you can dynamically specify the target temperature at run time?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "## Originality:\nThe paper seems to be a useful and novel contribution to the literature, namely the using GFNs to fine tune LLMs to solve formley intractable inference problems inside LLMs.  While the learning objective itself has been suggested before for LLMs, there does not appear to be any follow up work in applications like the low temperature sampling, infilling and learning chains of reasoning as in the current work.  It also appears to be a novel method for training chain of thought reasoning to arrive at a specific outcome.  \n\n## Quality:\nThe authors clearly reference the contemporary literature and they compare against suitable baselines in their experiments.  Overall, they have clearly demonstrated a variety of strong results against suitable baselines.\n\n## Clarity:\nThe paper is clearly written and structured in an easy to follow manner.  The descriptions of the methods and experimental setups are complete enough that the results can easily be reproduced by other interested parties.\n \n## Significance:\nLLMs are intrinsically highly significant at the moment and contain a huge amount of relevant information about the world which can't always reliably be extract out, so better methods to run interesting queries on them will have a practical effect. After all, they demonstrate that by this fine-tuning approach they can extract more valuable information from the same LLM, requiring only (presumably) extra inference time work not more data.   On the more theoretical level they demonstrate that GFNs can useful scale to large models, motivating further explorations of such methods in the current era of large models.\n\nAlso, in particular, learning better chains of reasoning could in particular have many interesting planning and reasoning applications, which could be unlocked with future research.", "weaknesses": "The datasets are small, and the inference only involved fine-tuning vs. training from scratch, which may unlock entirely different and interesting new global solutions.  This is an understandable limitation, but a more full exploration (which can be tackled as future work) might extend the power and reach of their method.  \n\nOne of the central pieces, the learning objective, has already been derived in a different soft-RL Q learning context, as pointed out by the authors.  There doesn’t appear to have been further explorations of downstream applications as performed in the current work however.  \n\nTheir method requires fine-tuning for different queries/inference problems.  It would be interesting to see if instead it would be possible to have a single network (as they suggest in future work) that can answer many different types of queries.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)