Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
Authors: Fangxu Yu
Lai Jiang
Haoqiang Kang
Shibo Hao
Lianhui Qin
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
reasoning
diversity
multi-step reasoning
Primary area: deep_learning->large_language_models
TL;DR: Flow of Reasoning (FoR) is a diversity-seeking finetuning method that enhances reasoning ability in large language models by using GFlowNets to discover diverse and accurate solutions through a Markovian flow on a directed acyclic graph.

Abstract: The ability to generate diverse solutions to a given problem is a hallmark of human creativity. This divergent reasoning is also crucial for machines, enhancing their robustness and enabling them to assist humans in many applications such as scientific discovery. However, existing approaches to multi-step reasoning with large language models (LLMs) have mostly focused only on reasoning accuracy, without further discovering more diverse valid solutions. For example, supervised fine-tuning improves reasoning quality but requires vast labeled data, while reward-maximizing reinforcement learning finds top-reward solutions while neglecting the solution diversity. To fill this gap, we propose Flow of Reasoning (FoR), an efficient diversity-seeking LLM finetuning method aimed at improving reasoning quality and diversity with minimal data. FoR formulates multi-step LLM reasoning as a Markovian flow on a DAG-structured reasoning graph. This formulation allows us to incorporate and adapt principled GFlowNet approaches, for finetuning LLMs to sample divergent paths with probabilities proportional to the (unnormalized) reward of target problems. Extensive experiments show that, with limited training examples (e.g., 15 examples), FoR enables the discovery of diverse, creative, high-quality solutions, greatly outperforming a wide range of existing inference and training methods across six challenging reasoning tasks, including BlocksWorld (embodied reasoning), Game24 (math puzzle solving), Rubik's Cube (spatial reasoning), 1D-ARC (abstraction reasoning), GSM8k (math reasoning), and ProntoQA (logical reasoning). Code is available at https://github.com/Yu-Fangxu/FoR.

Flow of Reasoning:
Training LLMs for Divergent Reasoning with Minimal Examples
Fangxu Yu" 1
Lai Jiang"
Haoqiang Kang"
Shibo Hao
Lianhui Qin
Abstract
1. Introduction
The ability to generate diverse solutions to a given problem hallmark of human creativ- ity. This divergent reasoning is also crucial for machines, enhancing their robustness and en- abling them to assist humans in many applica- tions such scientific discovery. However, ex- isting approaches to multi-step reasoning with large language models (LLMs) have mostly fo- cused only on reasoning accuracy, without further discovering more diverse valid solutions. For ex- ample, supervised fine-tuning improves reason- ing quality but requires vast labeled data, while reward-mas maximiiia reinforcement learning finds lop-reward solutions while neglecting the solu- tion diversity. To fill this gap, we propose Flow of Reasoning (FoR), an efficient diversity-sceking LLM finetuning method aimed at improving rea- soning quality and diversity with minimal data. FoR formulates multi-step LLM reasoning as a Markovian flow on DAG-structured reasoning graph. This formulation allows to incorporate and adapt principled GelowNet approaches, for finetuning LLMs to sample divergent paths with probabilities proportional to the (unnormalized) reward of target problems. Extensive experiments show that, with limited training examples (e.g.. 15 examples), FoR enables the discovery of diverse, creative, high-quality solutions, greatly outper- forming wide range of existing inference and training methods across six challenging reason- ing tasks, including BlocksWorld (embodied rea- soning), Game24 (math puzzle solving), Rubik's s Cube (spatial reasoning), ID-ARC (abstraction reasoning). GSM8k (math reasoning). and Pron- toQA (logical reasoning). Code is available httpss//github..om/ wtw..////ooaaaaaouuuo.....
Divergent problem solving the ability generate multiple diverse solutions to given problem (Runco, 1991; Runco & Acar, 2012). As a hallmark of human intelligence, this ability drives creativity by uncovering novel ways to ac- complish a task, providing more possibilities and adaptivity in different complex situations. Similarly, by encouraging machines to explore diverse solutions rather than confin- ing to one reasoning path, we not only enhance machines' robustness (e.g., by ranking aggregating different solu- tions) (Wang et al., 2022), but also empower automated systems that assist humans in generating ideas and thinking out-of-the-box, thereby potentially facilitating task comple- tion (Shinn et al., 2024), education (Li et al., 2023a), and scientific discovery (Jain et al., 2023a).
State-of-the-art reasoning with large language models (LLMs), however, has largely focused improving only the reasoning accuracy with the topmost solution, with- out moving a step further to discover more diverse valid solutions. Specifically, inference methods, such as CoT (chain of thought, Wei et al., 2022), ToT (Yao et al., 2024), RAP (Hao al., 2023), and others (Chen et al., 2024b; Besta et al., 2024), rely heavily on the underlying pretrained LLM'' capability and decoding algorithms to obtain diverse reasoning solutions. Moreover, the search-based inference (Yao et al., 2024; Hao et al., 2023) can be computationally costly when searching for multiple reasoning paths. On the other hand, the finetuning methods like supervised finetun- ing (SFT) (Yue et al., 2023; Yu et al., 2023c) often demands extensive supervision data to capture the full diversity of solutions, which can be costly to label in many applications. Alternatively, wwwrrdmmxiimization reinforcement learning (RL), such as proximal policy optimization (PPO, Schulman et al., 2017), trains LLMs to generate the highest-reward rea- soning solution and overlooks solution diversity. As shown in the case study in Figure I, the above-mentioned methods generate limited diverse solutions.
'qqual contribution Univeerstty of Califoria San Diego. Cor- respondence to: Lianhui Qin <lianhui @du..
To overcome the limitations, introduce Flow of Rea- soning (FOR), data-efficient approach that finetunes L.MMs for diverse reasoning with minimal training data. FOR draws inspirations from generative flow networks (GFlowNets) for amortized diverse sampling (Bengio et al.,
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
Coes study
Goal: orange block on top of the blue block
Examples
Julie is reading. She wants to read half the remaining pages tomorrow. How many pages should read?
Hypothesis: 51 is not imaginary
$1 i natural number
Pick blue
generated action
0
natural numbers are inteagers
11:4x7
So-477
Enviromentt generateed next state
s1
pages did she read todly??
pages has she read?
color the 7th into yallow
15
oz
Al: 24
integer
6+12m
S+35-18
irrm
color the 5th into red
$1 real number
left: 4.5
left: 15
left: 15.9
7:17
left:
A2:30
7+17-24
left:
Al: 42:42
imaginery
number
Game of 24: Use basic arithmetic operations (a-1/) reach
BlocksWorld:
Cube: Recover
ID-ARC; Identifying patterns of examples and applying Rubik's transformations
GSMBK: Grad School Math
PrOntoQA:
Monipulating blocks to achiove specific arrangements
Reach the hypothesis using reasoning
Figure 1: Multi-step LLM reasoning as a Markovian flow on six tasks, forming DAG-structured reasoning graphs. In the example of Game24 (left), we sample 20 reasoning paths from each method, respectively. Baseline methods such as SFT and CoT generate only one valid solution (leftmost path) repetitively (c.g., SFT generates this solution twice out of the 20 attempts), while our method FOR discovers three additional unique solutions.
2021) that have been studied in different domains like molecule synthesis (Koziarski et al., 2024) and operation scheduling (Zhang et al., 2023c). In particular, FoR enables diversity-seeking finetuning of multi-step LLM reasoning, to sample high-quality reasoning paths with probabilities proportional to the reward of target problems (as opposed to reward maximization in conventional RL like PPO). To this end, we formulate multi-step LLM reasoning from Markovian flow perspective (Figure 2), where each reason- ing step corresponds to an edge (action) that leads to the next node (state) in a flow graph. The reasoning process thus forms a flow that travels step-by-step from an initial state the terminal states of the target problem. Based on this new formulation, we use the trajectory- balanced objec- tive and adapt efficient exploration methods from the recent GelowNet studies, enabling finetuning of LLMs to generate more accurate and diverse solutions in reasoning tasks.
proach across six challenging benchmarks: BlocksWorld (embodied reasoning: Kambhampati et al., 2024), Game24 (math puzzles; Yao et al., 2024), Rubik's Cube (spatial rea- soning, Ding et al., 2023), ID-ARC (abstraction; Xu et al., 2023b), GSMSK (math reasoning; Cobbe et al., 2021), and ProntoQA (logical reasoning: Saparov & He, 2022; see Ap- pendix 4.7). Empirical results show that FOR, with limited (e.g. 15) training examples, generates diverse, high-quality solutions, greatly outperforming wide range of baselines with 20% 85% improvements, including supervised train- ing methods like SFT, reward-maxinizing RL like PPO, sversity-seckingg approaches GFN-CoT and various decod- ing methods, and advanced inference methods like CoT, ToT, GoT, and RAP. Ablation studies further validate the key designs in FoR that lead to robustness and effectiveness.
2. Related Work
FoR differs crucially from the recent GFlowNet applica- tions on autoregressive sequence generation with or without LLMs (Hu et al., 2023a; Malkin et al., 2022a). In partic- ular, contrary to the token-level modeling in the previous work (Hu al., 2023a), FoR introduces higher-level model- ing at the granularity of reasoning steps. This overcomes the limitations of aforementioned search-based LLM reasoning (Yao et al., 2024; Hao et al., 2023) by amortizing infer- ence into training. Additionally, GFlowNets' exploratory nature (Madan al., 2024) enhances both reasoning qual- ity and diversity, addressing the limitations of fine-tuning methods like SFT and PPO (see $4).
Divergent Thinking. Divergent thinking involves two branches: lateral and vertical (Waks, 1997b). Lateral think- ing explores unconventional ideas and challenges assump- tions, as seen in brain teasers (Zhong et al., 2024; Summers- Stay et al., 2023), while vertical thinking focuses on struc- tured, step-by-step reasoning. Our work focuses on vertical thinking, aiming to generate multiple different reasoning paths to solve a single problem.
LLM Reasoning. Recent LLMs (Achiam al., 2023) have shown strong potential in tackling complex reason- ing tasks (Hu et al., 2023c; Zhang et al., 2023d; Yu et al., 2023b). (1) Fine-tuning LLMs, including supervised fine- tuning (SFT) and
We evaluate the divergent reasoning capability of our ap-
reinforcement learn-
2

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
R
Ap
Local search
PE
Pr
TF
Pr
LLM
forward trajectory
PF
80
Training
PF
TF-U
P2 backward trajectory
aam F(an)
M(sa)
training object: trajectory balance
Figure 2: Left: The forward policy Pr(st g) in the flow-based formulation is parameterized LLM and finetuned with the trajectory balance objective (Eq.5) to achieve the desired flow F(s.) R(ss) on all terminal states 8,. Right: FOR incorporates local search with bestroy-and aud-recoostruction process to augment informative trajectories in training ($3.2). This facilitates efficient exploration and improves policy learning.
ing (RL), is key method for improving LLM reasoning abilities. SFT, leveraging large and high-quality datasets of reasoning chains, has proven highly effective (Yu et al., 2023c). RL techniques like PPO and DPO are widely used for optimizing reward-driven behavior in LLMS (Ouyang al., 2022; Bai et al., 2022). However, both approaches tend to limit solution diversity. (2) Prompting- ods engage LLMs in step-by-step thinking process. Chain- of of-Thought (CoT) (Wei et al., 2022) enhances LLMs per- formance by guiding them through intermediate steps to reach the final answer. Building on COT, the methods like ToT (Yao et al., 2024) and GoT (Besta et al., 2024) model reasoning as tree and graph searches, and RAP (Hao al., 2023) and XoT (Ding et al., 2023) use planning approaches to refine reasoning trajectories. DoT (Naik et al., 2023) enhances reasoning diversity through varied prompts.
develop an efficient LLM training method using GFlowNet objectives for multi-step reasoning tasks (section 3.2).
3.1. Multi-step LLM Reasoning Markovian Flow
Multi-Step Reasoning Problem as DAG. Consider multi- step reasoning problem that gives an initial state so and goal For example, in Blocks (Figure 2), an initial state is the starting configuration of the block stack, and goal describes the desired configuration of blocks. Rea- soning aims find complete paths (or trajectories) that lead from the initial state the state(s) that satisfy the goal. Given current state s, applying action it leads to the transition to the next state s', denoted as s - s'. For example, in Figure 2, the state 8o transits to after an action "pi ckup blue". A complete trajectory is thus a sequence of transitions T = (so - $1 where s, is the terminal state and trajectories. Given current state 8r, there could be multiple alternative next actions, resulting in different branches of the reasoning steps. Also, different sequences of actions can lead to the same intermediate//erminall states, as shown in Figure 1. As a result, we first formulate the multi-step reasoning problem as a directed acyelic graph (DAG).
meth-
s,,) T, the of all complete
GetowNets. GFlowNets (Bengio et al., 2021) were devel- oped to generate diverse, high-reward samples from unnor- malized distributions (Shen et al., 2023b; Roy et al., 2023; Zhang et al., 2023b), making them particularly effective in domains like molecule synthesis (Koziarski et al., 2024; Kim al., 2024a; Lu et al., 2024), where diversity is essential. Recently, GFlowNets have been integrated with L.Mss (Hu et al., 2023a; Kwon et al., 2024; Lee et al., 2024b; Song et al., 2024), but none of these have addressed multi-step reasoning problems. While Takase et al. (2024) is concur- rent work, its scope limited to math problems and does not extend to general reasoning problems. In contrast, FOR extends GelowNets to multi-step reasoning, modeling it Markovian flow through DAG, enabling the exploration of divergent reasoning paths across six reasoning benchmarks.
The reasoning graph consists of diverse trajectories that lead to different terminal states. crucial component often provided in reasoning tasks is the reward R(s,,) E Rzo- which assigns a numerical value to any terminal state 8n. For instance, a terminal state meeting the goal g receives a high reward. As discussed in $1, to generate diverse correct reasoning trajectories for solving task, we want to sample the trajectories with probabilities proportional the re- ward. This significantly differs from popular reinforcement learning methods (e.g.. PPO) and prompting based planning al algorithms (e.g.. RAP, ToT), which focus on optimizing for only the maximum-reward trajectory.
3. FOR for Divergent Reasoning
We first formulate the divergent multi-step reasoning tasks from Markovian flow perspective (section 3. Then,
3

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
The Flow Perspective. Sampling complex multi-step tra- jectories from the (often unnormalized) reward is partie- ularly challenging (LeCun et al., 2006; Qin et al., 2022). To overcome the difficulty, we consider the above rea- soning problem from a flow-hased viewpoint which was initially developed in (Bengio et al., 2021) and has been studied in other machine learning settings like molecule generation (Li al., 2023c; Lahlou et al., 2023; Li al., 2024a; He et al., 2024). Specifically, we define a trajectory flow function F- T - Rz Analogous to the classi- cal concept of flows in networks, the flow F(T) can be thought of as the volume of water traveling along this path T. Based on this, for any state 8, we can define the state flow F(s) = Emer F(T), and for any edge 8 ='8 8', the edge flow F(s - 55) -e-wer F()). These concepts of (unnor- malized) flow are connected the (normalized) probability distributions. Specifically, the flow trajectory determines distribution over trajectories:
other more recent extensions (Jang et al., 2023; Pan et al., 2023a) like subtrajectory balance (Madan et al., 2023) as future work.
LLM Parameterization. We parameterize the forward pol- icy Py with an LLM and finetune as described in the next section. Specifically, for a reasoning task, we express its goal g. action a, and state s as natural language (see Fig- ure I, BlocksWorld as an example). At each reasoning step f, the LLM generates an action da PLLM(a| f 9, c), where is an appropriate prompt (e.g., instructions or in- context demonstrations). The prompts used in the experi- ments are detailed in Appendix C. Once an action is gen- erated, the state transits to the next 8t+1 = =(ss, as) with a transition function T. Therefore, assuming that differ- ent actions applying to the same state St lead to different next states, and that action a leads to state St+1, we can write Pp(8(41) 0) (L &&;; 0, g, e)) In the exper- iments, we follow previous work and define T either by an LLM with appropriate prompts and greedy decoding (e.g.. Blocks World as in (Hao et al., 2023)) or by the environment (e.g., Rubik's Cube as in (Ding et al., 2023)).
P(r) F(r) Z Z= -.E.
TET F(T).
With Markov assumption, it can be shown that the distri- bution factorizes into step-wise distributions:
3.2. Efficient Diversity-Seeking Training of LLMs
P(T)= IT_ Po (s where (r(st(S(-1) = F(s Fe)/F(st-1).
The proposed flow-based formulation of LLM reasoning enables the adaptation from GFlowNet training methods by fine-tuning an LLM as the forward policy Pp.
That is, intuitively, (s characterizes the proportion of water at node 8 that travels toward node st. The distribution PF is also called the forward policy, which can generate a trajectory 7 by sampling a sequence of transitions step-by-step starting from the initial state sy- Equivalently (Bengio et al., 2023), there exists a backward policy that detines the distributions (-(- over the parents of each state s, (s = F(s-11 Fs.)
Training Objective. Since can sample complete trajec- tory, we derive our training objective based on the trajectory balance approach (Malkin et al., 2022a), which has shown improved efficiency than other alternatives (Bengio et al., 2023; 2021). Specifically, for any complete forward trajec- tory T (Sp - 81
- 8n), the trajectory balance constraint, with task goal g, says (Figure 2):
Z(5o.g) [. Pr((s(s--1g)) F( 1"_ Pa 19
Let T be the trajectory ending at the terminal state 8n- Re- call that our aim in diverse LLM reasoning is to obtain a forward policy (r(ss]s,_1) such that the resulting trajec- tory distribution is proportional to the reward. From the flow perspective, according to (1) and (2), this aim is equivalent to approximating a Markovian flow F such that F(sn) equals the reward (Bengio et al., 2021):
(4)
where we have used the fact that P(s.) F(s / ((so g) for the terminal state Sn, Plugging in the reward R, as motivated by (3.()), to provide supervision signals, the constraint leads to a loss function w.r.t the parameterized forward policy Pp:
F(sn) = H(sn), V terminal state Sn.
(3)
1(T;0,9) (log Z(5o. 9) Ih= Pr 11-1 1;00)) 2
R( Th PB Pa |1; 0,
The above flow- concepts provide a rich set of con- straints that can be converted into training objectives for learning the desired forward policy. For example, the detailed balance constraint F(ss-))Pp(s((sss F Pp(x.11s()))))) yields the respective objective used in molecule generation tasks (Bengio et al., 2023). In this work ($3.2), we devise the learning objective from the re- cent trajectory balance constraint shown to be more efficient (Malkin et al., 2022a). We consider the incorporation of
(5)
Ps P=
PP((s))
where [Pa(s:) denotes the number of parents of state st and (Malkin et al., 2022a) suggested a canonical choice of setting PB(-Is)) to be uniform over the parents. Note that Z is the total flow conditioning on each goal and initial state sp. The term log Z is challenging to learn accurately in an LLM setting (Hu et al., 2023a). We thus follow (Zhang
4

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
al., 2023c) to the log-variance approximation, which
4. Experiments
implicitly estimates log Z given each trajectory
4.1. Experimental Settings
((r; 0) log (ssn)) + E,_, log 18 1-1 log Pr (s(ss(se-1;00g)
0,9)
Baselines. We first compare our approach with several prompting-basee methods, including k-shot CoT (Wei et al., 2022). ToT (Yao et al., 2024), GoT (Besta et al., 2024), XoT (Ding al., 2023), and RAP (Hao et al., 2023). We use the Llama-3-8B (Dubey et al., 2024) as the base model for all tasks. For fine- fine-tuning based methods, we evaluate SFT with diversity-enhancing decoding strategies like Tempera- ture Sampling (Shi et al., 2024), Nucleus Sampling (Holtz- man et al., 2019) (1) = 0.95), Typical Sampling (Meister et al., 2023) (T 0.95), and diverse beam search (Vijayaku- mar et al., 2016) (DBS, beam width k 20). Additionally, we apply fine-tuning with PPO (Schulman et al., 2017) and GFN-CoT (Hu et al., 2023a). We also compare it against OpenAI-O1, which is the latest and strongest reasoning model. All finetuning methods are trained on the same dataset as FOR. In the BlocksWorld task, we evaluated most baselines for a broad comparison, but some methods (e.g., GFN-CoT) showed suboptimal performance, which informed our decision to selectively apply high-performin baselines in the subsequent tasks.
(6)
where 0( equals to true log Z in the optimal case. Our optimization goal then turns into minimizing the variance of (T; over different trajectories 7 with the loss:
(T; = (T); - F, (=]
where we sample trajectories r from a behavior policy [T; 0, g) for training, and B.[*((; 0)] is estimated with mini-batch of sampled trajectories. We have different con- figurations of æ, such as on-policy, off- off-policy, and mixed explorations, which could impact training efficiency (see ablation studies ($4)). If Lv (T; 0) is globally optimized, the resulting flow satisfies (3) and P= g) samples proportionally to the reward as desired.
Efficient Exploration. Training with the loss function in Eq. (7) requires trajectory collection, raising the challenge of how collect them efficiently and effectively. On-policy sampling, which generates trajectories via PF(T: g) and optimizes [y(r; 0) with a batch of trajectories, often struggles to explore the vast trajectory space due to lim- ited exploration capabilities. This can result in inadequate estimation of the distribution needed to capture diverse rea- soning paths. To address this, we want to set up a distri- bution (T;, 0) in Eq. (7) that enhances exploration while producing various training samples for policy optimization. Inspired by recent advances in GFlowNets (Vemgal et al., 2023; Hu et al., 2023a), we combine on- on-policy and off- policy strategies and further enhance exploration by adapt- ing local search (Kim et al., 2023) for LLM training.
Evaluation. We generate n solutions per problem and evaluate methods based on four key metrics: (Aec): A problem considered correct if at least one of the n sampled solutions is correct. (2) Diversity: For problems with at least one correct solution, we report the average number of semantically different correct solutions among the number of correct solutions, where higher is better (see Appendix for details). (3) Creativity: The proportion of unique correct solutions found by a method that are not discovered by others. (4) Runtime: The average time to generate one solution by using a single NVIDIA A100 80GB GPU is used as efficiency metric. See Appendix C.1 and C.2 for details of metrics.'
(1) Accuracy
4.2. Embodied Reasoning: Blocks
Blocks involves a set of blocks with unique colors that can be stacked on top of each other or moved around. The goal of this task is to enable LLMs to plan a sequence of actions to transform an initial configuration of blocks into a desired goal configuration using a series of actions. The actions are text instructions (STACK, UNSTACK, PUT, PICKUP) generated based on domain rules and block posi- tions, and a state is the current block orientation.
For on-policy exploration, we use both the online policy Pr(s)|s(1;;f 9) and tempered variant to generate trajec- tories given goal g and initial state so. Off-policy explo- ration leverages replay buffer prioritizing high-reward trajectories, e-sampling, and offline trajectory data (for Game24 in $4.3). To efficiently explore high-reward re- gions, we propose modified local search method (Figure 2). Specifically, we select the highest-rewand trajectory from each batch, truncate its latter portion, and reconstruct it us- ing random policy Pv. This avoids the computationally expensive forward process of L.MMs, enhancing efficiency. High-reward trajectories are more likely to reconstruct cor- rectly in early steps, with errors typically occurring later. Additional details on the full algorithm, exploration strate- gies, and local search are in Appendices E and F.
Setup. Blocksworld examples (Valmeekam et al., 2024) are grouped by the minimum number of required actions: 30
'In our experiments, we report the average accuracy and stan- dard deviation over three repetitions, except for search-based meth- ods and Ol Ol-series due to time and budget constraints. Creativity is not averaged over repetitions, as the performance exhibit low variance across runs.
5

---

Flow of Reasoning: Training LLMS for Divergent Reasoning with Minimal Examples
Table I: Results on BlocksWorts comparing prompting- and finetuning- baaedd methods on questions requiring two, four, and six steps, respectively. The standard deviations of three runs are shown in brackets. a is à temperature. indicates that results from multiple runs for ToT (DFS).
2-step Acc. (%)
4-step Diversity
6-step Diversity
Runtime (s)
Method
Acc, (%)
Creativity (5)
Acc. (%)
Creativity (%)
(6-step)
Prompting based methods
CoT (1-shol)
48.88 (8,31)
28.57 (5.83)
1.05 (0.04)
0.00
15.82 (2. 08)08)
1.05 (0.03)
0.00
3.57
CoT (5-shol)
68.89 (8.31)
42.86 (1.94)
1.04 (0.03)
0.00
29.63 (1.72)
1.02 (0.01)
0.00
3.68
CoT (15-shol)
(6.29)
06 (4.89)
1.03 (0.02)
(1.26)
(0.03)
5.32
CoT (GPT-4a, 1-shot)
93.33
54.76
1.08
0.00
67.67
1.06
0.79
3,92
ToT (DSS)
13.33
16.67
8.08
48.91
ToT (DFS)*
40.00
42.85
1.0
0.00
31.31
1.06
1.58
44.41
ToT (BFS)
13.33
14.28
5.05
398.74
RAP
100.00
92.86
69.70
466.09
01 O1-series methods
O1-mini (1-shot)* x
100.00
100.00
1.05
0.00
93.93
1.05
2.38
10.38
O1-preview (I-shot) x
100.00
95.24
78.79
36.61
Finetuming based methods
SFT (a=1.0)
44.44 (3.14)
42.06 (5.44)
1.05 (0.01)
0.00
34.68 (2.52)
1.04 (0.01)
4.76
4.05
SFT (a=0.5)
42.22 (3.14)
39.68 (2.24)
1.02 (0.02)
0.00
29.63 (1.90)
1.02 (0.02)
0.79
4.07
SFT (d=0.1)
26.67 (5.44)
26.20 (3.89)
1.00 (0.00)
0.00
17.51 (1.26)
1.00 (0.00)
0.00
4.08
SFT DBS
31.10
38.88 (1.12)
1.00 (0.00)
0.00
18.85 (1.25)
1.00 (0.00)
0.00
15.71
SFT Nucleus
48.89 (3.14)
53.97 (2.97)
(0.03)
0.00
42.08 (1.71)
1.12 (0.03)
0.00
4.21
SFT Typicul
53.33 (5.44)
48.41 (2.25)
1,08 (0.02)
0.00
37.71 (2.38)
1.08 (0.02)
0.00
3.65
SFT PPO
46.66 (5,44)
44.44 (2.24)
1.11 (0.05)
2.04
24.58 (1.72)
1.08 (0.03)
3.17
4.03
SFT + GFN-CoTT
48.89 (8.81)
44.42 (2.96)
1.00 (0.00)
0.00
40.73 (1.25)
1.05 (0.03)
0.00
4.08
FoR (Ours)
100.00 (0.00)
98.41 (1.12)
1.27 (0.02)
12.24
78.44 (4.54)
1.33 (0.03)
9.52
13.98
Table 2: Results on Game of 24.
examples for steps, 57 for 4 steps, and 114 for 6 steps, following Hao et al. (2023). We select the first 15 of each group as the training examples for FoR and the rest as test examples. We sampled 8, 20, and 40 times for the 2, 4, and 6-step datasets, respectively, to report diversity and creativity. All the baselines are included in 1.
Method
Acc. (%)
Diversity
Creativity (9)
Prompting based methods
CoT (5-shot)
23.00
1.04
6.60
CoT (GPT-4o, 5-shot)
59,00
1.61
49.42
XeT
20.00
ToT
21.00
Reward Design. We compose a terminal state reward and an augmented intermediate reward to evaluate trajectories. Terminal state reward is assigned to high positive value when a trajectory reaches the goal The augmented in- termediate reward assesses actions by using the LLM to estimate the confidence of actions in achieving their goals. natural choice is to use the log-likelihood of actions, log PLLM(0e(s-11122 g). However, the value of log nega- tive. To maintain monotonicity consistency and positive reward values, The total reward is defined as: R(s,) = w I(success) + A A-r-, -1 log ( where w is the success weight (set to 100 and the following tasks).
RAP
10.00
O1-series methods
Open ---iii -mini
94.00
Finetuning based methods
SFT 1.0)
29.00
1.18
11.22
FOR
48.00 (1.41)
1.30 (0.09)
29.15
and SFT-based methods, while being 30x faster than ToT (BFS) and RAP. Additional results of eeeesstty-encouragginn prompts, larger-size models, and out-of-distriibtion (OOD) generalization are in Appendix C.4, and training costs are detailed in Appendix C.3.
use -/ log PLLM(as) (a))))11.)) instead.
Results. As shown in Table 1, Our method outperforms all other baselines across all metries, showing that FoR not only achieves the best accuracy but also finds more diverse and unique solutions. Specifically, most baselines do not find any unique solutions, resulting in creativity score of 0. is worth noting that the O1 series improves the accuracy to a large extent, but still struggles to find diverse reason- ing paths. Additionally, FoR matches the inference speed of high-efficiency baselines like s-ot CoT, ToT (DFS),
4.3. Math Puzzle Solving: Game of 24
Game of 24 is mathematical reasoning task that may have multiple solutions. The objective of this task is to use 4 integers and 4 basic arithmetic operations (+, -, x, +) to reach 24, where each number can only be used once. Each action a, is defined as an equation composed of 2 numbers and an operator, and the state s, is the left number.
Setup. We use the LL.M-reaoone, dataset (Hao et al., 2024)
6

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
Table 3: Results on Rubik's ' Cube.
Table 4: Results on ID-ARC.
Method
Acc. (%)
Diversity
Creativity (%)
Method
Ace. (%)
Diversity
Creativity (%)
Prompting-based methods
10
10.37 (1.21)
CoT
0.00
0.00
0.00
CoT
39.51 (1.94)
1.04 (0.01)
1.45
CoT (GPT-4)
1.09
1.00
4.35
CoT (GPT-44)
40.00
1.00
0.00
ToT (BFS)
0.00
Program-Only
0.74
GoT
0.00
Hypo-Search
1.48
XoT
4.92
FoR
50.37 (1.60) 1.17 (0.02)
21.74
Finetuning-brit methods
SFT (a- - 1.0)
1.82 (0.06)
1.00 (0.00)
8.69
Results. As shown in Table 3, FoR outperforms all base- lines on the Rubik's Cube task, achieving absolute increase of 5.95% in accuracy than XoT. generating absolute 29% increase in diverse solutions, and discovering unique and creative solutions than the baselines.
SFT + PPO
0.55 (0.45)
1.00 (0.00)
0.00
FOR
10.87 (1.18)
1.29 (0.02)
82.61
and randomly select 20 examples for training and 100 exam- ples for testing. Since prior work shows that LLMs struggle to sample correct trajectories online in Game24 (Yao et al., 2024; Yu et al., 2023a), we generate offline ground-truie trajectories using Python code for fine-tuning. We sample 20 solutions for all methods.
4.5. Abstraction Reasoning: ID-ARC
ID-ARC is one-dimensional simplification of the ARC benchmark (Chollet, 2019), introduced in (Xu et al., 2023b), Each problem in the dataset contains set of training input- output 1D grid pairs that capture a specific underlying rule and a test case that measures the model correctly under- stands the rule. Following recent program search approaches (Wang et al., 2023b; Qiu et al., 2023; Butt et al., 2024), which frame the problem as a sequence of transformation functions, each action ar is function (e.g., horizontal mir- roring), with the intermediate grid as state St.
Reward Design. Similar to BlocksWorld. the success re- ward gives a high positive reward when a trajectory r suc- ceeds in reaching 24, and the augmented reward gives the product of the probability of correctness for each action as given its last state st-1 provided by the untrained LLM model: R(s.)= w I(success) + TT= Pantrained Puntranee(({((s_1))
Results. As shown in Table 2, FoR demonstrates superior accuracy and diversity in solving math puzzles compared to other baselines with the same base model. Not surprisingly, O1-mini and GPT-4o achieve better performance due to the stronger intrinsic mathematical knowledge and reasoning mechanism. We also investigate the fact that GPT-4o tends to use velf-eeriiiation and reflection during Game24's infer- ence. This may explain its superior performance in Game24.
Setup. We randomly select 5 examples from the ld_ @d_move_lp. 1d ld_padded_fill and 1d ld_denoising tasks, These tasks involve moving the color bar by pixel, filling in empty spaces enclosed by pixels, and removing noise- like pixels respectively. The 15 selected examples form the training set, while the remaining 45 examples from each task form the test dataset. We sample 20 solutions for each example during inference. See Appendix C.7 for details.
4.4. Spatial Reasoning: Rubik's Cube
The Rubik's Cube is well-known puzzle requiring multi- step spatial reasoning. The model's task is to plan a se- quence of rotations to restore a shuffled cube, where each state 8t represents the block arrangement, and each action aj is a layer rotation (e.g., 90 or 180 degrees).
Baselines. Since there are no complex reasoning baselines (e.g., ToT) evaluated on this task, we compare against Input- outpul (1O) prompting (Mirchandani al., 2023). Program Only and Hypothesis Search (Wang et al., 2023b) synthesize Python programs for transformation, with the latter first gen- erating language-based transformation hypotheses before program synthesis. Fine-tuning methods are not compared due to the lack of labeled reasoning data.
Setup. We randomly select 15 examples from the training dataset in from (Ding et al., 2023), and evaluate different methods on test set containing 183 examples. Each ex- ample can be solved in four steps. All the baselines are included in 1. For SFT, CoT and FOR, 10 solutions are sampled. See more details in Appendix C.6.
Reward Design. In addition to a success reward, we design the augmented rewards for actions based on how much they reduce the distance to the goal. Specifically, an action receives a higher reward if it reduces the distance between the current state and the goal. The total reward is R(sn)= w- I[ (success) + Σi= Σx= exp(ha(si where hal(:-) is hamming distance and K is the number of training input-output pairs.
Reward Design, Similar to previous tasks, we have suc- cess reward and augmented reward that is based on the difference in minimum steps required to restore the cube from its current state. Actions that reduce steps receive higher rewards, while others receive lower rewards. For- mally, R(s,)) = -o-I(sucess)+ + +"= exp(r(s(-1)-r(1)) where r(st) represents the remaining minimum steps.
ha (t(.gg) ))
Results. Results in Table 4 show that FoR substantially outperforms previous methods on all metrics, especially
7

---

Flow of Reasoning: Training LLMS for Divergent Reasoning with Minimal Examples
Table 5: Ablation results on BlocksWorld for different components in FOR with Llama-3-8b.
2-step
4-step
6-step
Method
Acc. (%)
Acc. (%)
Diversity
Acc. (%)
Diversity
FOR (Ours)
100.00 (0.00)
98.41 (1.12)
1.27 (0.02)
78.44 (4.54)
1.33 (0.03)
w/o local search
100.00 (0.00)
89.68 (2.97)
1.18 (0.02)
53.90 (2.10)
1.31 (0.03)
w/o augmented rewards
100.00 (0.00)
91.30 (1.10)
1.22 (0.02)
47.10 (1.30)
1.21 (0.01)
w/o replay buffer
100.00 (0.00)
94.44 (2.97)
1.24 (0.04)
72.38 (1.71)
1.24 (0.01)
w/o e-sampling
100.00 (0.00)
97.61 (1.95)
1.26 (0.03)
73.39 (2.38)
1.25 (0.04)
Table 6: Results on GSM8K.
Table 7: PrOntoQA Results. Pred Ace measures the ac-
curacy of the final conclusions, while Proof Ace evaluates the correctness of the reasoning process (e.g., no short- cuts/hallucinations)).
Method
Acc. (%)
Diversity-H
Diversity-G
CoT (2-shot)
45.72
1.12
1.60
CoT-SC (2-shot)
41.74
RAP
37.16
SFT (a = 1.0)
52.69
1.13
1.63
Method
Prod (%c
In- nn-Distribution
Proof (cc.(%)
Pred Acc.(%)
Out-of-Distrbbutin
Proof (oc.(%)
FoR
57.39
1.26
1.72
Prompring- based methods
FOR+ PRM
62.62
1.31
1.77
CoT
$2.20 (1
35.40 (1.86)
43,50 (1.48)
(1.91)
CoT (GPT-lo)
89.00
47,80
62.92
24.78
ToT (BFS)
49.80
32,20
diversity and creativity. Previous approaches (bypothesis search and program-only) generate programs at once, which easily results in errors during the intermediate process, lead- ing inferior performance.
RAP
50.70
39.50
Finctuning based methods
STaR
88.90
$4,00
50.10
24.60
FOR
88.73 (1.33)
54.60 64.60CC (1.50)
63.07 (1.71)
28.88 (2.36)
FOR +STaR
90.50 (1.89)
54,70 (1.41)
63.00 (2.13)
26.67 (2.80)
4.6. Mathematical Reasoning: GSM8K
4.7. Logical Reasoning: PrOntoQA
GSM8K consists of grade-school math problems (Cobbe al., 2021). Following RAP (Hao al., 2023), we define an action a intermediate sub-question and state as the history of all sub-questions and their corresponding answers.
PrOntoQA is a logical reasoning task. Each test case in- cludes question (goal), a list of facts A (action space), and an initial state 8o- A state is the conclusion derived from the previous state St-1. Performance is evaluated using two metrics: prediction accuracy and proof accuracy. Predic- tion accuracy refers to the correctness of the final answer, regardless of the reasoning process. Proof accuracy, on the other hand, measures the correctness of the entire reasoning chain, ensuring that each step leading to the final answer is accurate. Both metrics are calculated using rule-based string matching. The Diversity metric is not applicable, as each question has only one valid reasoning chain.
Setup. We use the last 50 training examples in the original training dataset, and we sample 4 times for every problem at inference. We follow the baseline implementations from LLM reasoners (Hao al., 2024). Since there is no standard metric for reasoning diversity, we manually annotate 50 test examples to evaluate the semantic difference between solutions as the diversity metric (Diversity-H)), and propose an automatic metric using GPT-4o to distinguish different solutions (Diversity-G), where the prompt is provided in Appendix C.9.
Setup. We randomly select 50 examples for the training set and 120 for the test set. The evaluation is conducted on both in- and out-of-distributtin (OOD) examples, with 32 samples drawn per problem during inference. In addition to the baselines described in 1,.1, we adopt STaR (Zelik- man et al., 2022), which applies SFT on correct examples through online sampling. We also evaluate FoR on top of the model fine-tuned by STaR. See Appendix C.8 for more experimental details.
Reward Design. We removed the success reward since the pretrained mathematical knowledge enables LLMs to perform relatively well compared to BlocksWorld Using LLMs to verify each state is computationally expensive. The reward is defined as R(s,) = w-I Issucesss). In addition, we augment the above reward with the process reward model (PRM) using to evaluate the intermediate reasoning steps: R(s.)= (ssuccess) + > PRM (1)))
1ew..5--MM----MMMM1- (Zhang et al., 2025)
Reward Design. We removed the success reward pre- vent the model from arriving at correct answers through flawed reasoning paths. Instead, we apply a rule-based augmented reward that evaluates the feasibility of a fact given the previous state $,-1, checking if they share the same ontology. Formally, the reward is defined as R(s,) = ni -E w I(8t-1, st),, where is hyperparameter and
Results. As shown in the table 6, in the task of GSMSK, FoR outperforms all baselines in both accuracy and di- versity. These results highlight the potential of FoR for extending to more open-ended reasoning tasks.
1
8

---

Flow of Reasoning: Training LLMS for Divergent Reasoning with Minimal Examples
Table 8: Ablation results of FoR on BlocksWorld without local search.
2-step Acc. (%)
4-step
6-step
Method
Acc. (%)
Diversity
Acc. (%)
Diversity
Ours w/o local search
100.00
89.68
1.18
53.90
1.31
w/o augmented rewards
100.00
71.43
1.04
30.30
1.17
w/o replay buffer
100.00
78.57
1.10
34.34
1.15
w/o e-sampling
100.00
83.33
1.12
49.49
1.14
I(st-1,8t) is an indicator function. 1(s--1, 8t) equals only when the transition (s St.) is part of the ground- truth reasoning path, ensuring no shortcuts are taken.
100
Accuracy (86
90
80
-13
Accuracy
60
70
1.1
4.7.1. RESULTS
60
11.0
4310
20
As shown in Table 7, FoR achieves superior results on both in- and out-of-distribuutin problems compared all baselines. While FoR slightly outperforms the SFT-based STaR for in-distribution tasks, its advantage is far greater for out-of- out-of-distribution tasks. Moreover, combining FoR with STaR enhances in-distribution performance while preserv- ing -uitt--disrribution success, revealing the complementary strengths of these methods.
50-
0.0
1.0
5.5
2.0
15
50
45
60
0.8
augmented reward weight
Number of training data
Figure 3: Additional analysis on Blocks Left: Ac- curacy of FoR across different step settings with varying intermediate reward weight (A). Right: Comparison of ac- curacy and diversity between SFT trained with varying data sizes and FoR trained on the test dataset.
4.8. Additional Analysis
from divergent reasoning trajectories, enhancing trajectory coverage and improving generalization to new cases.
Ablation Study. To further demonstrate the effectiveness of FoR, we conduct ablation studies to analyze the im- pact of individual components, focusing specifically on the BlocksWorld task. Table 5 and Table 8 summarize our component-wis ablations:
Discussion. FoR achieves higher reasoning accuracy and diversity by leveraging its exploratory nature (Madan et al., 2024) to sample multiple valid trajectories, covering a broader solution space without getting stuck on incorrect paths like SFT. This increases the likelihood of reaching a correct solution. A detailed case study that compares FoR with SFT is provided in Appendix G. Furthermore, FoR demonstrates robust performance across varying training set sizes, different LLM families, and larger model scales (see Appendix D).
1) Local search enhances exploration and high-reward tra- jectory collection its removal causes a 31.3% absolute drop in 6-step accuracy; 2) e-ammpling also aids exploration, though to a lesser degree; 3) Augmented intermediate re- wards are removial--emooing them yields a 51% decrease in 6-step accuracy and a 13% reduction in diversity, and tun- ing the reward weight À (left plot of Fig. 3) further refines performance (overly large À harms accuracy); 4) The replay buffer leverages past high-reward incurs a 4-6% loss with local search versus an 1 1-19% loss without, highlighting its greater importance when search absent. To isolate these effects, also remove local search and reassess each component: shown in Table 8, the replay buffer's impact grows without search (11-19% drop vs. 4 6%). and eliminating augmented rewards still causes the largest performance degradation, underscoring their effectiveness in FOR.
5. Conclusion
it
We introduce FOR, which efficiently trains L.MSs to gen- erate diverse, high-quality reasoning paths proportional to unnormalized rewards. By leveraging a based multi-step reasoning formulation, FOR adapts and improves GFlowNets training strategies. Across six reasoning tasks, FoR outperforms finetuning- and prompting-basee base- lines in both accuracy and diversity. Limitations are dis- cussed in Appendix! I.
Data-Efficiency. We test the final 20 examples from the 6-step test set and adjust the number of training examples for both FoR and SFT. As illustrated in the right plot of Figure 3, SFT's accuracy improves with additional training data, while its diversity remains stable. However, SFT's performance remains lower than FOR's 's under any amount of training data. This is attributed to FoR's ability to learn
Impact Statement
This study introduces FOR, a framework that enhances LLMs' ability to solve multi-step reasoning problems. This approach lays the foundation for scalable, principled L.LM reasoning, with potential applications in automated theorem proving, robotics, and scientific discovery, contributing to more robust and interpretable Al systems.

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
References
P., Zhang, R., Zhang, S., Zhang, S., Zhang, W., Zhang, W., Zhang, X., Zhang, X., Zhao, H., Zhao, Q., Zhao, X., Zhou, F., Zhou, Z., Zhuo, J., Zou, Y., Qiu, X., Qiao, Y., and Lin, D. Internlm2 technical report, 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Chen, Q. Zhang. B., Wang, G., and Wu, Q. Weak-eval- strong: Evaluating and eliciting lateral thinking of Ilms with situation puzzles. arXiv preprint arXiv:2410.06733, 2024a.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C.,, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint 2rXiv:22/2.08000 2022.
Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588 2022.
Bansal, H., Hosseini, A., Agarwal, R., Tran, V. Q., and Kazemi, M. Smaller, weaker, yet better: Training Ilm
reasoners via compute-optimal sampling, arXiv preprint arXiv:2408. 16737, 2024.
Chen, Y. and Mauch, L. Order-preserving gflownets. arXiv preprint arXiv:2310.00386, 2023.
Barke, S., Gonzalez, E. A., Kasibatla, S. R., Berg- Kirkpatrick, T., and Polikarpova, N. Hysynth: Context- free Ilm approximation for guiding program synthesis. arXiv preprint arXiv:2405. 15880, 2024.
Chen, Z. White. M., Mooney, R., Payani, A., Su, Y., and Sun, H. When is tree search useful for Ilm plan- ning? it depends on the discriminator. arXiv preprint arXiv:2402. 10890, 2024b.
Bengio, E., Jain, M., Korablyov, M., Precup, D., and Ben- gio, Y. Flow network based generative models for non- iterative diverse candidate generation. Advances in Neu- ral Information Processing Systems, 34:27381-27394, 2021.
Chollet, F. On the measure of intelligence. arXiv preprint arXiv:1911.0154,, 2019.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Bengio, Y., Lahlou, S., Deleu, T., Hu, J., Tiwari, M., and Bengio, E. Gflownet foundations. Journal of Machine Learning Research, 24(210):155,2022:
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Pod- stawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 38, pp. 17682 17690, 2024.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M.. Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10.2//1.144.. /4/68, 2021.
de Miquel, A. Are frr_le abstraction and reasoning chal- lenge kaggle competition code. https : httpps://tha //gbthhb. com/ ailejandrodemte www.eemcmueeeRRccccc Kaggle,, 2021. Accessed: 2024-09-29.
Butt, N., Manczak, B., Wiggers, A., Rainone, C., Zhang, D., Defferrard, M., and Cohen, T. Codeit: Self-improving language models with prioritized hindsight replay. arXiv preprint ar Xix.:22442.048555 04858, 2024.
Deleu, T. and Bengio, Y. Generative flow net- works: a markov chain perspective. arXiv preprint arXiv:2307.01422.2000 2023.
Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., Dong, X., Duan, H. Fan, Fei, Z., Gao, Y., Ge, J., Gu, C., Gu, Y., Gui, T., Guo, A., Guo, Q., He, C., Hu, Y., Huang, T., Jiang, T., Jiao, P., Jin, Z., Lei, Z., Li, J., Li, J., Li, L., Li, S., Li, W., Li. Y., Liu, H. Liu, J., Hong, J.. Liu, K., Liu, K., Liu, X., Lv, C., Lv, H., Lv, K., Ma, L., Ma. R., Ma, Z., Ning, W., Ouyang, L., Qiu, J., Qu, Y., Shang, F., Shao, Y. Song, D.,
Deleu, T., Góis, A., Emezue, C., Rankawat, M., Lacoste- Julien, S., Bauer, S., and Bengio. Y. Bayesian structure learning with generative flow networks. In Uncertainty in Artificial Intelligence, pp. 518-528. PMLR, 2022.
Song, Z., Sui, 7., Sun, P. Sun, Y., Tang, H., Wang, B., Wang, G., Wang, J., Wang, J., Wang. R., Wang, Y., Wang, Z., Wei, X., Weng. Q. WU, F,, Xiong, Y., Xu, C., Xu, R., Yan, H., Yan, Y., Yang, X., Ye, H., Ying, H., Yu, J., Yu, J., Zang, Y., Zhang, C., Zhang, L.,, Zhang, P., Zhang,
Deleu, T., Nishikawa-TTomey, M., Subramanian, J., Malkin, N., Charlin, L., and Bengio, Y. Joint bayesian inference of graphical structure and parameters with a single gen- erative flow network. Advances in Neural Information Processing Systems, 36, 2024.
10

---

Flow of Reasoning: Training LLMS for Divergent Reasoning with Minimal Examples
Ding, R., Zhang, C., Wang, L., Xu, Y., Ma, M., Zhang, W. Qin, S., Rajmohan, S., Lin, Q., and Zhang. D. Every- thing of thoughts: Defying the law of penrose triangle for thought generation. arXiv preprint 2rX04:2311.02254, 2023.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint wrXiv:2106.09683 2021.
Hu, E. J., Jain, M., Elmoznino, E., Kaddar, Y., Lajoie, G., Bengio, Y., and Malkin, N. Amortizing intractable inference in large language models. arXiv preprint arXiv:2310.04666 2023a.
Dubey, A., Jauhri, A.. Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama herd of models. arXiv preprint arXiv:2407.2178,, 2024.
Hu, E. J., Malkin, N., Jain, M., Everett, K. E., Graikos, A., and Bengio, Y. Gfflwnet-em for learning compositional latent variable models. In International Conference on Machine Learning, pp. 13528-13555 PMLR, 2023b.
Feng. X., Wan, Z., Wen, M., Wen, Y., Zhang, W., and Wang, Alphazero-like tree-search can guide large language model decoding and training. arXiv preprint Sifi:23009171799 2023.
Hu, Y., Yang, H., Lin, Z., and Zhang, M. Code prompt- ing: neural symbolic method for complex reasoning in large language models. arXiv preprint arXiv:2305. 200.:22775557777 2023c.
Ghari, P. M., Tseng, A., Eraslan, G., Lopez, R., Biancalani, T., Scalia, G., and Hajiramezanali, E. Generative flow networks assisted biological sequence editing. In NeurlPS 2023 Generative Al and Biology (GenBio) Workshop, 2023.
Huang. S., Ma. S., Li, Y., Huang, M., Zou, W., Zhang, W., and Zheng, H.-T. Lateval: An interactive Ilms evalua- tion benchmark with incomplete information from lateral thinking puzzles. arXiv preprint arXiv:2308. 10855, 2023.
Hao, S., Gu, Y., Ma, H., Hong, J. J., Wang, Z., Wang, D. Z., and Hu, Z. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305. 14992, 2023.
Ismayilzada, M., Paul, D., Bosselut, A., and van der Plas, L. Creativity in ai: Progresses and challenges. arXiv preprint arXiv:2410. 172/8, 2024.
Hao, S., Gu, Y., Luo, H., Liu, T., Shao, X., Wang. X., Xie, S.. Ma, H., Samavedhi, A., Gao, Q., et al. Llm reasoners: New evaluation, library, and analysis of step- by-step reasoning with large language models. arXiv preprint arXiv:2404.0522 2024.
Jain, M., Bengio, E., Harnaneez-Gariaa, A., Rector-Brooks J., Dossou, B. F., Ekbote, C. A., Fu, J., Zhang, T., Kil- gour, M., Zhang, D., et al. Biological sequence design with gflownets. In International Conference on Machine Learning, pp. 9786-9801 PMLR, 2022.
Havrilla, A., Du, Y., Raparthy, S. C., Nalmpantis, C. Dwivedi- Yu, J., Zhuravinskyi, M., Hambro, E., Sukhbaatar, S., and Raileanu, R. Teaching large language models to reason with reinforcement learning. arXiv 2024.
Jain, M., Deleu, T., Hartford, J., Liu, C.-H., Hernandez- Garcia, A., and Bengio, Y. Gflownets for ai-driven scien- tific discovery. Digital Discovery, 2(3):557-577, 2023a
preprint ar orXiv:2403.0466
He, H., Chang, C., Xu, H., and Pan, L. Looking backward: Retrospective backward synthesis for goal-conditioned gflownets. arXiv preprint arXiv:2406.01/500 2024.
Jain, M., Raparthy, S. C., Mernández-Gardd A., Rector- Brooks, J., Bengio, Y., Miret, S., and Bengio, E. Multi- objective gflownets. International conference on ma- chine learning, pp. 14631-1465: PMLR, 2023b.
Hendrycks, D.,, Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. arXiv preprint arXiv:2 2021.
Jang, H., Kim, M., and Ahn, S. Learning energy decompo- sitions for partial inference of gflownets. arXiv preprint arXiv:2310.0330/ 2023.
Samandez-Garciaa A., Saxena, N., Jain, M., Liu, C.-H., and Bengio, Y. Multi-fidelif active learning with gflownets. arXiv preprint ar arXiv:2306.1 11715, 2023.
Jiang, Y., Ilievski, F., Ma, K., and Sourati, Z. BRAIN- TEASER: Lateral thinking puzzles for large language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 14317-14332, Singapore, December 2023. Association for Computa- tional Linguistics. doi: 10.18653/w12003..mmmppp 885. URL https:/ /alss//aalaaaan wwc.anthhooo..oogoo/0000. 2023. emnlp-maine 1alp-mmann.. 885.
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. arXiv preprint 1rXiv:1904.09751, 2019.
Hosseini, A., Yuan, X., Malkin, N., Courville, A., Sordoni, A., and Agarwal, R. V-star: Training verifiers for self- taught reasoners. arXiv preprint orXiv:2402.00 00644777 2024.

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
Jin, C., Wu, Y., Cao, J., Xiang, J., Kuo, Y.-L., Hu, Z., Ullman, T., Torralba, A., Tenenbaum, J. B., and Shu, T. Mmtom-qa: Multimodal theory of mind question answering. arXiv preprint arXiv:2401.087443 2024.
Lahlou, S., Deleu, T., Lemos, P., Zhang, D., Volokhova, A., Hernández-Crareaa A., Ezzine, L., N., Bengio, Y., and Malkin, N. A theory of continuous generative flow net- works. In International Conference on Machine Learning, pp. 18269-18300. PMLR, 2023.
Jung. J., Qin, L.. Welleck, S., Brahman, F., Bhagavatula, C., Bras, R. L.., and Choi, Y. Maieutic prompting: Logically consistent reasoning with recursive explanations. arXiv preprint arXiv:2205.1182. 2022.
LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., and Huang, F. A tutorial on energy-based learning, Predicting struc- tured data, 1(0), 2006.
Kambhampati, S., Valmeekam, K., Guan, L., Stechly, K., Verma, M., Bhambri, S., Saldyt, L.,, and Murthy, A. Lims can't plan, but can help planning in Ilm-modulo frame- works. arXiv preprint wrXiv:2402.08//1 2024.
Lee, H., Kim, S., Lee, S., Hwang, S., Lee, J., Lee, B.-J., and Kim, S. Arcle: The abstraction and reasoning corpus learning environment for reinforcement learning, arXiv preprint arXiv:2407.20806 2024a.
Lee, S., Kim, M., Cherif, L., Dobre, D., Lee, J., Hwang, S. J., Kawaguchi, K., Gidel, G., Bengio, Y., Malkin, N., et al. Learning diverse attacks on large language models for robust red-teaming and safety tuning. arXiv preprint arXiv:2405. 185.0 2024b.
Kang, H., Sachdeva, E., Gupta, P., Bae, S., and Lee, K. Gflowvlm: Enhancing multi-step reasoning in vision- language models with generative flow networks. arXiv preprint urXiv:2503.065/4 2025.
Kim, H., Kim, M., Choi, S., and Park, J. Genetic-guided gflownets: Advancing in practical molecular optimization benchmark. arXiv preprint 2xXv:2402.00966, 2024a.
Li, C., Liu, C.-H., Liu, D., Cai, Q.. and Pan, L. Bi- furcated generative flow networks. arXiv preprint arXiv:2406.01901, 2024a.
Kim, M., Yun, T., Bengio, E., Zhang, D., Bengio, Y., Ahn, S., and Park, J. Local search gflownets. arXiv preprini arXiv:2310.02710, 2023.
Li, Q., Fu, L., Zhang, W., Chen, X., Yu, J. Xia, W., Zhang, W., Tang. R., and Yu, Y. Adapting large language models for education: Foundational capabilities, potentials, and challenges. arXiv preprint (rXXi:2401.08666 2023a.
Kim, M., Choi, S., Son, J., Kim, H., Park, J., and Bengio, Y. Ant colony sampling with gflownets for combinatorial optimization. arXiv preprint arXiv:2403.07044 2024b.
Li W. Li. Y., Li, Z., Hao, J., and Pang, Y. Dag matters! gflowntts enhanced explainer for graph neural networks. arXiv preprint arXiv:2303. 02448, 2023b.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Ad- vances in neural information processing systems, 35: 22199-22213, 2022.
Li, Y., Luo, S., Wang. H., and Hao, J. Cflownets: Continu- ous control with generative flow networks, arXiv preprint arXiv:2303.00 02430, 2023c.
Koziarski, M., Rekesh, A., Shevchuk, D., van der Sloot, A., Gainski, P., Bengio, Y., Liu, C.-H., Tyers, M., and Batey, R. A. Rgfn: Synthesizable molecular generation using gflownets. arXiv preprint arXiv:2406.0855 2024.
Li, Z., Chen, G., Shao, R., Jiang, D., and Nie, L. Enhancing the emotional generation capability of large language models via emotional chain-of of-uuhh arXiv preprint MXXi:24401006834 2024b.
Kraaijveld, K. Jiang. Y.. Ma, K., and llievski. F. Columbus: Evaluating cognitive lateral understand- ing through multiple-choice rebuses. arXiv preprint arXiv:2409.04053 2024.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint urXiv:2305.200540 2023.
Kumar, A., Zhuang, V., Agarwal, R., Su, Y., Co-Reyes, J. D., Singh, A., Baumli, K., Iqbal, S., Bishop, C., Roelofs, R., et al. Training language models to self-correct via reinforcement learning. arXiv preprint 2rXiv:2409.12917 2024.
Lin, B. Y., Wu, Z., Yang, Y., Lee, D.-H., and Ren, X. Rid- dleSense: Reasoning about riddle questions featuring lin- guistic creativity and commonsense knowledge. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1504-1515, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/11/2011 findings-acl. 131. URL https:/ acl aclanthology. org/2021 / findings-aci 131.
Kwon, O. J., Matsunaga, D. E., and Kim, K.-E. Gdpo: Learning to directly align language models with diversity using gflownets. arXiv preprint arXiv:2410. 15096, 2024.
12

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
Liu, D., Jain, M., Dossou, B. F., Shen, Q., Lahlou, S., Goyal, A., Malkin, N., Emezue, C. C., Zhang. D., Hassen, N., et al. Gflowout: Dropout with generative flow networks. In International Conference on Machine Learning, pp. 21715-21729. PMLR, 2023a.
Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., and Scialom, '. Gaia: benchmark for general ai assistants. arXiv preprint arXiv:2311 12983, 2023.
Michail, A., Clematide, S., and Opitz, J. Paraphrasus: A comprehensive benchmark for evaluating paraphrase de- tection models. arXiv preprint arXiv:2409. 22ti:2409.12000, 2024.
Liu, S., Cai, Q. He, Z., Sun, B., McAuley, J., Zheng, D., Jiang, P., and Gai, K. Generative flow network for list- wise recommendation. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1524-1534, 2023b.
Min, D., Hu, N., Jin, R., Lin, N., Chen, J., Chen, Y., Li, Y., Qi, G., Li, Y., Li, N., al. Exploring the impact of table-to-text methods augmenting Ilm-based ques- tion answering with domain hybrid data. arXiv preprint arXiv:2402. 2et.22242..88000 12869, 2024a.
Lu, S. Z.. Lu, Z., Hajiramezanali, E., Biancalani. T., Ben- gio, Y., Scalia, G., and Koziarski, M. Cell morphology- guided small molecule generation with gflownets. In ICML 2024 Workshop on Structured Probabilistic Infer- ence & Generative Modeling, 2024.
Min, D., Xu, Z., Qi. G., Huang, L., and You, C. Unihgkr: Unified instruction-ave heterogeneous knowledge re- trievers. arXiv preprint arXiv:2410.20163, 2024b.
Mirchandani, S., Xia, F., Florence, P., Ichter, B., Driess, D., Arenas, M. G., Rao, K., Sadigh, D., and Zeng, A. Large language models as general pattern machines. arXiv preprint arXiv:2307.0472) 2023.
Luong, T. Q., Zhang, X., Jie, Z., Sun, P., Jin, X., and Li, H. Reft: Reasoning with reinforced fine-tuning, arXiv preprint arXiv:2401.0896) 2024.
Ma, G., Bengio, E.,, Bengio, Y., and Zhang, D. Baking sym- metry into gflownets. arXiv preprint arXiv:2406.05426, 2024.
Mishra, S., Finlayson, M., Lu, P., Tang, L., Welleck, S., Baral, C., Rajpurohit, T., Tafjord, O., Sabharwal, A., Clark, P., et al. Lila: A unified benchmark for math- ematical reasoning. arXiv preprint arXiv:2210.17517, 2022.
Madaan, A., Tandon, N.. Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye. S., Yang, Y., et al. Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing Systems, 36, 2024.
Naik, R., Chandrasekaran, V., Yuksekgonul, M., Palangi, H., and Nushi, B. Diversity of thought improves reason- ing abilities of large language models. arXiv preprint arXiv:2310.0700 2023.
Madan, K., Rector-Brooks J., Korablyov, M., Bengio, E., Jain, M., Nica, A. C., Bosc, T., Bengio, Y., and Malkin, N. Learning gflownets from partial episodes for improved convergence and stability. International Conference on Machine Learning, pp. 23467-23483, PMLR, 2023.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730 27744, 2022.
Madan, K., Zhan, A., Lamb, A., Bengio, E., Pan, L., Berseth, G., and Bengio, Y. Goal2flownet: Learning diverse pol- icy covers using GFlownets for goal-conditioned RL, 2024. URL https: http//ppeeevviieeeeeeeeeeeeeeeee net. net//orumm? forum? id-VCscggkg2t.
Pan, L.,, Zhang, D., Courville, A., Huang, L., and Bengio, Y. Generative augmented flow networks. arXiv preprint arXiv:2210.03008, 2022.
Pan, L., Malkin, N., Zhang, D., and Bengio, Y. Better training of gflownets with local credit and incomplete trajectories. In International Conference on Machine Learning, pp. 26878-26890 PMLR, 2023a.
Malkin, N., Jain, M., Bengio, E., Sun, C., and Bengio, Y. Trajectory balance: Improved credit assignment in gflownets. Advances in Neural Information Processing Systems, 35:5955-5967, 2022a.
Pan, L., Zhang, D., Jain, M., Huang, L., and Bengio, Y. Stochastic generative flow networks. In Uncertainty in Artificial Intelligence, pp. 1628-1638. PMLR, 2023b.
Malkin, N., Lahlou, S., Deleu, T., Ji, X., Hu, E., Everett, K. Zhang, D., and Bengio, Y. Gflownets and variational inference. arXiv preprint arXiv:2210.00580 2022b.
Qin, L., Welleck, S., Khashabi, D., and Choi, Y. Cold decoding: Energy-based constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems, 35:9538 95531895511111 2022.
Meister, C., Pimentel, T., Wiher, G., and Cotterell, R. Lo- cally typical sampling. Transactions of the Association for Computational Linguistics, 11:1021221 2023.
13

---

Flow of Reasoning: Training LLMS for Divergent Reasoning with Minimal Examples
Qiu, L.., Jiang, L.,, Lu, X., Sclar, M., Pyatkin, V., Bhaga- vatula, C., Wang, B., Kim, Y., Choi, Y., Dziri, N., al. Phenomenal yet puzzling: Testing inductive reasoning ca- pabilities of language models with hypothesis refinement. arXiv preprint arXiv:2310.08555 2023.
Shridhar, M., Yuan, X., Côté, M.-A,, Bisk, Y., Trischler, A., and Hausknecht, M. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020.
Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Liu, P. J., Harrison, J., Lee, J., Xu, K., Parisi, A., et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585, 2023.
Rein, D., Hou, B. L.,, Stickland, A. C., Petty. J., Pang, R. Y.. Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-pron proof q&a benchmark. arXiv preprint ar Xiv:2311.11022. 2023.
Roy, J.. Bacon, P.-L., Pal. C., and Bengio, E. Goal- conditioned gflownets for controllable multi-objective molecular design. arXiv preprint arXiv:2306.04620, 2023.
Song, Z., Yang, C., Wang, C., An, B., and Li, S. Latent logic tree extraction for event sequence explanation from llms. arXiv preprint arXiv:2406.01 124, 2024.
Summers-Strr D., Voss, C. R., and Lukin, S. M. Brain- storm, then select: generative language model im- proves its creativity score. In The AAAI-23 Workshop on Creative Al Across Modalities, 2023, URL https: / openreview. net/ /
Runco, M. A. Divergent thinking. Ablex Publishing Corpo- ration Norwood, NJ, 1991.
Runco, M. A. and Acar, S. Divergent thinking as an indica- tor of creative potential. Creativity research journal, 24 (1):66-75, 2012.
Takase, R., Tsunokake, M., Tsuchiya, Y., and Inuzuka, S. Gflownet fine-tuning for diverse correct solu- tions in mathematical reasoning tasks. arXiv preprint arXiv:2410.20147, 2024.
Saparov, A. and He, H. Language models are greedy rea- soners: A systematic formal analysis of chain-of-thought. arXiv preprint arXiv:2210.012444 2022.
Tan, J. C. M. and Motani, M. Large language model (Ilm) as system of multiple expert agents: An approach to solve the abstraction and reasoning corpus (arc) challenge. arXiv preprint arXiv:23 146, 2023.
Saunders, W., Yeh, C., Wu, Bills, S., Ouyang, L.. Ward, J., and Leike, J. Self-critiquing models for assisting human evaluators, arXiv preprint (Xiv:2206600802. 2022.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634 2017.
Tian, Y., Peng, B., Song, L., Jin, L., Yu, D., Mi, H.. and Yu, D. Toward self-improvement of 1lms via imagination, searching, and criticizing. arXiv preprint arXiv:2404 12253, 2024.
Sendera, M., Kim, M., Mittal, S., Lemos, P., Scimeca, L., Sector-Brooks. J., Adam, A., Bengio, Y., and Malkin, N. On diffusion models for amortized inference: Bench- marking and improving stochastic control and sampling. arXiv preprint ar arXiv:2402.05098 2024.
Todd, G., Merino, T., Earle, S., and Togelius, J. Missed connections: Lateral thinking puzzles for large language models. arXiv preprint arXiv:2404.1 1730, 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Shen, M. W., Bengio, E., Hajiramezanali, E., Loukas, A., Cho, K., and Biancalani, T. Towards understanding and improving gflowee training. In International Conference on Machine Learning, pp. 30956-30975. PMLR, 2023a.
Shen, T., Pandey, M., and Ester, M. Tacogin: Target con- ditioned gflownet for srueture-based drug design. arXiv preprint arXiv:2310.03223, 2023b.
Trinh, H., Wu, Y., Le, Q. V., He, H., and Luong, Solv- ing olympiad geometry without human demonstrations. Nature, 625(7995):476-455 2024.
Shi, C., Yang, H., Cai, D., Zhang, Z., Wang, Y., Yang, Y., and Lam, W. A thorough examination of decoding meth- ods in the era of llms. arXiv preprint arXiv:2402.06925, 2024.
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, I.., Creswell, A., Irving, G., and Higgins. I. Solv- ing math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.142773 2022.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal rein- forcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Valmeekam, K., Marquez, M.. Sreedharan, S., and Kamb- hampati, S. On the planning abilities of large language models-a critical investigation. Advances in Neural Infor- mation Processing Systems, 36, 2024.
14

---

Flow of Reasoning: Training LLMS for Divergent Reasoning with Minimal Examples
Vemgal, N., Lau, E., and Precup, D. An empirical study of the effectiveness of using a replay buffer on mode discovery in gflownets. arXiv preprint arXiv:230..07674, 2023.
Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 37, pp- 41 4115-4122, 2023a.
Xu, Y., Li, W., Vaezipoor, P., Sanner, S., and Khalil, E. B. Lims and the abstraction and reasoning corpus: Suc- cesses, failures, and the importance of object- based rep- resentations. arXiv preprint arXiv:2305.18335 2023b.
Vijayakumar, K., Cogswell, M., Selvaraju, R. R., Sun, Q.. Lee, S., Crandall, D., and Batra, D. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint arXiv: 1rXiv:1610.02224 2016.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. technical report. arXiv preprint arXiv:2412.15115, 2024.
Waks, Lateral thinking and technology education. Jour- nal of Science Education and Technology, 6:245-255, 1997a. URL https:: /ttp::://appp.ee org/ htc//orppuSD::10865553
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao. Y. React: synergizing reasoning and acting in language models (2022). arXiv preprint
Waks, Lateral thinking and technology education, Jour-
arXiv:2210.03629, 2023.
nal of Science Education and Technology, 6(4):245-255,
1997b. doi: 10. 10.1023/A410222343100000 //doi. wwwg/////// 1tt.:122222211022333313111
URL https:
Yao. S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Wang, P., Li, I.., Shao, 7., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-shepherd: A label-free step-by-step verifier for Iims in mathematical reasoning. arXiv preprint ar 2xxv:2312.08933: 2023a.
Yu, F., Gao, A., and Wang, B. Dutcome-supervin verifiers for planning in mathematical reasoning. arXiv preprint arXiv:2311.099 1.09724, 2023a.
Wang, R., Zelikman, E., Poesia, G., Pu, Y., Haber, N., and Goodman, N. D. Hypothesis search: Inductive reasoning with language models. arXiv preprint arXiv:2309.05660 2023b.
Yu, F., Jiang, L., Huang, S., Wu, Z., and Dai, X. Persuasive- tom: A benchmark for evaluating machine theory of mind in persuasive dialogues. arXiv preprint arXiv:2502.21017, 2025.
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency im- proves chain of thought reasoning in language models. arXiv preprint arXiv:2203.
Yu, J. He, R., and Ying, R. Thought propagation: An analogical approach to complex reasoning with large lan- guage models. arXiv preprint arXiv:2310.0396,, 2023b.
2022.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F,, Chi, E., Le, Q. V., Zhou, D., al. Chain-of ofothhught prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:24424-248877 2022.
Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. arXiv preprint arXiv:2309. /2284, 2023c.
Yuan, L.,, Cui, G., Wang, H., Ding, N., Wang, X., Deng, J., Shan, B., Chen, H., Xie, R., Lin, Y., et al. Advancing Ilm reasoning generalists with preference trees. arXiv preprint M.XXi::244002077 2024a.
Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang, Y. An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.0m 00724, 2024.
Yuan, W., Pang, Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language models. arXiv preprint arXiv:2401.10020 2024b.
Xie, J., Zhang, K., Chen, J., Zhu, T., Lou, R., Tian, Y., Xiao, Y., and Su, Y. Travelplanner: benchmark for real-world planning with language agents. arXiv preprint arXiv:2402.01622 2024a.
Yuan, Z., Yuan, H., Li, C., Dong, G., Tan, C., and Zhou, C. Scaling relationship on learning mathematical rea- soning with large language models. arXiv preprint arXiv:2308.0183 2023.
Xie, Y., Kawaguchi, K., Zhao, Y., Zhao, J. X., Kan, M.- Y., He, J., and Xie, M. Self-evaluation guided beam search for reasoning. Advances in Neural Information Processing Systems, 36, 2024b.
Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.nm 05653,, 2023.
Xu, Y., Khalil, E. B., and Sanner, S. Graphs, constraints, and search for the abstraction and reasoning corpus. In
15

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. Star: Boot- strapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
Zhu, X., Wang, J., Zhang, L.,, Zhang, Y., Huang, Y., Gan, R., Zhang, J., and Yang. Y. Solving math word problems via cooperative reasoning induced language models. arXiv preprint wxXi:2220.16255/ 2022.
Zhang, D., Malkin, N., Liu, Z., Volokhova, A., Courville, A., and Bengio, Y. Generative flow networks for discrete probabilistic modeling. In International Conference on Machine Learning, pp. 2641 2-24428. PMLR, 2022.
Zimmermann, H., Lindsten, F., van de Meent, J.-W., and Naesseth, C. A. A variational perspective on generative flow networks. arXiv preprint wrXiv:2210.07990 2022.
Zhang, D., Dai, H., Malkin, N., Courville, A., Bengio, Y., and Pan, I.. Let the flows tell: Solving graph combinato- rial optimization problems with gflownets. arXiv preprint arXiv:2305. 17010, 2023a.
A. Additional Related Work
Reasoning with LLM. Recent LLMss (Achiam et al., 2023;
Touvron et al., 2023; Bai et al., 2022; Chowdhery et al., 2023) have demonstrated great potentials in tackling com- plex reasoning tasks (Cobbe et al., 2021; Mishra et al., 2022; Hendrycks et al., 2021; Rein et al., 2023; Mialon et al., 2023). (1) Fine-tuning LLMs is a primary way to enhance their reasoning abilities, including SFT and reward- maximization reinforcement learning (RL) approaches. SFT with large-scale and high-quality datasets of reasoning chains has proven very effective (Yu et al., 2023c; Yue et al., 2023; Yuan et al., 2024a). Various methods for constructing training samples have been proposed when ground truth rea- soning chains are not available. For example, STaR (Zelik- man et al., 2022) uses online sampling with self-correction to find positive samples. ReSTEM (Singh et al., 2023) and V-SSae(iosseini et al., 2024) filter samples with external verifiers. RL techniques, particularly reward-maximizin policy optimization methods like PPO, are widely employed in LLMs (Ouyang et al., 2022: Bai et al., 2022; Havrilla et al., 2024; Luong et al., 2024). However, both maximum likelihood training (i.e. SFT) and reward-maximinn icy optimization (e.g.. PPO) do not encourage models to generate diverse solutions. (2) prompting-
Zhang, D., Pan, L.,, Chen, R. T., Courville, A., and Bengio, Y. Distributional gflownets with quantile flows. arXiv preprint ar X1v::2302.05793, 2023b.
Zhang, D. W., Rainone, C., Peschl, M., and Bondesan,
R. Robust scheduling with gflownets. arXiv preprint 0rXiv:2302.05444 2023c.
Zhang, Z., Yao, Y., Zhang.
A., Tang, X., Ma, X., He, Z., Wang, Y., Gerstein, M., Wang, R., Liu, G., et al. Igniting language intelligence: The hitchhiker's guide from chain- of-thought reasoning to language agents. arXiv preprint arXiv:2311.11797, 2023d.
Zhang, Z., Zhang, A., Li, M., Zhao, H., Karypis, G., and Smola, A. Multimodal chain thouuht reasoning in lan- guage models. arXiv preprint arXiv:2302.000 00923, 2023e.
Zhang. Z., Zheng, C.,, Wu, Y., Zhang, B., Lin, R., Yu, B., Liu, D., Zhou, J., and Lin, J. The lessons of developing process reward models in mathematical reasoning. arXiv 2025.
pol-
preprint arXiv:2501.072:
Zhao, Z., Lee, W. S., and Hsu, D. Large language models as commonsense knowledge for large-scale task planning. Advances in Neural Information Processing Systems, 36, 2024.
reason- ing algorithms aim to better elicit the knowledge inside LLMs without tuming their parameters. Techniques such as CoT (Wei et al., 2022) and its variants (Chen et al., 2022; Li et al., 2024b; Zhang et al., 2023e; Zhou et al., 2022: Kojima et al., 2022) have improved LLM performance by enabling them to generate intermediate steps before arriving at final answer. To provide reasoning more guidance, self- evaluation (Xie et al., 2024b; Shinn et al., 2024; Madaan et al., 2024) and reward models are introduced to enhance reasoning process (Uesato et al., 2022; Lightmmnn et al., 2023) Besides, a more relevant series of works combine LLM reasoning capabilities with planning and search algo- rithms such as MCTS (Hao et al., 2023; Feng et al., 2023; Zhao al., 2024), tree and graph search (Jung et al., 2022: Zhu et al., 2022; Yao et al., 2024; Besta et al., 2024; Yao et al., 2023). Moreover, recent studies turn to amortizing computation for reasoning paths (Yuan et al., 2023; Wu et al., 2024; Bansal et al., 2024), such as self-correct (Kumar et al.,
Zhong, S., Huang, Z., Gao, S., Wen, W., Lin, L., Zitnik, M., and Zhou, P. Let's think outside the box: Exploring leap- of-thought in large language models with creative humor generation, 2024. URL https:: //a //txxvvvvvv////////////a/////av org/ abs/ 2312. 02439.
Zhou, D., Schärli, N., Hou, I.,, Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q. al. Least-to-mos: prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205. 10625, 2022.
Zhu, D., Li, Y., Shao, Y., Hao, J., Wu, F., Kuang, K., Xiao, J., and Wu, C. Generalized universal domain adaptation with generative flow networks. In Proceedings of the 31st ACM International Conference on Multimedia, pp-
2024; Saunders et al., 2022), self-improvemenn (Tian et al.,
8304-83155 8315, 2023.
2024; Yuan et al., 2024b).
16

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
GFlowNets. GelowNets (Bengio al., 2021) were origi- nally proposed to learn policies for sampling from unnor- malized distributions, with primary motivation from scien- tific discovery (Jain et al., 2023a), which requires generat- ing diverse high-reward samples (Shen et al., 2023b; Roy et al., 2023; Zhang et al., 2023b; Ma et al., 2024; Pan et al., 2023b), such as molecular generation (Koziarski et al., 2024; Kim et al., 2024a; Lu et al., 2024) and biological sequence generation (Ghari et al., 2023; Jain et al., 2022). Beyond the science domain, GelowNets have also been applied in various downstream applications such as recommendation systems (Liu et al., 2023b), domain adaptation (Zhu et al., 2023), combinatorial optimization (Zhang et al., 2023a; Kim et al., 2024b) and explainability of deep neural networks (Li et al., 2023b). Additionally, GFlowNets have proven to be suitable for sampling from posterior distributions (Hu et al., 2023b; Deleu et al., 2022: 2024; Zhang et al., 2022). As reinforcement learning method, prior works have incor- porated intermediate feedback with GelowNets to address sparse reward issues (Pan et al., 2023a; Jang et al., 2023; Pan et al., 2022) and multi oujttiobeecttve rewards (Jain et al., 2023b; Hernandez- Gareia et al., 2023; Chen & Mauch, 2023). There are also theoretical analyses treating GetooWeets as recurrent MCMC (Deleu & Bengio, 2023) and variational inference (Malkin et al., 2022b; Zimmermann et al., 2022) that are used to model the distribution over trajectories.
(DAG) g AS, where S is finite set of vertices (states), and A = 8 x S is set of directed edges (ac- tions). If 5 - 5' is an action, we say is parent of s' and s' is a child of s. There is exactly one state that has no in- coming edge, called the initial state 8n E S. States that have no outgoing edges are called terminal states. We denote by the set of terminal states. A complete trajectory is a sequence T = (s0 ... - sn) such that each s; - Si+1 is action and 8n complete trajectories and the terminal state Ta-
X. We denote by T the set of
Here we define the reward R : X - Rt, and define a forward transition probability function, or a forward policy, (-(()s), which a distribution over the children of every state 8 E S. The forward policy is typically parametrized by neural network that takes representation of s as input and produces the logits of a distribution over its children. Any forward policy Pp induces a distribution over complete trajectories т E T (denoted by Pr as well), which in turn defines marginal distribution over terminal states x E X:
PF(T) = PF(50 - ... + 5. II Pr(s441(s)) V
(8)
Given forward policy Pr, terminal states E X can be sampled from PF by sampling trajectories T from (p( and taking their final states 8,, GFlowNets aim to find a forward policy Pp such that the induced distribution Pf (z) is proportional to the reward function:
Diverse Thinking. Vertical and lateral thinking (Waks, 1997a; Ismayilzada et al., 2024) are two distinct approaches that differ significantly in their focus and methodology. Ver- tical thinking emphasizes logical, structured, and sequen- tial reasoning, often following a step-by-step approach to solve problems. Our work aligns with this paradigm to generate multiple correct, structured reasoning processes to achieve specific goals. In contrast, lateral thinking priori- tizes creativity and innovation, encouraging the exploration of unconventional perspectives and challenging established assumptions. There are multiple works that evaluate (Huang et al., 2023; Chen et al., 2024a; Kraaijveld et al., 2024; Todd et al., 2024) and promote(Zhong et al., 2024; Summers-Stay et al., 2023; Jiang et al., 2023; Lin et al., 2021) the lateral thinking ability of LLMs.
(x)
(9)
Training, Training GFlowNets considers achieving a consis- tent flow (Bengio et al., 2023: Malkin al., 2022a), which means the flow for the forward direction should equal to the flow for the backward direction. Below we introduce relevant objectives.
Detailed Balance (DB). The DB objective (Bengio et al., 2023) requires learning two objectives in addition to para- metrie forward policy Pr(-1s): 1. A Buckward policy, which is distribution PB(('|s;; 6) over the parents of any non- initial state. 2. A State flow function: F(::6) S Rzo- Then DB loss for single transition s' is defined as:
B. Preliminaries and Background
GFlowNets (Bengio et al., 2021: 2023; Liu et al., 2023a) class of models that amortize the cost of sampling from an intractable target distribution over terminal states approximation
-(08 F(s: F(s:0))P(((1:00) 2 ((s:0))P((ss:)0)
CDB = (log
(10)
X' by learning a neural of the target distribution using its unnormalized density or reward function. The task of sampling from this distribution resorts to a decision-male making process. Below, we introduce GFlowNets with more details.
if LDB is optimized to 0 for each transition, then the forward policy Pr satisfies 9.
Trajectory Balance (TB). Trajectory balance (Malkin et al., 2022a) introduces a backward policy Pp. which is a learned distribution Pa(-Is') over the parents of every state s E S. We have the following backward probability:
Settings. We are given a pointed directed acyclic graph
17

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
Table 9: Training time shown is seconds when training on the Blocks
n-1 PB(T) = PB(s0 - -... sn|sn)) = II
PB( VTET
Method
Runtime (s)
1-0
(11)
SFT
196.37
SFT+PPO
1740.96
Then we give the trajectory balance constraint for any com- plete trajectory (80 8n):
FoR
6833.37
[(;; > 1) is an indicator function that is if there is at least one successful trajectory found for the i-th question and otherwise. Thus, the denominator the number of exam- ples in which model finds at least one trajectory, and the nominator is the of all successful trajectories model finds across all examples. The smallest diversity is 1 when a method can only find at most one successful trajectory on average, and diversity = 1.5 indicates a method able to find 1.5 different successful trajectories on average.
((0)) II Pr P = (88n) I Pn(8i-1(8s) (12)
where we have P(s,)= = F(s.)/Z. To enforce this con- straint, we convert equation 12 into an objective for op- timization. Suppose that a model with parameters out- puts estimated forward policy (r(-|s; 0) and backward pol- icy PB[-|s;), for states s, well as a scalar Ze estimat- ing F(so). GFlowNets aim to approximate a Markovian Flow Fg that F(sn)= R(s,). For a complete trajectory (80 _ - 8a), define the trajectory balance loss as
C.2. Creativity Metric
follows:
We define the following metric to quantify the creativity of a reasoning method. Given the same number of samples during inference, we calculate the ratio of unique success- ful trajectories that method identifies in the test dataset Dteat, which not found by any other methods. Let M = (m,, M.... m)M represent the set of reasoning methods. For the i-th problem. the 1-th method has a so- lution set S% where 1 & 1 & |M]. The complete set of solutions across all methods is defined as:
2
(TB(T: 1(18g log
Zo
Pr(s 1 88; 0)
(13)
R(sn)
If LTB is made equal 0 for every complete trajectory T, then equation 9 satisfies for all x X and Z is the inverse constant of proportionality: Z Exex R(x).
Conditional GFlowNets. In GFlowNet, both the policy and reward function can be conditioned on additional infor- mation. For instance, the tasks we focus on, a GelowNet policy generates actions sequentially for an embodied rea- soning problem, starting from initial state 8o and a goal g. Furthermore, the allowable actions vary depending the specific so in each case. The conditional GFlowNet, we train achieve amortization by sharing the policy model across different so and g, enabling the model to generalize to initial states and targets that were not seen during training.
S
UU
S
(15)
Then we can define the creativity metric of method my as:
D
Creativity(mi) = [|]
Σ Σ I(s SES
(16)
where for the i-th problem, the solution E S is found only by method my and not by any other method mg (where k # 1), then I(s, i, 1. Otherwise, i(s,i,) = 0. The indicator function T(,, i, 1) is defined as:
C. Experimental details
C.1. Diversity Metric
We define the following metric to measure the diversity of reasoning paths found by different approaches. Under the same number of samplings at inference time, we count the number of different successful trajectories a policy finds for the successful example on average.
11 if Ukei St
[(s,i, 0, otherwise
(17)
C.3. Efficiency Analysis
E St 1(S, 1); EX I(S 1),
All experiments were conducted using server with single NVIDIA A100 GPU. Below we report the average of 3 times training for 6-step training cost on Blocks World dataset for 10 epochs. We compare with SFT, PPO and table 9 shows the results.
Diversity
(14)
where n is the total number of problems, S, is the number of successful trajectories found for the i-th question, and
18

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
PPO and FOR need much more training costs because they need exploration and interaction with environments col- lect trajectories for training, and SFT only trains on ground- truth trajectories which take less time.
(5) Proximal Policy Optimization (PPO) (Schulman et al., 2017): This is widely-used forcement learning method for LLM training. We de- sign the objective to encourage the LLM to generate so- lutions that satisfy the goal. Following the common prac- tice of previous work (Ouyang et al., 2022; Wang et al., 2023a), we penalize the policy if it deviates too much from the reference policy. Formally, the objective is max=, ET-Ty [R(E, v)] (1l. [TO(y 2>)]| (y) =:)]
rein-
C.4. Blocks
FoR Setup. During the training, we finetune the LLM with LoRA (Hu et al., 2021) with r= 32, a = 64, and 1.. We set e from 0.3 and decrease it to 0.01, 33 from 1 to 2, and the probability 8 using replay buffer increases from 0.3 to 0.5 throughout the iterations linearly. The learning rate is set to le-4 with a cosine annealing schedule, and the number of training iterations is set 10. Reward weight À is set to 1.5. In our ablation study when setting À = O, we add small number b = 0.5 to avoid log O. Table 7 shows the template we use for the forward policy in the 6-step setting, and its difference between 2- step and 4-step is only replacing the 6-step demonstration to 2-step and 4-step. During testing, we sample 8, 20, and 40 trajectories for 2, 4, and 6 steps respectively. As long as one trajectory reaches the goal, we label this instance solved, all the baselines conform to the same rule.
(6) GFN-CoT (Hu et al., 2023a): This approach adapts the GFlowNets training paradigm, which is a diversity- -eeekigg RL method, to enable posterior sampling of the intermediate reasoning process from LLMs.
Performance on OOD settings. We further assess perfor- mance on out-of-distribttion (OOD) settings. Specifically, train the model using FoR and SFT 2-step training set and evaluate it on a 4-step test set, and then train the model on the 4-step training set and evaluate it on the 6-step test set. This allows us to analyze their generalization on OOD problems. For prompting based baselines, we use step and 4-step examples as demonstrations, respectively.
According to the result in table 10, FoR maintains the high- est accuracy (71.43%) OOD tasks compared to other methods like CoT and SFT, which range from 9.52% to 14.28%. FOR also achieves greater diversity (by an abso- lute improvement of 0.2 over SFT), highlighting its superior generalization and solution exploration capabilities.
Additional details for baselines. We compare FoR the following baselines:
(1) of prompting (CoT) (Wei et al., 2022): It concatenates i problems with ground truth solutions and the test problem, and prompts the LLM to generate solu-
tion. We test the setting where k = 1, 5, 15, and pass the test cases to LLMs at the same times as FoR, and the test case is regarded as solved if at least one plan is correct.
Additional results on search- based approaches. To as- sess whether search-based methods can generate diverse solutions, we execute ToT (DFS) multiple times (equal to the number of runs performed for our method) and com- pare their performance. The results shown in Table 13 demonstrate that though search-based methods improve the diversity, they still underperform FOR.
(2) ThToughtsss prompting (ToT) (Yao et al., 2024): This approach constructs a tree of actions and searches for the solution with the highest reward. For each action, the reward includes (a) the likelihood of the LLM predicting the action and (b) self-evaluation, where the LLM is prompted with the question, "Is this action good?" and the answer is mapped to a reward value. We implement ToT with both breadth-firt search (BFS) and depth-first search (DFS), terminating after generating 10 solutions.
Additional baselin results with eeeersttt-enouurainng in- struction. To further stimulate the diverse reasoning ability in the baseline approaches, we add a diversity prompt as instruction:
Please carefully understand the goals and initial states, then come up with diverse solutions and think outside the
(3) Reasoning-vit via- Planning (RAP) (Hao al., 2023): This method also conducts tree search for the optimal solution. Different from ToT, it alternatively predicts the next action and predicts the resulting block arrangement. Besides the rewards used in ToT, if the predicted block arrangement matches the goal, a high reward will be assigned.
box.
We evaluate multiple baseline methods using LLama-3-8B as the base model, following the exact same settings as in Section 4.2. The results for BlocksWorle are reported in Table 11. The numbers in parentheses indicate the perfor- mance difference compared to the original prompt without the diversity- instruction.
(4) Supervised Fine-Tuning (SFT): We use problems in the training set and their corresponding ground truth solutions to finetune the LLM. Note that this is an easier setting than FoR which does not have access to ground truth solutions. We train LLM with the same iterations as FOR.
We observe that and SFT basclines lead to improvements in both diversity and accuracy, with average absolute gains of 0.03 and 5.1 1%,
prompts for the CoT
19

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
Table 10; OOD results on BlocksWorle
2-step to 4-step
4-step to 6-step
Method
Acc. (%)
Diversity
Creativity (%)
Acc. (%)
Diversity
Creativity (%)
CoT (1-shot)
9.52
1.00
3.12
2.02
1.00
0.00
CoT (S-shot)
14.28
1.00
3.12
12.12
1.08
3.45
CoT (15-shot)
11.90
1.00
3.12
8.08
1.00
0.00
ToT (BFS)
9.52
8.08
ToT (DFS)
4.76
6.06
RAP
80.95
34.34
SFT (a=1.0)
11.92
1.00
9.37 59.38
28.28
1.03
1.15
FOR (Ours)
71.43
1.20
65.65
1.25
60.92
Table 11: Baseline results with diversity-encouraging instruction prompt. "+" denotes the performance improvement compared to the absence of the prompt.
4-step
6-step Diversity
Method
Acc. (%)
Diversity
Creativity (%)
Acc. (%)
Creativity (%)
CoT (1-shot)
16.67 (-10.90)
1.00 (-0.05)
0.0 (0.00)
11.11 (-4.71)
1.09 (+0.04)
0.0 (0.00)
CoT (5-shot)
59.52 (+16.66)
1.12 (+0.08)
2.04 (+2.04)
33.33 (43.70)
1.03 (+0.00)
0.79 (+0.79)
CoT (15-shot)
52.38 (+12.32)
1.09 (+0.06)
0.0 (0.00)
13.13 (-6.40)
1.07 (+0.04)
0.0 (0.00)
SFT (a=1.0)
59.52 (+17.46)
1.10 (+0.05)
0.0 (0.00)
47.47 (+12.79)
1.10 (+0.06)
0.0 (0.00)
FoR (Ours)
98.41
1.27
12.24
78.44
1.33
9.52
respectively. However, FoR still outperforms them, achiev- ing average absolute improvements of 0.19 in diversity, 9.46% in creativity, and 34.93% in accuracy compared to the best baseline for each metric.
tion and refinement of thoughts during intermediate search stages. While this approach allows for more adaptable thought structures, it still requires several LLM inference calls for evaluation, leading to substantial computational expenses.
Additional case study. In Table 12, we show examples generated by FOR. We observe that after training, FoR can sample the terminal state with probability approximately proportional to the rewards, leading to an approximate sam- pling of different plans with the same probability. This empirically verifies the efficacy of the training objective.
of XoT) (Ding et al., 2023): XOT is a collaborative framework combining LLMs with MCTS to optimize the thought generation process, aiding LLMs in solving complex problems. It first trains a small network to explore the space fast while LLMs refine and correct the thoughts generated by MCTS.
C.5. Game of 24.
C.7. ID-ARC
FoR Setup. See Figure for the prompt template used in the experiment of the Game of 24. We use LoRA to train 32, 1..pout=0.1. We load the LLM in fp16, and set the hyperparameters as follows: batch size = 4, learning rate = 1e-5, number of epochs = 5, and the reward weight w = 100.
FoR Setups. Except that we train the model for 1 iteration, other training hyperparameters are identical to BlocksWorle We use the hand-crafted transformation functions in ARC Challenge 2nd-place (de Miquel, 2021) on Kaggle 2020. See Figure 11 for the prompt template of the ID-ARC task. Part of the prompt is adapted from (Tan & Motani, 2023). For CoT and FoR, we sampled 20 times. IO methods di- rectly predict the output grids without explicit reasoning process, while program-only and Hypothesis Search ap- proaches generate large number of candidate programs and choose the best candidates, which is time-consuming. As a result, we do not report diversity and creativity metrics for these methods.
the model with 8,
C.6. Rubik's Cube
FOR Setups. The training hyperparameters are identical to Blocks World. During testing, we sample 10 trajectories. See Figure for the prompt template of the Rubik's Cube task.
Additional details for baselines. Apart from the baselines in Blocksworld, we further compare them with GoT and XOT.
Additional details for baselines. In addition 10 and CoT, also compare approach with Hypothesis Search which belongs to discrete program search methods (Barke et al., 2024; Xu et al., 2023a; Lee et al., 2024a).
Graph-of Cruph-of-Thoughhe Thought (GoT) (Besta et al., 2024): GoT builds upon the ToT method by introducing the ability to create graph-like thought structures, achieved through the aggrega-
20

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
Table 12: An example of the probability of two trajectories to be sampled by FOR.
Initial State
goal
Trajectory
Terminal State
PF(T)
unstack the orange block from top of the blue block
2. put down the orange block
unstack the blue block from on top of the
the red block is on top of the blue block
red black
0.47
put down the blue black
5. pick up the red block
stack the red block on top of the blue block
unstack the orange block from on top of the blue block
2. put down the orange block
unstack the blue block from top of the red block
the red block is top of the blue block
0.46
stack the blue block top of the orange block
5. pick up the red block
stack the red block on top of the blue block
Table 13: Performance of Search-based methods with multi- ple runs.
RAP, we compare FoR with STaR (Zelikman et al., 2022), which uses online sampling to filter our positive examples consistent with ground truth trajectories to finetune the LLM. Note that this is an easier setting than FoR, which doesn't 't have access to ground truth solutions. It also indicates an upper bound of SFT methods that do not rely on ground truth solutions, like. All baselines use Llama3 8B as the base model.
denotes results from a single run.
Method
2-step Acc. (%)
4-step
6-step
Acc. (%)
Diversity
Acc. (9)
Diversity
ToT (DFS)
13.33
16.67
8.08
(DFS)
31.31
FoR
100.00
98.41
1,27
78.44
1.33
Hypothesis Search (Wang et al., 2023b); The method first generates multiple hypotheses describing the underlying transformation rules in natural language, and then selects a subset of potentially correct hypotheses. Based on these selected hypotheses, numerous Python programs are synthe- sized, which are subsequently tested on the training input- output pairs to verify whether they pass all the cases. If program successfully passes all the training input-output pairs, it is considered to have accurately captured the under- lying transformation rules.
C.9. Mathematical Reasoning
Automatic Diversity Metric. To distinguish different solu- tions, we adapt a paraphrase-based method (Michail et al., 2024) to automatically evaluate the diversity with GPT-4o measure the number of different solutions for each problem. The prompt is shown in Figure 13 Additional results. To further investigate the relationship between accuracy and the number of samples, we set the sampling counts to 1, 5, 10, and 20, and compared the performance under these configurations. As shown in Figure 4, FoR consistently out- performs SFT by a significant margin, ranging from 3.5% 6.6%. The higher accuracy demonstrates that aligning with the reward distribution of solutions enables FoR to better explore the reasoning space, increasing the possibility of reaching the correct solution.
OOD data creation. We separate the in-distribution and OOD data by topics and ontology. We use the animal-related problems as in-distribution examples and the number-related problems OOD examples.
C.8. Logical Reasoning
Setup. We use LoRA train the model with r 8, 32, 1ropout=0.1. We load the LLM in fp16, and set the hyperparameters follows: batch size = 4, learning rate 5e-6, number of epochs = 40, and the reward weight 100. See Table 10 for the prompt template of the logical reasoning task.
D. Additional Analysis Results
Performance with various LLMs. We conduct experi- ments to evaluate FoR's 's scalability and diversity gains us- ing larger Llama-3-70B model and different family models, Qwen2.5 (7B & 72B) (Yang et al., 2024) and Intern2.5.
Additional details for Baselines. Apart from CoT, ToT, and
21

---

Flow of Reasoning: Training LLMS for Divergent Reasoning with Minimal Examples
Table 14: Performance with larger models and different families of models.
Method
4-step
6-step
Aec. (%)
Diversity
Acc. (%)
Diversity
CoT 5-shot (Llama3-8B)
28.57
1.05
15.82
1.05
CoT 5-shot (Llama3-70B)
45.23
1.05
46.46
1.11
FOR (Llama3-8B)
98.41
1.27
78.44
1.33
FoR (Llama3-70B)
100.00
1.38
87.65
1,40
FOR (Qwen2.5-7B)
100.00
1.24
86.86
1.36
FOR (Qwen2.5-72B)
100.00
1.41
90.13
1.46
FOR (IntemL_M2.5-7B-Chha))
100.00
1.26
83.83
1.31
2. Offline training: (1) Experience replay represents a significant advancement in reinforcement learning, of- fering enhancements in both learning efficiency and stability, as evidenced by recent empirical studies in GFlowNets (Vemgal al., 2023; Shen et al., 2023a). To optimize the utility of the trajectories collected. we set up prioritized replay buffer (PRB). This buffer facilitates the sampling of trajectories in proportion to their reward value, R(()). or its logarithmic value, thereby prioritizing potentially more informative experiences. (2) For tasks (e.g. Game of 24) that have large space, online sam- pling diverse trajectories with LLMs is computationally expensive. Therefore, we integrate the offline trajecto- ries to have larger coverage of space and improve the efficiency, which means 8 = 0. 7B-Chat (Cai et al., 2024) on the BlocksWorld task. Re- sults shown in Table 14 indicate that FoR consistently im- proves accuracy and diversity with larger models. Com- pared to CoT baselines, FOR exhibits greater diversity gains as model size increases. This suggests that the gains of FoR become more pronounced as model capacity increases. Even with minimal data (15 examples), FOR yields clear improvements, highlighting its potential for robust perfor- mance across different base models.
100
SFT (Mean Std)
80
FoR (Mean Std)
Accuracy (%)
40
20
1
10
20
Number of solutions
Figure 4: Accuracy of FoR and SFT across the different number of sampled solutions.
Algorithm describes the training framework.
F. Modified Local Search
Local search is simple data augmentation technique for GFlowNets (Kim al., 2023; Zhang et al., 2022; Sendera et al., 2024), which is designed to enhance training ef- ficiency. Different from the original local search which is conducted on each sampled trajectory, we select the trajectory in a batch with the highest reward to conduct local search. Here we denote the trajectory reward R(T) ) as the reward of the terminal state of the trajectory R(T (So
E. Exploration and Training
FoR employs the following techniques to explore during the training phase:
sn)) R(sx)- More specifically,
we illustrate our modified local search for one instance as follows:
1. Online training: (1) we employ the online policy (p a), and its tempered version (2) Similar to c-greedy, we sample action at step t by Pr with probabil- ity e, and sample with uniform distribution over action space (acls(st-) with (1 1-c e) probability. (3) To fur- ther explore the high-reward region, we modified the local search (Kim et al., 2023; Zhang et al., 2022). More specifically, we select the trajectory with the highest reward in a batch and conduct a destroy and reconstruc- tion process for augmenting the trajectories to enable a higher probability of sampling successful trajectories, referring to Appendix F for more details.
Sampling: Sample set of complete trajectories [TI..., TM) } using forward policy Pe and select the Tm with the largest reward R((,))
We destroy Tm by backtracking K-step into a partial trajectory and reconstruct the complete trajec- tory from the partial trajectory:
Searching:
Tdestroy (s0 - ... - 8nnk),
(18)
Trecon = (5 ... - 8n)
22

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
Algorithm 1 FoR Training
1: Input: I: number of iterations, PF: initial LLM policy, D: Prioritized Replay Buffer, M: Batch-size, 8: online-offine ratio, & Training Dataset, O: offline Data
2: Output: Trained policy Pr
3: for - to do
4: Sample from training dataset with initial state 5g and goal
5: Sample u ~ [0
6: if then
7: // Exploration
8:
Sample . online trajectories (TI..... TM] with forward policy Pf Select trajectory Tm E (TI...., TM) with the largest R(Tm)
9:
10:
[T...... -xx+ - Local Search(T,, Fm))
11:
Update D DU (T....., TM} u (T T
else
13:
// Exploitation
14:
if is Game24 then
15:
Sample M offline trajectories from Offline Data O else
16:
17:
Sample M offline trajectories from
18:
Exploit M (with N') trajectories to compute the objective function Eq.
19:
Update the parameter Pp with respect to Eq.
20:
return PF
We obtain the local searched trajectory 7':
capability in reaching successful final steps from various middle steps. For example, FoR successfully transitions from intermediate steps (3,6,15) to target 24, whereas SFT fails to so. This highlights the effectiveness of FoR design in simultancously promoting diversity and ensuring accuracy.
-' (50 8
(19)
Where the Trecon is completed by the random policy Py which randomly selects a feasible action for effi- ciency. We can obtain set of reconstructed trajectories (k
Better robustness due to exploratory nature. According to Figure 6, we use the problem (7,9,9,1)) to demonstrate the robustness of FOR. As shown in the figure, SFT repeat- edly fails by getting stuck in a single second state of 19,3,16) 20 times, while FoR successfully discovers multiple diverse trajectories leading the correct solution. This robustness can be attributed to the exploratory nature of FoR's 's train- ing objective, which encourages the model to sample di- verse successful trajectories. By expanding the search space through high-reward exploration, FoR increases the chance of finding successful outcomes. This capability not only improves the robustness of the model but also enhances its generalization new scenarios, showcasing the effective- ness of FoR in addressing complex reasoning tasks.
Filtering:
We now need evaluate the collected recon- structed trajectories TN and determine whether to accept reject 7' E (Ti...,,Th J. Specifically, we accept r' as follows:
A(T,T') (10)=1R((s>> [R(F)>B(T)
(20)
This means greedily filter out the candidates (T()...Te)) c (Tí....,''s Tx) that have higher reward than Tm, which has higher possibility of reaching the goal. Then we return these trajectories and add them into the replay buffer D.
Other examples Figure 14 shows generated samples for the Blocks Figure 15 for Game24, and Table 15 for PrOntoQA, respectively.
G. Case Study
Balance between diversity and accuracy. According to Figure 5, we use the problem (3,4,6,17) to show how FOR achieves such high performance while focusing on diversity. As illustrated in the figure, we compare trajectories sampled 20 times by both SFT and FoR. While both methods pro- duce diverse trajectories initially, FoR demonstrates better
H. Prompts.
Below, detail all the prompts used in all settings from Figure 7, 8, 9, 10, 11, 12.
23

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
FoR
SFT
3 6 11
left:
left: 7.6
left:
left:
left:
left: 3 6
left: 3
left: 4 17
left: 3 4.5
3
6 11
3 4
left: 3
left: 10
left:
left:
left:
left:
15
7
15
left: 10
4
left:
15
left: 15 9
left:
4
22
left: 27
left:
left:
left: 24
left: 19
laft:
left:
left:
left: 135
left:
left:
45
Figure 5: Problem (3,4,6,11). Green blocks represent the states that can achieve 24. Blue numbers represent the sample times of trajectories bigger than 1. This shows that FoR can achieve 24 in multiple different states while SFT usually fails to do so.
Table 15: Examples for PrOntoQA.
Future work can extend FOR to broader reasoning tasks, including reasoning on structured data (Min al., 2024a;b), social reasoning (Jin al., 2024; Yu al., 2025), and mul- timodal reasoning (Zhang et al., 2023e; Kang al., 2025).
Query: True or false:
31 is not imaginary. (OOD)
State
Action
31 a natural number.
Natural numbers integers.
31 is an integer.
Integers are real numbers.
31 a real number.
Real numbers imaginary. Finish.
31 is not imaginary.
Query: True or false: Wren not bony. (In-distrihution) State
Action
Wren is painted lady.
Each painted lady is butterfly.
Wren is butterfly.
Each buttertly a lepidopteran.
Wren is lepidopteran.
Each lepidopteran is insect.
Wren is an insect.
Each insect an arthropod.
Wren is arthropod.
Each arthropod is not bony. Finish.
Wren is not bony.
Limitations and Future Work
Future work should further address the two limitations of FoR.
The first is acquisition of a large number of trajectories efficiently. Online sampling with LLMs is computationally expensive, leading to more efficient and effective strate- gies to explore more complex settings, such as real-world settings, AlfWorld (Shridhar et al., 2020), and TraveIPlan- ner (Xie et al., 2024a) to be further studied.
The second is faciliating FoR long-range steps reasoning. LLMs fall short in long- range planning and reasoning, thus methods like MCTS (Feng et al., 2023) or an automatic reasoning system (Trinh et al., 2024) can be combined with LLMss for long- mngghorizon divergent reasoning.
24

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
FoR
SFT
799 13
799
left: 13 16
left: 9 13 2
left: 18 13
left: 79 22
left:
13 16
9 20
left: 13
left: 31
left:
left:
left:
left: 16 4
left: 13
9
22 16
13
5
22
*14
left: 20
left: 38
left: 20
left:
left: 64
left: 38
91
Figure 6: Problem (7,9,9,13). Green blocks represent the states that can achieve 24. Blue numbers represent the sample times of trajectories bigger than 1. This shows that FoR achieve 24 in multiple different states while SFT usually fails to do
25

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
Prompt BlocksWor World
am playing with of blocks where need to arrange the blocks into stacks.
Here are the actions I can do
Pick up block
Unstack block from top of another block
Put down block
Stack block on top another block
have the following restrictions
actions:
I can only pick up unstack one block at time.
I can only pick up unstack block if my hand is empty.
can only pick up a block the block on table and the black clear.
block clear the block has no other blocks on top of and if the block is not picked up. I can only unstack block from top of another block if the block
I am unstacking
really top of the other block.
I can only unstack block from top another block the block am unstacking is clear. Once I pick up or unstack a block, am holding the block.
I can only put down a block that am holding.
I can only stack block top of another block if holding the block being stacked.
I can only stack block top of another block the block onto which stacking the block is clear.
Once put down or stack block, my hand becomes empty.
[STATEMENT]
As initial conditions have that, the orange block is clear, the hand empty, the red block is top of the blue block, the orange block the blue block on the table. My goal is to have that the blue block is on top of the orange block.
block on top of
My plan is as follows:
[PLAN]
unstack the orange block from top of the red block put down the orange block unstack the red block from on top of the blue block put down the red block
pick up the blue block stack the blue block on top of the orange block [PLAN END]
[STATEMENT]
As initial conditions have that, <eurrent state>
My goal is to My goal is to have that <goals>
My plan is as follows:
[PLAN]
<action>
Figure Prompt template for the embodied reasoning task (6-step).
26

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
Prompt Game of
Use numbers and basic arithmetic operations /) to obtain 24. For each step. you are only allowed to choose two of the remaining numbers to obtain new number.
Input: 4 6
Steps:
8 = (left: 12)
6-4= (left: 12)
2 12 24 (left: 24)
Input: 299 10 12
Steps: 12 24 (left: 10 24)
-0-9= 1 (left: 24)
24 1 = 24 (left: 24)
Input: 49 10 13
Steps:
13- = 3 (left: 3 349 9)
9-3= (left: 4 6)
4 24 (left: 24)
Input: 848
Steps:
8/4=2 (left: 2 8) 1+2=3 (left: 3 8)
= 24 (left: 24)
Input: 5559
Steps:
5+ 5 10 (left: 5 10)
1 15 (left: 9 15)
9 24 (left: 24)
Input: <input>
Steps:
<action>
Figure 8: Prompt template for the mathematical puzzle task.
Prompt for Rubik's Cube
You are virtual expert in solving 2x2 Pocket Cube. Your task is restore scrambled Rubik's Cube to its original state. the given problems can be solved to 4 moves. You cannot exceed more than 11 moves. Provide the sequence of moves required for the restoration. Please follow the instructions and rules below to complete the solving: 1. A 2x2 Pocket Cube has six faces, namely: [Upper, Front, Bottom, Left, Right, Back] Each consisting of 2x2 grid of squares, with each square having its color.
Colors the Cube are represented numbers: [0, 1, 1, 3, 4,
3. You must make a move to the Cube to achieve Restored State. Note that we just need each face to have the same numbers, no matter which face has which color.
A restoration of Pocket Cube is to move squares in each face to have the numbers.
You are only allowed to use following moves [U, U'. U2, R, R', R2, F", F2]. Now strictly follow the above process to form Restoration Moves.
[STATEMENT] As initial state of the cube, I have that [Initial Cube State]:
<current state>
[Process]:
[Step 1]
[Move] <action>
Figure 9: Prompt template for the spatial Reasoning task.
27

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
Prompt for PrOntoQA
Given a list of facts, a current claim, output one possible as the next step ONLY BASED THE LAST CLAIM without using your knowledge. Be sure to copy the EXACT sentence in the facts. Do NOT change any wording. Do NOT create your own words. Give me the next step ONLY.
Facts Each lepidopteran an insect. Each arthropod is protostome. Every animal is multicellular. Protostomes are invertebrates. Each whale is bony. Each painted lady is a butterfly. Invertebrates are animals. Butterflies are lepidopterans, Each insect is six-legged. Every insect is an arthropod. Arthropods are not bony.
Query I: True or false: Sally is bony.
Claim 1.1: Sally
insect.
Next 1.1: Every insect is an arthropod.
Claim 1.2: Sally is arthropod.
Next 1.2: Arthropods are not bony.
Claim 1.3: Sally not bony.
Next 1.3: Finish.
Facts 2: Lepidopterans insects. Every animal is multicellular. Each insect is arthropod. Each invertchrate is animal. Insects six-legged, Arthropods small, Arthropods invertehrates. Each butterfly is lepidopteran. Whales are not small.
Query 2: True or false: Polly is not small.
Claim 2.1: Polly is lepidopteran.
Next 2.1: Lepidopterans are insects.
Claim 2.2: Polly
insect.
Next 2.2: Each inseet is an arthropod.
Claim 2.3: Polly is an arthropod.
Next 2.3: Arthropods are small.
Claim 2.4: Polly small.
Next 2.4: Finish.
Facts 3: <facts>
Query 3: <query>
Claim 3.1: <initial state>
Next 3.1: <action>
Figure 10: Prompt template for logical reasoning task.
Prompt ID-ARC
You provided with series of input-output pairs, where each value from 'a' to represents different color, and '.' denotes blank cell. example, a'.""""
represents a grid with rows and columns, where color "a" at position (1.0) and
color "b' is at position (2.1). Coordinates are expressed in 2D positions (row, col), with 'row" indicating the row number and 'col' indicating the column number, both using zero-based indexing. The input-output pairs may not cover all possibilities, you should infer the simplest possible relationship between them.
Your task is to reason through sequence Python functions that can transform the input grid into the output grid. Please strictly follow this process to form the appropriate Python function.
[STATEMENT]
You have the following input-output pairs:
[Initial Grid State]:
<init state>
Based on the provided list of Python functions, select the appropriate function to achieve the transformation from the input to the
output:
<python function>
frrthon_unntion>>>>>
Now, please choose one function from above list:
<action>
Figure 11: Prompt template for abstraction reasoning task.
28

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
Prompt for GSM8K
Given a question, please decompose into sub- cub-questions. For each sub qubstioesno please answer a complete sentence, ending with "The answer is". When the original question is answerable, please start the subquestion with "Now we can answer the question:
Question I: Four years ago, Kody was only half as old Mohamed. If Mohamed is currently twice as 30 years old, how old is Kody?
Question 1.1: How old is Mohamed?
Answer 1.1: He is currently 30 =2 60 years old. The answer is 60.
Question 1.2: How old was Mohamed four years ago?
Answer 1. 1.2: Four years ago, he must have been 60 4 56 years old. The answer is 56.
Question 1.3: How old was Kody four years ago?
Answer 1. 1.3: Kody was half as old as Mohamed four years ago. Thus, Kody was 56 / 2 28 years old. The answer 28.
Question 1.4: Now we can answer the question: How old is Kody?
Answer 1.4: She is currently 28 + 4 32 years old. The answer is 32.
Question 2: On moonless night, three fireflies danced in the evening breeze. They were joined by four less than dozen fireffies before two the fireflies flew away. How many fireflies remained?
Question 2.1: 1: How many fireflies joined?
Answer 2.1: The fireflies were joined by four less than a dozen more fireflies, which are 12 4 8 fireflies. The answer is 8.
Question 2.2: Now we answer the question: How many fireflies remained?
Answer 2.2: Three fireflies were dancing originally. They were joined fireflies before of them flew away. So there remaining. The answer
Question 3: <question>
Figure 12: Prompt template for the mathematical Reasoning task.
Prompt for counting different solutions
Analyze the given problem-solving approaches and identify the number of distinct methods used to arrive at the solution. Consider variations in reasoning, calculation steps, and logical structuring. Directly return the total count of unique solution approaches without additional explanations.
Solution 1: "Question 3: Claire makes a 3 egg omelet every morning for breakfast. How many dozens of eggs will she eat in weeks?
Question 3.1: How many eggs does she cat per day?
Answer 3:1: She makes egg omelet every morning for breakfast. So she cats eggs per day. The answer is 3.
Question 3.2: How many days are in week?
Answer 3.2: There are 7 days in a week. The answer is 7."
Solution 2: "Question 3: Claire makes 3 egg omelet every morning for breakfast. How many dozens of eggs will she eat in 4
weeks?
Question 3.1: How many eggs does cat a day?
Answer 3.1: She eats eggs in a day. The answer is 3.
Question 3.2: How many days are there 4 weeks?
Answer 3.2: There are 4 7 28 days in weeks. The answer is 28.
Question 3.3: How many eggs does she cat in 4 weeks?
Answer 3.3: She eats eggs in day. So eats 3 28 84 eggs 4 weeks. answer 84. Question 3.4: Now we can answer the question: How many dozens of eggs will she eat 4 weeks?
Answer 3.4: She eats 84 eges in 4 weeks. 84 eggs is 84 / 12 =7 dozens. The answer is 7."
Number:
Solution Three of the 16 eggs go Janet breakfast. Four of the remaining 13 eggs go to her muffins. That leaves eggs sell. The eggs sell for 2each, soshemakes912 =18. The answer is 18.
Solution 2: Janet lays 16 eggs a day. She eats three eggs for breakfast and bakes muffins with four this, so she has 16 3 4 eggs left. She sells these eggs at 2 each, so she make 9 2 = 18 at the market. The answer is 18.
Number:
Solution <solution1>
Solution 2: <solution2>
Number:
Figure 13: Prompt template measures the number of different solutions.
29

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
x
Goal: red block top of the blue block
Figure 14: Example of BlocksWorld for 6-step planning.
30

---

Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
4
left:
left:
24
15
10
left: 11
left: 10
X
11 66
left: 66
15
left:
X
=1111
11 00
left:
66
left:
990
X
left:
0
6
left:
24
input: 11
4 2
4-11 --
left:
2 2-0=
left:
X
24 0
24 O 24
left:
24
11 11
left: 6 11 -7
11-(-7) 18
left:
6 18 24
left:
18
24
17
17 17-(-7)
left: 4 6 22
left:
17
24
-7 17 - -19
4 6 10
22 4 18
left:
X
-119
left: 10 22
left:
18 24
6
left:
22
24
=
10 22 220
left: 12
X
left:
X
left:
X
Figure 15: Example of game 24.
How many times is the whiteboard cleaned in day?
Q2: How... day? A2: 813=24
How many lessons will be held in the classroom?
Q1: How... classroom?
A1:
How many times is the whiteboard cleaned per day?
0
How many teachers share the whitebourd?
whitaboard? A2: 4
How por day?
A3: 4'2"3-24
How many lessons do the teachers have per day?
Q1: How..per per today?
classroom has whiteboard which ...how many times the whiteboard cleaned day?
How many times does each teacher clean the board per day?
Q2: How. per day? A2: 2"3=6
Q3: How...per day? A3: 4"8=24
many times whitehoard cleaned per day?
Figure 16: Example of GSMSK.
31


Reviews:
[{"review_id": "EVdlPoMpMU", "reviewer": "Reviewer_C9xd", "summary": "The paper introduces Flow of Reasoning (FOR), a method for training Large Language Models (LLMs) to generate diverse, high-quality reasoning paths with minimal training examples. The authors formulate multi-step LLM reasoning as a Markovian flow on a DAG-structured reasoning graph, adapting Generative Flow Networks (GFlowNets) to train LLMs to sample reasoning paths with probabilities proportional to their rewards.\nThe key innovation is enabling \"divergent reasoning\" - generating multiple valid solutions to a problem rather than just maximizing rewards for a single solution path. FOR incorporates local search with destroy-and-reconstruction processes to augment training trajectories, and uses both online and offline exploration strategies including replays and ε-sampling.\nThe method demonstrates superior performance across six challenging reasoning tasks (BlocksWorld, Game24, Rubik's Cube, 1D-ARC, GSM8k, and ProntoQA), outperforming prompting-based methods and fine-tuning approaches in both accuracy and solution diversity with only ~15 training examples.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our innovative Markovian flow approach to multi-step LLM reasoning with GFlowNets, our strong performance across six benchmarks, and our method’s data efficiency.\n\n- Q1: More complex real-world tasks?\n\nA1: We evaluate FoR on six benchmarks that pose significant challenges for current LLMs (e.g., embodied reasoning in BlocksWorld and math reasoning in GSM8K). These tasks capture core difficulties encountered in real-world scenarios. For instance, the BlocksWorld task requires not only spatial reasoning but also practical world knowledge, closely mirroring real-world planning problems.\n\n- Q2 The diversity metric to capture semantic differences?\n\nA2: Thank you for your suggestion. We intentionally use human annotation for GSM8K to ensure the evaluation is accurate and reliable, despite the high cost of it. Here, we run additional experiments using a paraphrase-based method [1] to automatically evaluate the diversity with GPT-4o to measure the number of different solutions for each problem. The results on GSM8K are shown below:\n\n|Method|Acc. (%)|Div. (Human)|Div. (GPT-4o)|\n|-|-|-|-|\n|CoT (2-shot)|45.72|1.12|1.60|\n|SFT (α = 1.0)|52.69|1.13|1.63|\n|FoR|57.39|1.26|1.72|\n|FoR w/ Process Reward Model (PRM)|62.62|1.31|1.77|\n\nCompared to human annotation, GPT-4o overestimates the reasoning diversity, indicating that these automatic metrics currently do NOT provide more reliable evaluation. So, we leave this evaluation method as future work.\n\n- Q3: Reward formulations that enhance diversity?\n\nA3: Thanks for your inspiring comment. We agree that explicitly incorporating diversity into the reward formulation may further enhance solution diversity. We will discuss this in the next version.\n\n- Q4: Analysis of diversity truly benefits downstream applications\n\nA4: Thanks, we'll further highlight this in the revision. We'd like to clarify that diversity indeed benefits downstream tasks by improving accuracy, as mentioned in the Introduction (Lines 101-104). For a detailed analysis, please refer to Section 4.7 (Experiment Discussion) and Appendix F. We find that diversity promotes exploratory behaviors (e.g. in Game 24, it explores a large trajectory space), thereby enhancing the robustness of the model (Fig. 6).\n\n- Q5: Scale with larger models, and diversity gains compare to baselines?\n\nA5: Given the time and resource constraints in our research lab during the rebuttal period, we ran additional experiments to evaluate FoR's scalability and diversity gains using LLaMA-3 (8B & 70B) and Qwen2.5 (7B & 72B) on the BlocksWorld task. Results, shown below, indicate that FoR consistently improves accuracy and diversity with larger models. Compared to CoT baselines, FoR exhibits greater diversity gains as model size increases. This suggests that the gains of FoR become more pronounced as model capacity increases. Even with minimal data (15 examples), FoR yields clear improvements, highlighting its potential for robust performance across different base models. The results are shown below and will be included in the revision.\n\n|Model|4-Step Acc. (%)|4-Step Div.|6-Step Acc. (%)|6-Step Div.|\n|-|-|-|-|-|\n|CoT 5-shot (Llama3-8B)|28.57|1.05|15.82|1.05|\n|CoT 5-shot (Llama3-70B)|45.23|1.05|46.46|1.11|\n|FoR (Llama3-8B)|98.41|1.27|78.44|1.33|\n|FoR (Llama3-70B)|100.00|1.38|87.65|1.40|\n|FoR (Qwen2.5-7B)|100.00|1.24|86.86|1.36|\n|FoR (Qwen2.5-72B)|100.00|1.41|90.13|1.46|\n|FoR (InternLM2.5-7B-Chat)|100.00|1.26|83.83|1.31|\n\n- Q6: Iterative reasoning refinement based on feedback?\n\nA6: Yes, FoR supports iterative refinement. During both training and inference, by incorporating feedback—from the model itself or an additional model—into the state, subsequent state predictions refine solutions iteratively. This aligns with R1/O1's approach, where each step either refines the current state or advances the reasoning. For example, in BlocksWorld, given a current block position, the model predicts an action like 'move blue onto yellow,' and then generates feedback on whether to proceed or refine. Based on the state including this feedback and block positions, the model predicts the next state, iteratively refining its actions or proceeding. While we haven't tested this, iterative refinement based on feedback is a promising future direction. \n\n- Q7: A meta-learning approach to reduce the need for task-specific reward engineering?\n\nA7: While we haven't directly explored meta-learning approaches, we ran an additional experiment on an existing process reward model, Qwen2.5-Math-PRM-7B, for the GSM8K task. Table in A2 indicates that using this reward model with FoR improves performance over the previous 0/1 outcome reward function, achieving a 9% relative increase in accuracy and a 4% relative gain in diversity. This shows a promising step towards reducing the need for task-specific reward engineering.\n\nReferences: \n\n[1] Michail et al. (2025) \"PARAPHRASUS: A Comprehensive Benchmark for Evaluating Paraphrase Detection Models.\" COLING 2025.", "claims_and_evidence": "The claims are well-supported by comprehensive experiments:\n\n1. Performance comparisons with numerous baselines (CoT, ToT, RAP, SFT, PPO, GFN-CoT)\n2. Detailed ablation studies demonstrating the contribution of each component\n3. Quantitative metrics for accuracy, diversity, and creativity\n4. Analysis of data efficiency showing FOR's effectiveness with limited examples\n5. Well-designed case studies illustrating how FOR discovers multiple correct solutions\n\nThe methodology is sound and the empirical results convincingly demonstrate FOR's advantages.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are appropriate for the problem:\n\nThe six reasoning tasks cover diverse domains (embodied, mathematical, spatial, abstraction, etc.)\nThe metrics (accuracy, diversity, creativity) directly address the paper's goal of divergent reasoning\nThe baselines represent state-of-the-art methods in both prompting and fine-tuning approaches\nThe ablation studies isolate the contributions of individual components\n\nThe reward designs for each task are thoughtfully crafted to balance accuracy and exploration.", "theoretical_claims": "There doesn't seems to has concrete theoretical claim", "experimental_designs_or_analyses": "Report performance across multiple runs with standard deviations\nDefine clear metrics that directly measure their objectives\nInclude OOD testing to demonstrate generalization capabilities\nProvide detailed implementation specifications in the supplementary material", "relation_to_broader_scientific_literature": "The paper builds on three critical areas:\n\nLLM reasoning: Extends CoT/ToT approaches by enabling diverse reasoning paths\nGFlowNets: Adapts them from molecule generation to structured reasoning\nDiverse sampling methods: Provides a principled alternative to beam search\n\nUnlike previous applications of GFlowNets with LLMs, FOR implements higher-level modeling at the reasoning step granularity rather than token level, which is key to its success in reasoning tasks.", "essential_references_not_discussed": "The paper generally covers relevant literature thoroughly.", "other_strengths_and_weaknesses": "Strengths:\n\nNovel formulation of LLM reasoning as a Markovian flow problem\nData efficiency (works with ~15 examples)\nComprehensive evaluation across diverse reasoning tasks\nClear ablation studies demonstrating component contributions\nPractical runtime comparable to efficient baseline methods\n\nWeaknesses:\n\nThe paper doesn't address integration with larger models or more complex real-world tasks\nThe trajectory diversity metric could be more sophisticated to better capture semantic differences\nLimited exploration of other reward formulations that might further enhance diversity\nNo analysis of whether diversity truly benefits downstream applications", "other_comments_or_suggestions": "Nothing critical to comment", "questions_for_authors": "How would the method perform with even larger models (70B+)?\nWould the gains in diversity increase or decrease relative to baselines?\nCould the approach be extended to iterative reasoning refinement settings where solutions are revised based on feedback?\nHave you explored whether a meta-learning approach could reduce the need for task-specific reward engineering?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mD54cf8rGk", "reviewer": "Reviewer_KgwZ", "summary": "The paper introduces a novel fine-tuning method called Flow of Reasoning (FOR) that trains large language models (LLMs) to generate diverse, high-quality multi-step reasoning paths using minimal training examples. The key idea is to formulate the reasoning process as a Markovian flow on a directed acyclic graph (DAG) and to leverage GFlowNet-inspired training objectives. By assigning probabilities to entire reasoning trajectories proportional to an (unnormalized) reward, FOR encourages the discovery of multiple valid solution paths. Extensive experiments on six reasoning tasks demonstrate that FOR not only improves overall accuracy but also significantly enhances the diversity and creativity of the generated solutions compared to standard methods such as supervised fine-tuning (SFT), reward-maximizing reinforcement learning , and various prompting-based approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for recognizing our strong performance across 6 benchmarks, ablation studies, data efficiency, and case studies.\n\n- Q1: Hyperparameters selection and analysis? Guidelines for designing rewards? Handcrafted rewards?\n\nA1: Please refer to Section 4.7 and Figure 3 for the hyperparameter analysis on a small training set (i.e., 15). We search for the best-performing λ values on each task's training set.\n\nWe further assess how the intermediate reward weight λ affects test set performance. Results for varying λ values are shown below:\n\n|λ|0||0.5||1||1.5||2||\n|-|-|-|-|-|-|-|-|-|-|-|\n|FoR|Acc.(%)|Div.|Acc.(%)|Div.|Acc.(%)|Div.|Acc.(%)|Div.|Acc.(%)|Div.|\n|4-step|90.5|1.2|92.9|1.2|97.6|1.2|98.5|1.3|95.2|1.3|\n|6-step|47.1|1.2|71.7|1.3|75.8|1.3|78.4|1.3|76.8|1.3|\n\nThe results show that performance consistently improves as λ increases up to 1.5, but drops a bit at λ= 2 across 4- to 6-step settings. The influence of λ becomes more pronounced with more steps.\n\nFor the guidelines, we use the generalizable common principles: 1. For tasks with likely correct outputs (e.g., GSM8K), a binary success reward suffices. 2. For challenging reasoning tasks (e.g., BlocksWorld, Game24) with sparse rewards, intermediate rewards from prior works (Hao et al., 2023) (e.g., LLM log-likelihood, rule-based) help.\n\nRegarding handcrafted rewards, in GSM8K, we simply use a standard outcome reward (0/1). Additionally, we use the reward model Qwen2.5-Math-PRM-7B, achieving 62.62% accuracy and 1.31 diversity (please see A2 to reviewer c9xd due to page limit). These two reward designs show that FoR does not rely on sophisticated reward design, supporting general applicability.\n\nWe will include all these in the revision.\n\n- Q2: Varying amounts of training data.\n\nA2: We ran additional experiments on the 6-step BlocksWorld task using Llama-3-8B with varying training sizes {1, 15, 30, 45, 60}, following the same setup as Section 4.7. The results are shown below:\n\n|Method|1||15||30||45||60||\n|-|-|-|-|-|-|-|-|-|-|-|\n||Acc.(%)|Div.|Acc.(%)|Div.|Acc.(%)|Div.|Acc.(%)|Div.|Acc.(%)|Div.|\n|SFT|15.0|1.0|40.0|1.0|50.0|1.0|60.0|1.1|70.0|1.0|\n|FoR|45.0|1.3|80.0|1.3|85.0|1.3|90.0|1.3|90.0|1.4|\n\nFoR consistently outperforms SFT across all data sizes—for example, with just one training example, FoR achieves a relative 200% higher accuracy and a relative 30% higher diversity. This highlights FoR’s data efficiency even in low-resource settings.\n\n- Q3: Theoretical derivations.\n\nA3: Please refer to Appendix B for background information on GFlowNets.\n\nFor the trajectory balance (TB) constraint, please see Section 3.2. Below is a short derivation of the TB constraint:\n\nLet $P_F(τ) = \\prod_{t=1}^{n} P_F(s_t \\mid s_{t-1})$ be the forward trajectory distribution, and $P_B(τ) = \\prod_{t=1}^{n} P_B(s_{t-1} \\mid s_t)$ be the backward trajectory distribution.\n\nThe **TB constraint** requires: $P_F(τ) \\cdot Z(s_0) = P_B(τ) \\cdot R(s_n)$. \n\nThis constraint aligns the flow allocated to $τ$ under the forward policy with reward-scaled backward probability, enabling proper credit assignment.\n\nWe will add these in the revision.\n\n- Q4: Scalability and open-ended tasks.\n\nA4: For larger models, we run additional experiments with LLaMA-3-70B and Qwen2.5-72B, and FoR consistently achieves better accuracy and diversity (see A5 to reviewer c9xd due to page limit). On the open-ended GSM8K benchmark, FoR also achieves stronger performance, highlighting its scalability.\n\n- Q5: Limitations and potential failure modes.\n\nA5: Please see the Limitations in Appendix H and will move it to the main text. For potential failure modes, FoR, like other on-policy RL methods, may struggle in sparse-reward settings. However, FoR mitigates this by the augmented intermediate rewards (Sec. 4.7), which provide dense guidance, and by leveraging off-policy data (Sec. 4.3 Game24). \n\n- Q6: More reasoning tasks, such as Math and Code.\n\nA6: We’d like to clarify that FoR has already been evaluated on GSM8K, a math benchmark (Sec. 4.6), as well as on ARC-1D (Sec. 4.5), which involves Python program synthesis.\n\n- Q7: Optimizations to be computationally efficient?\n\nA7: We'd like to clarify that although FoR’s training is slower, it is data-efficient (Section 4.7), requiring only ~15 training examples—keeping the overall training cost low. Two potential further optimization directions are: 1. Off-Policy Data: Parallel off-policy training can reduce the computational cost (Appendix H). 2. Intermediate rewards: accurate reward functions (e.g., robust reward models) enable faster convergence.\n\n- Q8: A large number of reasoning paths? Scaling limitations?\n\nA8: FoR performs well with large reasoning spaces. For example, the Game24 task involves ~8,000 distinct reasoning trajectories per sample, showing that FoR could scale to complex reasoning tasks.\n\n- Q9: Other techniques like verifiers?\n\nA9: We use rule-based rewards as verifiers in Rubik’s Cube and 1D-ARC (Sec. 4.4&4.5). Your suggestion is also a promising direction.", "claims_and_evidence": "The claims made in the submission are well-supported by empirical evidence:\n\n1. The superiority of FOR over baselines is demonstrated across six diverse reasoning tasks with consistent improvements in accuracy, diversity, and creativity metrics (Tables 1-6).\n\n2. The ablation studies (Table 5) convincingly show the contribution of each component (local search, augmented rewards, replay buffer, ε-sampling).\n\n3. Data efficiency is well-demonstrated through comparisons with SFT trained on varying amounts of data (Figure 3).\n\n4. The case studies in Figures 5-6 provide qualitative evidence of FOR's ability to discover diverse solution paths.", "methods_and_evaluation_criteria": "The proposed method’s formulation of multi-step reasoning as a Markovian flow is both novel and well-motivated. Using the trajectory balance constraint to link the flow to rewards is theoretically sound and aligns with recent advances in GFlowNets. The evaluation criteria—accuracy, diversity (semantic differences among correct solutions), creativity (unique solutions), and runtime efficiency—are appropriate for the task at hand and provide a comprehensive picture of the method’s strengths and limitations.", "theoretical_claims": "The paper primarily presents a conceptual framework rather than making formal theoretical claims requiring proofs. The authors adapt existing GFlowNet theory to the domain of reasoning steps in a sound manner.", "experimental_designs_or_analyses": "The experimental designs are robust:\n\n* Task Diversity: Evaluations span across several reasoning domains (embodied, mathematical, spatial, abstraction), which reinforces the generality of the approach.\n* Baselines and Metrics: The comparisons with both prompting-based methods and various fine-tuning strategies, along with clearly defined evaluation metrics, add rigor to the analysis.\n\nOne point for further improvement would be a deeper analysis of how hyperparameters (especially in reward design) affect the outcomes. Additionally, exploring the method’s performance with varying amounts of training data could further highlight its data efficiency.", "relation_to_broader_scientific_literature": "The key contributions of the paper are related to the current research LRMs, which could potentially be an approach to refine the thinking and exploration process of the current reasoning trajectory.", "essential_references_not_discussed": "To my knowledge, there are no essential references not discussed.", "other_strengths_and_weaknesses": "Strengths:\n\n* Novel Formulation: The flow-based perspective and the integration of GFlowNet techniques are both innovative and promising.\n* Comprehensive Evaluation: The extensive experimental validation across multiple reasoning tasks is a significant strength.\n* Data Efficiency: The method’s ability to work with minimal training data is particularly compelling.\n\nWeaknesses:\n\n* Complexity in Derivations: Some of the theoretical derivations might be challenging for readers not already familiar with GFlowNets.\n* Hyperparameter Sensitivity: The reliance on handcrafted reward designs and specific hyperparameters might limit the method’s generalizability without further analysis.\n* Scalability: While the approach works well on the tasks presented, its performance when scaling to larger models or more open-ended tasks is not fully explored.", "other_comments_or_suggestions": "Suggestions:\n* Consider including a more detailed discussion on the sensitivity of the method to different hyperparameter settings, especially in the reward design.\n* A discussion on potential computational overhead or scalability challenges when applying the method to larger models or datasets would strengthen the paper.\n* Clarifying the limitations and potential failure modes of the method could provide a balanced view of its applicability.\n* Try to generalize the current FoR to wider range of challenging reasoning tasks, such as Math and Code.", "questions_for_authors": "Question:\n\n1. How sensitive is FOR to the reward design? Could you provide guidelines for designing rewards for new tasks to help understand the generalizability of your approach?\n\n2. The training cost of FOR is significantly higher than SFT (Table 7). Are there optimizations that could make the approach more computationally efficient without sacrificing performance?\n\n3. How does FOR perform when the number of possible reasoning paths grows very large? Are there scaling limitations when applying this to more complex reasoning tasks?\n\n4. Have you explored combining FOR with other techniques like verifiers or self-consistency approaches? This might address some of the limitations for longer-range reasoning.\n\n5. How did you determine the reward weights (e.g., λ values) for each task? Is there a systematic approach to tuning these hyperparameters?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L9q6Z4CZHa", "reviewer": "Reviewer_PdTd", "summary": "This paper introduces \"Flow of Reasoning\" (FOR), a novel, data-efficient method for finetuning large language models (LLMs) to achieve divergent reasoning, i.e., generating multiple, diverse, and valid solutions to multi-step reasoning problems. FOR formulates LLM reasoning as a Markovian flow on a directed acyclic graph and adapts principles from Generative Flow Networks (GFlowNets) to train the LLM to sample reasoning paths with probabilities proportional to the problem's reward. The authors show, across six challenging reasoning benchmarks (including BlocksWorld, Game24, Rubik's Cube, 1D-ARC, GSM8k, and ProntoQA), that FOR, with minimal training examples (around 15), significantly outperforms various baselines including supervised finetuning, reward-maximization reinforcement learning, and prompting-based methods, in terms of both solution accuracy and diversity. The method leverages efficient exploration techniques including local search and a prioritized replay buffer to improve the training process.\n\n\n## update after rebuttal\nAs discussed during the rebuttal period, I think some minor issues still exist, and the authors need to add some details to the paper. I would like to maintain my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments and suggestions.\n- Q1. How representative are the benchmarks? Are there reasoning types where it fails?\n\nA1: Some tasks like GSM8K and Game24, are popular LLM benchmarks, while others, like BlocksWorld and 1D-ARC, are known to be challenging for LLMs. FoR remains a general framework for multi-step reasoning: whenever a problem can be decomposed into intermediate states and has a well-defined reward, we expect FoR to be applicable.\n\n- Q2: Tradeoff between amortized inference vs. slower inference with faster training. When training cost of FoR is justified by its better inference performance? \n\nA2: Unlike methods like ToT/RAP without diversity, FoR enables efficient amortized inference with diversity. Though training time is long, it eliminates abundant inference costs in ToT/RAP, keeping the overall cost low when the number of tasks needed to be solved is numerous. Once trained, it avoids repeated, expensive searches. In such cases, the initial training cost is largely offset by the benefits of efficient inference. Compared to SFT/PPO, FoR has higher training costs but is highly data-efficient, needing only ~15 examples, reducing data collection costs. \n\n- Q3: Diversity metric (human annotation & automatic)\n\nA3: Following your advice, we use a paraphrase-based method [1] with GPT-4o to automatically evaluate solution diversity on GSM8K. Please see detailed results in A2 to reviewer C9xd due to page limit. The results show that FoR’s diversity is still higher than baselines by using GPT4o, although it consistently overestimates.\n\n- Q4: Evaluation of diversity for search-based methods and O1-series.\n\nA4: We ran ToT-DFS multiple times on the Blocksworld task with the same setup as in the paper. The results are shown below:\n\n||2-step|4-step||6-step||\n|-|-|-|-|-|-|\n|Method|Acc.(%)|Acc.(%)|Diversity|Acc.(%)|Diversity|\n|ToT-DFS|40.0|42.9|1.0|31.3|1.1|\n|FoR|100|98.4|1.3|78.4|1.3|\n\nSearch methods (ToT/RAP) show limited diversity despite multiple runs. Table 1 in the paper shows ToT/RAP needs 4-40× FoR's inference time per sample, making equal-sample comparison time-unfair. We'd like to clarify that Table 1 shows that O1-mini achieves decent accuracy but limited diversity across multiple runs.\n\n- Q5: Replay buffer is more important without local search? \n\nA5: We add an additional ablation study by removing the local search to assess the impact of the other components in FoR. See the results below:\n\n|Method|4-step Acc.(%)|4-step Div.|6-step Acc.(%)|6-step Div.|\n|-|-|-|-|-|\n|FoR w/o local search|89.7|1.2|53.9|1.3|\n|w/o replaybuffer|78.6|1.1|34.3|1.2|\n|w/o ϵ-sampling|83.3|1.1|49.5|1.1|\n|w/o augmented reward|71.4|1.0|30.3|1.2|\n\nTable 5 in the paper and the above results show that removing the replay buffer causes a smaller performance drop in FoR with local search (4-6%) than without it (11-19%), indicating that the replay buffer is more critical when the local search is removed. \n\n- Q6: While applying GFlowNets to LLM reasoning is novel, core techniques are adapted from existing GFlowNet literature.\n\nA6: Thanks for recognizing our novel formulation, which allows us to adapt existing approaches to the new multi-step LLM reasoning domain. This ability to reuse and generalize, rather than reinvent everything from scratch, is a key advantage of our work.\n\n- Q7: Related works.\n\nA7: We'll add more discussion on Takase et al. (2024) in the revision. Their work generates diverse math solutions via token-level diversity, similar to Hu et al. (2024) (mentioned in our Introduction, Lines 95–96). Oh et al. (2024) [2] follow these lines and apply GFlowNets to LLM preference optimization. In contrast, FoR targets broader reasoning tasks beyond math or preference learning, modeling thought process structures with GFlowNets.\n\n- Q8: Details on training data sampling and replay buffer update mechanism.\n\nA8: We used random sampling from the full training set. For the replay buffer, we set the capacity to store 50 trajectories, new trajectories will replace the lowest-reward ones when the buffer is full. These details will be added to Appendix D.\n\n- Q9: More methods (e.g. efficient exploration) to reduce the training cost.\n\nA9: We use (1) local search and (2) ϵ-sampling (Sec. 4.7) to encourage efficient exploration. In Game24, we use offline data to reduce model's exploration costs. Other methods to accelerate training are a promising future direction.\n\n- Q10: Different base LLMs.\n\nA10: We test LLama-3-8B, Qwen-2.5-7B, and InternLM2.5-7B-Chat on Blocksworld (one task due to rebuttal constraints), and FoR consistently outperforms baselines (see A5 to reviewer c9xd due to page limit).\n\nReferences:\n\n[1] Michail et al. \"PARAPHRASUS: A Comprehensive Benchmark for Evaluating Paraphrase Detection Models.\" COLING 2025.\n\n[2] Oh Joon Kwon, et al. GDPO: Learning to Directly Align Language Models with Diversity Using GFlowNets. EMNLP 2024", "claims_and_evidence": "1. The paper focuses on 6 main benchmark tasks. It also performs some testing of OOD transfer from smaller to large problems. While testing these benchmarks, how representative are they of all multi-step reasoning problems? There is limited discussion on the types of reasoning these benchmarks don't cover. For example, are there types of reasoning (e.g. causal, counterfactual) for which this approach is expected not to be a good choice?\n2. Computational Cost of GFlowNets vs. Alternatives: The introduction and related work sections highlight the potential computational cost of search-based inference methods (ToT, RAP). While FOR amortizes inference cost into training, the paper acknowledges in Appendix C.3 that FOR's training time is significantly higher than SFT and even PPO (Table 7: FOR 6833s, SFT 196s, PPO 1740s). This crucial point needs to be discussed more prominently in the main text, not just the appendix. The trade-off between inference speed and training cost should be explicitly addressed. The claim of efficiency is somewhat misleading without this context. The paper argues for amortized inference, but a user concerned with overall computational cost (including training) might still prefer a slower inference method with much faster training.", "methods_and_evaluation_criteria": "1. The paper uses a diversity metric based on manually annotating 50 test examples to evaluate semantic differences between solutions in GSM8K. While this is a reasonable approach, manual annotation can be subjective, and 50 examples might be insufficient for a robust evaluation of diversity in a task as open-ended as mathematical reasoning. The paper should acknowledge the limitations of this approach and, if possible, explore alternative or supplementary diversity metrics that are less reliant on manual annotation, or increase sample size. For instance, using different paraphrases, could be an automated way to detect different solution paths.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. The paper mentions that search-based methods (ToT, RAP) and O1-series are evaluated with limited runs due to time and budget constraints. This is a significant limitation, as it can lead to an inaccurate assessment of their performance, especially regarding diversity and creativity, which are inherently stochastic. The paper should either conduct more runs (ideally) or clearly acknowledge the potential for underestimating these baselines' capabilities.\n2. The ablation study analyzes the impact of removing each component individually. It would be helpful to also discuss potential interactions between these components. For example, is the replay buffer more important when local search is not used? A more comprehensive ablation study, though potentially computationally expensive, would explore removing combinations of components.", "relation_to_broader_scientific_literature": "The paper leverages GFlowNets, extending their application beyond established domains like molecule generation to multi-step reasoning, similar to but distinct from concurrent work like GFN-CoT by focusing on reasoning steps rather than token-level generation. The work incorporates ideas from reinforcement learning exploration, adapting methods like local search and prioritized replay buffers to the GFlowNet framework for LLMs. Finally, it positions itself within the broader context of LLM prompting and fine-tuning, presenting an alternative approach that attempts to address data requirements of supervised methods and the lack of diversity focus in reward-maximization reinforcement learning.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "1. While applying GFlowNets to LLM reasoning is novel in this specific formulation, the core techniques (trajectory balance, local search, replay buffer) are adapted from existing GFlowNet literature.\n2. The paper acknowledges Takase et al. (2024) as concurrent work but states it is limited to math problems. A more thorough comparison, even if brief, would be valuable. Are there other concurrent works on applying GFlowNets or similar diversity-seeking methods to LLMs that should be acknowledged and differentiated?\n3. The algorithm description could be more precise. For example, the \"Sample from training dataset\" step needs more detail. How are examples selected? Randomly? With specific criteria? The update of the replay buffer D also needs clarification. Are all sampled trajectories added, or only those exceeding a certain reward threshold?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. The paper acknowledges in the appendix that FOR's training time is significantly higher than SFT and PPO. Could you discuss this trade-off between inference speed and training cost more prominently in the main text? In what scenarios would the increased training cost of FOR be justified by the improved inference performance?\n2. Have you explored any methods to reduce the training cost of FOR, such as more efficient exploration strategies or approximations of the GFlowNet objectives?\n3. How does the performance of FOR change with different base LLMs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors introduce Flow of Reasoning (FOR), a method for training LLMs to generate diverse, high-quality reasoning paths using minimal training examples by modeling multi-step reasoning as a Markovian flow on a DAG-structured graph, leveraging Generative Flow Networks (GFlowNets). It claims superior performance in accuracy and solution diversity across six reasoning tasks compared to baselines like CoT, ToT, and SFT. Strengths include its novel formulation, data efficiency, and comprehensive evaluation across diverse tasks with clear ablation studies. Weaknesses include limited exploration of scalability to larger models or complex real-world tasks, a basic diversity metric that may not fully capture semantic differences, and high training costs (e.g., 6833s vs. 196s for SFT) not prominently discussed in the main text. The submission could benefit from deeper analysis of hyperparameter sensitivity and applicability to other reasoning types like causal or counterfactual.

During the rebuttal, reviewers raised concerns about training cost trade-offs, diversity metric robustness, scalability, and benchmark representativeness. The authors provided additional experiments, including scalability tests with larger models (LLaMA-3-70B, Qwen2.5-72B) showing consistent gains, a paraphrase-based diversity evaluation with GPT-4o (though less reliable than human annotation), and hyperparameter analysis for reward weights. They clarified FOR’s applicability to math and code tasks and detailed replay buffer mechanics. Reviewer C9xd maintained a strong accept (score 4), appreciating the thorough responses. Reviewer KgwZ, initially a weak accept (score 3), upheld their score, satisfied with most clarifications. Reviewer PdTd, also weak accept (score 3), noted minor unresolved issues like training cost discussion but did not change their score. For the acceptance recommendation, I weighed the robust empirical results and novelty heavily, with the authors’ rebuttal addressing scalability and diversity concerns sufficiently. The training cost issue, while valid, is mitigated by data efficiency and amortized inference benefits. This would be a solid contribution to ICML
Decision: Accept (poster)