Flow of Reasoning: Training LLMs for Divergent Problem Solving with Minimal Examples

Authors: Fangxu Yu, Lai Jiang, Haoqiang Kang, Shibo Hao, Lianhui Qin
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Large Language Models, reasoning, diversity, multi-step reasoning, planning, creativity

Abstract:
The ability to generate diverse solutions to a given problem is a hallmark of human creativity. This divergent reasoning is also crucial for machines, enhancing their robustness and enabling them to assist humans in many applications such as scientific discovery. However, existing approaches to multi-step reasoning with large language models (LLMs) have mostly focused only on the reasoning accuracy, without further discovering more diverse valid solutions. For example, supervised fine-tuning can improve LLM reasoning quality, but requires extensive supervised data to capture the full range of possible solutions. Reinforcement learning aims to find limited highest-reward solutions while neglecting the solution diversity. To fill this gap, we propose Flow of Reasoning (FoR), an efficient diversity-seeking LLM finetuning method aimed at improving reasoning quality and diversity with minimal data. FoR formulates multi-step LLM reasoning as a Markovian flow on a DAG-structured reasoning graph. This formulation allows us to incorporate and adapt principled GFlowNet approaches, for finetuning LLMs to sample diverse reasoning paths with probabilities proportional to the (unnormalized) reward of target problems. Extensive experiments show that, with limited training examples (e.g., 15 examples), FoR enables the discovery of diverse, creative, high-quality solutions, greatly outperforming a wide range of existing inference and training methods across five challenging puzzle-solving tasks, including BlocksWorld (embodied reasoning), Game24 (math puzzle solving), PrOntoQA (logical reasoning), Rubik's Cube (spatial reasoning), and 1D-ARC (abstraction reasoning).

Decision: reject
Reviewer scores: 8, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730472404328

thread_type:
official_review

<thread_object>
creator: reviewer_iyfq

<content>
summary:
The paper introduces FOR, a novel method designed to enhance the divergent reasoning capabilities of LLMs. FOR addresses the challenge of generating diverse solutions to complex problems by formulating multi-step reasoning as a Markovian flow on a DAG structured reasoning graph. This approach allows the incorporation of GFlowNet techniques to train LLMs to sample diverse and high-quality reasoning paths proportional to the unnormalized reward of target problems. The paper demonstrates FOR's effectiveness across five diverse reasoning tasks, showing significant improvements in discovering creative and high-quality solutions with minimal training data compared to existing methods.

strengths:
- well-written and easy to follow
- The idea of GFlowNet is quite interesting

weaknesses:
I believe this is an interesting piece of work with a highly valuable topic. However, I think the benchmarks considered in this paper, such as Game24, PrOntoQA, Rubik's Cube, etc., do not actually require much divergent thinking. I strongly recommend directly considering some works on lateral thinking [1][2][3] to assess whether the proposed methods are truly effective. I understand that it might not be possible to fully attempt these benchmarks during the rebuttal stage, but it could be possible to try some examples or add a related works section for lateral thinking or creativity.



[1] Let’s Think Outside the Box: Exploring Leap-of-Thought in LLMs with Creative Humor Generation, CVPR24

[2] Lateral Thinking Puzzles for Large Language Models, EMNLP23

[3] RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge, ACL21

questions:
see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730682795380

thread_type:
official_review

<thread_object>
creator: reviewer_h3sb

<content>
summary:
This paper proposes Flow of Reasoning (FOR), a data-efficient finetuning method that enables large language models (LLMs) to generate diverse, high-quality solutions for multi-step reasoning tasks. Unlike existing methods that prioritize accuracy or highest-reward solutions, FOR promotes solution diversity by modeling reasoning as a Markovian flow on a directed acyclic graph (DAG), applying principles from Generative Flow Networks (GFlowNets) to sample multiple reasoning paths based on reward. With minimal data requirements (around 15 examples), FOR significantly outperforms traditional approaches across five complex puzzle-solving tasks, demonstrating its effectiveness in improving both reasoning quality and diversity.

strengths:
The paper effectively addresses a gap in existing LLM approaches by focusing on solution diversity in multi-step reasoning tasks, which enhances robustness and creativity in applications like scientific discovery. 

The FOR framework introduces a unique approach by modeling multi-step reasoning as a Markovian flow on a directed acyclic graph (DAG), using Generative Flow Networks (GFlowNets) principles. This higher-level, step-based approach enables more efficient sampling of diverse reasoning paths, distinguishing FOR from traditional token-level models.

FOR demonstrates impressive data efficiency, requiring only about 15 training examples to achieve competitive results. This efficiency makes the method more feasible for practical applications where extensive labeled data is limited.

Through comprehensive testing across five complex tasks (embodied reasoning, logical reasoning, spatial reasoning, etc.), FOR outperforms multiple baselines by 20%–85%, showing versatility and robustness across different reasoning challenges.

weaknesses:
The primary issue I find with this paper is the lack of clarity regarding any original insights or core observations the authors provide about large language models in the context of reasoning with diversity. Throughout the paper, it seems that all key observations, points, and techniques are primarily derived from existing works, without presenting novel contributions from the authors. For example, the authors note that existing LLMs focus on generating the reasoning path with the highest reward. However, LLMs inherently exhibit randomness and uncertainty in their reasoning processes, which could naturally encourage exploration across diverse reasoning paths. In additional papers, such as TR [1] and CR [2], they all build a DAG toward getting solutions. 

The second issue is that there are plenty of high-level ideas and "big words" in the explanation parts of the paper. For instance, in Line 172, what does "As discussed in §1, to generate diverse, high-quality reasoning trajectories for solving a task, we want to sample the trajectories with probabilities proportional to the reward. " Why sampling in such a way can generate not only diverse but also high-quality reasoning paths? What is the logic between this strong conclusion and its previous and subsequent sentences? 

The third issue is that this paper introduces too many existing methods, ideas, equations, and conclusions from prior works. However, the way the authors organize these elements does not integrate them effectively, resulting in a lack of clear and logical explanations to support an understanding of their own contribution. While reading the paper, I found the frequent citations, numerous "See appendix" references, and lack of important insights to be distracting and disruptive to my understanding. For example, the authors do not provide clear or consistent notation descriptions. It appears that notations are created on the spot whenever needed, leading to confusion and a lack of coherence throughout the paper.

The fourth issue is that some of the questions posed in the paper are unclear, leading to confusion about their intent and relevance. For example, (1), in Line 188, why the distribution directly equals to F(\tau)/Z? To me, this should be an approximation.; (2), in Line 191, where is the P(s_0) that is the prior in this step-wise distribution? (3). in Line 202, when F is defined for the state flow, it's unclear why F(s_n) is to be equal to R. 

The final issue is that the experiment section does not provide sufficient persuasiveness for the paper. For example, if the goal of this paper is to fine-tune the LLM to generate diverse reasoning paths, the main content should provide a more specific and detailed explanation of how to fine-tune LLMs according to the proposed objectives. Besides, how the authors define diversity is unclear to me. They may need to count successful trajectories a policy finds for the successful example. But how the two trajectories can be compared when there are many ones? Thus, during training, how can you ensure that the LLM can generate two different paths from one state? How about comparing your methods with other similar (using DAG) methods [1], [2]? 

I was not impressed by the optimal results in the table due to the lack of sufficient explanation, in-depth analysis, and insights in the paper.

[1]. Chen, et.al., Toward Adaptive Reasoning in Large Language Models with Thought Rollback, ICML24. 

[2]. Zhang, Cumulative reasoning with large language models, ICLR Workshop24.

questions:
See Weaknesses Above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684784163

thread_type:
official_review

<thread_object>
creator: reviewer_3ete

<content>
summary:
Paper proposed a new approach for generating diverse reasoning traces using GFlowNets.

strengths:
- Paper is organized cleanly and easy to understanding
- Experiments involve a variety of tasks and baselines
- Approach shows improvements to accuracy and diversity of model generations

weaknesses:
Although a variety of tasks were evaluated, all of the tasks are relatively structured and uses constrained action spaced (such as STACK, UNSTACK, PUT, PICKUP in BlocksWorld). In contrast, reasoning tasks that are useful to real users are typically much more open ended, with a more diverse variety of both questions and potential answer formats. In order to show that the proposed approach is scalable to these more open ended reasoning tasks, it would be great to also evaluate on benchmarks like MATH or GSM8k.

The proposed approach requires a reward function, which, in the experiments, required different kinds of reward shaping for each task. Designing specialized reward functions for each task is not practical when training general-purpose LLMs on a wide range of different tasks. It would be great to either show that the proposed approach leads to improvements in accuracy and diversity with just the sparse "success" reward, or use a shaped reward that can work with all tasks. 

It would be great to show qualitative examples of the different responses generated by each approach, and highlight how they differ from standard approaches

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697267434

thread_type:
official_review

<thread_object>
creator: reviewer_waxe

<content>
summary:
The paper presents Flow of Reasoning (FoR), a novel fine-tuning approach for LLMs that aims to enhance solution diversity. FoR frames the reasoning process as a Markovian flow, sampling diverse reasoning paths with probabilities aligned to the reward associated with each target problem similar to GFlowNet. Extensive experiments across multiple domains and baselines demonstrate that the proposed approach helps LLMs discover more diverse and higher-quality answers, showcasing the effectiveness of the algorithm.

strengths:
1. The paper addresses the important challenge of encouraging diverse reasoning paths, a key problem in advancing LLMs’ capability for creative and robust problem-solving.
2. The experimental setup is solid, with evaluations across a wide range of domains and baseline models. Additionally, the comprehensive ablation studies effectively demonstrate the effectiveness of the design choices.
3. Strong performance is demonstrated, with the proposed approach outperforming state-of-the-art methods, including both prompt-based and fine-tuned models. In some tasks, the method even achieves results comparable to the advanced o1 reasoning model, despite using a significantly smaller Llama 3 model.

weaknesses:
### The Presentation of the Paper Can Be Improved

- The motivation of the method is not clear enough. While the authors emphasize formulating the problem as a Markovian flow and sampling reasoning paths proportionally to rewards by drawing from GFlowNets, it is not clear why these specific elements are essential or expected to be effective in addressing the problem of diverse reasoning paths.
- The authors could consider presenting the motivation and broader rationale behind key design choices before delving into technical and mathematical specifics. For instance, Section 3.2 is challenging to follow without a detailed explanation of why and how choices like the trajectory balance approach, log-variance approximation, or local search are adopted. It would also be helpful to clarify the reasoning behind Equations (5) and (7) and the role of local search within the approach.
- While the empirical results are impressive, the paper would benefit from additional analysis to strengthen its scientific arguments. For example, a case study illustrating the diverse reasoning paths generated by FoR, which other methods fail to find, would be informative. Additionally, a discussion on why FoR achieves such high performance while focusing on diversity would be valuable, especially considering that in most experiments, it outperforms baselines that emphasize mostly reasoning path quality. It would also be interesting to analyze how diverse reasoning paths may contribute to FoR’s effectiveness in out-of-distribution tasks.

### Limitations of the Markov Assumption and Markovian Flow Setup

The core setup of the algorithm relies on Markov assumptions and a Markovian flow structure, which are suitable only for domains that can accommodate state/action flows under these constraints. However, many complex real-world reasoning tasks do not necessarily align with the Markov assumption, having significant partial observability challenges or making it challenging to define clear “state” and “action” flows. More discussions of these limitations would help readers better understand the specific types of domains where the proposed method is most applicable and where it may face constraints.

### Challenges of Hard Exploration and Sparse Rewards

The method’s core approach—sampling reasoning paths proportionally to rewards—faces inherent challenges in many real-world decision-making problems, where exploration is difficult, and rewards are sparse. In scenarios where only a binary success/failure reward is available, for example, the proposed method may converge toward behavior similar to traditional reinforcement learning, as it becomes difficult to maintain diverse reasoning paths with limited reward signals.

To clarify, I recognize that it is unrealistic to expect a single paper to address all the challenges present in the field, and this paper has already made valuable contributions to a specific class of problems. However, discussing these broader challenges would benefit readers by clarifying how this method could perform in more complex, real-world applications. Could the current approach address such challenges, or might these issues require further research and adaptation of the proposed method? Alternatively, is this method perhaps less suitable for tasks with particularly sparse rewards?

### See other questions below. Some of those questions may be worth discussing.

questions:
1. Why does only the Game of 24 domain use a different model?
1. Could the authors clarify which components are borrowed from existing works and which are newly proposed in this method? Many core components (sampling proportional to rewards, trajectory balance objective, adaptive efficient exploration) seem to be derived from existing approaches.
1. Could the authors clarify the usage of examples in the test dataset? Is it used for few-shot prompting during inference, or is it only used for calculating creativity metrics?
1. In Line 301, what is the value of N for each domain? The choice of N seems to not be specified for domains like Blocks World or Game of 24.
1. While it's clear that each fine-tuning method shares the same offline dataset, has the size of the online dataset also been controlled across methods?
1. The proposed method seems to not include search during inference. Why then is its runtime significantly higher than other COT/SFT methods that also do not include search?
1. For baseline evaluation, do the prompts explicitly include keywords to encourage diverse responses? If not, this could be worth investigating.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732176642689

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response - Part 2

comment:
## Qualitative Results

Our paper already provides case study examples in both Figure 1 and Appendix F. In Figure 1’s (left) Game24 example, we show that the SFT and CoT methods can only find one trajectory, while our FoR can find 4 different trajectories. Appendix F provides more examples of FoR among different tasks. 

We would like to present two additional qualitative results on the Game24 task to highlight the strengths of our method. This part would also be added to our revision.


- **Balance between diversity and accuracy**: For the first case study (please see the figure [here](https://anonymous.4open.science/r/iclr25-432C/casestudy.jpg)), we use the problem *(3,4,6,11)* to show how FoR achieves such high performance while focusing on diversity. As illustrated in the figure, we compare trajectories sampled 20 times by both SFT and FoR. While both methods produce diverse trajectories initially, FoR demonstrates better capability in reaching successful final steps from various middle steps. For example, FoR successfully transitions from intermediate steps such as *(3,6,15)* to the target 24, whereas SFT fails to do so. This highlights the effectiveness of FoR’s design in simultaneously promoting diversity and ensuring accuracy.

- **Better robustness due to exploratory nature**: For the second case study (please see the figure [here](https://anonymous.4open.science/r/iclr25-432C/casestudy2.jpg)), we use the problem *(7,9,9,13)* to demonstrate the robustness of FoR. As shown in the figure, SFT repeatedly fails by getting stuck in a single second state of *(9,3,16)* 20 times, while FoR successfully discovers multiple diverse trajectories leading to the correct solution. This robustness can be attributed to the exploratory nature of FoR's training objective, which encourages the model to sample diverse successful trajectories. By expanding the search space through high-reward exploration, FoR increases the chance to find successful outcomes. This capability not only improves the robustness of the model but also enhances its generalization to new scenarios, showcasing the effectiveness of FoR in addressing complex reasoning tasks.

---

References

[1] Hu, Edward J., et al. "Amortizing intractable inference in large language models." The Thirteenth International Conference on Learning Representations 12 (2024) Oral.

[2] Yao, Shunyu, et al. "Tree of thoughts: Deliberate problem solving with large language models." Advances in Neural Information Processing Systems 36 (2024).

[3] Shibo Hao, et al. Reasoning with Language Model is Planning with World Model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8154–8173, Singapore. Association for Computational Linguistics.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732176690938

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response - Part 1

comment:
We sincerely thank all reviewers for their encouraging feedback on our paper. We are grateful for the recognition of our contributions, including our novel focus on the **diversity of multi-step LLM reasoning**, an area previously unexplored, and the effective solution to this challenge (*h3sb*); our innovative idea of modeling multi-step reasoning as Markovian flow and applying GFlowNets to the LLM reasoning tasks (*iYFq*), which leads to significant improvements of success rate from 20% to 80% for a 7B model, even surpassing OpenAI-o1 model in some tasks (*h3sb*, *3ETe*, *waXE*). We are pleased that the reviewers recognize the robustness of our solid experiments across 5 benchmarks and ablation studies (*waXE*, *3ETe*, *h3sb*), as well as the well-organized and easy-to-follow presentation of our paper (*3ETe*, *iYFq*). We also appreciate the acknowledgment of the data efficiency demonstrated by our method (*h3sb*). 



## Contributions

We would like to highlight again the **contributions** of our work that differ from previous works as follows:

1. **Novel problem:** We are the first to introduce the problem of **diverse** LLM reasoning. This problem is fundamental to building more intelligent machines and is crucial for applications such as creative problem-solving, robustness, and scientific discovery.  

2. **Technical contributions (I):** We are the first to formulate multi-step LLM reasoning from a Markovian flow perspective. Unlike the latest work (such as [1]) which focuses on token-level modeling (Line.92), our state-action formulation is more general and better suited for complex multi-step reasoning settings. On The Other Hand, compared to recent Reinforcement Learning-based LLM reasoning (which also adopts state-action formulation and thus is similarly general) [2,3], our Markovian flow perspective allows for promoting diverse reasoning solutions, rather than solely optimizing for the maximum-reward solution.

3. **Technical contributions (II)**: Based on the above Markovian flow formulation, we manage to adapt and integrate a series of the latest GFlowNets approaches to our problem setting, such as *trajectory balance* objective with *log-variance* approximation for tractability, and design new *local search* method for better solution exploration.

4. **Empirical contributions:** We conduct extensive experiments across **5 benchmarks** challenging for LLMs and approximately **20 baseline methods**, showing our approach achieves substantial improvements not only in accuracy scores (by a 20% to 80% margin) but also in diversity and creativity. Note that *we are the first to systematically measure diversity and creativity in multi-step LLM reasoning*. In the task of Blocksworld, our approach with a 7B model either approaches or exceeds the performance of OpenAI-o1.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents Flow of Reasoning (FoR), a fine-tuning approach for LLMs that aims to enhance solution diversity. FoR frames the reasoning process as a Markovian flow, sampling diverse reasoning paths with probabilities aligned to the reward associated with each target problem similar to GFlowNet. Extensive experiments across multiple domains and baselines demonstrate that the proposed approach helps LLMs discover more diverse and higher-quality answers, showcasing the effectiveness of the algorithm.

On the positive side, the paper approach appears to be novel and new. However, some of the writing is not clear, and additional baselines would be helpful. In addition, when I carefully read the paper, the authors make a big deal above sampling trajectories according to the boltzmann distribution over rewards. However -- this is the exact density that is used in KL regularized RL such as DPO and PPO so it would be helpful to clarify in the paper how flow of reasoning is different than other existing approaches.