HDFlow: Enhancing LLM Complex Problem-Solving with Hybrid Thinking and Dynamic Workflows

## Abstract

Abstract Despite recent advancements in large language models (LLMs), their performance on complex reasoning problems requiring multi-step thinking and combining various skills is still limited. To address this, we propose a novel framework HDFlow for complex reasoning with LLMs that combines fast and slow thinking modes in an adaptive manner. Our approach consists of two key components: 1) a new approach for slow, deliberate reasoning called Dynamic Workflow, which automatically decomposes complex problems into more manageable sub-tasks and dynamically designs a workflow to assemble specialized LLM or symbolic reasoning tools to solve sub-tasks; 2) Hybrid Thinking, a general framework that dynamically combines fast and slow thinking based on problem complexity.
Finally, we propose an easy-to-scale method for automatically synthesizing a large-scale dataset of 27K challenging reasoning problems for complex reasoning and a hybrid thinking tuning method that trains smaller LLMs on this dataset to internalize the fast/slow hybrid reasoning strategies.
Experiments on four reasoning benchmark datasets demonstrate that our slow thinking with dynamic workflows significantly outperforms Chain-of-Thought, and hybrid thinking achieves the highest accuracy while providing an effective balance between computational efficiency and performance. Fine-tuning using our hybrid thinking approach also significantly boosts the complex reasoning capabilities of open-source language models. The results showcase the promise of slow thinking, dynamic workflows, and hybrid thinking in expanding the frontier of complex problem-solving with LLMs 1 1 1 Code and data will be released at https://github.com/wenlinyao/HDFlow . .

## 1 Introduction

Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks, from code generation and mathematical reasoning to natural language understanding and generation. However, their performance on complex reasoning problems that require multi-step thinking and various skills is still limited. Recent advancements in symbolic reasoning and tool usage, such as AlphaCode (Li et al., ; ), AlphaGeometry (Trinh et al., ), and AlphaProof (), have shown significant improvements in specific domains by integrating LLMs with specialized procedures and symbolic reasoning engines. Various prompting strategies, such as Chain-of-Thought (CoT) (Wei et al., ), Tree of Thoughts (ToT) (Yao et al., ), and Graph of Thoughts (GoT) (Besta et al., ), have been developed to enable different reasoning topologies to enhance LLM problem-solving capabilities. Despite these advancements, enhancing the reasoning abilities of LLMs to solve challenging problems across diverse domains in a unified framework remains crucial for expanding their real-world applicability.

Existing methods for complex reasoning with LLMs have several limitations.
First, complex problem-solving often requires combining various knowledge domains, skills, and tool usage.
While previous approaches such as AlphaCodium (Ridnik et al., ) and Alphageometry (Trinh et al., ) have demonstrated the potential of combining language models and symbolic reasoning to solve complex problems, they rely on manually designed workflows tailored to specific domains (i.e., competitive programming or geometry theorem proving). The language model and symbolic engine take predefined turns in a rigid problem-solving process. This limits the applicability and adaptability of these systems to broader domains. Thus, we aim to enhance the generic problem-solving capabilities of LLMs by dynamically alternating between natural language reasoning in the “text space” and symbolic reasoning in the “symbolic space” based on the problem at hand. This dynamic integration of the two reasoning modes enables the system to address a much broader range of problems and adapt the problem-solving process to the unique requirements of each task.
Second, traditional approaches to complex problem-solving with LLMs often rely on a single mode of thinking, which may struggle with more intricate tasks that demand a deliberate, analytical approach.
For example, many approaches employ a fixed reasoning strategy, such as CoT prompting, regardless of the problem’s complexity. For instance, OpenAI’s most recent o1 model(^2^22o1-preview model tested on Sept.24, 2024. o1-preview model thinks for a few seconds to users’ casual conversational queries such as How are you?) only engages in a singular deep thinking mode despite the complexity of the user’s query.
This can lead to suboptimal performance on tasks that require a more deliberate, multi-step approach.
While multi-agent frameworks such as AutoGPT (), ReAct Yao et al. (), and AutoGen (Wu et al., ) have addressed some aspects of this challenge by enabling recursive goal decomposition, interleaving reasoning and acting, and state-driven workflows, they do not fully exploit the potential of thinking approaches that can switch between intuitive thinking and more analytical thinking modes based on problem complexity. Finally, as problem complexity increases, the performance of existing approaches tends to degrade significantly, highlighting the need for frameworks that can scale to handle even the most challenging reasoning problems. Recently, OpenAI o1 model () demonstrates the potential to consistently improve LLM performance of complex reasoning with compute scaling in inference-time through deep thinking.

To address these limitations, we propose a novel framework for complex reasoning with LLMs that combines fast (system I) and more analytical slow thinking (system II) adaptively, inspired by the dual process theory of human cognition (Daniel, ).
Our approach consists of two key components. First, we introduce a new approach for slow, deliberate reasoning called Dynamic Workflow, which automatically decomposes complex problems into more manageable sub-tasks. It then dynamically designs a workflow to assemble specialized LLM or symbolic tools to solve each sub-task.
To achieve this, the dynamic workflow orchestrates a team of specialized LLM experts, each contributing unique domain knowledge or tool usage, to solve the sub-tasks in a structured manner. Second, we propose Hybrid Thinking, a general framework that dynamically combines fast and slow thinking based on problem complexity. For simpler tasks, the model defaults to a fast-thinking mode using CoT strategy. When the model’s confidence in the fast thinking output is low, it automatically switches to slow thinking with dynamic workflow, allowing for more efficient and more accurate problem-solving.
Finally, to train local LLMs for complex reasoning, we present an easy-to-scale method for automatically synthesizing a large-scale dataset of 27K challenging reasoning problems
and propose a hybrid thinking tuning approach that finetunes open-source LLMs on this dataset, enabling them to internalize the fast/slow hybrid reasoning strategies.

We conduct experiments on four reasoning benchmark datasets (i.e., BBH (Suzgun et al., ), MATH (Hendrycks et al., ), Game of 24 Yao et al. (), DeepMind Math (Saxton et al., ). Experiments using GPT-4-Turbo reveal that slow thinking with dynamic workflows significantly outperformed CoT, with an average accuracy improvement of 22.4%. Hybrid thinking, which combines fast and slow thinking, achieved the highest accuracy on three of the four datasets. While slow thinking required the most inference tokens, hybrid thinking struck an effective balance between computational efficiency and performance. Furthermore, fine-tuning Llama-3-8B-Instruct using hybrid thinking significantly boosted performance across all datasets compared to the original model. Hybrid thinking after fine-tuning yielded accuracy gains of 10-23% over CoT prompting, with broad improvements across different subject areas in MATH. Overall, the results demonstrate the promise of slow thinking with dynamic workflows and hybrid thinking in enhancing the complex problem-solving abilities of LLMs.

## 2 Related Work

Symbolic Reasoning and Tool Usage.
Bridging LLMs with symbolic reasoning and tool usage has demonstrated significant improvements across various domains. AlphaCode (Li et al., ; ) combines LLMs with a specialized search and reranking mechanism, achieving top-tier performance in competitive programming. Similarly,
AlphaCodium (Ridnik et al., ) improves AlphaCode’s performance by applying a predefined multi-stage process of problem analysis, solution generation, and iterative testing and bug fixing.
By using an evolutionary search procedure guided by an LLM, FunSearch (Romera-Paredes et al., ) can discover new mathematical constructions and algorithmic heuristics.
AlphaGeometry (Trinh et al., ) leverages a neuro-symbolic system trained on synthetic data to guide a symbolic deduction engine, achieving near-expert performance in geometry theorem proving.
Chain of Code (Li et al., ) encourages LLMs to write pseudocode for challenging sub-problems, which is then executed by the LM itself when it cannot be handled by a standard interpreter.
These approaches rely on carefully designing when and how to integrate symbolic reasoning for each task domain.

Prompting Strategies.
Various prompting strategies have been developed to enable different reasoning topologies (Besta et al., ) for enhancing LLM problem-solving capabilities. Chain-of-Thought (CoT) prompting (Wei et al., ) first introduced the concept of generating intermediate reasoning steps to improve performance on complex tasks. Building upon this, the Tree of Thoughts (ToT) (Yao et al., ) enables the exploration of multiple potential reasoning paths and incorporates deliberate decision-making through self-evaluation and backtracking. Graph of Thoughts (GoT) (Besta et al., ), models LLM-generated information as an arbitrary graph where thoughts are vertices and dependencies are edges, allowing for more complex reasoning structures and outcomes. In a different direction, Program of Thoughts (PoT) approach (Chen et al., ) disentangles computation from reasoning by expressing the reasoning process as a program, with external computation handling numerical operations. SELF-DISCOVER (Zhou et al., ) introduces a self-discovery process where LLMs autonomously select and compose multiple atomic reasoning modules into explicit reasoning structures. Our hybrid thinking approach allows for the efficient resolution of tasks within the LLM’s core capabilities through direct reasoning, while adaptively engaging in deeper, multi-step workflows for more complex problems.

Multi-Agent Frameworks for Task-Solving.
Recent advancements also led to the development of various frameworks for complex task-solving and multi-agent collaboration. AutoGPT () pioneered the idea of using LLMs for recursive goal decomposition and task completion, where sub-tasks are then performed sequentially to yield a larger result. ReAct (Yao et al., ) introduced a method for interleaving reasoning and acting, allowing LLMs to generate both reasoning traces and task-specific actions. Reflexion (Shinn et al., ) further enhanced language agents’ capabilities by incorporating verbal reinforcement learning, enabling them to reflect on feedback and improve decision-making. MetaGPT (Hong et al., ) addressed the challenge of LLM hallucination in multi-agent systems by incorporating human workflows and standardized operating procedures into the framework. AutoGen (Wu et al., ) presented a flexible multi-agent conversation framework that allows for customizable, conversable agents with human participation. CAMEL (Li et al., ) introduced a role-playing approach to facilitate autonomous cooperation among communicative agents. Finally, StateFlow (Wu et al., ) proposed a state-driven workflow that conceptualizes complex task-solving processes as state machines, enhancing control and interpretability. In contrast to these existing works, our approach uniquely integrates hybrid thinking, combining fast and slow thinking modes with automatic workflows, to enhance LLMs’ ability to tackle complex reasoning problems more effectively and with greater adaptability.

## 3 Overview of The Hybrid Thinking Approach

Figure: Figure 1: Overview of our HDFlow approach for complex problem-solving. Overall, it is a dual-path hybrid thinking approach, beginning with a CoT solver for initial fast reasoning followed by verification of each reasoning step. If verification fails, the process transitions to a slower, more deliberate ”Dynamic Workflow Solver.” This solver iterates until a verified answer is obtained, incorporating a final verification step before concluding with a solution.
Refer to caption: x1.png

Our hybrid thinking approach (Figure 1) combines the strengths of fast and slow thinking modes to enable LLMs to more effectively solve complex reasoning problems. It consists of the following three key components.
1) Fast Thinking with Direct CoT. In the fast thinking mode, the LLM uses a direct chain of thought (CoT) approach to quickly solve the task query if possible. This leverages the LLM’s core abilities to perform certain types of reasoning efficiently by directly generating the rationale and the final answer.
2) Adaptive Combination of Fast and Slow Thinking. Next, we employ a self-verification mechanism where the LLM examines each step of the fast-thinking CoT reasoning to assess its confidence in the generated answer. This is achieved by applying the LLM to analyze the coherence, logical consistency, and correctness of each reasoning step in the context of the given query. If the LLM detects any inconsistencies, errors, or low-confidence steps during this self-verification process, it triggers a switch to the slow-thinking mode.
3) Slow Thinking with Dynamic Workflow. To tackle highly complex tasks, we propose a novel slow-thinking mechanism called Dynamic Workflow (Figure 2), which automatically decomposes the original task into sub-tasks and dynamically switches between verbal reasoning and symbolic reasoning to solve each sub-task. Our approach starts with multi-level problem reflection and decomposition. It then designs a workflow to assemble specialized LLM skills or symbolic tools for sub-tasks. Next, we dynamically chain together the sub-task reasoning steps into a multi-step workflow and execute the workflow.
Finally, all sub-task results are aggregated into the final answer to the original query. We will present details in Section 4.

By first attempting fast thinking, our hybrid thinking approach can efficiently handle queries that are within the LLM’s core capabilities. When the query exceeds what fast thinking alone can confidently handle, the hybrid thinking will smoothly transition to a slow thinking workflow to enable the LLM to tackle a broader range of challenges accurately.

## 4 Slow Thinking with Dynamic Workflow

Figure: Figure 2: Three-Stage Framework of Dynamic Workflow. The dynamic workflow design begins with Problem Reflection, where key elements are analyzed and sub-tasks identified. Stage 2 focuses on Expert Design, utilizing a variety of specialists and tools to architect an optimal workflow. Stage 3 involves constructing and executing the workflow graph to get the final result.
Refer to caption: x2.png

In contrast to the rapid responses of fast thinking (e.g., CoT), our new slow-thinking mechanism applies dynamic workflow to enable a more deliberate, analytical approach to complex problem-solving (see Figure 2). It allows an LLM to dynamically transition between reasoning in the text space (natural language reasoning) and the symbolic space (symbolic reasoning).
The high-level idea is we first let the LLM decompose the original reasoning problem into several more manageable sub-tasks and solve each sub-task to form the final solution.
When necessary, the LLM Engine will translate the sub-problem from the text space into the symbolic space, enabling the symbolic engine(^3^33In this paper, we mainly use program to achieve symbolic reasoning.) to perform precise symbolic reasoning. The results are then mapped back into natural language using the LLM Engine.
By decomposing the problem, combining the strengths of both natural language and symbolic reasoning in a tailored workflow, and executing it from start to finish, LLMs can tackle very hard problems that require multiple steps of accurate reasoning. Appendix B presents a complete example solution using our dynamic workflow approach and compares with the solution using OpenAI o1-preview. Prompts used are listed in Appendix C.

### 4.1 Breaking Down Complexity: Problem Analysis and Decomposition (Stage 1)

The first step in our slow thinking is problem analysis and planning. We aim to break down the original problem statement into more manageable sub-tasks. Specifically, the LLM is asked to analyze the key elements of the query, such as available information, constraints, and the desired output. It then identifies logical sub-goals needed to progress from the initial state to the solution. This decomposition allows the LLM to approach the problem in a structured manner, focusing on one part at a time. Therefore, the LLM can catch gaps in reasoning and handle complex problems that the fast thinking of CoT alone would struggle with.

Problem Reflection.
The first step in tackling complex problems is conducting a thorough problem reflection. This involves the LLM analyzing the original problem and restating it in its own words to demonstrate understanding. Our problem reflection includes two parts:
1) Identifying the core objective or question posed by the problem.
2) Recognizing any constraints, assumptions, or special conditions mentioned.
By internalizing the problem through reflection, the LLM can gain a solid understanding of what needs to be accomplished before proceeding to decomposition.

Subtask Decomposition.
Once the problem is well understood, the LLM is instructed to perform a multi-level decomposition to break it down into some tractable sub-problems. The LLM is asked to follow four principles to achieve an optimal decomposition.
Sequential dependency. The sub-problems are organized in a logical sequence, such that the outputs of earlier steps feed into subsequent ones, creating a structured workflow from start to finish.
Non-overlapping. Each sub-problem represents a distinct portion of the original problem, with no duplication of work between sub-problems. This keeps the overall solution efficient.
Proper Decomposition. The sub-problems are decomposed to the optimal level of granularity - not so small that there are too many to track and coordinate, but not so large that they are still struggling to solve.
Modular. Where appropriate, sub-problems are defined in a generalizable, modular way, such that the logic and code used to solve them can potentially be reused to solve similar problems in other contexts.

Integrating Symbolic Reasoning.
Another key aspect of our approach is leveraging the symbolic engines to modularize the solution and handle well-defined sub-tasks more accurately. For example, some sub-tasks in the decomposition can often be addressed by writing code functions.
Therefore, we explicitly instruct the LLM to consider sub-tasks that can be well handled by writing and executing modular code in subtask decomposition.

### 4.2 Orchestrating Expertise: Workflow Design (Stage 2)

With the problem decomposed into sub-tasks, our approach next proposes a team of specialized experts, each contributing unique skills and tools, arranged in a dynamic workflow. The central component is a Meta-Expert, initialized from the foundation LLM, designs the expert team, and coordinates their efforts.
The orchestration process consists of four steps.

- 1.
Design of Experts. Based on the identified sub-tasks, the Meta-Expert designs a team of specialized experts with one expert solving one sub-task. Each expert is assigned a unique name and a clear description of their specific skills, knowledge, and responsibilities(^4^44Our implementation leverages JSON for efficient data management and extraction across the system.).
The dynamic workflow leverages two types of experts to handle each sub-task, enabling a seamless integration of verbal and symbolic reasoning. The first type are specialized experts initiated from LLMs, such as linguists, mathematicians, and data scientists. These experts bring domain-specific knowledge and skills to the workflow, allowing for sophisticated verbal reasoning and analysis within their fields.
The second type of expert focuses on symbolic reasoning, particularly using programming or other symbolic engines(^5^55We mainly use Python code interpreter as the symbolic engine in our experiments, but our approach can be extended to other symbolic engines, such as the symbolic deduction engines used in AlphaGeometry (Trinh et al., ) to solve Euclidean geometry problems.).
For example, some sub-tasks can often be addressed by writing compact, targeted code functions.
This allows the LLM to handle common operations such as mathematical calculations, data parsing and manipulation, and so on without bringing errors.
- 2.
Workflow Arrangement. The Meta-Expert arranges the experts into an efficient workflow sequence. Each expert’s output serves as the input for the next, progressively moving towards the final solution. The Meta-Expert ensures there is no redundancy of functions across experts.
- 3.
Collaboration and Iteration. As the experts work through the problem, the Meta-Expert facilitates collaboration and puts together their inputs and outputs. For sub-tasks involving logical reasoning, mathematical operations, data structures, or programming, the Meta-Expert provides strategic guidance and sends the implementation details to the corresponding symbolic reasoning experts. These experts utilize LLMs to generate code, which is then executed to perform symbolic reasoning in Stage 3.
- 4.
Final Review and Conclusion. The last expert in the workflow, often an LLM specialist, is tasked with holistically reviewing the findings of the previous experts and generating the final answer to the original problem.

By combining the power of specialized LLMs and the usage of tools into a thoughtfully designed, adaptable workflow, our approach can tackle complex problems that are beyond the capabilities of the original model. The Meta-Expert serves as the intelligent connector, analyzing the unique needs of each problem and dynamically assembling the optimal workflow.
Our approach creates a bridge between natural language reasoning and rule-governed symbolic reasoning.

### 4.3 Flow Execution: Constructing and Running Workflows (Stage 3)

With the workflow graph generated, our approach finally proceeds to execute the graph to get the final result.
The execution follows the dependency order, ensuring the correct flow of data between experts.
To ensure robust execution, if any of the generated code encounters errors, the corresponding symbolic reasoning experts will trace the issue, use the error message to repair the code, and rerun it.
As the workflow progresses, the downstream experts continually update their memory with the intermediate results and insights generated by previous experts.
Upon completion of the workflow execution, the last LLM expert analyzes the results, identifies key findings, and summarizes them into a final answer to the original problem.
The workflow execution is not a one-time process. The LLM continually assesses the quality and correctness of the final generated solutions and identifies potential errors. It engages in iterative rerun by applying a different problem decomposition, expert assignments, or adjusting the workflow structure.

## 5 Model Tuning of Hybrid Thinking

Figure: Figure 3: Data Synthesis of Complex Reasoning Problems. The creation and refinement of reasoning problems contain three steps. Step 1 involves brainstorming and generating high-level descriptions of new reasoning tasks, either inspired by human-written tasks or directly writing puzzle tasks. Step 1 produces 45K descriptions of reasoning tasks. Step 2 performs semantic matching and deduplication and results in 18K reasoning task descriptions. The final Step 3 writes concrete questions based on task descriptions and applies a CoT validation process to filter or refine the tasks down to 27k valid reasoning problems.
Refer to caption: x3.png

Figure: Figure 4: Three example reasoning problems generated by our data synthesis approach.

In our experiments, we observed that open-source language models (typically those with around 7B parameters) often struggle with advanced meta-planning and problem-solving skills required for solving difficult reasoning tasks.
To address this limitation and develop local smaller models with hybrid thinking abilities comparable to the large models, we construct a comprehensive training dataset and propose hybrid thinking tuning to improve the complex reasoning abilities of local models. We define “local” models as models that can be trained and deployed on local hardware with limited computational resources, such as the Llama-3 model (Meta, ). Our goal is to improve the complex reasoning abilities of these local models through our proposed approach.

The primary challenge lies in constructing a large-scale dataset of reasoning problems that are sufficiently diverse, high-quality, and difficult. Such a dataset is crucial for teaching smaller local models to perform complex reasoning tasks.
However, manually curating such a dataset presents significant difficulties in ensuring a wide range of problem domains and maintaining high standards in problem formulation.
As a result, it is extremely time-consuming and expensive to ask human experts to consistently generate problems meeting all criteria.
Therefore, we propose a novel approach for
automatically generate a variety of reasoning problems and collect solutions of hybrid thinking, which can then be used to train our local LLMs.

### 5.1 Reasoning Problems Synthesis

To enhance reasoning task diversity and coverage, our data synthesis pipeline consists of three steps (Figure 3). In the first step, we strategically leverage human-authored seed tasks to inspire the creation of new reasoning problems (similar to Self-Instruct (Wang et al., )) or let the LLM brainstorm
reasoning puzzles that cover a variety of task formats, difficulty levels, and problem domains.
This step only focuses on generating high-level task descriptions to encourage diversity. In the second step, we apply deduplication to remove near-identical tasks.
Finally, we apply LLMs again to write three specific problems based on the task descriptions and validate those problems.

Task Generation Inspired by Seed Tasks. The first step of our reasoning data synthesis pipeline is generating an expanded set of reasoning tasks.
We augment the few-shot prompts with 10 high-level task descriptions randomly sampled from the 214 BigBench tasks (Srivastava et al., ). Next, we employ the 10 seed tasks as in-context examples to prompt LLMs(^6^66We use both GPT-4-0125 and Claude-3-Opus to encourage diversity. We find Claude-3-Opus does generate very different reasoning tasks compared with GPT-4-0125.) to generate 10 new tasks inspired by seed tasks.
To encourage additional diversity in the generated tasks, we also let the LLM to brainstorm different genres of puzzles, such as crossword puzzles, math puzzles, number puzzles, relational puzzles, logic puzzles, etc. By repeating two strategies, we produce an expanded pool of 45K candidate reasoning tasks that creatively cover diverse reasoning types and scenarios.

Data Filtering and Deduplication.
The previous task generation step produces a sizable pool of candidate reasoning tasks. However, the generated data is likely to contain duplicate or highly similar entries.
To address this, we employ a comprehensive data filtering and deduplication process. First, we apply n-gram to identify nearly identical tasks.
Next, we filter out any tasks or problems that fail to meet our quality criteria, such as insufficient complexity (e.g., trivial one-step questions), or ambiguity in the description by prompting GPT-4-Turbo. This helps ensure that only high-quality, unambiguous reasoning tasks are retained in the final dataset.
Through this rigorous deduplication and filtering process, we condense the pool of 45K generated tasks down to 18K deduplicated tasks.

Reasoning Problem Synthesis.
In the last step, we aim to synthesize multiple concrete reasoning problems for each of the 18K tasks produced by the previous task generation and deduplication steps. Taking each task’s description as input, we prompt an LLM to generate 3 distinct questions or problems that test the specified reasoning skill. This enables us to turn each high-level task into a set of actual solvable questions, resulting in a pool of 54k reasoning problems.
To ensure the generated problems are well-posed and solvable, we employ a chain-of-thought (CoT) based validation step. We prompt GPT-4-Turbo to apply CoT to each synthesized problem and analyze if the resulting reasoning steps coherently lead to a definite answer.
Problems for which the model fails to converge to a clear solution or exhibits inconsistent reasoning are filtered out.
This results in the final 27K reasoning problems. Figure 4 provides three examples of reasoning problems generated.

### 5.2 Finetuning Open-Source Models on Synthesized Data

To prepare the training data for enhancing the open-source models’ complex problem-solving abilities, we utilize the GPT-4-turbo model to collect reasoning trajectories on the dataset of synthesized and mathematical problems.
For each problem, GPT-4-turbo generates one or several fast/slow reasoning trajectories using the hybrid thinking approach.
Each reasoning trajectory consists of a sequence of (query, answer) pairs representing the model’s step-wise hybrid thinking process. Therefore, we use all (query, answer) pairs from the reasoning trajectories to construct the training data, capturing the complete problem-solving process.
When multiple reasoning trajectories are produced (iterative retry), only the solution trajectory that passes the verification process is retained in the training set to optimize the model’s problem-solving capabilities, while the verification results for all trajectories are kept to enhance the model’s self-verification abilities.

The Llama-3 models have demonstrated superior performance compared to other models of similar size due to significant enhancements in both pretraining and post-training (Meta, ).
Therefore, we choose the Llama-3-8B-Instruct model as the foundation model for our hybrid thinking tuning experiments.
Specifically, The Llama-3-8B-Instruct model was fine-tuned using 8 A100 GPUs with bf16 precision(^7^77We adopt LitGPT (AI, ) in our model training.). The training utilized a global batch size of 128, spanning 4 epochs. The model employed the AdamW optimizer of a learning rate of 2.0e-5, with a maximum sequence length of 4096 tokens and a maximum of 2048 new tokens generated.

## 6 Experiment

### 6.1 Reasoning Benchmark Datasets

BIG-Bench Hard (BBH) (Suzgun et al., ): A subset of 27 challenging tasks from the BIG-Bench benchmark (Srivastava et al., ), which aims to measure the capabilities and limitations of language models across diverse text-based tasks.
MATH (Hendrycks et al., ): A dataset consisting of 5,000 test problems from mathematics competitions. These problems assess the mathematical problem-solving ability and often require the application of problem-solving techniques and heuristics beyond standard K-12 mathematics tools.
Game of 24 (Yao et al., ): A mathematical reasoning challenge dataset containing 1,362 games sorted by human solving time. The goal is to use four given numbers and basic arithmetic operations (+ - * /) to obtain 24.
DeepMind Math (Saxton et al., ): A dataset consisting of various types of mathematics questions, released with both generation code and pre-generated questions. This dataset provides an additional measure of algebraic generalization abilities.

**Table 1: Accuracy (%) of GPT-4-Turbo-0125 across different reasoning modes on various datasets. We show the accuracy of the model using Chain of Thought (CoT) v.s. slow thinking (with dynamic workflow) and Hybrid Thinking approaches proposed by us. The Fast/Slow indicates the ratio of Fast and Slow Thinking contributions in the Hybrid approach. Results are derived from the top 100 instances for each sub-category in BBH (27 sub-tasks), MATH (7 sub-domains), and GameOf24 (3 difficulty levels) to reduce API cost and ensure replicability. For the DeepMind Math dataset, the top 10 instances from each of the 56 sub-domains were used.**
| Methods | BBH | MATH | DeepMind Math | GameOf24 | Avg. |
| --- | --- | --- | --- | --- | --- |
| CoT (Fast Think.) | 77.8 | 62.6 | 53.4 | 9.3 | 50.8 |
| Slow Think. | 87.1 (+9.3) | 67.6 (+4.6) | 67.7 (+14.3) | 70.3 (+61.0) | 73.2 (+22.4) |
| Hybrid Think. | 87.8 (+10.0) | 70.0 (+7.9) | 59.6 (+6.2) | 72.0 (+62.7) | 72.4 (+21.6) |

**Table 2: Average number of inference tokens of GPT-4-Turbo-0125 using different reasoning modes on various datasets. Performance is reported in Table 1.**
| Methods | BBH | MATH | DeepMind Math | GameOf24 | Avg. Tokens |
| --- | --- | --- | --- | --- | --- |
| CoT (Fast Think.) | 351 | 992 | 581 | 387 | 577.8 |
| Slow Think. | 3227 | 5694 | 3562 | 5246 | 4432.0 |
| Hybrid Think. | 1299 | 4398 | 1742 | 4983 | 3105.5 |

### 6.2 Results Based on Prompting

We first conduct experiments by prompting GPT-4-Turbo-0125(^8^88[https://platform.openai.com/docs/models](https://platform.openai.com/docs/models). A full list of prompts can be found in Appendix C.) to achieve three reasoning modes: Chain of Thought (CoT), Slow Thinking with Dynamic Workflow, and Hybrid Thinking across four benchmark datasets. Table 1 shows that slow thinking with dynamic workflow significantly outperforms CoT by 22.4% on average across four benchmarks.
It also reveals that Hybrid Thinking achieves the best accuracy on three datasets BBH, MATH and GameOf24. Notably, both Slow Thinking and Hybrid Thinking consistently outperform CoT across all datasets, with the most dramatic improvements seen in GameOf24, where gains are 61.0% and 62.7% respectively.

Table 2 illustrates the average number of inference tokens used by each method. CoT consistently used the fewest tokens (average 577.8), while Slow Thinking required the most (4432.0 on average). Hybrid Thinking struck a balance with an average of 3105.5 tokens.
A clear trade-off emerged between computational efficiency and performance, with CoT using the fewest tokens but achieving the lowest accuracy. Hybrid Thinking demonstrated a good balance, achieving high accuracy with moderate token usage. These findings suggest that incorporating dynamic workflows and combining fast and slow thinking processes can enhance the reasoning capabilities of LLMs, with Hybrid Thinking emerging as a particularly promising approach.

**Table 3: Performance comparison of the original Llama-3-8B-Instruct model and the Llama-3-8B-Instruct after our hybrid thinking tuning. We show the accuracy (%) of the model using Chain of Thought (CoT) v.s. slow thinking (with dynamic workflow) and Hybrid Thinking approaches proposed by us. The Fast/Slow indicates the ratio of Fast and Slow Thinking contributions in the Hybrid approach. Results are derived from all test instances in BBH, MATH, DeepMind Math and GameOf24.**
| Methods | BBH | MATH | DeepMind Math | GameOf24 | Avg. |
| --- | --- | --- | --- | --- | --- |
| Llama-3-8B-Instruct (Original) |  |  |  |  |  |
| CoT | 51.7 | 30.0 | 18.6 | 2.7 | 25.8 |
| Llama-3-8B-Instruct (After Hybrid Thinking Tuning) |  |  |  |  |  |
| CoT (Fast Think.) | 58.5 (+6.8) | 37.0 (+7.0) | 34.2 (+15.6) | 5.1 (+2.4) | 33.7 (+7.9) |
| Slow Think. | 61.2 (+9.5) | 37.8 (+7.8) | 48.8 (+30.2) | 15.4 (+12.7) | 40.8 (+15.0) |
| Hybrid Think. | 62.3 (+10.6) | 40.2 (+10.2) | 41.7 (+23.1) | 16.0 (+13.3) | 40.5 (+14.7) |

**Table 4: Average number of inference tokens of the original Llama-3-8B-Instruct model and the Llama-3-8B-Instruct after our hybrid thinking tuning on various datasets. Performance is reported in Table 3.**
| Methods | BBH | MATH | DeepMind Math | GameOf24 | Avg. Tokens |
| --- | --- | --- | --- | --- | --- |
| Llama-3-8B-Instruct (Original) |  |  |  |  |  |
| CoT | 356 | 496 | 359 | 510 | 430.2 |
| Llama-3-8B-Instruct (After Hybrid Thinking Tuning) |  |  |  |  |  |
| CoT (Fast Think.) | 720 | 985 | 770 | 1384 | 964.7 |
| Slow Think. | 3901 | 5743 | 4395 | 6714 | 5188.2 |
| Hybrid Think. | 2521 | 4414 | 2577 | 6371 | 3970.7 |

Figure: Figure 5: Proportion of fast thinking (CoT) and slow thinking (dynamic workflow) applied in hybrid thinking across four datasets. The left is GPT-4-Turbo (performance is shown in Table 1), while the right is Llama-3-8B-Instruct after our hybrid thinking tuning (Table 3).
Refer to caption: x4.png

### 6.3 Results of Hybrid Thinking Tuning

We next compare the performance of the original Llama-3-8B-Instruct model and the model after our hybrid thinking tuning.
As shown in Table 3, the Llama-3-8B-Instruct model after hybrid thinking tuning significantly outperforms the baseline model on all datasets.
Examining the different thinking modes, hybrid thinking consistently provided the best tradeoff between performance and efficiency. Compared to the CoT baseline, hybrid thinking improved accuracy by 10.6%, 10.2%, 23.1% and 13.3% on the BBH, MATH, DeepMind Math and GameOf24 datasets respectively. Interestingly, we also observe that hybrid thinking tuning enhances Llama-3’s fast thinking (CoT) performance across all reasoning tasks at the cost of increased model inference tokens.

**Table 5: Accuracy comparison of the original Llama-3-8B-Instruct model and the Llama-3-8B-Instruct after our hybrid thinking tuning on different domains of the MATH dataset. “Count. and Prob.” and “Inter. Algebra” represents “Counting and Probability” and “Intermediate Algebra”.**
| MATH Subsets | Llama-3-8B-Ins. | Llama-3-8B-Ins. (After Hybrid Thinking Tuning) |  |  |  |
| --- | --- | --- | --- | --- | --- |
| CoT | CoT (Fast Think.) | Slow Think. | Hybrid Think. | Fast/Slow |  |
| Prealgebra | 43.2% | 58.9% | 59.7% | 63.3% | 0.69/0.31 |
| Algebra | 30.2% | 53.6% | 52.7% | 56.1% | 0.68/0.32 |
| Number Theory | 15.0% | 31.1% | 37.6% | 38.0% | 0.52/0.48 |
| Count. and Prob. | 21.1% | 32.5% | 34.2% | 35.9% | 0.48/0.52 |
| Geometry | 13.4% | 24.8% | 23.6% | 26.3% | 0.33/0.67 |
| Precalculus | 12.5% | 22.0% | 21.8% | 24.5% | 0.35/0.65 |
| Inter. Algebra | 9.1% | 15.6% | 16.3% | 17.3% | 0.30/0.70 |

Table 5 breaks down performance on the MATH dataset into specific subject areas. Again, the Llama-3-8B-Instruct model after hybrid thinking tuning outperforms the original model on all subsets, with gains ranging from 8% on intermediate Algebra to 23% on Number Theory. Hybrid thinking yielded the highest accuracy in each domain, demonstrating its broad applicability.

### 6.4 Fast/Slow Routing Analysis

Figure 5 illustrates the proportion of fast thinking and slow thinking (orange) approaches applied by both models when solving complex problems across the datasets. The GPT-4-Turbo model demonstrates a higher reliance on fast thinking for BBH, DeepMind MATH, and Game of 24 tasks compared with Llama-3-8B-Instruct model.
This observation can be attributed to the fact that GPT-4-Turbo’s fast thinking (in the form of CoT) is more reliable and effective compared to Llama-3-8B-Instruct. As a result, hybrid thinking in GPT-4-Turbo tends to apply more fast thinking since it is sufficient to achieve a correct solution in many cases. In contrast, Llama-3-8B-Instruct after tuning exhibits a greater reliance on slow thinking strategies, particularly in complex tasks, where fast thinking alone may not yield the desired results. This highlights the importance of hybrid thinking to improve problem-solving efficiency, suggesting that our method can dynamically adjust the optimal balance between fast and slow thinking based on the model’s downstream reasoning capabilities.

In summary, the dynamic combination of fast and slow thinking modes greatly enhanced the model’s problem-solving capabilities. Our results showcase the potential of hybrid thinking approaches to expand the frontier of what LLMs can achieve on challenging tasks.

## 7 Discussion and Future Work

Limitations and Potential Improvements. One promising direction is to incorporate a value network that scores the successfulness or quality of completing each sub-task within the dynamic workflow. By integrating such a value network, we can formulate the problem-solving process as a reinforcement learning task, enabling the optimization and search for the best solution trajectory. This enhancement could lead to more efficient and effective problem-solving strategies, as the model learns to prioritize and select the most promising decompositions and workflows based on predicted values.

Generalization to Other Reasoning Tasks. Constructing high-quality and sufficiently challenging reasoning problems for training still remains a significant challenge. While our data synthesis approach offers a scalable solution, ensuring the validity and difficulty of each generated reasoning problem is crucial for effective model development. One potential improvement is to involve human experts in the data synthesis process, allowing them to verify, modify, and curate the generated problems.

Integration with Symbolic Reasoning Systems.
Our dynamic workflow approach seamlessly integrates specialized language models and symbolic reasoning tools, enabling LLMs to tackle complex problems more effectively. However, there is significant potential to extend this integration to more advanced symbolic reasoning systems, such as Lean(^9^99https://lean-lang.org/) for mathematical theorem proving or other domain-specific tools. Moreover, integrating our approach with tools such as search engines and web browsers could enable LLMs to access and utilize external resources, further amplifying their problem-solving abilities to broader applications.
By incorporating more powerful tools into the dynamic workflow, we can expand the range of problems that LLMs can solve.

## 8 Conclusion

This paper introduces a novel framework HDFlow for enhancing the complex problem-solving capabilities of LLMs through hybrid thinking and dynamic workflows.
The dynamic workflow mechanism enables LLMs to decompose complex problems into manageable sub-tasks and integrate specialized language models and symbolic reasoning tools, while hybrid thinking strategically engages deeper, multi-step reasoning for challenging problems that exceed the capabilities of fast thinking alone.
Extensive experiments demonstrate the significant advantages of our approach, with slow thinking with dynamic workflow greatly outperforming CoT and hybrid thinking achieving the highest overall accuracy by balancing efficiency and performance.

## Appendix A Appendix

## Appendix B Dynamic Workflow Example Solution

In this section, we demonstrate how our dynamic workflow approach works by presenting an example solution to a reasoning problem. We also present the solution provided by the OpenAI o1-preview model to compare with our approach. We can see the solution produced by our approach contains many sub-goals (sub-tasks), which is very similar to the o1-preview solution. Our approach also leverages the symbolic tool - Python to enhance the accuracy of the final answer. The reasoning problem is generated using our data synthesis method.

### B.1 Reasoning Problem:

### B.2 Solution using dynamic workflow:

Stage 1. Problem Reflection

Stage 2. Workflow Design

Stage 3. Graph Construction and Execution

### B.3 Solution using OpenAI o1-preview

Hidden Thought Process (thought for 12 seconds)

Final solution

## Appendix C Prompts Used in Our Dynamic Workflow Approach

Note that placeholders enclosed in {} are used within the prompts to represent variables that will be substituted with actual values at LLM query time.

## Appendix D Data Synthesis of Reasoning Problems

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: N/A
soundness: 2
strengths: 1. strong performance improvement compared with direct CoT thinking
summary: This paper presents a fast-slow thinking mechanism where the fast thinking is direct CoT and slow thinking is a dynamic workflow method. It also utilizes a dataset containing fast thinking process and slow thinking process to train a model to internalize the fast/slow thinking strategy.
weaknesses: 1. missing citation and discussion for System-1.x: Learning to Balance Fast and Slow Planning with Language Models, which also talks about the combination of fast thinking and slow thinking
2. This paper basically use cot as fast thinking and agentic planning as slow thinking. I feel like there's not much novelty here
3. missing baselines such as the method from System-1.x: Learning to Balance Fast and Slow Planning with Language Models

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: I will read authors' rebuttal and discuss more about the paper.
soundness: 3
strengths: ## Strengths

1. The paper presents a new approach that facilitates deliberate, slow reasoning. (Compared to previous methods like CoT/PAL, ) this method automatically breaks down complex problems into smaller sub-tasks and designs a dynamic workflow to solve each sub-task using specialized LLMs or symbolic reasoning tools.
2. The proposed HDFlow is tested on 4 reasoning benchmark datasets. The Slow Thinking approach with Dynamic Workflow outperformed traditional CoT-like methods, achieving a notable average accuracy improvement.
3. Authors introduces an easy-to-scale method for automatically generating a large-scale dataset of ~27K reasoning problems. Using this dataset, they propose a hybrid thinking tuning approach to fine-tune smaller, open-source LLMs.
summary: This paper introduces a framework called HDFlow aimed at enhancing the complex reasoning abilities of LLMs. HDFlow combines fast & slow thinking modes in an adaptive manner to tackle problems that require multi-step reasoning and the integration of various skills. The framework is designed to automatically decompose complex problems into manageable sub-tasks and dynamically assemble specialized LLMs or symbolic reasoning tools to solve them, thereby improving both efficiency and accuracy in problem-solving.
weaknesses: ## Weakness

Major Concerns:

1. The whole framework seems like an engineering design, which incorporates adaptive modules and workflows to address some complex reasoning problems. It lacks the detailed technical contributions of a well-established research paper. I suggest the authors provide more explanations on the technical novelty.
2. The authors claim that the framework is novel. However, there exist many previous works, combining fast and slow thinking to solve complex scenarios. Such as "SWIFTSAGE: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks" (it is just one of the examples). Could you please make a comparison with these previous baselines in the experiments? CoT baselines seem a little weak in 2024.

Minor concern:

1. CoT is considered to be fast thinking in this paper. It is quite different from the definitions in other works. Because CoT can also involve deliberate trial and error, or self-reflection. Could you provide some explanations on this point?

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: Minor comments:
1. The last sentence in the second paragraph of the introduction feels awkward.
2. The captions for Tables 1 and 3 mention a Fast/Slow ratio, which is not found in the Tables.
3. The last sentence of the first paragraph in sec 6.3 mentions an interesting finding. This could be further discussed for more insights.
4. There seems to be a contradiction in section 6.4 regarding the reliance on fast thinking, as the statement does not match the results in Figure 5.
soundness: 4
strengths: - The paper is well-written, clearly conveying the core ideas and methodology.
- It presents a comprehensive process, covering theoretical framework, data synthesis, fine-tuning, and evaluation. This entire process provides strong evidence supporting the superiorty of HDFlow compared to existing methods.
summary: The paper presents HDFlow, a novel framework designed to enhance complex reasoning in large language models (LLMs) by integrating fast and slow thinking modes. Inspired by dual process theory, HDFlow features two main components: Dynamic Workflow and Hybrid Thinking. Dynamic Workflow breaks down complex problems into sub-tasks, using specialized LLMs and symbolic tools to solve them. Hybrid Thinking adapts between fast and slow reasoning based on task complexity, improving efficiency and accuracy. The authors also developed a large-scale dataset of 27K challenging reasoning problems to train LLMs in these strategies. Experiments on four benchmark datasets show that HDFlow significantly outperforms existing methods like Chain-of-Thought, with Hybrid Thinking achieving the highest accuracy **on three out of four benchmarks**. This approach demonstrates the potential of combining dynamic workflows and hybrid thinking to advance LLMs' problem-solving capabilities.
weaknesses: - In the "Reasoning Problem Synthesis" section, using GPT-4-Turbo with CoT to filter synthesized problems may limit the dataset's ability to enhance slow thinking, as all problems are solvable with GPT-4 + CoT?
- A contamination test is needed to ensure training data differs sufficiently from evaluation datasets. If the result is not promising, please decontaminate your training data.
- The claim that "hybrid thinking achieves the highest overall accuracy" is misleading, as it only tops three out of four benchmarks and does not have the highest average accuracy. This statement should be revised for precision.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: It would be appreciated if solve the questions mentioned in the weaknesses. Besides, there is a question about the workflow design:

**Additional**: Where are the graph-related illustrations used in this paper? It is suggested that this missing part be added.
soundness: 3
strengths: **+1.** This paper introduces a reaonable method to enhance LLM reasoning.

**+2.** The experiments show that Hybrid Thinking outperforms Slow Thinking and original LLM baselines (COT).

**+3.** The paper is clearly written and easy to understand.
summary: The paper introduces HDFlow, a framework designed to improve complex reasoning in large language models (LLMs) by adapting task-solving strategies from simple to more complex problems. According to the authors' ablation studies, the system achieved better results compared to the setting without the proposed modules.
weaknesses: **-1.** Although the concept of slow and fast thinking is fancy, the authors did not clearly define what constitutes slow and fast thinking. The proposed method fails to capture the full complexity of human cognition. I suggest either clarifying the related claims or reducing them if they do not strongly align with the method. Simply labeling quick responses as "fast thinking" and more detailed problem-solving as "slow thinking" seems to be an incorrect interpretation of the book [1].

[1] Kahneman, Daniel. Thinking, Fast and Slow. 2017.

Need more reasonable claims and demonstrations to support `To address these limitations, we propose a novel framework for complex reasoning with LLMs that combines fast (System I) and more analytical slow thinking (System II) adaptively, inspired by the dual-process theory of human cognition (Kahneman, 2017).`

**-2.** I suggest that the authors conduct a more careful and comprehensive literature review. Based on the reviewer's experience, several important and key references have been missed (published at least six months prior), such as [2], [3], and [4]. Additionally,  recent [5] provides a useful summary of  (many)  related similar work that the authors could refer to.

[2] A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration.

[3] GPTSwarm: Language Agents as Optimizable Graphs.

[4] XAgent: An Autonomous Agent for Complex Task Solving.

[5] Automated Design of Agentic Systems.

**-3.** I suggest adding more baselines beyond the self-produced ablations. The current experiments are weak and less convincing without at least two additional public-available baselines included.

Meta-review: This paper proposes a new method for LLM reasoning. Specificallly, the authors propose a new approach for slow, deliberate reasoning called Dynamic Workflow, which automatically decomposes complex problems into more manageable sub-tasks and dynamically designs a workflow to assemble specialized LLM or symbolic reasoning tools to solve sub-tasks. Besides, they propose Hybrid Thinking, a general framework that dynamically combines fast and slow thinking based on problem complexity. The reviewers consider that: 1) similar idea has been presented by several previous works; and 2) the proposed method is somehow a bit engineering.

Decision: Reject