Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
Authors: Lakshmi Nair
Ian Trase
J. Mark Kim
Venue: ICML 2025 poster (2025)
Keywords: LLM Agents
AutoML
LLM reasoning
Primary area: deep_learning->large_language_models

Abstract: We present a novel reasoning approach called Flow-of-Options (FoO), designed to address intrinsic biases in Large Language Models (LLMs). Flow-of-Options enables LLMs to systematically explore a diverse range of possibilities in their reasoning, as demonstrated by an FoO-based agentic framework developed for autonomously solving Machine Learning (ML) tasks. FoO enforces diversity in LLM solutions through compressed and interpretable task representations, resulting in improvements of 38.2% - 69.2% on standard data science tasks, and 37.4% - 47.9% on therapeutic chemistry tasks, as compared to state-of-the-art baselines. With an overall operation cost under $1 per task, our framework is well-suited for cost-sensitive applications. Going beyond tabular classification and regression, we show the broader applicability of our FoO-based agentic system to tasks such as reinforcement learning and image generation. Our code is open-sourced at: https://github.com/flagshippioneering/Flow-of-Options.

Flow-of-Options:
Diversified and Improved LLM Reasoning by Thinking Through Options
Lakshmi Nair
lan Trase
J. Mark Kim
Abstract
Update
allee
CODE 'A
We present a novel reasoning approach called Flow-of-Options (FoO), designed address in- trinsic biases in Large Language Models (LLMs). Flow-of-Options enables LLMS to systematically explore a diverse range of possibilities in their rea- soning, as demonstrated by an FoO-based agentic framework developed for autonomously solving Machine Learning (ML) tasks. FoO enforces diversity in LLM solutions through compressed and interpretable task representations, resulting in improvements of 38.2% 69.2% on standard data science tasks, and 37.4% 47.9% on therapeutic chemistry tasks, as compared to state-of-the. baselines. With an overall operation cost under $1 per task, our framework is well-suited for cost-sensitive applications. Going beyond tabular classification and regression, show the broader applicability of our FoO-based agentic system to tasks such as reinforcement learning and image generation. Our code is open-sourced at: ht tps: / //github.. com/ slagshippicneering/
Input task
Planner
Opt Generator
Flow-of Nlow-of-Options
Plan Executor
[Explore options]
[CBR]
Develop
Deplay
Figure 1: Overview of our Flow-of-Option based agentic framework incorporating Case-Basee Reasoning (CBR).
(FoO)'. Given a task, FoO enables LLMs to "think through" the options available for executing each step in the task,
prior to the actual execution. Broadly, FoO is a network data structure that explicitly enumerates options for each step in the task, as nodes in the network. Thus, FoO forces the LLM be aware of, and to explore, a broader spectrum of possibilities for completing the task. We show the practi- cal value of FoO by incorporating it in an agentic framework for automating Machine Learning (ML) tasks (Figure 1).
Recent studies show that existing agentic frameworks, such as AutoGPT (Significant Gravitas, 2023) and LangChain (Chase, 2022), often struggle with ML tasks when compared to the base LLMs (Huang al., 2023). Alternatives involv- ing fine-tuning LLMs (Carta al., 2023; Christianos et al., 2023), although promising, incur significant computational effort making it challenging to adapt to broader tasks the same way that a reasoning strategy like CoT or ToT can.
1. Introduction
Large Language Models (LLMs) have impacted automation on wide spectrum of tasks, motivating a growing body of work in agentic system design, e.g., web-browsing (Yu et al., 2025), idea generation (Li et al., 2024b), and data science (Guo et al., 2024; Grosnit et al., 2024). Other ap- proaches have focused on improving the inherent reasoning capabilities of the LLMs, through Tree-of-Thoughhe (ToT) (CoT) (Wei et al.,
Recent OpenAl models have focused on improving outputs by increasing test-time compute; fine-tuning the LLMs "think deeper" through CoT-style reasoning (OpenAI, 2024). However, LLMs show strong biases towards options favored in their pre- pre-training data when proposing solutions to tasks, thus impacting the agentic systems built on top of them. In Figure 2, we see that zero-shot GPT-4o and ol have a bias towards random forest when using the Scikit-learn (Sklearn) package (Pedregosa et al., 2011). limiting the diversity of the solutions and preventing the LLMS from exploring better options for the task. Our experiments further confirm this
(Yao et al., 2024) or Shaii-of-Thhuseee 2022). This paper seeks to bridge the two areas, exploring how "thought-based" approaches can be extended im- prove agentic systems. Hence, propose Flow-of-Optioos
Flagship Pioneering, Cambridge MA, USA. Correspondence to: Lakshmi Nair <lnair@ blngshippionsering.om..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Similar flow-of-thought, i.c. stream of consciousness (Chafe, 2017)

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
GoT-4o
GPT + Fao
1.
16
Randomfforest
RandomFFoestt
GradeentBoossinng
Random
Rand RandorfFin
RandomForestR
RandomForeste
GradienBBoostingg
Proportion
Decision Treak TreiR
Weighborsc
LogisticR
Decisionreed
6:00
044
3:15
Lineark
SVC
ElasticNle
Lineat
2.10-
SYR
0.-
0.-
Figure 2: Proportion and diversity of LLM outputs for solving therapeutic chemistry tasks using Sklearn. FoO boosts the LLM's output diversity, enabling the LLM to explore a wider range solutions for a task (R regressor, - classifier).
bias across multiple tasks. Given this pre-training bias, the model will never "step out of its comfort zone" without coercion, and propose FoO as new strategy do this.
(MNIST), and in Appendix clustering (unsupervised ML), machine translation, and traveling salesman problem.
A recently proposed approach, DS-Agent (Guo et al., 2024) offers an alternative workaround that uses case-based rea- soning (CBR) (Kolodner, 1992) to incorporate a repository of human-generatee generated solutions within an agentic framework. CBR retrieves past solutions to problems, reuses them for the current problem, and evaluates their effectiveness, re- taining the successful solutions. Through the retrieval and reuse of human insights in DS-Agent, model is forced to explore alternatives, thus overcoming its intrinsic biases. While DS-Agent demonstrates improved performance, the system requires the use of a carefully engineered repository of human insights (expert knowledge obtained from Kaggle) summarizing the steps that worked/failed for collection of problems. This requirement complicates the application of DS-Agent to most tasks, where Kaggle-like, refined, large- scale repository of human insights is usually unavailable.
2. Flow-of-Options
Construction of FoO: Given a task T consisting of exe- cution steps T {81, 82, ..., 8n we sample & options for each step i using an LLM ( po(T, 0 (11111)). We prompt the LLM to emphasize diversity in its generations. By conditioning the LLM outputs on the task, the current step of the task, and the previously generated options, we ensure that the LLM outputs are consistent with the task and the options generated in previous steps of the plan, Given the LLM outputs, Flow-of-Options can be represented as a network J (V, E,r) directed-acyclic graph) of depth n. A node in this network represents option. An edge connects two option nodes, with associated values r (initial- ized to a small fixed value at the beginning). To formulate the network, we first instantiate a dummy root node. Then, beginning at the root, the network is constructed with the options corresponding to each step i of the plan, placed at the corresponding depth Note that LLMs only used generate options, and not to construct the FoO structure. Let (i) = )o (1) (i) denote nodes at depth i. The ..... o
We find that the intrinsic knowledge of L.LMs, a sufficient proxy for human insights, provided that the knowledge can be effectively extracted. This motivates the value of Flow- of-Options, demonstrated through the design of an agentic framework for automated MI. (Figure 3). We empirically demonstrate the benefits of our FoO-based agentic frame- work on 16 typical data science tasks and 20 therapeutic chemistry tasks (across ADME-Tox, drug-drug interaction, drug-target interaction, and chemical bond type prediction). ML. methods designed by our approach achieves the high- est overall rank compared to those designed by existing fate-ooftthearr baselines, with 38.2% 69.2% 9% rank im- provement in typical data science tasks and a 37, 7,44% 47.9% rank improvement in therapeutic chemistry tasks. Per-task deployment and development costs using GPT-4o are each under $1. Unlike the majority of existing work on agentic systems for data science, we further show the broader appli- cability of our work, by successfully deploying our system in domains beyond tabular classification and regression: re- inforcement learning (cartpole balancing), image generation
edges are directed as follows:
= [(u,,, v) u v(ii), E (00 ViE [1,
Where, u denotes options at depth - 1, v denotes options at depth i, and (u, u) represents directed edge from to Visually, our FoO network can be likened to a fully connected neural network, where every option at depth i i- is connected to every option at depth i. Importantly, there are no connections between options within the same depth, i.e., (u E V((), E (()) & F. We update the values the edges by traversing the FoO network.
Traversal of FoO: Given a network F, a walk W represents sequence of options, one from each depth:
(2) W (of ,0 .... 0g (n)) where oz p(
(6)
2

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
DEVELOPNENT
Update Case
Update Values
Planner
FLow-o -p0ttions
Dlee AAH
User inputs task
Expty
CODE/
0000g WAH
Plan Executor
Not-erpty
Option Generator
Generation with Consistency Checker
Case Bank
Retrieve and adapt
DEPLOYMENT
Figure 3: Overview of Flow-of-Options incorporated into an agentic framework for automating ML tasks. When starting with an empty case bank, a task plan and FoO is generated, evaluated and updated in the case bank (development). With a non-empty case bank, the closest case is retrieved and adapted to the new task. Further, either new options are explored (development), or the adapted FoO and task plan are directly used (deployment), based on the user's preference.
Our current implementation naively samples options, and some walks may be repeatedly generated from one iteration to another. In our future work, we will explore alternative in- formed sampling approaches. Given, a walk W ([W) n),, we evaluate the sequence of options for the task to obtain an output value or metric R = S(W). We propagate the value to the edges connecting the options using a max update:
Coseet naaues slvve cako sepggrettt uiiiue Brigeei
spaclping bugg ir
J- Usa Hangn parpiiss sala shows as diruuer genemeed FER
fosturts capt streclune intomuation sheal mesiiiile
en comete tamule compleslt redeecs desrrpio furture affectietyy
stubileds (8d vellig velinatcn acphoah the ling
r(op (1) o ()+1)) max(r(o))) (1) o ( R) of W
of coe
Where r(u, v) represents the value associated with the edge (u, v). Hence, each option in W will have their edge value to the subsequent option updated by the max operator. An illustrative example shown in Figure 5.
5: TTann dee Locaoo raais inddi saing preaecciseed dete
dkts.
108.0
Once the network is constructed and initial values are updated for the edges in F, it would be beneficial to explore alternative combinations of the highest- valued options when generating walks. Prior work has demonstrated the value of beam search in the context of im- proving the output generations of language models (Graves, 2012; Franceschelli & Musolesi, 2024), by maintaining several hypotheses, eventually choosing the one with the highest probability. We follow a similar approach, and gen- erate walks, by sampling options from the highest valued options at every step of the task plan. We refer to b as the beam width. For identifying the top-b options, we compute the highest value associated with each option E Vii), This is the maximum value of all the edges coming into from options in (ii1),
Beam Traversal:
Figure 4: Example of a Flow-of-Options network (k = 2, n = 3). Red arrows denote the edges that are inconsistent as identified by the consistency checker LLM. The best walk in this case follows options + 1 5 - (R 0.899).
When b 1, the walk will sample the highest valued option for each step 8;, resulting in the best sequence of options for solving the task. Setting b = k corresponds to uniform sampling of all options at each step (naive). Intermediate settings explore combinations of high-valued options.
Consistency checking: Conceptually, the walks explore all combinations of options at different steps in the task. How- ever, some combinations can lead inconsistencies in the solution. For instance, if RandomF orest o Regressor is chosen at depth i, subsequent options in the same walk should not reference a different model. To ensure this con- sistency, we pass the walk W through an L-sd- consis- tency checker. If the LLM identifies an inconsistent walk, it dropped (unevaluated) and new walk is re-sampled.
value(v) = max(r(u, v)) Vu Yuc E V(i-1)
Then, the top-b options in ( (Top;(V())) are the op- tions in (ii) that have the highest value. We sample from the top-b options for generating walks.
Wheam (o oz (n)) where of" (V())
3

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
Figure 4 shows an example Flow-of-Options network. The consisteney checker LLM identifies paths from 5 - 4 and 3 + 6 as inconsistent due to the discrepancy in models chosen at these steps. Next, we discuss the benefits of FoO, followed by our FoO-based agentic framework design.
Planner
Option Generator
plan:
Filtered plan:
Options:
Ieport Librarses
1.
Process features
vei [FL,
2.
Select model
(RandosForest
Select rodel
(BFI,GradaentBosttan (GB))
Train model
5.
Print results
Construct Flownof-Options
Repeat for iters
Batch kalks (Traversal)
2.1. Benefits of Flow-of-Options
Propageet De col ucing max updare
Ø
Flow-of- oft-ttff--sossssst is a special case of a directed acyclic graph (DAG) that is similar in structure to fully connected neural network. Why is this formulation useful, as opposed to a typical tree or directed acyclic graph? We discuss this in the context of two existing methods that are closely related: SELA (Chi al., 2024) (using trees) and Data Interpreter (Hong et al., 2024) (using DAGs). In addition to distinguish- ing our work from existing methods, these differences also highlight key challenges of building agentic systems for ML (Hutter et al., 2019), that we seek to alleviate with FoO.
-1.8
-1.0
-1.8
F1
F2
-1.8
.
-2.0
-2,
-1.0
-88
RF
GE
DF
GB
Flye-of-options
Updated Flau-ot-
Plan Executor
Figure 5: Illustrative example of our agentic workflow. 4 + F/2 retains the max value of 0.98 observed on that edge.
SELA (Chi et al., 2024) decomposes tasks with pre-specified steps into tree, combined with Moonte-Carlo Tree Search (MCTS) to find optimal paths. FoO offers two improve- ments over this work: First, by definition, child node in tree can only have one parent node. This eliminates the pos- sibility of exploring different combinations, e.g., of features and models. Converting Figure 4 to a tree, can cause com- binations of nodes, e.g., N2 - N3, to never be explored if only node N1 is a parent of node 33; Second, we argue that MCTS may be an overkill for AutoML-style problems, since it requires significant number of rollouts for conver- gence, that are computationally intensive (requiring code execution). Although SELA proposes modified upper con- fidence bound for trees (UCT) to mitigate this impact, SELA takes the longest among the baselines in our experiments. We believe AuL can be treated differently: A "poor" rollout of a feature F, with ML model Mi does not have to impact FI'S 's value (as with UCT), if better combination of F1 with M2 exists. If we discover once, that F1 and M2 performs well, we can stick to that path with a max update, regardless of whether Fj is (on average) a "good state".
possible combinations of feature engineering techniques with ML models, that could lead to accuracy improvements. Lastly, Data Interpreter does not enforce the exploration of different options for executing sub-tasks, often sticking to RandomForest or XGB models. In contrast, FoO enforces diversity in the options explored, resulting in the discovery of superior solutions. In summary, FoO introduces an im- proved, more specialized, data structure to tackle range of ML. problems. Empirically, we show that our FoO-based system yields benefits over both SELA and Data Interpreter.
3. Agentic Framework Using FoO
An overview of our proposed agentic framework with Flow- of- Options and Case-based Reasoning (CBR) is shown in Figure We first discuss CBR and how it is incorporated with FoO before discussing the framework in more detail.
3.1. Case-Based Reasoning with FoO
Alleviating some of the shortcomings of SELA, Data In- terpreter (Hong et al., 2024) utilizes an LLM to produce a directed acyelic graph (DAG) decomposing task into sub- tasks. Data Interpreter then seeks the most optimal graph based on an output performance measure. FoO improves over this approach in three ways: First, the LLM -eMeeeeeratedd graph in Data Interpreter is not guaranteed to be acyclic (as the authors note). In contrast, FoO is constructed without LLMs (only L.M--generate options). Since LLMs are not involved in the construction of the network itself, cycles are explicitly avoided. Secondly, the DAG generated by Data Interpreter, does not enforce the "fully connected" structure of FoO. As a result, Data Interpreter is not guaranteed to explore combinations of nodes at each depth, e.g.. exploring
Prior work has demonstrated the benefits of Case-Based Reasoning (CBR) in enhancing the problem-solving capa- bilities of LLMs, while also improving efficiency in terms of computational resources (Guo et al., 2024). Motivated by these benefits, we further incorporate CBR with Flow-of- Options, by retrieving and reusing previously generated FoO networks. CBR involves a case bank consisting of individ- ual cases c. Specifically, we denote a case c c= (T,F,R) R*), where T denotes the task, F denotes the FoO network gen- erated for the task, and It denotes the best reward achieved with the network for this task. Unlike the long-form descrip- tions of task solutions in DS-Agent (Guo et al., 2024), F represents compressed and relevant information on the case.
For the same task, if a new F with higher reward is discov-
4

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
ered, then the case is updated with the new F and the corre- sponding reward. For new task T', we retrieve the closest case e that maximizes the cosine similarity of its correspond- ing task with T', i.e., c = argmax, s im(EETe). E(T')) Here, Ef-) denotes a pretrained embedding model. For the new task, we reuse the original task plan Te. and the corre- sponding F. This ensures consistency between the retrieved Flow-of- oft-ooon and the task plan, with the implicit assump- tion that similar task plan can be applied to similar tasks. We threshold the retrieval based on the similarity score, to ensure the validity of this assumption. Cases with similarity scores below a threshold will not be reused, and instead a completely new FoO will be developed for the task.
its past explorations. With the updated F, the framework proceeds with walk generations and updates values based on the corresponding code executions. The resultant Fis is then updated in the case bank. We can repeat development on the same task, in which case the previously generated plan and F for the task will be retrieved, allowing generation and exploration of new options. This can lead to the discovery of better options (improvement in accuracy), or retention of the past best performing options (stability in accuracy).
The deployment phase is a computationally efficient phase with low resource requirements, Given a new task, deployment occurs when the case bank has a similar task with a high enough similarity. First, the retrieved task plan and F are adapted to the new task. Then deployment is analogous to development with the following setting: k = 0 (new options are not generated), 6 = J. = 1. and T = 1 (n same as retrieved F). This setting directly samples the best-valued walk, reducing deployment time and cost. The outcomes for the new task are then updated in the case bank.
Deployment:
3.2. Framework
Inspired by prior work (Guo et al., 2024), we incorporate two phases in our framework: Development, and Deploy- ment. Our framework begins with a user input as a string prompt, followed by either development or deployment.
Improving Computational Efficiency: To boost the practi- cal applicability of our framework, particularly in the case of large FoOs, we implement three techniques. First, when the task plan is generated, we use an LLM to filter out the most important subset of n steps that can impact accuracy on the task. Identifying this subset allows us to improve computational efficiency by restricting the depth of F to few, relevant steps in the plan. Additionally, this helps narrow the exploration to the steps in the plan that matter, e.g., the different ways of importing a package is not crit- ical to accuracy, and can be safely ignored. Secondly, we prune low-value options in F to prevent an explosion of the FoO size. While this risks removal of some options from "memory", we find that pruning two options per depth yields good balance between computational efficiency and the exploration of a reasonable number of unique options. Lastly, within each iteration, we parallelize the j walks and their corresponding code executions. This is similar to batch processing of inputs. At the end of the iteration, the values in the batch are then used to update F.
Development takes one of two paths depend- ing on whether the case bank is empty or has data. If the case bank is empty, a task Planner (an LLM), generates a sequence of steps for completing the task. Given the task plan, the Option Generator generates options for each step of the plan. The generated options are then converted into FoO network F as described in Section 2. Once the network is generated, we traverse through the network for T itera- tions, where each iteration uses a fixed beam width that is reduced at later iterations to encourage exploration over the high-valued states (Appendix Figure 15 demonstrates the value of reducing beam width over subsequent iterations). Each iteration performs a set of j walks in batch. At the end of j walks, the Plan Executor converts each walk into code, reflectively debugs any errors, and executes the code to extract the final metric R₃- = fW,)). The metric is prop- agated to the nodes in the walk W, to update F. Once T iterations are complete, F is added into the case bank.
Development:
If the case bank is non-empty, the closest case to the user input is retrieved from the case bank. If the similarity (as described in Section 3.1) is below pre-specified thresh- old, the framework reverts to the empty case bank work- flow. If the similarity exceeds the threshold. the frame- work reuses the corresponding F and task plan for the cur- rent problem. First, an Adapter agent, adapts the F and corresponding task plan to the new problem, e.g., mod- ifying regression - classification, but reusing the same class of model Botaddinnttoostinggeeee.. Gradien Pnddieenoooottoii alssingllsssiiissmis f er) (Appendix Figures 19 - 21 show additional examples of adaptations). Once adapted, the Option Generator generates new options for F in context of the past options in it. This enables the model to continue exploring the space with some "memory" of
4. Experiments and Results
Baseline performance comparison on classification and regression: We evaluate our framework on 16 tasks ob- tained from (Guo et al., 2024). Our baselines include DS- Agent (Guo et al., 2024). AutoGluon (Erickson et al., 2020). SELA (Chi et al., 2024), Data Interpreter (DI) (Hong et al., 2024), Autogen (Wu et al., 2024), and zero-shot with Chain- of of-Thought (CoT) (Wei et al., 2022). We also evaluate on 17 ADME-To- Tox tasks using Therapeutic Data Commons (TDC) (Huang et al., 2021). We exclude DS- DS-Agent from TDC tasks, since DS-Agent requires a repository of human in- sights (similar to Kaggle) which currently unavailable
5

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
Development
Deployment
Avg.
WB
MC
ES
EC
AR
ST
1LI
88
MH
W
SD
CA
CS
HB
WR
Rank
(4)
(4)
(1)
(t)
(L)
(1)
(4)
(1)
(C)
(L)
(1)
(4)
(4)
(4)
(1)
(t)
(J)
DS-Agent
304
0.30
0.40
0.27
4.47
0.78
6.49
0.99
0.34
0.34
0.80
0.67
0.73
11.7
0.68
0.65
3.69
AutoGluon
322
0.28
0.61
0.80
0.89
0.54
0.69
1.36
11.4
4.67
SELA
321
0.29
0.71
1.19
0.51
0.72
0.32
0.85
0.81
1.39
11.8
0.75
4.17
DI
314
0.30
0.98
0.43
1,11
0.82
1.05
0.88
0.06
X
0.82
0.98
0.40
9.88
0.76
0.75
2.33
Autogen
309
0.30
0.67
0.37
1.67
0.80
2.86
0.90
0.40
0.52
0.85
0.69
1.38
10.3
0.72
0.83
3.19
Zero-shot
263
0.26
0.80
0.35
1.91
0.78
5.19
0.83
0.50
0.38
0.81
0.79
1.16
10.0
0.72
0.83
3.19
Ours
182
0.18
0.98
0.43
1.59
0.82
1.53
0.99
0.29
0.36
0.98
0.67
0.73
9.18
0.76
0.80
1.44
Table 1: Performance of our framework compared to baselines on DS tasks (Task abbr. in Table 5). Results are reported over three independent runs (same prompts and fixed seed for all methods). Each run uses different seed. Development and Deployment applies to DS-Agent and our work. Best results are in bold, with second best underlined. Systems that cannot flexibly handle non-tabular tasks are marked as X denotes that DI failed to generate working code in any of the three runs.
Development
Deployment
Avg.
CW
HH
B1
PG
LI
BB
PP
VD
C2
C3
C2S
C3S
HO
CH
A
hE
DI
Rank
())
(f)
(1)
(1)
(4)
(t)
(1)
(1)
(1)
(1)
(1)
(1)
(7)
(t)
(f)
(*)
(1)
(4)
DeepMol
0.35
0.87
0.50
0.82
0.69
0.72
8.34
0.33
0.20
0.64
0.38
0.63
0.49
0.14
0.65
0.76
0.79
2.35
Autogen
0.42
0.77
0.50
0.86
0.77
0.73
10.3
0.43
0.36
0.63
0.42
0.59
0.32
0.28
0.76
0.64
0.72
2.71
Zero-shot
0.44
0.81
0.56
0.81
0.76
0.77
11.2
0.29
0.34
0.64
0.39
0.59
0.28
0.32
0.71
0.69
0.72
2.82
Ours
0.34
0.91
0.58
0.89
0.75
0.78
9.51
0.52
0.57
0.80
0.62
0.62
0.26
0.32
0.76
0.68
0.84
1.47
Table 2: Performance of our FoO-based framework compared baselines on TDC ADME-Tox tasks (Task abbr. in Table 6). Development is performed over Absorption and Distribution tasks. Deployment is performed on Metabolism, Excretion, and Toxicity tasks. Results are reported over three independent runs. Best results are in bold, with second best underlined.
for TDC. Without these insights DS-Agent performs poorly (Guo et al., 2024). Similarly, we exclude AutoGluon from TDC tasks (requiring use of packages like RDKit), and lan- guage model tasks, since AutoGluon cannot handle them. We also exclude SELA from forecasting and TDC tasks and DI from TDC tasks, as they are not flexibly supported in their MetaGPT implementations (DeepWisdom, 2024). For TDC, we compare against DeepMol (Correia et al., 2024), an AutoML approach that is specialized for ADME-Tox.
eterization for AutoGluon. We run SELA for 5 rollouts and ASen for development iterations (deployment direct). We run DeepMol for trials?. For development, we use T 5 iterations, j 3 (walks per batch), k 4 (DS) or k = 3 (TDC), and filter tasks to use n = 2 (DS) or n = 3 (TDC). For Tables 1 and 2, we start with an empty case bank, and disable CBR for all development tasks. FoO for each development task is then added to the case bank together at the end. For deployment, we enable CBR - FoO are retrieved and reused from the case bank (no new options explored). Please refer to Appendix A for task descriptions.
Scaling to computationally intensive scenarios: We eval- uate our scalability via drug-drug combination, drug-target interaction prediction (TDC), and chemical bond prediction (Loschen, 2018) with = 100x data than the previous tasks.
4.1. General Data Science (DS) Tasks
Generalizing beyond tabular classification and regres- sion: We evaluate our work on: a) reinforcement learning (RL) task of cartpole balancing, and b) image generation using MNIST. In the context of these tasks, we also inves- tigate the capabilities of the Adapter LLM to adapt prior FoOs on the same task to atypical instantiations of the tasks.
We show results for 16 data science tasks in Table (arrows show whether lower (L) or higher (f) metric preferred), For DS-Agent and our framework, tasks, and deploy on the rest. We see that MI. approaches designed by our framework outperforms the baselines with an average rank of 1.44 (best possible rank is 1.0, and worst is 7.0). a 38.2% to 69.2% improvement in ranks compared to baselines. All approaches, except DI, succeeded in 100% of the cases. Our approach produces high-performing solutions in 15/16 tasks (except AR). We see the accuracy benefits of FoO even in the absence of CBR, in the development phase.
develop on 7/16
Similar to AS-Agent, we retain a separate Derain. with test- ing on Dtest- In all cases, we use GPTT-4o (Hurst et al., 2024) as the foundational LLM. For Autogen, we follow the offi- cial documentation to construct system consisting of two LLM agents: one that produces a task plan and code, and another that critiques the output suggest improvements ("reflection") (Microsoft, 2024). We use the default param-
*Typically run for Rs 100 trials for leaderboards, but we use the same number of trials for all approaches for fair comparison.
6

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
Metric Spaceshi--Tianic task
Average Cost (S)
Average Time (mins)
SELA
0.17
21.01
0.80
DS-Agent (Develop)
1.53
12.03
0.78
DS-Agent (Deploy)
0.06
0.74
Data-Interpreter
0.2
3.39
0.74
Autagen
Ours (Develop)
0.62
13.29
Autetian
Ours (Deploy)
0.03
0.91
0.22
0.525
Table 3: Resource comparisons between LLM-based agentic systems. Autogen and zero-shot consume low time and cost. Our framework has negligible deployment time and cost.
0.500
Matric on HiAaHou task
31.1% 16
Foo exploration
27
Planner 7.9%
Consistency Checker
2.8%
Dus
Zero shot
Autogen
Plan Filter) 9.4%
Deophrol
Beeation
Figure 6: Our approach shows the capacity to improve. DeepMol failed in the first three iterations (returning -inf) and Data- Interpreter failed one iteration (Iteration 2).
48.8%
Option Generator
Figure 7: Ablation of total cost. Percentage of total cost con-
tributed by each component in our FoO-based framework.
We note that Data Interpreter (DI), while competitive, ex- hibits two key disadvantages compared our work: a) DI failed to produce code in 1/3 runs for 5/16 tasks; b) DI uses highly specific, hand-crafted prompts for the tasks (Ap- pendix M) compared to the general guidance provided to our FoO-based system (Appendix N). The specificity of the prompts may, at least in part, contribute to the competitive- ness of DI. Despite the more general prompts, FoO enables our approach to outperform the baselines. It is also worth noting that in spite of explicitly specifying several models for tabular tasks in the prompt, DI almost always used XGB or RandomForest only (See Appendix Figure 17) indicating the potential value of enumerating options in FoO form.
new options in iterations 0-3, 3, the agent finds improved out- comes. In iteration 4, it fails to find better options, achieving stability by retaining the best past options it has found.
Resource costs for our framework on DS tasks are in Table 3. Although costs vary, they are generally under $1. Since the repetitive LLM querying from prior work (Guo et al., 2024; Wu et al., 2024) is replaced with some non-LLM operations on FoO networks, the costs are low. Although our approach takes longer develop, it quick once initial case bank is produced. SELA took the longest time compared to other baselines, possibly due to expensive MCTS rollouts. In Figure 7, we measure the percentage of total cost contributed by each component in our FoO-based framework. Majority of the total cost is attributed to the option generator and FoO exploration, which entails generating and debugging code using the LLM,, for given sequence of options (FoO walk).
We demonstrate the capacity of our approach to improve on task, by repeating development on the same task (as described in Section 3.2). Here, we run development repeat- edly on the same task with T' = 1 for five separate runs, saving the FoO at each run into the case bank. At each run, the past FoO is retrieved and new options are explored (we prune two options). We also run the remaining approaches for five independent runs. At each iteration I, DS-Agent reflects on past code from iteration t In Figure 6, Auto- gen, SELA, DI, and zero-shot performances fluctuate (since they do not reuse past experiences, leading to randomness). While DS-Agent generally improves in performance, there are cases where the agent's "reflection" results in worse outcomes (e.g.. in iteration 2). Our framework reuses the past Flow-of- Opttino network, which includes the best so- lution found thus far (nodes with highest values). Exploring
4.2. TDC Tasks ADME-ToX
Results are shown in Table Our framework outperforms the baselines, achieving an average rank of 1.47 (37.4% to 47.9% improvement over baselines), consistently producing high-performing solutions (15/17 tasks except HO and hE).
1.2.1. COMPUTATIONALLY INTENSE TASKS
Our FoO-based agentic framework involves the construction and traversal of FoO networks. In the event that the training data and models are large, it would be beneficial to scale
7

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
Task
Zero-shot
Autogen
Ours
Ze dero-shet
Pandtont oreat
Rerdienfoven
DC (TDC) (L)
16.61
13.88
11.85
DTI (TDC) (t)
0.18
0.32
0.42
CB (T)
0.996
0.998
0.998
Avg. Rank (1)
2.0
1.0
Cradenndbutting
Decissorrorr
Table 4: Performance of our FoO-based agentic system
on computationally intensive therapeutic chemistry tasks (development). Results are reported on three runs.
Ours
Rondom
Dediannhee
our approach to more computationally intensive scenarios. We present one potential strategy that uses coreset selection (Guo et al., 2022) to overcome this problem. Coreset selec- tion involves selecting subset of the data that reasonably re- flects performance on the full training data. We demonstrate the performance of this strategy on tasks with about x00x more data than the previous sections. We use a simple data selection strategy that applies stratified sampling on binned label values, although more sophisticated approaches could be applied here (Guo et al., 2022). We apply development with our framework the reduced subset of data 1/50th of the original dataset size), that took = 15.8 mins. Once the best performing code is generated, we apply it the complete training set and report performances on the test set. Our results are shown in Table 4. MI. approaches de- signed by our framework performs well across the three tasks, outperforming the baselines. Additionally, our ap- proach explores wider range of possibilities, as shown by the diversity of options in Figure 8, highlighting key benefit of FoO. ML approaches designed by our framework achieves 80% 90% of the performance of human- designed approaches in 12/19 TDC tasks (Appendix Figure 14).
Figure 8: Proportion and diversity of ML models explored by the different Al agents across tasks from Section 4.2.1.
Autrggnn
Zero-shot
200
Noation
190
100
Epicodes
Figure 9: Reward comparisons for the different methods. Internal: Cartpole positions of our method for the left- constrained variant compared to the original problem.
4.3. Beyond Tabular Classification and Regression
In contrast to existing AutoML frameworks, we demonstrate our approach beyond tabular classification and regression tasks. Our basclines are Autogen and zero-shot with CoT owing to their flexibility handling wider range of tasks.
the original problem, successfully adapted with a reward modification, and deployed to the new, constrained problem. We compare the poses of the model outputs for the original and modified problems in Figure 9. We see that the adapted solution of modifying the REINFORCE reward function performs well, underscoring the capacity of our framework to effectively adapt to novel variants of past tasks.
Reinforcement Learning (Cartpole Balancing): We show results of our approach on the classic cartpole balancing problem from OpenAl Gym (Brockman, 2016) in Figure 9, compared to baselines. We see a distinct performance difference between our approach and baselines on this task. Our framework selects the more sophisticated REINFORCE algorithm (Zhang et al., 2021) compared to either using a simple Q-table (Autogen) or random policy (zero-shot).
Synthetic Image Generation using MNIST data: We show results of our framework on synthesis MNIST im- ages in Figure 10, with SSIM (Structural Similarity Index Metric) comparing our framework the baselines. We measure SSIM between the original MNIST dataset and the generated images as measure of similarity of the synthetic generations to the original data. Higher SSIM is preferred. We see a marked improvement in the quality the synthetic generations for the approach developed by our framework, both quantitatively (in terms of SSIM), and qualitatively.
We further evaluate the capacity of our framework to adapt (via the Adapter) to novel, uncanonical variants of the task, as inspired by prior work (Lake et al., 2017). Specifically, we added the constraint of staying within the left zone of the arena. Our prompts for this task is shown in Appendix O. Appendix Figure 21 shows how the FoO developed on
8

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
0.40
Measured
0.25
3
6 67 11493
0.30
9 6
29
2940
Curs: MISST generations uncer constraint
FD 1
4
T-11.08
thight
0.15
1
T-0.89
T-2.68
k- number of options
0.05
00
Autogen
Figure 10: Output generations and SSIM values for the different methods, including output of method for the color-constrained variant of the problem (right).
T-4.84
T=0.68
T=1.54
Zero-shot generation uses a Generative Adversarial Network and Autogen uses a convolutional (VAE). Our framework evaluates two different architectures for each in terms of activation functions.
2
T=0.44
T=0.87
T-1.32
number of steps
We also evaluated our framework with the constraint that the generated digits should be red. The adapter proposes adding a post-processing step to the generated images to modify the channel data to generate red images (Appendix Figure 22). The results are shown to the right of Figure 10. Alongside image generation and RL, we demonstrate the application of our work to range of additional tasks such as clustering and machine translation in Appendix B.
Figure 11: Scaling behavior of FoO. Measured time taken (I' in mins) vs. expected time for FoO exploration.
et al., 2024; Grosnit al., 2024). Prior work has looked at Graph of Thoughts (Besta et al., 2024) to tackle reasoning problems (different from AutoML), but do not enforce the notion of "options" option diversity. For the most closely related works, SELA (Chi et al., 2024) and Data Interpreter (Hong et al., 2024), we discussed data structure differences in Section 2.1. We highlight some additional framework- level differences here. In contrast to our work, SELA (Chi et al., 2024) predefines steps in the pipeline within the LLM prompt, e.g., data analysis, feature engineering, and model selection. Data Interpreter predefines set of "task types" with a detailed set of instructions for each task type. This ranges from specific prompts for "feature engineering" tasks and "model training" tasks, to highly specific prompts for conversion" tasks. This limits their applica- bility to broader problems. Additionally, SELA and Data Interpreter do not leverage past experiences for new tasks.
5. On the Scalability of Flow-of-Options
In section 3.2, we noted parallclization and pruning as two strategies for improving the computational efficiency of FoO. Further, we note that the consistency checker implicitly prunes paths that are invalid in the FoO (as shown in Figure 4). As n increases, the proportion of invalid paths is also higher. Empirically, we found that the consistency checker leads to R 22.8% reduction in the number paths explored. Hence, in practice, the computational complexity of FoO is often lesser than the theoretical complexity of (I+)), where is the number of options, and is the depth of the FoO. In Figure 11, empirically measured the time taken k (averaged across three runs the CW task from Table 2). From 7 1, A 3 to 3, k 3, the total number of paths is 9x higher. However, the time taken scales by R 7x (this includes parallelization and consistency checking, but excludes pruning which can further improve the efficiency). Moreover, the consistency checker's scaling benefit comes low dollar cost (2.8% of the total cost in Figure 7).
"image-to-
and
For therapeutic tasks, prior work has introduced DrugAgent (Liu et al., 2024). Similar in spirit to our work, DrugAgent explores an "idea space" of different models but specifi- cally designed for the task drug discovery only, and does not leverage structured thinking like FoO. In contrast, Flow- of Options offers improved flexibility and performance.
A Acknowledgements
6. Related Work
We would like to thank Mickey Atwal, Armen Mkrtchyan, and our Pioneering Intelligence teammates at Flagship Pio- necring for their valuable feedback. We also thank our team- mate Drew Dresser for reviewing our open-source code.
Existing agentic designs, including recent frameworks over- fit to a narrow spectrum of tasks data science by assuming an overall workflow (Guo al., 2024; Wu al., 2024; Baek

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
Impact Statement
Christianos, F,, Papoudakis, G., Zimmer, M., Coste, T., Wu, Z., Chen, J., Khandelwal, K., Doran, J., Feng. X., Liu, J., et al. Pangu-agent: A fine-tunable generalist agent with structured reasoning, arXiv preprint arXiv:2312.14878, 2023.
Our paper seeks to advance the field of Machine Learning. Our work is intended assist (not replace) ML scientists and data scientists that use LLMs in their work, by improv- ing the quality and value of LLM-generated outputs to user prompts. Complete autonomous usage of the system with- out any human supervision can pose critical security and privacy issues. Hence, precautions must be taken regard- ing the execution of -eMeraaed code, to prevent the automatic generation and execution of potentially malicious software, or the accidental transmission of sensitive data to external servers in the case of API usage. We executed our methods within environments that have the necessary safeguards in place. The use of LLMs that adhere to strict safety and privacy regulations is heavily emphasized.
Correia, J., Capela, J., and Rocha, M. Deepmol: an auto- mated machine and deep learning framework for compu- tational chemistry. Journal of Cheminformatics, 16(1): 1-17, 2024.
DeepWisdom. Metagpt. petps://githie com/
geekan/MetaGPP 2024. Accessed: 2024-12-22.
DeepWisdom. Metagpt. https://github. com/ haakka///eetaG///bbbo///miaa//// prompts/task_type.oo 2025. 2025-01-09.
Accessed:
References
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, I.., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. Prottrans: towards cracking the language of life's code through sule-supe learning. IEEE Transactions on Patlern Analysis and Machine In- telligence, 44:7112-7127, 2021.
Back, J., Jauhar, S. K., Cucerzan, S., and Hwang, S. J. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprint 004702240..07738 2024.
Besta, M.. Blach, N.. Kubicek, A., Gerstenberger, R., Pod- stawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 38, pp. 17682-17690, 2024.
Erickson, N., Mueller, J., Shirkov, A., Zhang, H., Larroy, P., Li, M., and Smola, A. Autogluon-tabular: Robust and accurate AutoML for structured data, arXiv preprint arXi::2003.06500 2020.
Franceschelli, G. and Musolesi, M. Creative beam search. Proceedings of the Fifteenth International Conference on Computational Creativity, pp. 364 -368, 2024.
Brockman, G. Openai gym. 2016.
arXiv preprint
Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., and Oudeyer, P.-Y. Grounding large language models in interactive environments with online reinforcement learn- ing. In International Conference on Machine Learning, pp. 3676-3713. PMLR, 2023.
Graves, A. Sequence transduction with recurrent neural networks. arXiv preprint arXiv:1211.37111 2012.
Grosnit, A., Maraval, A., Doran, J., Paolo, G., Thomas, A., Beevi, R. S. H. N., Gonzalez, J., Khandelwal, K., la- cobacci, I., Benechehab, A., et al. Large language models orchestrating structured reasoning achieve kaggle grand- master level. arXiv preprint arXiv:2411.03562 2024.
Chae, H., Kim, Y., Kim, S., Ong, K. T.-i., Kwak, B.-w., Kim, M., Kim, S., Kwon, T., Chung, J., Yu, Y., and Yeo, J. Language models as compilers: Simulating pseu- docode execution improves algorithmic reasoning in lan- guage models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 22471-22502. November 2024.
Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., et al. A survey on LLM-as-a- judge. arXiv preprint arXiv:241 15594, 2024.
Guo, C., Zhao, B., and Bai, Y. Deepcore: A comprehensive library for coreset selection in deep learning. In Inter- national Conference on Database and Expert Systems Applications, pp. 181-195. Springer, 2022.
Chafe, W. Language and the flow of thought. In The psychology of language, pp. 93-111. Routledge, 2017.
Chase, H. LangChain, October 2022. URL https:// github. com/ wwwp:/aanahaaaaaaaaaaaaaaa aaw.aaiaiiaagaain..nn
Guo, S., Deng, C., Wen, Y., Chen, H., Chang, Y., and Wang, J. DS-agent: Automated data science by empowering large language models with case- case-based reasoning. In Pro- ceedings of the 41st International Conference on Machine Learning, pp. 16813 18813-16448, 2024.
Chi, Y., Lin, Y., Hong, S., Pan, D., Fei, Y., Mei, G., Liu, B., Pang, T., Kwok, J., Zhang, C., et al. Sela: Tree-search enhanced LLM agents for automated machine learning. arXiv preprint 0rXiv:2410.122.. 2024.
10

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
Hong, S., Lin, Y., Liu, B., Liu, B., Wu, B., Zhang, C., Wei, C., Li, D., Chen, J., Zhang, J., al. Data inter- preter: An LLM agent for data science. arXiv preprint arXiv:2402.1867,, 2024.
Microsoft. LLM reflection. https://microsoft,
github httpb//o//uuto//////// prompting and -ndeeeiinnnnnn--------- reassnn/g//eelllltttioo.. 2024. Accessed: 2024-04-12.
Huang, K., Fu, T., Gao, W., Zhao, Y., Roohani, Y., Leskovec, J., Coley, C. W., Xiao, C., Sun, J., and Zitnik, M. Ther- apeutics data commons: Machine learning datasets and tasks for drug discovery and development. Proceed- ings of Neural Information Processing Systems, NeurIPS Datasets and Benchmarks, 2021.
OpenAL. LLMs.
Learning to reason with bttps://opente com/ index/ Accessed: 2024-11-26.
2024.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V.. Vanderplas, J., Passos, A., Cour- napeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-leam: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2011.
Huang, Q. Vora, J., Liang, P., and Leskovec, J. Bench- marking large language models as Al research agents. In NeuriPS 2023 Foundation Models for Decision Making Workshop, 2023.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. GPT-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Raieli, S. Clustering techniques with gene expres- sion data for acute myeloid leukemia. https: / //medium. peeedumm.cm//Lukkeaaaaeeeeeerr c expression-dant data-aaaaaaa--aa (ar.-oo-ddaa-4335504550 4b3500488755, 2024. Ac- cessed: 2024-12-16.
Hutter, F., Kotthoff, L., and Vanschoren, Automated ma- chine learning: methods, systems, challenges. Springer Nature, 2019.
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Out, M., Zitnick, C. L.., Ma, J., et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, (irrnness,11(115555 118(15)ee216223115 2021.
Manczys-Dowmud M., Grundkiewicz, R., Dwojak, T., Hoang, H., Heafield, K., Neckermann, T., Seide, F., Ger- mann, U., Aji, A. F. Bogoychev, N., et al. Marian: Fast neural machine translation in C++. arXiv preprint arXiv: TEt.:1800.0034,, 2018.
Kolodner, L. An introduction case-based reasoning. Artificial intelligence review, 6(1):3-34, 1992.
Roberts, A., Raffel, C., Lee, K., Matena, M., Shazeer, N., Liu, J., Narang, S., Li, W., and Zhou, Y. Exploring the limits of transfer learning with unified text-to-text transformer. Google, Tech. Rep., 2019.
Lake, B. M., Ullman, D., Tenenbaum, J. B., and Gersh- man, S. J. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
Shmoys, D. B. and Williamson, D. P. Analyzing the held- karp tsp bound: A monotonicity property with application. Information Processing Letters, 35(6):281-285, 1990.
Li, C., Liang, J., Zeng, A., Chen, X., Hausman, K., Sadigh, D., Levine, S., Fei-Fei, L., Xia, F., and Ichter, B. Chain of code: Reasoning with language model--ugmeeted code emulator. In Proceedings of the 41 International Con- ference Machine Leurning, pp. 28259-28277, 2024a.
Significant Gravitas. AutoGPT, 2023. URL https:
//github. www//iigifiiaaaaaaaacaaaimaaaaaa Gravitas/ AutoGPT.
Li, L., Xu, W., Guo, J., Zhao, R., Li, X., Yuan, Y., Zhang, B., Jiang, Y., Xin, Y., Dang, R., et al. Chain ideas: Revolutionizing research via novel idea development with LLM agents. arXiv preprint arXiv:2410.133 13185, 2024b.
Tiedemann, J.. Aulamo, M.. Bakshandaeva, D., Boggia, M., Grönroos, S.-A., Nieminen, T., Raganato, A., Scher- rer, Y., Vazquez, R., and Virpioja, S. Democratiz- ing machine translation with Opus-mt. arXiv preprint arXiv:2212.01936, 2022.
Liu, S., Lu, Y.. Chen, S., Hu, X., Zhao, J., Fu, T., and Zhao, Y. Drugagent: Automating ai-aided drug discovery programming through LLM multi-agent collaboration. arXiv preprint 1.XXv:2411.15600 2024.
Varnat-Herresthal S., Perrakis, K., Taschler, B., Becker, M., Baßler, K., Beyer, M., Günther, P., Schutte-Screepping J., Seep, L.,, Klee, K., et al. Scalable prediction acute myeloid leukemia using high- high-dimensionall machine leam- ing and blood transcriptomics. Iscience, 23(1), 2020.
Loschen, C. Perception of chemical bonds via machine learning. Chemrxiv, 2018

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F,, Chi, E., Le, Q. V., Zhou, D., al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35; 24824-24837 2022.
Wu, Q. Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E.,, Li, B., Jiang, L., Zhang, X., and Wang, C. Autogen: Enabling next-gen LLM applications via multi-agent conversation framework. First Conference on Language Modeling, 2024.
Yao, S., Yu, D., Zhao, J., Shafran, L. Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Yu, X., Peng. B., Vajipey, V., Cheng, H., Galley, M., Gao, J., and Yu, Z. Exact: Teaching Al agents to explore with reflective-mcts and exploratory learning. Proceed- ings of the 13th International Conference on Learning Representations, 2025.
Zhang, J., Kim, J., O' Donoghue, B., and Boyd, S. Sample efficient reinforcement learning with reinforce. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 35, pp. 10887-10895, 2021.
12

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
Appendix Contents
Page No.
A
Task descriptions and hyperparameters
14
Tasks beyond classification and regression
14
B.1 Unsupervised learning clustering
14
B.2 Machine translation
14
B.3 Traveling salesman problem
15
B.4 Case study with non-code generation problem
15
Flow-of-Options with additional LLMs
16
Comparison our framework human generated code baselines
16
Why beam traversal helpful?
17
Additional component ablation study
17
Future work
17
G.1 Tool-use with Flow-of-Options
17
G.2 Nested Flow-of-Options
17
H Relative advantages and disadvantages of the different baselines
18
I Limitations of the overall framework
19
L.1 LLM-module specific challenges
19
J Code generated for the math case study
20
K Additional Flow-of-Options network examples
21
Task adaptation using LLM adapter
21
M Data interpreter prompts
24
LLM prompts used in our framework
25
User inputs to our framework
28
13

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
A. Task descriptions and hyperparameters
B. Tasks beyond classification and regression
We investigate the application of Flow-of-Options to tasks beyond classification and regression, namely: clustering, machine translation, traveling salesman problem, and case- study FoO applied to non-code generation task, L.e., mathematical reasoning problem.
Dataset
Abbr.
Task
Metric
Wild-blucberry-yiiie
WB
Regression
MAE
Mediaa-ampaign-oon
MC
Regression
RMLSE
Fazyme-substrate
ES
Classification
AUROC
Eihanol-concentration
EC
Classification
Accuracy
Airline-reviews
AR
Regression
RMSE
Spaceship-titnnic
ST
Classification
Accuracy
Expression
ILI
Forecasting
MSE
Smoker-status
SS
Classification
AUROC
Mohs-hardness
MH
Regression
MedAE
Weather
W
Forecasting
MSE
20-
Software-defects
SD
Classification
AUROC
Jigsaw
Regression
RMSE
PCX Componen
Crab-age
CA
Regression
MAE
Concrete-strength
CS
Regression
RMSE
Heartbeat
HB
Classification
Accuracy
WebMD-revvews
WR
Classification
Accuracy
Table 5: Tasks and corresponding abbreviations for DS- Agent tasks along with their metries.
Dataset
Abbr.
Task
Metric
PCA component
Taco2-Wang
CW
Regression
MAE
Figure 12: Visualization of clusters for the AMI gene ex- pression data generated by our system.
H[A-Hou
HH
Classification
AUROC
Pgp-broccatelli
PG
Classification
AUROC
Bioavailability
Classification
AUROC
Lipophilicity
LI
Regression
MAB
B.1. Unsupervised learning Clustering
BRB-Martinz
Classification
AUROC
PPBR A7.
PP
Regression
MAR
We applied framework the problem of clustering gene expression data Acute Myeloid Leukemia (AML) Herrest-Hrresthhl et al., 2020). We pre-specified the num- ber clusters to be in the input user prompt (See Ap- pendix O). We follow an existing tutorial for pre-processing the data (Raieli, 2024) provided to our approach. The plan- ner produces task plan with the following filtered steps (recall that approach filters the steps in the task plan to the most relevant steps): 1. Pre-process the data if neces- sary, ensuring that it is in suitable format for clustering; 2. Select an appropriate clustering algorithm from the Sklearn library, such as KMeans, since the number of clusters is pre- defined; Fit the clustering algorithm to the gene expres- sion data perform the clustering. The resultant clustering output for the model chosen by our framework is in Figure 12. Our framework explores different options, such as scal- ing the data before clustering, followed by agglomerative clustering or k-means clustering, The final method chosen consists of min-max scaling of the data and k++ centroid initialization with k-means clustering to yield a Silhouctte index of 0.118.
VDss Lombardo
Regression
Spearman
CYP2D6 Veith
C2
Classification
AUPRC
CYP3A4 Veith
C3
Classification
AUPRC
CYP2D6 Substrate
C2S
Classification
AUPRC
CYP3A4 Substrate
C3S
Classification
AUPRC
Half Life Obach
HO
Regression
Spearman
Clearance Hepatocyte
CH
Regression
Spearman
AMES
Classification
AUROC
hERG
hE
Classification
AUROC
DILI
Classification
AUROC
Drug-drug combination
DC
Regression
MAE
Drug-target interaction
DTI
Conelattinn
Pearson
Com
Bond Prediction
Classification
Accuracy
Table 6: Tasks and corresponding abbreviations for TDC ADMET and additional therapeutic chemistry tasks.
T (iters)
Task
(FoO depth)
(num options)
(walks/batch)
RL
Image Generation
Clustering
3
Machine Translation
3
3
3
Triveling
B.2. Machine translation
Table 7: Hyperparameter settings of our framework.
We applied our framework to the problem of translating from English to French. We found that our framework ran
14

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
Rouge-1
Rouge-2
Rouge-L
Our approach
0.55
0.37
0.52
Table 8: Rouge scores of the ML model generated by our
subotitute the values a. and into the quadratic syrtil esprecsion under equare root convwrting d Meuumes Theorem find not
framework for translating from English to French.
calouate the sausee of 41 polor ferm, then conver rectanguda into quadiatic tammul findt heo solutions for by
negative
Square root
Zero-shot CoT
Autogen
Ours 3/3
use properties comples contugatess deternise the parts sakation since confiicient quadrase equation are real, the sauriens co comples for both solutions.
Math problem
0/3
0/3
Table 9: Success rate on three runs of the math problem case study. Our approach succeeds in 100% of the runs.
TM
Figure 13: FoO generated for the math case study. The hyperparameter settings produce a single chain of reasoning.
into issues with the download and use of Opus- 100 dataset using HuggingFace API. Hence, we passed template in our prompt to enable to model to correctly load and use the dataset API. Once provided with the API, our framework proposes the fine-tuning of various pre-trained models, such as T5 (Roberts et al., 2019). MarianMT (Junezys-Dommunt et al., 2018), and Opus-mt (Tiedemann et al., 2022) models, available on HuggingFace. The finalized approach fine- tunes b5-base from HuggingFace on Opus- 100 for English to French translation, achieving the Rouge scores reported in Table 8. We note that besides tokenization, there is no fea- ture engineering involved in this task. Our Planner accounts for this specificity and generates an uppropriate task plan focused on model selection, while the Option Generator produces appropriate options for the model choice.
B.4. Case study with non-code generation problem
We performed a small case study on applying our FoO-based agentic system on a math problem. The prompt provided is: "The equation x:² + 2x = i has two complex solutions. Determine the product of their real parts". This case is in- teresting for two reasons: 1. doesn't 't necessarily involve the notion of "options" (e.g., multiple ML approaches or feature engineering), and 2. the correct evaluation metric to use (metric R used to evaluate the walks is unclear). Al- though, existing approaches have used LLMs for scoring and evaluation (Gu et al., 2024), we circumvent the need for an evaluator for the purposes of this case study, by setting j -= 1, n = 3, k 1 (number of options), and T setting results in a reasoning chain analogous to COT. We find that our approach is able to produce the correct solution. The filtered task plan includes: 1. Substitute the values of a, b, and e into the quadratic formula; 2. Substitute the value of the square rool back into the quadratic formula to find the two solutions for x; 3. Calculate the real parts of the two solutions obtained from the quadratic formula. The corresponding FoO is shown in Figure 13. The model correctly produces the solution of -0.207 ((1 v(2)) /2). We repeated this experiment for three runs by disabling case- base- reasoning. Hence, FoOs were not saved nor re- trieved. All three runs are therefore independent. We also apply Autogen and zero-shot for this task, across three inde- pendent runs. Table shows the success rates of the three methods on the task, across the three runs. We note that both Autogen and zero-shot failed produce the correct result in any of the three runs, often producing 0.0 or 1.0 as the result. We note that our approach produced "code" (the generated code is shown in J), whereas the other two approaches produced written out formulae. For instance, our approach used the cmath package. whereas Autogen and zero-shot explicitly described the formulae for some of
This
B.3. Traveling salesman problem
We applied our framework to the optimization task of solv- ing the traveling salesman problem (TSP). For this experi- ment, we fixed a seed and generated a random distance ma- trix for the TSP with 10 cities. We note that unlike typical ML problems, there is no model selection or feature engi- neering involved here. Our planner produces a task plan that accounts for the nature of this task and filters the following step for option generation: Implement a method to find the optimal path that visits each city exactly once and returns to the starting city with the minimum total distance. Our framework explored different options for this step, including a brute-force method, genetic algorithm, and dynamic pro- gramming using HelldKKarp (Shmoys& & Williamson, 1990). We evaluated the paths in the FoO (the metric R) using time taken solve the task preferring lower metrics. Based on this metric, our framework chose the Held-Karp Karp dynamic programming method, which completed in Rv 0.16 seconds, compared to > 5 seconds with the other approaches. This task in particular demonstrates the potential applicability of our FoO-based framework to tasks beyond data science, provided scoring metric is available.
15

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
CW ())
HH (1)
PG (1)
BI (1)
BB (1)
PP (L)
GPT-3.5 (FoO)
0.43
0.76
0.85
0.57
0.69
10.8
GPT-3.5 (Zero-shot)
0.62
0.82
0.48
GPT-3.5 (Autogen)
0.43
0.64
0.82
0.50
0.61
11.0
GPT-4o (FoO)
0.34
0.91
0.89
0.58
0.78
9.51
GPT-4o (Zero-shot)
0.42
0.77
0.86
0.56
0.77
11.2
GPT-4o (Autogen)
0.44
0.81
0.81
0.50
0.73
10.3
1erp:-5-Coode--22-IInttte (FoO)
0.33
0.89
0.88
0.58
0.72
10.2
Wween25--odde-32BB-nmsss (Zero-shot)
0.40
0.77
0.86
0.68
10.3
1wen2.55-Coderr322--nsstt (Autogen)
0.41
0.79
0.86
0.70
10.8
DeepMol
0.35
0.87
0.82
0.50
0.70
8.34
Table 10: Results are obtained on a subset of ADME-Tox tasks of TDC, averaged across three runs ("-" indicates that model failed to find working solution within three runs). Qwen is an open-source LLM.
the functions that cmath provides, leading to errors. It is possible that the Option Generator, having been prompted to generate options (albeit one option), was more likely to think in terms of different packages that could be used. The code generation here loosely simulates prior approaches "thinking through code" in LLMs (Li et al., 2024a; Chae et al., 2024), where enumerating code or pseudocode-like steps has resulted in improvements in reasoning tasks. Our future work seeks evaluate this hypothesis more exhaus- tively on range of similar problems.
D. Comparison of our framework to human generated code baselines
For the TDC tasks, we compare the output of our frame- work to the results the TDC leaderboard (Accessed on December 1st, 2024) in Figure 14. We note what percent- age ML approaches generated by our framework achieves compared to the output metric of human-generated baseline code at #1 position on the TDC leaderboard (for ADME- Tox tasks, drug-drug, and drug-target interaction). Figure shows that our approach mostly achieves > 80% of an expert human's performance. In other words, on average, it is comparable (though not always better than) a human expert. In some cases, it outperforms the human baseline. For instance, in the drug combination (DC) task from Table 4, our approach outperforms the human baseline on the TDC leaderboard, helping democratize ML for naive users.
C. FoO with additional LLMs
In this section, we demonstrate the results of using GPT- 3.5 and 1eie25--Coder-222B--1
in our FoO-based agentic framework on a subset of the TDC tasks. We see in Table 10, that GPT-3.5 combined with our FoO-based agentic framework performs better than GPT-3.55 zero-shot and Autogen variants. Additionally, GPT-3.5 with FoO performs comparably to zero-shot GPT-4o. However, we note that code generation was a key challenge with GPT-3.5, leading to failures in some code executions. Most failures resulted from the use of outdated APIs (such as outdated API of RDKit) or incorrect import statements. Apart from the debugging and code generation components, GPT-3.5 was successful at generating the task plan and options, well as consistency checking. In addition to models that are good at code generation and debugging, we note that our framework is most successful with models that are good at following instructions to produce structured output (c.g., output formatted as JSON with a specific set of keys). However, with the pace of research in open-source LLMs that are competitive with closed-source LLMs like GPT-4o, we believe that these requirements will be easily satisfied by open-source LLMs in the future. Overall, FoO consistently improves performance compared to Autogen and zero-shot baselines. FoO also helps mitigate failure rates in the cases of GPT-3.5 and Qwen models.
Performances of our framework human baselines
19.0% 0%
25.0%
Performance
90%
25.00 .0.0%
80- 90%
70%
70%
31.0%
Comparing the accuracy of our framework to human baselines on the TDC leaderboard. The chart shows what percentages of the human baseline code metric, our framework is able to achieve.
Figure 14:
16

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
E. Why is beam traversal helpful?
sesccii tes de buce bese tocle wrick
In our experiments, we reduce the beam width in later it- erations of the development phase of our framework. We find that this annealing of beam width, enables our approach to discover new combinations of the high-valued options, leading to performance improvements. We start with the full beam width 6 = k, and reduce it to b = k/2 at iteration 3. In Figure 15, we see that better solutions are discovered when beam width is halved across two subsequent itera- tions. Even though the FoO structure and options are the same, the reduced beam width encourages exploration of combinations of high-valuee options (e.g., combination of high-valued feature with a high-valued ML model), leading to discovery of improved solutions.
dalee cleiing oiferte Meniln Repecced
Aniast thrt
triee meciee de vetral
Figure 16: Example of FoO generated by incorporating external paper retrieval.
HIA-Hou
Cacož-cell permesbility
0.48
G. Future work
In this section, we discuss two potential avenues for future work with Flow-of-Options: a) Tool-use with FoOs, and b) Nested Flow-of-Options.
0.80
0.02
0:38
0.70
0.38
G.1. Tool use with Flow-of-Options
In our paper, FoO seeks to improve the base reasoning capa- bilitics of the LLMs even in the absence of tools. However, tools can be "force multiplier" with FoO. We conducted a small experiment to illustrate this further. We incorporate external research paper retrieval as a tool with FoO. The tool retrieves and provides two papers context into the Option Generator LLM, for the example task of fine-tuning protein language models. We note the two retrieved papers incorporated into the option generator's context: ProtBERT (Elnaggar et al., 2021) and ESM (Rives et al., 2021). Figure 16 shows the resultant FoO. We see that the nodes I and 2 in the FoO incorporates the information in the papers. Combining external tools with Flow-of-Options can further improve the capacity of the agent to reason about tasks, and we seek to explore this synergistic potential in the future.
Figure 15: Improvement in metrics from iteration 1 to 2 when beam width reduced by 50% in the second iteration.
F. Additional component ablation study
We measured the reduction in execution time when consis- tency checker is added to identify invalid paths (as opposed to w/o it), and planner when the adapter is added (as op- posed to w/o it). We also measure costs for each component when it is added. We measured these on average for tasks from Table 1. In Table 11, we see that both the consistency checker and plan adapter components offer benefits in terms of execution time, at minor additional cost. Without the consistency checker, the framework would explore poten- tially invalid paths, increasing execution costs. Similarly, by adapting the task plan (and options) from previous task to a new task, deployment is faster. Without the adapter, new task plan and new options would have to be explored from scratch for a new task, adding to the execution cost.
G.2. Nested Flow-of-Options
Our current formulation of flow-of-Options follows sin- gle task plan. The task plan constructed by the planner is sufficiently high-level, that conceptually distinct options are supported via the option generator LLM. For instance, in the case of processing SMILES strings, the "feature process- ing" step specification in the task plan is abstract enough to accommodate RDKit-style processing and NLP-style vector embeddings as options, even though they are conceptually different. However, in some cases, a given task can be de- composed in multiple ways resulting in multiple distinct task plans that cannot be captured with a single high-level plan. In our future work, we envision a nested srow-of-Optione to
Consistency Checker
Planner (Adapter)
% reduction Cost ($) 22.8
% reduction
Cost ($)
0.02
93.1
0.05
Table 11: % Reduction in execution time offered by the consistency checker and plan adapter components of our framework and their corresponding dollar costs.
17

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
Autogen
Zero shot
SELA
DS Agent
Data Interpreter
Ours
20-
Random
10
RandomForest
0.8
RandomiFo ForooFoorest
RandomForest
Bandomfarrest
Proportion
XGB
noree mmrorstt
Randem mFrrest
Gradiem vadientooestin
GradientBoostrn
GradientBoosing
Ela Supporteecto
XGB
Logistic
Decision Tree
Catboost
Limeal Decis Decision ceeeeeeeeee
Decisian
Linear
Suppo
XGB
Logistic
Calboest
Logies
XGB
Figure 17: Diversity of explored solutions across LLM- agents for tabular tasks from Table 1 (shown for subset of labular tasks WB, MC, ES, EC, and ST - for visual clarity). Y-axis denotes proportion of explored options across the tasks.
support such cases by implementing the FoO data structure over task plans. In this case, the depth will be n and width will denote the different task decompositions. Fur- ther, each task plan "node" can be itself represented as the FoO described in this work. This would enable the agent to reason about ML strategies, beyond just ML methods.
builds an initial case bank, a user can rely on deployment alone, which is faster. We found that SELA took = 21 mins average across the tabular tasks the largest amount of time taken compared to the other baselines. This was at- tributable to the time taken by the Monte-Carlo Tree Search approach. With cost (Table 3), our framework improves on DS-Agent by replacing repeated L.LM queries by operations on FoO. Lastly, our framework demonstrates the capacity for improvements over time through the incorporation of case-based reasoning with FoO (Figure 6). While DS-Agent also reflections to improve over past iterations, some of the reflections resulted in drops in performance across some iterations. The other LLM-based systems do not exhibit the capacity to improve over time.
H. Relative advantages and the different baselines
of
Autagen
AutoML
SELA
DS-Agent
DI
Ours
Metric Diversity Time
2
1 (Deploy) (Develop)
(Deploy) (Develop)
Cost
2
1
6
Improvement
We note some advantages and disadvantages of the different approaches on some qualitative measures: a) flexibility the ability of the approach to adapt beyond specific set of tasks, e.g., only TDC tabular tasks, b) ease of use ease of using the framework or approach, c) interpretability- ability for user to understand the decision making process of the LLM.
Qualitative:
Overall Rank
2.8
3.6
3.6
3,2
3.0
2.0
Table 12: Ranking relative advantages and disadvantages of the methods. DI refers Data Interpererrr..r..
Quantitative: We note the key advantages and disadvan- tages of the six different systems on the following dimen- sions: a) Metric, i.e. accuracy or error, b) Diversity - di- versity of generated solutions, c) time time taken for the system generate solution, d) cost $ cost of application of the framework, and e) Improvement ability improve over time to derive better solutions as the system gains more experience. Based on our results, we rank the approaches from (best) - (worst) on each metric (Table 12) for a summarized view of the relative advantages of the methods. AutoML refers to DeepMol and AutoGluon.
Existing AutoML approaches overfit to tabular ML tasks by specifying the steps in the pipeline, limiting their flexibility. Data- Pate-Interpreter and SELA make underlying assumptions regarding the overall task workflows (either task types steps in the pipeline), limiting their applicability to broader range of tasks. Neither DS-Agent nor our approach makes similar assumptions. In terms of ease, DS-Agent requires a curated repository of human insights, unlike our framework. However, there are certain hyper-paramene work that may complicate its application for some users (e.g., number of options generate, number of steps in the task plan to filter etc.). SELA, DI, DS-Agent, and our frame- work offer interpretability in the thinking process of the L.Mss. DS-Agent offers interpretability via the reflection logs of the LLMs showing the results of each iteration and the improvements applied in subsequent iterations based on
to our frame-
The key benefits of our approach are in terms of metric, diversity, and improvement. Our approach outperforms the baselines in terms of the output solutions (Tables 1, 2, 4), and also exhibits greater diversity in generated solutions compared to other -ased agents (Figures 8, 17). In terms of time (Table 3). development takes longer both with DS-Agent and our approach. However, once the system
18

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
the previous result. SELA, DI, and our framework offers interpretability through the underlying data structures (tree, DAG, and FoO).
retrieve the best case for a new task, instead retrieving alternative case that is not well suited to the cur- rent task. The tasks explored in our work do not seem particularly sensitive to this issue owing to the Adapter L.M. However, improving the retrieval module can further improve the performance our framework.
I. Limitations of the overall framework
In this section, we discuss some key limitations our work. First, our approach assumes the existence of "some" metric that be used to evaluate the options. In the event that the metric is not clearly defined, alternative evaluators, like LLMs, can be used as proxy for evaluating the outcomes and generating a corresponding reward (Gu et al., 2024). We also assume that the user provides some input data for each task (or it is obtained from a corresponding dataset or benchmark). If precise data is unavailable, connecting the LLM data loading tools, can help provide the data necessary for framework.
Since code is generated from lan- guage descriptions of code implementations, e.g., spec- ifying "Use Gradient Boost i MagClassiin fier", there are slight variations from one code generation to the next, when the plan executor interprets the options. Although we found these variations to be insignificant with GPT-4o, using code snippets (e.g., code snippet of loading and using the model) when generating options, can potentially eliminate these differences.
Plan Executor:
Option Generator: In few cases, we noted that the option generator would produce differently phrased, but similar options in spite of being prompted to courage diversity (See Figure 1 18). Introducing an addi- tional strategy to have the LLM options be regenerated, or prompting the LLM to modify similar options may help mitigate this issue.
Second, although our approach successfully generates more diverse options for solving different tasks, we still see some residual bias for Random Forest. In the future, we seek to improve the diversity of the option generations to ensure a more uniform distribution over a range of methods. We also note that although the intrinsic knowledge of LLMs contain several methods, the LLMs are not capable of identifying more novel approaches (e.g., ChemProp for TDC tasks), that we hope to rectify through external retrieval tools.
Lastly, our approach currently naively samples walks and this results in some walks being re-sampled from one itera- tion to another, leading to repetition in the runs. In our future work, seek to explore more sophisticated strategies for generating walks, such as tracking of already explored walks or updating the values to weight unexplored paths higher.
We further discuss some challenges associate with each LLM module in our framework.
L1. LLM-module specific challenges
While GPT-4o based frameworks were able to successfully execute tasks with 100% success rate, we note some engi- neering challenges for the modules our framework:
Consistency Checker: There were few cases where the consistency checker identified an otherwise consis- tent pathway as inconsistent. Although the agent would simply pursue other paths, thus producing "some" working solution in each case, this mistake could cause some viable and promising paths to remain unexplored. Adding an additional "evaluator" LLM to verify the consistency response, may help mitigate this issue.
Retriever:
We use the same retrieval mechanism as AS-Agent (using few cases where the case retrieval module did not
There were
19

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
J. Code generated for the math case study
Case study: Math Problem
import cmath
Step 1: Icentify the given equation: x + 2x i.
Recognize that the equation is quadratic equation in the form of ax bx =0, where a L, b 2, and c
= b 2 -1j
step 2: Recall the quadratic formula for solving ax bx c 0, which is x
(-b +- sqrt (b (ac)] (2a).
Implementation detail 0: Substitute the values of b, and into the quadratic formula:
sqrt 41)) /
Simplify fy the expression under the square root by converting 4i into polar form and using De Moivre's s Theorem to find the square root.
delta = b++2 4*a*c
to polar form - abs (delta)
Convert
theta - cmath.phase (delta)
Calculate the square root in polar form sqrt_r cmath. moath.sqrt(a (r)
sqrt_theta theta
Convert back rectangular form aqrt_delta- sqrt_r (cmath.cos (sqrt_theta) 1j cmath. mnath.sin(sgrrt_hheeaa
Step 3: Substitute this value into the quadratic formula to find the two solutions for x by evaluating both the positive and negative cases of the square root.
x1 (-b + sert_ delta) (2+a)
x2 (-b sqrt_delta) (2+a)
step 4: Use the properties of complex conjugates to determine the real parts of the solutions.
Since the coefficients of the quadratic equation are real, the solutions are complex conjugates. Calculate the real part of one solutien and use it for beth solutions. real reai_part_ae xe.real real part_ xeareal
Step 5: Determine the product of the real parts of the two solutions. proauct_of_rellparrss real_part_xl reat_part_x2
Step 6: Conclude with the final result, which the product the real parts of the solutions.
print (1 The product of the real parts of the solutions is:', eroduct_of_reai_pete
20

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
K. Additional Flow-of-Options network examples
tirplement di lewming algerthor crall seural estwalk input the retrort lo preket O walaa for each usionn rewart dtep and stom in variable call sonu yeasert.
gIFe soprisoon wileet withe The IA
AURENITY WEAEE THE
Tho sear derd delie sud gine coprtatie Settin eeuoree
and
sarple haurtiti based policy where action rnteriied
pale
per epuiddilty
Hormulaor the dite more thi Wan to short gome
sunnning the imvadd cirained at aack dhyy
L:Tue
TIe
30 indeenent deep acterne
predent (00m0)
the the siberr dpeetim from Glean with the onderr
redwal la remnoses lie utter whech vudpel centaciua the cameil ctate as nout ond eattuts the actor la take bratl ponrr.
the tor sncordedd enteree pognni froe Sulore
Lse Elieat alpe from tikune with dacaars cat
cesteal
siaus neteork pallo that sakas the curnet state as lipet doods
left or light. Trcck the sstl memard apsidde by lamiiiing called total yeward
rihturn duoddi fer per pen Arth the mererue sunder denal bas allie
firecteed
sarria Mgarttre the noamii cata anbitttt le das of deviicss
Figure 18: Examples of flow-of-option -ottions networks with varying depths (shown for the RL and clustering tasks). Paths with
value of - 10000.0 0 are unexplored paths (either due to consistency check failures or because the walks were not sampled), The tree on the right demonstrates one of the system's 's limitations regarding generation of similar options in some cases (e.g..
Nodes 1 and 3).
L. Task adaptation using LLM adapter
In this section, we show examples of how the LLM Adapter adapts a network from one task to another under different scenarios. This is shown in Figures 19, 20, 21, and 22. The figures show the description of the nodes in the Flow-of-Options network.
0:
0:
Utilize Morgan fingerprints (circular fingerprints) generated using RDKit with a radius 2 and a bit vector size of 1024 convert SMILES strings into numerical features.
Morgan fingerprints icircular fingerprints) using RoKe size of convert SMILES strings numerical features.
and
Apply Mordred package calculate comprehensive descriptors, including 2D descriptors, SMILES strings capture diverse properties.
2: Apply the Mordred package to calculace comprehansive set molecular descriptors including 20 and 3D descrigtors. from SMILES strings capture diverse chemical properties.
Use fandommoossteegrestr from sidearn with 100 estimators the regression model predict the Caco-2 cell effective permeability.
nandomntoresttegresss from skleam with 100 estimators the regression model to predict the solubiltt activity.
Adapting task description
4: Use the Randommioeesieggrsse model from sideam. which can
4: Use the RandomForest Rrstteeressee model from skdeam, which can capture non-linear relat relattonhhpss and interactions between molecular descrigtors, potentially improving prediction accuracy.
capture non-linear relacionshis and interactions between moliecular descriptors, potentially improving prediction accuracy.
5: Irain the Randommoorsttegrresss model using the training and generalization.
5: Train the SandomforessRegresse model using the training data with bootstrapping enabled improve model robustness and generalization.
data with bootstrapping enabled to improve model robustness
Irain the handommrorsstiegreeee model using the training focus ensemble learning, Train multiple RandomFoorssffe Reatmmoooreffeeeessee
Train the anddomfores/Regressso
using training
RandomForestRee Regssrero ensemble leamning multiple subsets training predictions and improve robustness the help reduce
seeds
saeds
subsets the training data, and then average can pradictions to form a final prediction This help reduce variance and improve the robusiness of the models predictions.
average their approach
model's predictions.
Figure 19: Example of the adapter L.MM adapting the task description from Caco-2 cell permeability to Solubility prediction task in TDC. Each number marks a node in the network along with its description. The network structure is unchanged since the network is reused, but the description of the nodes are adapted to the new task.
21

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
0:
Morgan fingerprints generated using ROKit with radius and a bit vector size 1024 to convert SMILES strings into numerical features.
fingerprints)
Utilize Morgan fingerprints (circular fingerpritt) generated using RCKE with radius of and bit vector size 1024 to convert SMILES strings into numerical features.
Apply package
molecular descriptors including compreheraive set descriptors, from strings capture diverse properties.
2: Apply the Mordred package to calculate comprehensive set of molecular descriptors, including 2D and 3D descriptors, from the SMILES strings to capture diverse chemical properties.
FandomForesFeegressor
100 effective
3: Us RandomForestCiassiffr rom skieam with 100 estimators predi the binary activity HIA.
weweeeereooonnnnn pommssiiity.
Randomforeesheereesr ridde from sklearn which
RandommooretCiaassfee nodel from skleam, which
interactions between
Adapting model
captore momertcr renaioosrpps and
molecular descriptors, potentially improving prediction accuracy.
molecular descriptors, potentially improving prediction accuracy.
Train RandonforestRegg Seaeonooorreeeeeesseeeeee ressor n rodel using the training
Train RandonnForetClassffee nodel using the training
data with veuee emobleu morov model robustness
data with boutxrappann apping en to rngrvve model robustriess and generalization.
and generaliation
Train Handomionne wrestorrrssssseeee essor odel using the training
6: Train the RandomForettiaassher r model using the training
RandomForecRPageee models with different random seeds or subsets of the training data, and than average their predictions to form a final prediction. This approach can help reduce variance and improve the robustness of the model's preciction.
data Weerarlowwr
ing. Train multiple
data widra a rocus on motembbe rearning Trann multiple RandomForestClan fier models with different random seeds or subsets of the training data, and then average their predictions to form final prediction. This approach can help reduce variance and improve the robustness of the model's predictions.
Figure 20: Example of the adapter LLM adapting the model from regression task a classification task.
1: Implement a baskc lcaming algorthm with small noural network the function assroximator Use the current state as input the network predict O-values each action. the
0:
Q-learning algorithm with small neural network as the function approximator. Use the current state as input to the network predict o-values for each action. and
the half of the area, modily the rosard function to penalize the cart for being In the right halt. Track the total reword per episode by summing the each step
highest v-vale. each stop and store in variable called total, reward.
action
rewards
2: Use a simple heurstic based policy where the action determined by sign the angle: postiive push the cart to the left. Track the total rewerd per episode by summinn the rewards obtained at each step and store in total reward.
2: Use simple heuristic-based policy where the action dotermined by sign the pole angle: if pole anglo the cart the left, Additionally introduce a bas towards pushing the cart the left encourage staying in the LEFT half the area. Track total reward per episode by summing rewards vertable called
angle regative,
3: implement a deep determmistic policy gradient (DDPG) algorithi using an actor critic architecture. Use neural network represent evaluates the action value function. The actor network takes the current state as input and outputs is the action to take. per opisods by summing the rewards step tstal reward.
outputs continuous
Adapting new constraint
implement deep deterministi policy gradient (DDPO) algoritim using
neural architecture. continuous
represent actions, and another neural network for the crtti, which evalustes the action value function The actor network takes inpur outputs
rage Ihu LEFT fill of artl track total roward per
4: a policy gradient method such as REIMFORCE, Implement neural policy distribution
as input
episode by summing the mwards obtained at each step and total, roward.
variable
an action from this distribution decide whether to move left or right. Track the total reward per opisade by summing each stop
outputs
actions.
Use policy gradient method such REINFORCE, Implement neural network policy that takes the current state ipput. and outputs probabilit distr Bution over actions. Sample ection Moaify cart to stay In the LEP half of the area, Track the total reward per episode by summing the rewards obtained each and store
called
decide
called
Figure 21: Example of the adapter LLM adapting to constraints of the new task in the case cartpole balancing. From the original problem, the adapter modifies the methods bias the to stay in the left of the arena.
22

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
Veriational architecture: Use encoder with two convolutiona layers followed a fully connected layer output the moan and log variance
1: implement a Variational Autoencoder (VAE) with simple architecture: Use an encoder with two convolutional layers followed by fully connected layer to output the mean and log variance
fully connected convolutional layers to reconstruct the 28x28 image. Use decoder
iollowed by transposed
latent space, and decoder with connected layer followed by transposed convolutional layers encoder and decoder, final ReLU activations the decoder range [0,
Sigmoid range (o. 1), Addiiionllyy modil the decoder to output channel by setting first channel
image,
images
Sigmoid activation
immmll ipplement Variational Autoencoder (VAE) with
decoder
2: implement a Variational Autoencoder (VAE) with a convolutional encoder and decoder: Use an encoder with two convolutional layers followed by a fully connected layer to space. map layer 28k28 transposed convolutional and include KL divergence loss
convolitional layers followed by a fully connected layer to map latert space, and deceder convolutional layers reconstruct the 20120 image. Use ReLL activations and include KL divegeence loss term Additionllly modily
with
actiaaiion
the first charnel and the others to
Adapting constraint
3: Implement Generative Adversarial Network (GAN) with simple architecture: Use generator with two fully connected layers followed by two transposed convolutioal layers upsample
simple arc archtecture: generato with beo fully
Adversarial
connected layers followed two crnspooed convolutonal
upsample
images. discriminator should have convolutional layers
have convolutional layers followed two
by fully images.
connected layers classily real vs. fake images.
layers classify
addiconally modily gener abor output images with
settine the chancel
implement Generative Adversarial Network (GAN) with simple architecture: Use generator with two fully connected layers followed by two transposed convolutional layers to upsample to 28x28 images. The discriminator should have convolutional layers followed by fully connected layers classify LeakyReLU activations batch normalization
10.0
Impl Generative Adversarial Network (GAN)
simple archtectures Use generator with two fully connected upsample 28x20 transposed convolutioal
discriinater should
images.
have two convolational layers followed by two fully connected layers classily real fake images, normalization generator Additionall. modil the generator output. images thannel setting finet channel
the
Figure 22: Example of the adapter LLM adapting to constraints of the new task in the case of MNIST image generation. From the original problem, the adapter modifies the decoder output with post-processing.
23

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
M. Data interpreter prompts
obtained these prompts from the Github repository of MetaGPT implementing the Data Interpreter (DeepWisdom, 2025). Note the specificity in the task types, with task-specific prompts such image2web conversion (IMAGEZMEBPAGE) New task types would require similar manually constructed prompts. Also note the hand-crafted details such as model choices in MODEL TRAIN PROMPT. Despite the specific model choice instantiations, Data Interpreter mostly resolved to XGB and RF models, demonstrating the benefit of encoding options in more explicit data structure like FoO. Note the in-depth, specific details provided for each task type, such as how and which columns to process, or which models to use for specific tasks. In contrast, the next section demonstrates our LLM prompts which are more general in comparison.
Prompt: Data-Interpreter
Prompt for taking on "eda" tasks EDA_FROMPT
The current task is about exploratory data analysis, please note the fo following: Distinguist column types with 'select_dtypes' for tailored analysis and visuali ialaiizaioon such as correlation
Remember to 'import numpy as np" before using Numpy functions.
Prompt for taking on "data_preprocess"" tasks PATA_PRRPROCESS_PRE PROMPPTTPPPPPP---- ""*
The current task is about data preprocessing, please note the following: Monitor data types per column, applying appropriate methods. on existing dataset columns.
Ensure operations
Avoid writing processed data to files.
**ATTENTION** Do NOT make any changes to the label column, such as standardization, etc.
Prefer alternatives one-hot encoding for categorical data.
Only encode scale necessary columns allow for potential feature-spect fic engineering tasks (like dime_extract, binning, extraction etc.) later.
Each step do data preprocessing to train, must do same for test separately at the same time.
Always copy the DataFrame before processing it and the copy to process. **"
Prompt for taking on "feature_engineerri tasks FEATURE [EATURE_ENGINEERIIG ENGINERRING_PROOP The current task is about feature engineering. when performing it, please adhere to the follewing principles:
Generate as diverse features as possible to improve the model's performance step-by-step.
Use available feature engineering toels if they potential impactful.
Avoid creating redundant or excessively numerous features one step. Exclude ID columns from feature generation and remove them.
Each feature engineering operation performed on the train set must also applies to the dev/test separately the same time.
**TTENTION Do NOT use the label column to create features, except for encoding.
Use the data from previous task result if exist, do not mack or reload data yourself.
Always copy the DataFrame before processing it and the copy to process. **н
Prompt for taking on "model_traine tasks MODEI_TRAIN_PROW
The task is about training medel, please ensure high performance: For tabular catasets you have access XGBoost, CatBoost, random forest, extremely randomized trees, k-nearest neighbors, linear regression, etc. For image datasets you have access to Swin Transformer, ViI, ResNet, EfficientNet, etc.
text datasets you have access to Electra, DeBERTa, GPT-2, BERT, etc.
24

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
Avoid the use of SVM because of its high training time.
Keep in mind that your user prioritizes results and is highly focused on model performance. Se, when needed, feel free to use models of any complexity to improve effectiveness, such XGHoost, CatRoost, etc.
non-numeric columns exist, perform label encode together with all steps.
Use the data from previous task result directly, do not mock or reload data yourself.
set suitable hyperparameters for the model, make metrics high possible. u*"
Prompt for taking on "model_evaluatee tasks BOBBUALEVALAEE_PRRRRA PROMPT
The current task is about evaluating model, please note the following:
Ensure that the evaluated data same processed as the training data. If not, remember use object in 'Done Tasks' to transform the data. Use trained model from previous task result directly, do not mock reload model yourself.
***
Prompt for taking on "image2webpage" tasks IMAGE2MEBPAGE, PROMPT TROEBP =
The current task is about converting image into webpage code. please note the following:
Single-Step Code Generation:
Exccute the entire code generation process in a single step, encompassing HTML, css, and JavaScript. Avoid fragment ing the code generation into multiple separate steps to maintain consistency and simplify the development. workflow.
Save webpages: Be sure to use the save method provided.
N. LLM prompts used in our framework
We present the prompts passed into the different LLM-based components of our framework. Note that our prompts are more general in comparison to frameworks like the Data Interpreter. Importantly, we do not make any assumptions about task types like the Data Interpreter.
Prompt: Planner
You are expert ML scientist provided with the following task: (task)
Think through the task step-by-step and produce highly detailed list steps involved in solving this task.
Do not include any code in your response.
Prompt: Planner Filter Mode
You expert scientist provided with task and high-level plan for solving the task.
Task: (task) High-level plan: (plan)
Your goal to rank the steps in the high-level plan based on their relevance accuracy on the task.
First, you must assess each step in the plan and identify whether it can impact accuracy on the task.
25

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
Second, for all the steps that can impact accuracy on the task, you must rank the steps from most impactful to least impactful.
Prompt: Adapter
You are proviced with task. You also provided with high-level plan for solving fifernnt, but related task, Your goal adapt the steps in the plan for the new task.
Task: (task)
Plan: (plan)
For each step in the plan, you must identify if that step needs to be adapted for the current task.
adaptation necessary, modify the step to adapt the new task. If no adaptation is necessary, leave the step as is.
Please do not add remove any steps. Return the adapted plan.
Prompt: Option Generator
You are expert scientist provided with task, and one step from high-level plan for solving this task. You are alsc given implementation details for of the steps prior to the current step.
task: (task) prior steps: (s)
current step: (step)
Your goal is to generate diverse options for implementing the current step if it impacts final accuracy.
Firstly, you must assess whether the implementation of the current step can impact final accuracy on the task.
IF is expected impact final accuracy, you must:
First, think as broadly as you can to generate nuumopptins]] diverse and distinct options for implementing the current step, while taking into account the prior steps.
Second, you must verify that each of your generated options da not conflict with any of the prior steps.
Third, you must regenerate alternatives for any options that conflict with prior steps.
Ploase follow these instructions in generating your response:
1. You must maximize the diversity acress all your generated options.
2. Your choices must be very specific so that a programmer can implement it.
For e.g., instead of specifying "Usee features", you must speci "Use features
x, Y, 2."
3. Your choices should not modify any the prior steps.
4. Do not repeat any of the prior steps.
5. Do not include code in your regponse.
26

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
Prompt: Consistency Checker
You are given a plan of steps for solving a task, and a new step. Your goal is to verify if the information contained in the new step contradicts any of previsus steps of the plan.
plan: (plan)
new step: (step)
The following
examples of contradiction:
1. the new step references different model than the previous steps the plan.
2. the new step references different feature than the previous steps in the plan.
First, assess whether the step contradicts previous steps in the plan described above.
If the new step contradicts other steps in the plan, return the word: yes If the step does not contradict other steps in the plan, return the word:
Prompt: Code Generator
are an expert programmer provided with task and plan steps for aecomplishing the task:
task: (task)
plan: (plan)
You are also given specific implementation details for some of the steps in the plan: (walk)
Your goal is stitch the plan together create complete Python code.
You must ensure to incorporate the specific implementation details into your final code.
You must follow the template provided the task description as closely as possible and integrate the impl implementation details into the provided template.
You must not make any funstional changes the provided implementation details themselves.
Your code must be complete and should not leave any additional steps for the user nor raise any errors.
Prompt: Code Generator - Reflective Debugging Mode
You are provided with code and errer message from running the code. Please debug the code systematically
fix the error.
For additional context, you are also provided with the original user prompt indicating the task that the code is trying to achieve.
Original prompt: (task),
Error message: (crror),
Codo: (code)
The may also be syntactical in which case you fix the syntax appropriately.
27

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
O. User inputs to our framework
show examples user imputs for the different tasks. We provide code template in our prompts (similar to DS-Agent (Guo al., 2024)) to: a) standardize evaluations between baselines by removing differences in how the models get evaluated; b) to avoid issues with models not knowing what column names exist the dataset, how to access them other API issues. This can be alleviated by connecting the model to external tools that would allow it inspect datasets, information contained therein, to reason about APIs. Similarly, we pre-specify packages to use, such Sklearn or Pytorch to prevent import errors when executing code (and to prevent autonomous installation of packages in our environment).
User Input Example for TDC task
Your task to write Python code for the following task. The dataset name task description is provided below.
Dataset name: (task_name
Task description: (description)
Please follow these instructions:
1. Follow the template in the example code shown below, and de not add new print statements to it.
2. Your code must complete and executable without needing additional user intervention.
3. Please use Sklearn or Pytorch packages only for your ML implementations.
Here is an example code snippet showing how load and evaluate dataset with the name Caco2_Wang"
from tdo. wew.bbnncmrrk_grauu import admet_group
group admet_ demmt_ (path- 'data/") predictions__ine []
For reprocucibility
for seed in [1, 3, 5]: benchmark getup.get
get Caco2 ('Cao2__aagg')
all benchmark names in a benchmark group are stored in group. droup.dataset_nama. predictions = ()
name = (aacchmark'namme""
train_val, test
('traan_rrrriaiaaallllaaa benchmark('test!
train, valid 'default', seed seed)
(benchmark name, split_type
NOTE: For the dataset, column names are 'Drug' (for the input SMILES strings) and "y" (fer the output labels)
Train your model using train, valid, test Save test prediction in t_sred_test variable
predictions [name] p_pre___peeeee predictions_list appond
results group. hwwp:o/cwalutttnnnay/ooccccotooo print (results)
User Input Example for RL task optional constraint specification shown
Your task is to solve classic reinfcrcement learning problem. where the goal is to balance pole upright on cart that can move left right.
28

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
We add this for the variant of the problem However, an important constraint in this problem is that your cart must try atay LEPT half of the
much possible.
Follow the template below for your code:
import gym
import numpy as np import torch import random
from syym.rrappes..moittoe www.tttorrnn.vieeeeeeeeeooo video_ import VideoRecorder
import patplotlib.pyplo. as pit
For reproducibility
SEED
random. seed (SEED)
torch. torch.manual_se www.mmaaallaeeeeeeee (SEED)
np. random. seed (SEED)
Create the environment env gym. CartPole-vi',, rende__nodo""rgb_attaa video VideoRccorder (env, "Cartpolc-video..me mpp")
State space has four values [cart pesition, cart velocity, pole angle, pole angular velccity]
Action space has two values -- [0: "push cart to the left", 1: "push cart to the right")
for episode in range (500):
state,
reset (seed=SEED)
total_reward - 0.0
FILL OUT: Solution for the cartpele problem FILL OUT: Predict action using model
next_state, state, reward, done,
env. sav.steplee (action)
# FILL OUT:
Track total reward per episode in variable called total_reward
Plot total reward episode ("Iteration") plt.ylabel ("Total Reward") plt.savefig (attoaaa ("totaalreeeaard.png
Print total reward from the last episode print ("Final episode reward: ", total reward)
state, env reset (SEed-SEED)
done False
while not done:
# FILL OUT: Predict action using model state, _ done, - env.step (actien)
revce()) video.capture__ram frame
video.c ()
env.close ()
111
29

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
User Input Example for Clustering Task
Your task is to perform clustering of gene expression data for a given dataset. You are proviced the gene expression profiles for 2321 patients, on 14208 genes. You must cluster them into 7 clusters.
Please visualize the generated clusters and save the visualizations.
Please follow the provided template to complete this task. You must restrict yourself to Pytorch Sklearn packages for this problem. Please ensure that your code does not require any additional steps from the user.
Here's 8 template ate code to follow:
import pancas pd import numpy as np from sklearn.metrich import silhouette_scon score
def compute_ssi (data, predisted_labcls)
print (f"Final silhouette Score: (silhouette_score (data, prodictoo[labcls)]
def load_dataset ():
data = wpw.reed_teee aaaae ("Leukemia--lean. txt", sep "(t")
datal"oisease" whewhere (datal "diseasee Dbiabetes_Iype_11 "Diabetes",
data "atas""iseas""
datal"cisease" = np.where (datal "disease"] "Diabetes_ Typee___p_________________ II" "I" "Diabetes",
data data["diseasee"]
other - htt::///iiiiilllylooooaaaaaaoaad
'MDS',
DDS_ oes.aaaseeeuuyyllppoooleeeeeeeee
data df data.drrop(diisesse axis=1)
df df ("GS"" axis=1) df df cro. axis=1) return cf.toonumpy()
There are 14208 columns (features) and 2321 samples data = load_dataset,
TO FILL Periorm clustering
TO FILL Predict labels for each sample and save into Ladiitted_labe.
compute ssi (cata, predicted labels)
TO FILL --- Save the clustering visualizations into a file called "clusters.png pnng
User Input Example for English-French Translation Task
We would like build language model that can translate from English to French (en-fr).
Please use ROUGE score to evaluate the result.
Print the final ROUGE score as "Final Reuge Score: <print final rouge metric>".
1. You must load and use the opus-100 dataset (en-fr) as shown below.
2. You must restrict yourself to Pytorch or HuggingFace packages only.
3. Your code must be complete, and should not 1eave additicnal steps for the user.
4.
Please restrict any training or fine-tuning to epochs. tokenzatiens on the GPU.
5.
Please run all models
Example of leading the dataset: from datasets import load_dataset from torch. forch.utils. utils. data import Dataset, DataLoader
30

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
dataset load_datasse (*Mellsinkk--Nii/gpu---0 "en- fr*)
modi fied needed.
class CustomDataset (Dataset):
def init__self,, dataset, transform=None)
Args:
data (array-like); Data samples.
targets (array-like); The corresponding labels for the data samples.
transform (callable, opticnal): Optional transform. to be applied on sample.
***
self.cata dataset #[samplel translationn) ['en'] for sample dataset] #self.targets transtran transform
][' fr' for sample in dataset]
def
***Reus the total number of samples. return len (self.data)
def www.ggttteem... (self, idx): "*"Generates one sample of data. "."
sample self.data translaticnn [idx] ['on'] target data/ translaticnn [idx] ['fr')
if self. transform:
sample sel f. wol...ransformmccc. (sample)
return sample, target
train dataset CustemDataset (dataset I' trrain' 1)
val_dataset Custombataset (dataset [' validationn)
cest_dataset = CustomDataset (dataset [' test'])
User Input Example for Traveling Salesman Problem
You must solve traveling salesman problem: Given matrix distance [1 [] of size (i.e., number of cities) where distance(ii] [i] denotes the distance from city to city j.
The task is to complete tour from city (0-based index) to all other cities such that we visit each city exactly once and then come back to starting city at minimum total distance.
You should evaluate your approach on time taken to complete.
You can follow the template below for guidance:
import numpy np import torch import random
For fixing the TSP instance generated by generate_tsp() function SEED - 42
random. seed (SEED)
torch. parch.manual. manual seed (SEED) (SEED)
np. random. seed (SEED)
def get het__pat__eenghae (path, distances): Fath is list of city numbers indicating path follow
31

---

Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
total path Length 0
for i in range (1en (path) - 1): start path[i] end path 1]
total_ Length distances |start] | [end] return total patall__at___ggtt
def generate_tap (nun_cities):
"**Generates a random TSP instance with given number cities. ".*
Create random coordinates for cities coordinates = np. random. rand (num_cities, 2) 100
Calculate distances between cities using Euelidean distance distances = np.zeros( (num_cities, num_cities)) for i in range (num_ cimm_)i
for in range (num_cities)=
if !- j:
distances [i, j] np. ap.linalg norm norm (coordinates [i] coordinates (j1)
return distances
Example usage:
oum_cities 10
distances = generate_tsp (num_citics)
optimal_path path None
time_taken None Track time taken by the proposed approach
TO PTLL -- Predict optimal path, where optimal paatmal_pathhhhaa path is 1ist of city numbers ranging from to num_cities
assert len (optimal path) num cities + 1, "Path does not visit all the cities" num_cities, "Path does not visit all cities once"
assert len (set print ("Optimal imal Path: ", opt imal__apatth) print ("FINAL Metric Time taken: ", time_taken)
32


Reviews:
[{"review_id": "RdeOiU356s", "reviewer": "Reviewer_utUi", "summary": "This paper proposes Flow-of-Options, a planning method for LLM agents, that can effectively track an optimal path over the combinations of possible options. More formally, Flow-of-Options can be represented as a directed-acyclic graph (DAG) of depth n, where a node is an option and an edge is a path between options in a sequence. Flow-of-Options finds an optimal path by evaluating possible paths and updating values (the return of the path) in edges. This paper evaluates Flow-of-Options on 16 Data Science (DS) tasks and 17 Therapeutic Data Commons (TDC) tasks. Experiment results show that Flow-of-Options can achieve better scores than SELA (utilizing a MCTS-based planner) and Data Interpreter (utilizing a DAG-based planner) in DS tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and will incorporate these suggestions into our updated paper.\n\n# W1\nWhile our paper is on *Application-Driven Machine Learning*, **FoO does not explicitly specify steps such as model selection or feature engineering, but rather adapts to the task plan that is produced**. We show this generalizability (beyond classification/regression of sections 4.1 and 4.2) as follows:\n- RL and Image Generation (Section 4.3) - uses a different ML pipeline than tasks in 4.1 and 4.2 (e.g., reward formulation)\n- Unsupervised clustering (Appendix B.2.1) – is a different model than the data science tasks\n- Machine Translation (Appendix B.2.2) – does not involve a specific feature engineering like the data science tasks (only tokenization)\n- Traveling Salesman Problem (Appendix B.2.3) – involves neither ML model selection nor feature engineering\n- Case study on a math problem for solving the complex solutions to $x^2 + 2x = i$ using FoO (Appendix B.3) - does not involve an ML model selection nor any feature engineering.\n\nIn particular, the TSP and math problems are larger deviations from the typical data science pipeline.\n\n# W2\n**FoO enforces diversity in LLM solutions through compressed, interpretable representations that support memory of past explorations when combined with case-based reasoning**. Specifically, FoO offers the following advantages over exhaustive combinatorial search:\n- FoO acts as “memory” of key info on previously explored task solutions - can be saved, reused and adapted from one task to another via deployment (Section 3.2). Reusing FoO from task $T_1$ to $T_2$ is fast and achieves good results (Tables 1, 2 and 3). Hence, prior knowledge encapsulated in the FoO can be effectively reused for $T_2$, whereas an exhaustive search would have to be repeated for each task.\n- Exhaustive combinatorial search w/o FoO assumes expert knowledge to set up the combinations themselves. An expert ML scientist can enumerate $k$ options for each step in the task, but a naive user may lack this knowledge. FoO encapsulates the knowledge of LLMs enabling even naive users to specify their problem in natural language, without demanding expertise. Even for experts, FoO can serve as a \"force multiplier”, potentially enumerating options that the expert may not have thought of.\n- The formulation of FoO also supports integration with tool use (Please see Reviewer cGuq Point 2).\n\n# Q1\nAverage number of options $k = 4$ and average number of steps $n = 3$. These are specified as hyperparameters into the framework. The computational complexity of FoO is indeed dependent on $n$ and $k$ and we have currently implemented the following solutions to mitigate the computational complexity (From Section 3.2, Page 5):\n- *Parallelization*: We parallelize the executions of walks through the FoO so that even if there are a large number of walks, the computational time taken is reduced.\n- *Pruning*: We prune some of the low-scoring edges of the FoO (so that they are not explored) which also helps reduce the computational complexity.\n\nLastly, the consistency checker (Section 2, page 3) identifies invalid paths in the FoO. Although the total number of paths are $k^n$, not all paths are valid (as $n$ increases, the proportion of invalid paths is also higher). Example of invalid paths are shown in Fig 4 (Page 3). Consistency checker empirically results in $\\approx 22.8$% reduction in the number of paths to explore (Empirical results noted in response to Reviewer zYfD Q1). Hence, in practice, computational complexity scales quite differently. Please see measured time performance with $n$ and $k$ (averaged across three runs on the CW TDC task of Table 2): https://ibb.co/0VvqpBR5. From $n=1, k=3$ to $n=3, k=3$, the number of paths is $9\\times$ more. However, corresponding time only scales by $\\approx 7\\times$ (this includes parallelization and consistency checking, but excludes pruning which can further improve the efficiency). \n\n# Q2\nThe development phase took about 13.29 mins on average in our work for the data science tasks. The development time is dependent on the complexity of the task based on $n$ and $k$ as noted in Q1.\n\n# Q3\nIn beam search, our goal is to narrow the search to the most effective set of options, by selecting the top $b$ options at each level, and exploring paths between them to discover potentially improved combinations of the top options. This can be visualized as exploration over a reduced FoO where the nodes are just the top performing options found thus far.\n\nIn our experiments, we start with a full beam width of 100% (exploring all the options), and reduce it to the top 50% of the options in the last two iterations of development. In Appendix C (Figure 13), we demonstrate that beam search can discover new combinations of top performing options, resulting in improvements in the final performance of the methods.", "claims_and_evidence": "This paper proposes Flow-of-Options, a planning method for LLM agents. And, it provides comprehensive evaluation results on 16 Data Science (DS) tasks and 17 Therapeutic Data Commons (TDC) tasks. Also, the evaluation results support that Flow-of-Options can achieve better scores than other baselines such as DS-Agent, AutoGluon, SELA, Data Interpreter, and AutoGen on DS tasks.", "methods_and_evaluation_criteria": "### Strengths of Methods:\n\nS1. Flow-of-Options has a structure that can effectively tack an optimal path over possible combinations.\n\n### Weaknesses of Methods:\n\nW1. Flow-of-Options seems to be overly customized for data science tasks which goal to find an optimal path consisting of feature engineering and model selection. I am not sure if Flow-of-Options generally works well over diverse tasks that is more complex.\n\nW2. I am not sure what is a key advantage of Flow-of-Options over the exhaustive search over all combinations.", "theoretical_claims": "This paper mainly proposes a method for effective reasoning of LLMs. It does not provide any theoretical claims.", "experimental_designs_or_analyses": "This paper provides comprehensive experiment results on 16 Data Science (DS) tasks and 17 Therapeutic Data Commons (TDC) tasks. Also, it provide the details on experiments in Section B in the Appendix. It seems that the experiment design is sound and valid. However, it seems that the experiments mainly deal with data science and chemical reaction tasks.", "relation_to_broader_scientific_literature": "This paper introduces Flow-of-Options, a DAG-based planning method for effective LLM agents. Designing an effective planning algorithm for LLM agents is one of important research areas, since LLM agents are widely applied into diverse domains.", "essential_references_not_discussed": "This paper properly discusses related works.", "other_strengths_and_weaknesses": "### Other Strengths:\nN/A\n\n### Other Weaknesses:\nN/A", "other_comments_or_suggestions": "### Other Comments:\nN/A", "questions_for_authors": "### Questions:\n\nQ1. Can you provide the computational complexity of finding an optimal solution in Flow-of-Options? If the number of options (k) is large and the number of steps (n) is long, the computational complexity can be high, since the number of walks increases exponentially (k^n). Can you provide the average number of options in some example tasks? How do you control the number of options in each step?\n\nQ2. In the Development phase of the FoO-based agent framework, how long does it take for the values in edges to converge?\n\nQ3. Can you provide some details on beam search over Flow-of-Options?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ytWxulYGLw", "reviewer": "Reviewer_4BTz", "summary": "This paper proposes FoO (flow of options) approach to diverse the LLM's reasoning paths. An FoO-based agentic system is developed for solving traditional machine learning tasks including regression, classification, reinforcement learning, and image generation tasks. The authors show that their framework outperforms the existing methods by a large margin with lower cost.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback. We will incorporate the suggestions into our updated paper.\n\n# Experimental Designs or Analyses\n## Point 1\nThe documentation for DS-Agent, AutoGluon, and SELA note that they are indeed suited to tabular tasks similar to the ones explored in our experiments. This is noted in their corresponding papers. Our dataset is not specifically curated by us, but is rather an existing dataset obtained from (Guo et al. 2024). We note the following potential reasons for the Table 1 observations:\n- *AutoGluon*: AutoGluon explores a fixed set of models which is well suited for some, but not all the data science tasks. Indeed some of the problems in the dataset, such as language-based tasks, are currently not supported by AutoGluon (which is a shortcoming of this method). Nevertheless, AutoGluon is a quite popular framework and is also an example of a non-LLM based AutoML framework. Hence we felt it would be useful to incorporate it as one of the baselines in our experiments.\n- *SELA and DS-Agent*: Both these frameworks incorporate *self-correction* over the LLM proposed methods. The LLM proposes a method and then repeatedly reflects on it to modify the method and improve the result. In “Large Language Models Cannot Self-Correct Reasoning Yet” by Huang et al. ICLR 2024, the authors note LLM performance can degrade after self-correction. Accuracy drops as the number of repeated self-correction calls increase (Table 3 of the cited paper). Both SELA and DS-Agent perform $>1$ self-correction calls. Besides zero-shot, Data Interpreter (DI) does not incorporate this form of self-correction, and tends to perform better among the baselines (although it has other shortcomings as noted in the paper). **LLM self-correction is not needed with Flow-of-Options, where the different options are already encapsulated and systematically explored via the FoO data structure. However, this does not mean that SELA and DS-Agent are not suited for the types of tasks in our experiments, but rather that there is room for improvement here, which we believe Flow-of-Options contributes.** It is possible that if self-correction were removed from DS-Agent and SELA their performances *could* improve. However, we do not change the baseline implementations available on Github for the purposes of comparison.\n\n## Point 2\nThis is correct with the exception that DS-Agent (alongside FoO) also incorporates mechanisms to improve upon past iterations. The other approaches do not have this design element. Our intention with Figure 6 is to:\n- Add experimental and *quantitative* evidence to our claims of being able to improve over subsequent iterations.\n- Demonstrate that some of the baseline approaches fail in certain iterations (e.g., DeepMol and data interpreter).  \n\nWe wanted Figure 6 to demonstrate the benefits of our approach at the agentic design level in comparison to the baselines.\n\n# W1\nWe seek to mitigate this with the beam search (Section 2: beam traversal, page 2), which selects the top $b$ options for exploration, thereby limiting the explored paths to the more promising options. **This prevents the bad options from being revisited**. We note additional computational improvements in Section 3.2, Page 5 (Also note response to Reviewer utUi Q1).\n\n# W2\nWe hope that our response to Point 1 above regarding baselines in Table 1 offers additional context around our compared baselines. Please also note additional experiments reported for W4 to strengthen our evaluation.\n\n# W3\nCurrently, in addition to the classification/regression tasks, the RL and image generation tasks, we explored the following tasks in Appendix B.2 and B.3 to further demonstrate generalizability:\n- Clustering\n- Machine Translation\n- Traveling Salesman task\n- Case study on a math problem\n\nWe hope that these additional tasks help demonstrate broader generalizability of our framework.\n\n# W4\nWe provide additional results for LLMs (including an open-source LLM) over a subset of the TDC tasks from Table 2: https://ibb.co/NRMNTw4. Arrows indicate whether lower or higher metrics are preferred. Our approach helps consistently improve performance across the newly added LLMs and also outperforms the baselines for the tasks. Note that for some cases \"--\" indicates that the model failed to produce working code within three attempts. Hence, we see that **in addition to improving the overall task performance compared to baselines, FoO can also help mitigate the failure rates in code generation**.\n\n# Q. SELA cost\nA lot of the time consumed in SELA is for the execution of the code from the MCTS rollouts. Each code execution is performed sequentially (it does not support parallelization like our framework -- parallelization is discussed in Section 3.2, Page 5) and therefore takes a significant amount of time. However, the code execution does not involve LLMs per se, therefore it does not reflect in the cost, which is associated with querying LLMs.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There is no theoretical claims in this submission.", "experimental_designs_or_analyses": "This paper evaluate their framework on many traditional machine learning tasks with other baseline methods. I have some questions as below:\n\n1. For Table 1, it seems that zero-shot is better than DS-Agent, AutoGluon, and SELA. This is a bit confused and raises concerns that these frameworks may not suitable for the considered tasks, and make me question about the fair comparison among different approaches?\n2. Fig 6 seems irrelevant as FoO-based approach by design should improve upon the iterations. However, other approaches are kinda independent among different trials (please correct me if I am wrong here).", "relation_to_broader_scientific_literature": "I believe the proposed methods can be related to broader applications other than traditional ML tasks.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "**Strengths**:\n1. This work use LLMs to generate diverse options to explore.\n2. The propose approach can effectively explore the optimal path for a give task. \n3. The authors explore the tasks beyond classification and regression, although just two cases. \n\n**Weaknesses**:\n1. The development part might be not efficient as the proposed methods explore the every path in the graph. When the width of the tree is larger, some bad options will also be visited many times. \n2. The evaluation needs to be more rigorous and can be further improved. The comparison may not fair enough based on the current results.\n3. More RL or other ML tasks should be explored to demonstrate the generalization of the proposed method.\n4. Only GPT-4o is evaluated, how about other open-sourced models?", "other_comments_or_suggestions": "NA", "questions_for_authors": "In table 3, I am a bit confused by the cost of SELA, it seems that SELA takes the longest time but the cost is not the highest?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nF72YX5ijM", "reviewer": "Reviewer_zYfD", "summary": "The paper introduces Flow-of-Options (FOO), an agentic system designed for auto-ML. The core contribution is a framework based on fully-connected network structures of step-by-step solution paths generated by LLMs.  The framework is evaluated comprehensively on multiple domains including standard data science tasks, therapeutic chemistry tasks, reinforcement learning, and image generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback. We will incorporate the suggestions into our paper.\n\n# Q1\nWe measure the reduction in execution time when consistency checker is added to identify invalid paths (as opposed to w/o it), and planner when the adapter is added (as opposed to w/o it). We also measure costs for each component when it is added. We measured these on average for tasks from Table 1. The results are noted here: https://ibb.co/7d4vmw7f. Adding these elements offer performance benefits at minor additional cost. We also performed a cost ablation for all components: https://ibb.co/LXgW5wBs \n\n# Q2\nPlease note our scaling performance response to Reviewer utUi (Q1)\n\n# Q3\nThis is a good question. We believe there are two cases in this scenario:\n- Case 1: A single task plan suffices, but has conceptually different option decompositions: For instance, a string denoting a drug molecule can be processed using the specialized RDKit package which computes chemical properties of the molecule such as molecular weight. Alternatively, we can convert the strings to vectors via NLP methods. These are conceptually different. However, the task plan constructed by **the planner is sufficiently high-level that conceptually distinct options are supported via the option generator LLM**. See the example FoO: https://ibb.co/GXXgj5p. Options 1 and 2 are feature processing, denoting the RDKit method and \"NLP style\" vector embedding respectively.\n- Case 2: A single task plan cannot denote the different decompositions: The experiments in our paper currently do not fall into this category. However, it is possible to envision a *nested* Flow-of-Options in our future work, with FoO data structure over task plans as well. In this case, the depth will be $n=1$ and width $k$ denotes the different task decompositions. Internally, each task plan node can be captured similar to our work in this paper. Hence, a nested version of FoO can be explored for such problems.\n\n# Expanded evaluation on LLM backbones\nWe provide additional results here: https://ibb.co/NRMNTw4. Please see response to Reviewer 4BTz (W4) for more details. We will expand further details around this in our paper as well.\n\n# Comparison to the tree and DAG\nThe current implementation of Data Interpreter does not appear to save the produced data structure.  SELA produces a text formatted tree shown in the excerpt below:\n```\n[Node 0-2]\nEngineer features if necessary to improve model performance. Additionally, generate a correlation matrix heatmap to identify highly correlated features, which might be candidates for removal or transformation.\n```\nWe convert this to a visualization of the tree for a data processing task and demonstrate it alongside FoO for the same task. Tree: https://ibb.co/NdzGtFYW. FoO: https://ibb.co/mVqnMpk4. We note the qualitative option diversity and connectivity improvements of FoO here.\n\n# WordClouds\nWe chose WordClouds as a compact representation of the frequency and diversity of the model choices made by our approach. We'd be happy to replace the wordclouds with bar charts (for proportion of options, i.e. frequency) which is more quantitative: https://ibb.co/3yVLzv0d (For Fig 2), https://ibb.co/tTS4Zwmz (Fig 5)\n\n# Hyperparams of baselines\nWe have performed hyperparameter tuning on the key *number of iterations* parameters for SELA, DS-Agent, and Data Interpreter prior to running them (the other frameworks do not include hyperparameters apart from LLM backbone to use). \n- *SELA*: Increasing the number of iterations would lead to a significant explosion in terms of time (5 → 6 iterations increased time from ~21 mins → to ~57 mins on average), but we did not note a corresponding improvement in accuracy beyond 5 iterations. This could be related to the complexity of the problems in our experiments. More complex problems may require more iterations of SELA (albeit at a significant time cost).\n- *Data Interpreter*: Increasing the number of iterations increases the amount of time (to a much lesser degree than SELA), but like SELA, it did not impact accuracy beyond 5 iterations. It is also worth noting that the execution failures in Data Interpreter did not correlate to the number of iterations (possibly because failures are related to the presence of cycles in the LLM-built DAG, noted as one of the shortcomings of DI).\n- *DS-Agent*: The number of iterations increases the amount of time, and we also noted an improvement in accuracy however, that would stabilize at about iterations 4 to 5. We set this to 5 to be consistent in our cost/time comparisons across all methods.\n- *Our approach*: For our approach, the number of iterations increases the amount of time with improvement in accuracy. For consistency with all the baselines, we fix number of iterations to 5.\n\nWe generally chose 5 iterations to maintain consistency across all our comparisons on time and cost assessments. We note relative advantages of the different methods in Appendix F.", "claims_and_evidence": "The paper presents several compelling results but some key claims lack sufficient supporting evidence:\n\n* The claim about the framework's effectiveness would be strengthened by including more comprehensive ablation studies and hyperparameter analysis in experiments. For example, it is important to include comparisons across different LLM backbones, as well as sensitivity analysis of the hyperparameters for both the baseline methods and the proposed approach. Without these analyses, it's difficult to assess the robustness and generalizability of the performance improvements.\n* Some claims in the paper would be better-supported by control studies, such as directly comparing the proposed fully-connected network structure against alternative architectures (trees, standard DAGs).\n* Figures 2 and 7, which aim to use word clouds to show improved solution diversity, are weak and unconvincing. A more systematic quantitative analysis of solution diversity would be necessary to support this central claim of the paper.", "methods_and_evaluation_criteria": "* The method is sound and presented in detail. The network structure design choices are well-justified in comparison to previous works. \n\n* However, it is not clear if the paper properly tunes hyperparameters for the baseline methods, which raises concerns about the fairness of comparisons.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "* The evaluation is primarily conducted with GPT-4o as the underlying LLM. More comprehensive evaluation across different LLM architectures would be needed. (There are some very rough comparisons with GPT 3.5 in the supplemental, but it didn't compare with other baselines and lacks enough details).\n* There are more ablation studies that can be done, such as evaluating the performance of the planner component and the consistency checker. It will let the readers have a better idea of which aspects of the framework are most critical to its success.\n* Overall, I do not feel like I learned enough insights from the results beyond performance numbers. The paper should design more experiments that provide deeper understanding of the system.", "relation_to_broader_scientific_literature": "The paper proposes a fully-connected network to build agentic systems for auto-ML. It improves upon existing approaches in several specific ways:\n\n* Compared to SELA FoO offers greater expressivity by using a fully-connected network structure instead of a tree structure and replaces SELA's computationally expensive Monte Carlo Tree Search with a more efficient traversal mechanism.\n\n* Compared with Data Interpreter, FoO provides a guarantee of acyclicity in its network structure since LLMs are only used to generate options, not to construct the network itself.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "* Can you design experiments to show the effectiveness of the planner and consistency checker and their impact on the performance?\n* How does the performance scale with the depth and width of the network?\n* How do you deal with the scenario when there are multiple ways to decompose the problem?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rGMezfhqKV", "reviewer": "Reviewer_cGuq", "summary": "The paper introduces the Flow-of-Options (FoO) framework, a structured reasoning approach for large language models (LLMs) that systematically generates and evaluates multiple decision options at each step. Instead of following a single reasoning path, FoO constructs a directed acyclic graph (DAG) where each node represents an option, and edges capture transitions between different decisions. The framework is applied to a range of tasks, including machine learning automation, therapeutic chemistry, reinforcement learning, and symbolic reasoning, aiming to improve decision-making by diversifying the exploration process. Additionally, FoO integrates case-based reasoning (CBR) to reuse past solutions for efficiency. The proposed approach is tested across multiple domains, comparing its performance against existing LLM-based agent systems and AutoML frameworks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We have consolidated the reviewer concerns into the following topics and will include them in our updated paper.\n\n# 1. Computational cost compared to fine-tuning\nFine-tuning introduces two additional implicit costs that do not impact our approach:\n- Fine-tuning requires a dataset that is reasonably large and diverse. Ensuring quality, quantity, and diversity of the dataset can present a high cost.\n- Fine-tuning also requires the model weights to be accessible which is not always the case. This enables a broader applicability of our approach.\n\n# 2. Tool use\n**Tool use is synergistic with Flow-of-Options**. In our paper, **FoO seeks to improve the base reasoning capabilities of the LLMs _even in the absence of tools_**. However, tools can be a *force multiplier* in our work. We conducted a small experiment to illustrate this further. We incorporate external research paper retrieval as a tool with FoO. The tool retrieves and provides two papers as context into the Option Generator LLM, for the example task of fine-tuning protein language models. We note the two retrieved papers incorporated into the option generator’s context: ProtBERT (Elnaggar et al. 2021) and ESM (Rives et al. 2021). Following is the resultant FoO produced: https://ibb.co/GQL0P06N. We see that the nodes 1 and 2 in the FoO incorporates the information in the papers when proposing options. In this way, tool use is synergistic with our work.\n\n# 3. Creativity and Discovery\nThis is indeed an interesting point. We believe Flow-of-Options incorporates “combinational creativity” as described in Boden 1998. For instance, in the case of the Drug-target Interaction problem (DTI task from Table 4), our approach proposes the following ML model architecture option: Linear → Swish → dropout → Linear → GeLU → Linear (for regression). This is not an existing model per se. Although the individual components, such as the linear and activation layers exist, the specific combination of these layers can be considered novel and performs well on the task.\n\nThe individual nodes can also be combinations of existing methods. For instance, in the Drug Combination problem (DC task from Table 4), existing human baseline on the leaderboard computes features of the drug molecules using packages such as RDKit, combined with a feed-forward neural network for prediction. In contrast, our approach proposes a novel ML pipeline that combines feature extraction using ChemBERTa (an existing language model for computing the embeddings of drug molecules), with feed-forward neural network for prediction. Although the individual components exist, this combination can be considered novel. In this sense, Flow-of-Options can support combinational creativity and discovery.\n\n# 4. Synergy with humans\n\n## Q1\nFigure 14 compares the performance of the ML approaches produced by FoO vs. human designed baselines on the TDC leaderboard. The figure shows that our approach mostly achieves > 80% of an expert human’s performance. In other words, on average, it is comparable to (though not always better than) a human expert. In some cases, it outperforms the human baseline. For instance, in the drug combination (DC) task from Table 4, our approach outperforms the human baseline on the TDC leaderboard. For naive users however, FoO can significantly democratize ML.\n\n## Q2\nCurrently, the LLM proposes the options. However, these options could be informed through human guidance. Similar to our demonstration on tool use incorporating external references, the option generation can be conditioned on user inputs. In this way, the Flow-of-Options data structure can be a synergy of LLM and human knowledge leading to discovery of novel combinations.\n\n# 5. FoO with learning\nOur current implementation of Flow-of-Options seeks to improve the base reasoning capabilities of LLMs. However, it may be complementary to approaches that incorporate closed feedback learning. For instance, it may be possible to learn over walks produced by the Flow-of-Options data structure. Since each walk is associated with a corresponding metric, it can be treated as a supervised learning problem, or incorporated with reinforcement learning. This would make for an interesting future exploration.\n\n# 6. Domain expertise solutions\nWe explore two such algorithms (neither use LLM-based agents): AutoGluon (Erickson et al. 2020) for data science tasks in Section 4.1, and DeepMol (Correia et al. 2024) for TDC ADME-Tox tasks in section 4.2. DeepMol is a specialized framework for ADME-Tox domain, and AutoGluon is a specialized framework for typical Data Science domain. Both DeepMol and AutoGluon optimize for ML models without using any LLM agents. AutoGluon has demonstrated improvements over Bayesian Optimization based AutoML such as Auto-Weka and other AutoML optimization frameworks. Hence we chose AutoGluon as the sota model. DeepMol is currently one of the sota AutoML methods for TDC tasks.", "claims_and_evidence": "This paper has several claims about the performance of FoO framework, including performance, structured reasoning, generalization, and computational costs. The authors used a variety of tasks and other frameworks to compare, showing a robust advantage in performance and capacity in structured reasoning.\n\nHowever, regarding computational costs, it may be oversimplistic to compare the LLM costs simply. Though the paper claims each task cost less than $1, however, implementing and running solutions in parallel can sometimes be very computationally intensive as well. Finetuning a language model is also expensive, but it may directly or quickly provide a good enough solution without testing all possible candidates. Therefore, it's hard to say which one is more computationally efficient.\n\nThe authors also mention such a framework has a stronger generalizability. However, this framework does not generate new insights. By asking LLM to list out options, this framework is more like implementing these options and validating their performance to choose the best one. This will largely depend on the capacity of the base LLM. If the LLM is not working creatively or has a really less representative or out-of-distribution task scenario, it would largely limit the performance of this framework in a more challenging, unsolved, and even unseen task. Though other frameworks may not overcome this difficulty as well, as an agent workflow, it should not only validate their actions, but also learn from a closed feedback loop to refine their policy, which is lacking in the current paper.", "methods_and_evaluation_criteria": "The paper adopts a wide range of tasks, and the evaluation criteria are consistent with the task property. The workflow shows a generalizability of task domains. However, just I mentioned, if working on an even more challenging and creative task, this may limit the task performance.", "theoretical_claims": "The paper is mainly empirical. The limited theoretical claims come at the definition of FoF as well as the DAG nodes. I don't see any problems there.", "experimental_designs_or_analyses": "The experimental design and analyses are robust within a variety of tasks as well as model/agents comparison. However, the tasks can be more challenging (for new knowledge/solution discovery) to emphasize the value of this framework.", "relation_to_broader_scientific_literature": "This paper proposes a FoO (flow-of-options) agent framework, optimizing multiple tasks of domain performance compared to the previous traditional frameworks. Though the agent workflow shows promising advantages in performance among these tasks, it does not essentially implement a learning and actively interacting agent, which is important for the stronger AI development. Current pipeline still primarily remains on using a known structure of knowledge to solve known structure of problems, which are more like picking up a best known solution, not coming up with an even better one. In this sense, the impact is limited.", "essential_references_not_discussed": "Some discussions about optimal algorithms in each task (not simply compare the existing agent framework), as well as human cognitive findings would be necessary.\n\nFor example, when talking about varying different tasks, what if adding reasoning models with tool use like o3-mini-high or deepseek-R1? For some specific tasks, what if using Bayesian Optimization in ML tasks, rather than asking the model to propose different options? There are some known other model pipelines as well as algorithms to achieve sota performance in relevant tasks. The agent framework may need to consider how this agent framework compares with those domain expertise solutions, but not only limited to agent frameworks.\n\nOn the other hand, discussion about how humans solve this task and potentially introducing a human baseline is also meaningful. What if humans can guide the model better by step-by-step feedback? What kind of heuristics do humans have that may be useful or harmful to the agent execution? \n\nExample references:\nKahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica.\nEvans, J. S. B. T., & Stanovich, K. E. (2013). Dual-process theories of higher cognition: Advancing the debate. Perspectives on Psychological Science.", "other_strengths_and_weaknesses": "Weakness: There are really a wide range of tasks, especially across domains. If in the main figures' captions, those short names could be specified, it would be much easier for readers to understand what the tasks mean.", "other_comments_or_suggestions": "I have no other comments or suggestions.", "questions_for_authors": "1. I noticed that in the supplementary information, there is one baseline about human coding on TDC leader board (Figure 14). I don;t quite get what the figure tries to reveal, are they saying this agent framework is better or worse than the human baseline?\n\n2. I was wondering what if asking humans to guide the model to do similar tasks, for example, give LLMs compositional problems, ask LLMs to list out options and implement in parallel, what would it be better or worse than the current case? This question wants to ask how much the agent framework can work in the real practical problems compared with humans, with the same core LLM that implements codes or proposes options. Since we can always replace the core model with stronger ones, but we need to figure out how much role the workflow currently plays in the problems.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Flow-of-Options, a planning approach aimed at enhancing LLM reasoning for auto-ML applications.
The system works by representing plans as DAGs, iteratively evaluating potential paths, and updating values along each edge to determine the best sequence of options. Experiments across 16 Data Science and 17 Therapeutic Data Commons tasks show that Flow-of-Options outperforms traditional planners like SELA (based on a MCTS planner) and Data Interpreter (which also utilizes a DAG-planner).

Reviewers appreciated the method's soundness and detailed presentation, found the main design choices well-motivated and justified, and praised the comprehensive experimental validation. Initial concerns about the narrow customization of the method to narrow data science tasks were addressed by the authors, who provided satisfactory clarifications about the generalizability of the method.
Other inquiries by the reviewers including on computational cost, scalability, and relation to other agentic paradigms like tool use were also satisfactorily addressed by the rebuttals, following which all reviewers indicated scores in the accept range.
Decision: Accept (poster)