DiscoveryBench: Towards Data-Driven Discovery with Large Language Models
Authors: Bodhisattwa Prasad Majumder
Harshit Surana
Dhruv Agarwal
Bhavana Dalvi Mishra
Abhijeetsingh Meena
Aryan Prakhar
Tirth Vora
Tushar Khot
Ashish Sabharwal
Peter Clark
Venue: ICLR 2025 Poster (2025)
Keywords: scientific discovery
data-driven discovery
data analysis
large language models
hypothesis generation
hypothesis verification
Primary area: datasets and benchmarks
TL;DR: We present DiscoveryBench, the first comprehensive benchmark that formalizes the multi-step process of data-driven discovery designed to systematically assess current LLM capabilities in discovery tasks.

Abstract: Can the rapid advances in code generation, function calling, and data analysis using large language models (LLMs) help automate the search and verification of hypotheses purely from a set of provided datasets? To evaluate this question, we present DiscoveryBench, the first comprehensive benchmark that formalizes the multi-step process of data-driven discovery. The benchmark is designed to systematically assess current model capabilities in discovery tasks and provide a useful resource for improving them. Our benchmark contains 264 tasks collected across 6 diverse domains, such as sociology and engineering, by manually deriving discovery workflows from published papers to approximate the real-world challenges faced by researchers, where each task is defined by a dataset, its metadata, and a discovery goal in natural language. We additionally provide 903 synthetic tasks to conduct controlled evaluations on data-driven workflows that are not covered in the manually collected split. Furthermore, our structured formalism of data-driven discovery enables a facet-based evaluation that provides useful insights into different failure modes. We evaluate several popular LLM-based reasoning frameworks using both open and closed LLMs as baselines on DiscoveryBench and find that even the best system scores only 25%. Our benchmark, thus, illustrates the challenges in autonomous data-driven discovery and serves as a valuable resource for the community to make progress.

Published as a conference paper at ICLR 2025
DISCOVERYBENCH: TOWARDS DATA-DRIVEN DISCOV- ERY WITH LARGE LANGUAGE MODELS
Bodhisattwa Prasad Majumder" Harshit Surana Dhruv Agarwal"
Bhavana Dalvi Mishra Abhijeetsingh Meena x
Aryan Prakhar" Tirth Vora
Tushar Khot Ashish Sabharwal Peter Clark
Allen Institute for AI OOpenLocus University of Massachusetts Amherst
Website: https://github com/ alleeaa/ddssooverybent equal contributions
ABSTRACT
Can the rapid advances code generation, function calling, and data analysis using large language models (LLMs) help automate the search and verification of hypotheses purely from a set of provided datasets? To evaluate this question, we present DISCOVERY BENCH, the first comprehensive benchmark that formalizes the multi-step process of data-driven discovery. The benchmark designed to systematically assess current model capabilities in discovery tasks and provide a useful resource for improving them. Our benchmark contains 264 tasks collected across diverse domains, such as sociology and engincering. by manually deriving discovery workflows from published papers to approximate the real-world chal- lenges faced by researchers, where each task is defined by a dataset, its metadata, and discovery goal in natural language. We additionally provide 903 synthetic tasks to conduct controlled evaluations on data-driven workflows that are not cov- ered in the manually collected split. Furthermore, structured formalism of data-driven discovery enables a facet-based evaluation that provides useful insights into different failure modes. We evaluate several popular LLM-based reasoning frameworks using both open and closed L.Ms as baselines on DISCOVERY BENCH and find that even the best system scores only 25%. Our benchmark, thus, illus- trates the challenges in autonomous data-driven discovery and serves as valuable resource for the community to make progress.
INTRODUCTION
Knowledge discovery via the scientific process has been a catalyst for human progress for centuries but has, thus far, been predominantly manual pursuit (Glass & Hall 2008). Recent breakthroughs in capabilities of large language models (LLMs) to reason and interface with the world using code (Chen cet al.] Roziere al. (023), external tools (Schick et al. 2024). and interactive agents (aao et al. 2023 Majumder et al. 2023), however, now suggest the possibility of realizing a discovery system that fully autonomous. Indeed, recent works (Majumder et al. (024) provide initial evidence for this paradigm within the setting of data-driven discovery, where both search and verification of hypotheses may be carried out using dataset alone (i.e., after physical experiments and data collection [T but the extent of this ability remains unclear. We, therefore, aim to systematically evaluate the following question:
How capable are current state-of-the-ar LLMs at automated data-driven discovery?
Answering this question is hard, data-driven discovery in the wild (real-world) is diverse across domains and subject areas, which in turn makes it difficult to build a robust evaluation framework to measure progress. We address this using pragmatic formalization of data-driven discovery, namely the search for a relationship that may hold between variables in a context, where (importantly) the description of those facets may not in the language of the dataset. A data-driven discovery task
In practice, experiments and analysis are interleaved, not sequential. Our concern in this work, however, is systematically studying the data analysis part of the (interleaved) pipeline.

---

Published as a conference paper at ICLR 2025
Dataset
Data Loading & Transformation
Regression Analysis
habitat
nannative pardening
unintentiooaal
nannative agriforest
elevation....
alyylogenttie_rgresssiiee
craplands
5
0
675
read_csv()
interactions factor_ combinations
wetlands
urban
2
3
329
Model Fit Selection & Check
Collinearity Check
compare _r__est( aice_values pgls_clinactie()
Goal
How urban use affect the invasion of different types of introduced plants in Catalonia?
inflation ractor
Final Answer: Urban land use increased invasion by plants over gardening introduced ones Catalonia.
DiscoveryBench Task
Ideal Discovery Agent Workflow
Each DISCOVERY BENCH task consists of a goal and dataset(s) (left). Solving the task requires both statistical analysis and scientific semantic reasoning. The realistic requirement of domain-specific tools some tasks adds more complexity. An idcal discovery system should implement discovery programs in Python.
Figure 1:
then has one of these components missing, e.g., "How did urban land use affect the invasion of introduced plants Catalonia?", Importantly, this formalization allows for systematic, reproducible evaluation over wide variety of real-world problems, by leveraging these facets.
Unlike prior datasets for statistical analysis (Liu et al. 2024) or AutoMI. (Zhang al., 2023 Gijshers et al. 2022). DISCOVERY BENCH tasks also require scientific semantic reasoning, for instance, deciding which of the many possible analysis techniques are appropriate for the domain (e.g.. phylogenetic regression for plant invasion, [. how clean and/or normalize the data, and how to map goal terms to dataset terms (e.g.. "land use to "habitat type"). Task solutions typically requires multistep workflow (Fig [ right). In this way, DISCOVERY BENCH is the first large-scale dataset to address the broader data-driven discovery pipeline, not just the statistical analysis component, and explore LLMs' capacity for this.
jutt
Given this framework, we created DISCOVERY by manually extracting 264 discovery tasks, i... goal dataset(s), from over 20 published papers, as well as creating real-world discovery workflows that solve each task. We additionally provide 903 synthetic tasks across 48 domains generated using LLMs to mimic the real-world discovery process. The synthetic benchmark is complimentary, capturing a diverse range of data-driven workflows that could not be included in the manually collected split of our benchmark due to lack of reproducibility. Our contributions are thus:
DISCOvERY EERDOVERRBENCHH,, the first comprehensive benchmark to formalize the multi-step (code-based) process of data-driven hypothesis search and verification, covering many real-world discovery tasks plus additional synthetic tasks.
A pragmatic formalism for data-driven discovery. flexible enough to characterize many real- world tasks while constrained enough to allow for rigorous, reproducible evaluation.
A comprehensive evaluation across faate-of-the-ee LLM- reasoning methods ("discovery agents"). We find performance peaks at 25% demonstrating the challenging nature of our task.
These suggest that DISCOVERY BENCH may be a valuable resource for helping make progress on autonomous, data-driven discovery.
2 RELATED WORK
Automated data-driven discovery has been a long-standing dream of AI (Majumder et al., 2024, Kitano 2016). Although there have been a range of data-driven discovery systems, from early ones that fit equations idealized data, e.g., Bacon (Langley) 198T), more modern ones handling complex real-world problems, c.g., AlphaFold (Jumper et al. 2021). their associated datasets are task- specific and customized to a builb pipeline. In contrast, DISCOVERY BENCH aims be general, including testing whether systems can design appropriate workflows themselves over multiple tasks.
Several datasets and tools are available for AutoML, related technology aimed at automating workflows for building optimal machine learning models (Jin et al.) 20033] Zhang et al.) 2023] LeDell & Poirier 2020). AutoML tools include packages like Scikit (Feurer et al. 2015),, and embedded in cloud platforms such Google Cloud Platform, Microsoft Azure, and Amazon Web Services.
2

---

small set, Published as a conference paper at ICLR 2025
However, associated datasets for AutoML are primarily used for training models, rather than for open-ended discovery tasks.
Similarly, there several datasets that test statistical analysis in various fields, c.g..
Feature
Ours
QRData
DSBench
BLADE
File handling Statistical tools
(Shao et al. 2023 Li al. 2024 Yang et al.
/
2022). Software packages like Tableaux,
Data Semantics
SAS, and R also support users in that task. However, these datasets and tools are de- signed specifically for data analysis, while DISCOVERY BENCH aims to automate the broader pipeline including ideation, semantic reasoning, and pipeline design, where statistical analysis is just one component.
Hypothesis Discovery
Domain Knowledge
X
Domain-specie Tools
Scientific Hypothesis
Table Comparison with other data-analysis benchmarks
One recent dataset similar in spirit to ours is QRData (Liu et al., (024). QRData also explores LLM capabilities but targets statistical/causal analysis for well-defined (mainly) textbook questions that have unique, (mainly) numeric gold answers. DSBench (Jing et al., 2024) proposes data science tasks that require agents understand multimodal input files, generate complex code, and interpret and process them to solve a multiple-choice or short direct-answer task. Unlike DISCOVERY BENCH. these tasks do not require domain knowledge or hypothesis discovery. Recent work, BLADE (Gu et al. 2024), presents complex hypothesis verification as a multiple-choice QA task. Even though expected workflows are multi-step, at each step, the space of operations is limited e.g., select, map, group-by, etc. In contrast, DISCOVERY BENCH has no prescribed boundaries on statistical techniques apply, uses open-ended questions and answers, and complex tasks drawn from state-of-the-arl published work. Further, DISCOVERY BARRYBENC ENCI poses a high-level research question that requires an agent to do a hypothesis search in addition verification. Table summarizes a list of distinguishing features of DISCOVERYBENCE which promotes thorough, systematic evaluation of (L.LM-driven) discovery systems in the challenging but realistic task of data-driven discovery.
3 FORMALIZATION
We begin by formalizing what we mean by a data-driven hypothesis and how the structure of a complex hypothesis may be viewed hypothesis semantic tree.
data-driven hypothesis in H. (the space of such hypotheses) is a declarative sentence about the state of the world whose truth value may be inferred from a given dataset D using verification procedure Vn= H - (supported, unsupported]. For our benchmark, the space of valid Vn is potentially any executable Python program.
Inspired by recent work of Thompson & Skau 2023), we additionally introduce a structured formal- ism that breaks hypothesis down into three hypothesis dimensions:
Contexts (c): Boundary conditions that limit the scope of hypothesis. E.g., "for men over the age of 30" or "in Asia and Europe" or unbounded/full dataset when not specified.
Variables (v): Known of concepts that interact in a meaningful way under a given context to produce the hypothesis. E.g., gender, age, or income. Note that each hypothesis is associated with target variable and of independent variables.
Relationships (r): Interactions between given set of variables under given context that produces the hypothesis. E.g., "quadratic relationship". "inversely proportional" " or piecewise conditionals.
With slight abuse of notation, we can now cquivalently define hypothesis (.).:.)) is generative function that returns the declarative sentence "under context c, variables v have relationship r." For instance, for the hypothesis from Fig == "urban land use; plant type agriforest and gardening", u == "invasion rate", and == "invasion rate is higher in agriforest plants than gardening plants." Note that the hypothesis itself is sentence, not tuple.
(ic, u, r). where
4 BESCH
We now introduce a novel benchmark, DISCOVERYBENCHH for discovering data-driven hypotheses. In this benchmark, data-driven discovery task is defined follows: Given one more task
3

---

Published as a conference paper at ICLR 2025
filtering based deaographica for soctology data
taking outtier health social values for nia data
combining wor Idbank data ebout incose, sevinga, education, exporte, echool enro Liment
statistical teats to non-native
Data Preparation (569)
Statissts (158)
statistical tests
Summary stattsrics
cedaplicatiin for sete analymia replication studtes
selection
feature anga create sarasave vartables
Feature Engg. (112)
regression explatn spatial vartation spectation rates
data deaning
time analysia pee logical
data matching
applted
time quantile regraaion to atudy effect of wealth tevels of incarceration across devogr ophica
regrassion
data doduplicatio
pca
data integratten
Cparcle
Widda
econowtric modeling for world bank indicators temigration data
staffstical sampling
gin
coee
Jober
Advanced Statistics (104)
Other (41)
Domain Specific (26)
Figure 2: Workflow categories in DB-REAL with representative examples.
dataset(s) D and a discovery goal G, derive hypothesis addressing G with the highest specificity for the context variables and relationship supported by D. Optionally, a workflow for deriving a hypothesis can be output to augment information already present in the hypothesis. DISCOVERYBENCH has two components: OB-REAL encompassing data-driven hypotheses and workflows derived from published scientific papers and DB-SYNIE capturing systematic variations in data-driven hypotheses and workflows obtained from synthetically generated datasets. The benchmark released at: ht /ttp://::tthi com/a a 1 lenai /ii s acover reveerhyhcc The licenses permitting the re-distribution of datasets collected from past work provided in Appendix B and Table 3
4.1 DB-REAL: COLLECTING DATA-DRIVE HYPOTHESES IN THE WILD
Our goal is to replicate the scientific process undertaken by researchers to search for and validate a hypothesis from one more datasets. We focus six scientific domains where data-driven research is the cornerstone of scientific progress: sociology, biology, humanities, economics, engineering. and meta-science. Our gold trajectories to solve discovery task carefully follow the published papers' workflows in respective domains. As most of the papers are highly cited, peer-reviewed, and from top venues from the domains, it is reasonable assume the published workflows are scientifically valid.
To ensure that we correctly reproduced the methods and code implementations and reached the correct hypotheses: (a) We restrict ourselves to cases where the complete code was available (e.g., in GitHub) from the original authors, so we could directly reuse it; (b) For the Economics and Sociology related hypotheses, we consulted domain experts (Economies PhDs) to verify that our interpretation, implemented workflows, and replicated hypotheses make sense; (c) For ML engineering, we relied on our own expertise in the domain. See Appendix B for the original sources of data and workflows.
While validating novel discoveries is incredibly challenging and requires domain experts to verify, the goal of DISCOvERY BEnCH not to make new discoveries rather to test if a discovery system can at least make known, validated discoveries correctly as first step the broader goal of automated discovery-an an easier bar, and one which our benchmark can assess systematically. We detail our data collection process, which follows either data-first code-first approach.
For the data-first approach: 1) we filter papers based on open public datasets (D) such as National Longitudinal Surveys (NLS), Global Biodiversity Information Facility (GBIF), and World Bank Open Data (WBOD) that have workflow details; 2) we then try to replicate these workflows in Python. For this data-first approach, replication took up 90 person-hours per dataset, often (30%) not resulting
4

---

Published as a conference paper at ICLR 2025
This highlights building data-driver discovery benchmarks from real studies is not only challenging and time- consuminn, but automating discovery can also be key for scientific progress and reproducibility.
in success.
The data-first approach by design is limited to well-known aforementioned public datasets. To improve diversity in domains, datasets (D), and workflows, we also adopted code-first approach to look beyond popular public datasets. In this approach, we 1) search for code repositories based on scientific papers with available datasets and 2) attempt to replicate them in Python with existing code or from scratch with interpretation of the associated paper. We looked at 785 data points in Zenodo. EU's Open Research Repository, with filter for computational notebooks. Over 85% of the repositories either had missing code, code that could not be easily translated to Python, or proprietary/non-opee dataset. We finalized candidate list of 14 repositories, but in the end, 3 of them passed all our checks for their hypotheses to be included in the benchmark
Upon replication of the result or implementation of the full procedure as described in the paper, we include the (dataset D, hypothesis h, implementation workflow tuple to the benchmark.
During the process, the implementation workflow may lead to other hypotheses that are not directly reported in the paper but can be supported by the data. We included them in DISCOVERY BENCH, which leads to a good mix of already reported science- worthy hypotheses as well as novel hypotheses grounded in datasets. This is particularly useful as goal is to evaluate LLMs' ability to solve discovery task that is realistic but never reported before.
Finally, the task datasets are supplemented with dataset description, natural language descriptions of the columns, and additional background knowledge related to the domain or the datasets. Some of tasks, for instance, archaeology, require domain knowledge to derive particular hypothesis.
Inferring task difficulty. Inferring the hypothesis space given a task dataset alone is impractical due to incomplete a priori information about unobserved intermediate variables and their association with observed ones. To infer task difficulty, we, therefore, use the length of the implementation workflow required derive target hypothesis. Each step in the workflow contributes to workflow length. In some cases, we derive two tasks: easy and hard from the same hypothesis, where for easy, we provide the derived variables as observed variables in the dataset (e.g., BMI), and for hard, it would require deriving intermediate variables (BMI from height and weight) to reach the target. Intuitively, discovery becomes harder as the hypothesis search space increases. In practice, this setting is observed when a task requires access to multiple datasets.
unit length
Forming discovery goals. By definition, each hypothesis can be fully specified by the declarative sentence as h == (,,,, r). To systematically construct the discovery goals for the task, we first mask one of each dimension, context c, variable v, relationship r, and generate a discovery goal to identify the masked information given the rest of the hypothesis and the task dataset(s). For instance, for a target hypothesis, "The effect of socioeconomic status on college degree completion is higher for females (0.4995) than males (0.4467) we form a discovery goal "How does socioeconomic status impact on college degree completion for females compared to males?" seeking the relationship to be discovered from the dataset(s) given the relevant variables and context c. Additionally, ensure each discovery goal leads to only one answer, i.e., the target hypothesis.
4.1.1 FEATURES OF DB-REAL BENCHMARK
Train
Test
BENC incorporates a broad landscape of data-driven discovery. With over 500 instances of data preparation activities such as cleaning, deduplication, and integration, captures the complexity of real-world data pre- processing for discovery. Tasks also demand a spectrum of statistical methods, from statistical tests to mixture models, and include domain approaches in econometric and ecological modeling, as reflected in the Fig 2
# tasks # unique hypotheses # tasks need > dataset # domains
25
239
14
144
4
110
3
6
Table Statistics DB-REAL
"Some repositorics include hypotheses from multiple papers as their background. "Implementation workflows are descriptions of implementations to reproduce the results from original works. 'A task may require multiple data preparation and analytical activities.
5

---

Published as a conference paper at ICLR 2025
Table 2 shows the diversity of tasks both in train and test split for DB-REAL. Most importantly, the benchmark incorporates 1 114 (4 + 110) tasks that require more than one related datasets to be analyzed, with maximum of 6 datasets for a task. Each workflow within the dataset can be viewed as a composition of unit actions-such as code generation for statistical tests-thar L.Ms excel at, showing how our tasks require the chaining of such atomic actions to address complex scenarios for data-driven discovery. We measure the complexity of these workflows by quantifying the number of unit actions involved, referring to this as the workflow length, whose distribution can be seen in Fig
4.2 DB-SYNTH: GENERATING DATA-DRIVEN HYPOTHESES USING LLMS
While DB-REAL covers a broad set of domains and workflows, several other types of workflows (e.g., nonlinear regression, cluster analysis) could not be included in the benchmark due to difficulty in replication or oft-of-scope domains. However, to provide a comprehensive view of data-driven discovery, we introduce complimentary split of our benchmark, DB-SYNTH, which captures synthetic task examples constructed from inspirational workflows from published works (Galbraith) ct al., 2010). Note, the workflow categories (e.g., symbolic regression, equation discovery, and function approximation) in DB-SYNTII are taken from a broader set of papers beyond what we considered in DB-REAL; hence the difficulty and solutions of the tasks broadly vary from the real split of our benchmark.
Our goal is to reverse-engineer the process of hypothesis discovery synthesize datasets and discovery tasks that require analysis workflows similar to those that are broadly observed in published works. Our approach leverages the broad pre-trained knowledge of L.MSs in four stages:
Domain sampling: First, we prompt the LLM to generate a list of diverse domains along with their natural language descriptions. E.g.. "Ancient architecture" + "Related to architectural marvels, ancient building and construction techniques." Note, such domain is not covered in DB-REAL.
Semantic tree construction: For each domain, we then build a hypothesis semantic tree Th, which is tree of variables depicting the dependencies between the vari- ables in the dataset (leaf nodes), intermediate variables, and the target (dependent) variable of a primary hypoth- esis h (root node). Note, Th, is not the hypothesis itself.
goloxy vis liiiiyy
vo
Amemto Rige
Specifically, we prompt the LLM with the domain and a sampled real-world workflow (e.g.. "within-cluster anal- ysis") to generate a hypothesis and its target variable. To encourage diversity in plausible (as per the L.M) workflows and to increase coverage (by accounting for uncovered workflows in DB- RE-RE,L), we allow LLMs combine several real- workfwoww steps (type of variables, type of relationships, etc). This significantly improves the likelihood of generating synthetic workflows that closely resemble the real ones. Setting the target variable as root (e.g., ga alayyvvvv wwiiiiiiiiii. lityy, we then derive child nodes by generating the independent variables required to verify h using VV-). as seen in 3ig[] We operationalize this by generating a column name and description for each child node (along with data type and range) and a pandas expression (Wes McKinney/ (010)) over only independent variables in TA such that its execution results in the target variable. We repeat this with each leaf in T as the root of new semantic sub-tree, generating intermediate hypotheses and new set of variables until the desired height of T is reached We also generate set of distractor columns disjoint from nodes in T, thus resulting in synthetic semantic forest F.
U1
U2
palaxp_sia
dousiyyy
U4
Figure 3: Hypothesis Semantic Tree
Data generation: We then construct a task dataset by generating synthetic data in a bottom-up manner (i.e., from leaves to root) for each node in F. Starting with various sampling strategies for leaf nodes (sec more in Appendix C), for each subsequent level in F, we create new columns for nodes by simply executing their pandas expressions. Finally, to mimic real-world challenges in data
"The pandas expression encodes the structured hypothesis (e(c, r).
'In practice, with probability 0.6, we choose whether node is derived further or marked Icaf.

---

Published as a conference paper at ICLR 2025
collection, we probabilistically perturb each instance x x; by adding noise dropping values to create missing data Note that at this stage, D contains a column for each node in F.
Task generation: For each internal node h in F, we now create multiple task datasets D from D, varying the difficulty of the discovery task based on the path length between h and the observed independent variables in F. Finally, we follow the same strategy for goal formulation as DB-REAL, We generate 903 tasks over 48 diverse domains and assign them to development and test sets using an 80/20 split, where each task is additionally tagged with difficulty level from 1-4.
See Appendix for the compositional details on both real and synthetic splits of DISCOVERY ASCOVERRYEENCIL BENCH. We also list some limitations and ethical considerations around our benchmark in Appendix A
4.3 EVALUATION
We evaluate task performance by measuring the alignment of the predicted and gold hypotheses in natural language. We take inspiration from recent works in LLM benchmarking (Shashidhar it al.] 20223 Zeng et al. 2023: Yuan et al., 2023; Fu et al.) 2023 Li et al. 2023 Lin et al.) 2024) and design model-based evaluation strategy using gpt preview 0125 as the evaluator, conditioned on our structured formalism of data-driven hypotheses. Critically, the evaluator assesses entailments//qquivlleeess between linguistic elements of predicted and gold hypothesis pair, folow- ing from several LM-based language entailment as automatic tools for scientific claim verification
(Wadden et al. 2022 Min et al. 2023 Schlichtkrull et al.).
Recall the declarative form h == (0,,, v, r) of hypothesis h that constitutes of three hypothesis dimensions: context (c), variables (v), and relationships (r). Given this structured representation of the gold and predicted hypotheses, we then compute hypothesis match score (HMS), which measures the degree to which two hypotheses align on each dimension, as follows:
context: We first use our GPT-4 based evaluator to independently extract the contexts for the gold (h²) and predicted (h") hypotheses (prompt in Listing []. Then, we compute a context alignment score atxalign, which takes value 1 if a GPT-4 based evaluator deems the contexts extracted from h and h² as semantically equivalent, otherwise 0 (prompt in Listing (). Note, instead of comparing the full hypotheses, we employ a much simpler faceted textual entailment (or semantic similarity) problem between two text segments, which GPT-4 is very strong at (Sanyal et al. [002)..
variables: Using our LLM evaluator, we extract the set of interacting variables in the gold and predicted hypotheses using the GPT-4 based evaluator (prompt in Listing 3 We compute the alignment between these two sets of variables as an Fl score, varf1, capturing how aligned the variables from he with the variables from h?.
relationships: Our GPT-4 evaluator computes a relationship accuracy with reference to the relationship between the gold variables (relace) based on evaluator judgments using the following scoring heuristic: 1 if there is an exact match of the relation, 0.5 when the predicted relationship is broader than the gold relationship but encompasses it and otherwise (prompt in Listing 4
Two authors independently evaluated the step-by-step L.M M-baeeed evaluation process on 200 examples. After resolving disagreements, we find GPT-4 was 97% correct in context extraction, 99% correct in context matching. 94% correct in variable extraction, and 99% correct in assessing relationship accuracy, exhibiting sufficiently strong performance in entailment-basse evaluation. Finally, we compute HMS (scaled) E [0, 100] as the alignment of the variable and relationship dimensions over montext-matched hypotheses weighted by the overall context alignment:
HMS((P, h19) = etxalign (h²,, 11)) X varFi (hP, 119) x relace (h², , 1/9)
Is the untcome-based evaluation sufficient? Yes. The gold hypotheses, which a discovery agent does not see the test time, are intentionally designed to be very specific, making it highly improbable to arrive at them by chance or via invalid means. F.g., while a discovery agent might guess a positive correlation via invalid means, it would be unable to compute the precise target correlation coefficient without using the correct statistical analysis. The HMS metric is designed to ensure that all aspects of the target discovery are satisfied in order to receive full reward. Most significantly, the very low evaluation scores (in Fig 4 substantiate this-even the best agents perform poorly, suggesting that random guess or invalid shortcuts do not lead to the target result.
Fach value is noised independently; therefore, each row has sufficient data useful for discovery.
7

---

Published as a conference paper at ICLR 2025
GPT-4o
GPT-ip
Llama-3
GPT44 Cocedion
GPT-4o tV GPT-4p CodeGian
GPI React 4PT-4p DV
Uimm CodeGenn
Ulma
DB-REAL
GPT44 Reast
Llamad Blact
NoDataGuess
0.0
4.7
11.5
CodeGen
20.9
19.3
18.9
React
24.4
22.4
18.4
DatuVoyager
20.8
20.4
11.6
6.15
Reflesion (Oracle)
24.6
23.1
22.5
relac
DB-SYNTH
CodeeGen
14.1
8.7
10.9
React
11.6
7.4
12.0
DatuVoyager
10.1
6.9
11.7
0.36
6:40
CM
0.30
0.36
Reflexion (Oracle)
15.7
12.9
23.2
CErukgn
varri
Figure 4: (Left) Hypothesis Matching Scores (HMS) across LLM pairs in DB-REA and DB -YNT.. (Right) Scatter plots for mean ctXalign vs mean vatF1 mean varrI vs mean relace. The plots show that agent-LLM combinations that are capable of identifying correct context have higher likelihood of obtaining the right set of interacting variables. Similarly, identifying correct variables leads to better identification of relationships between them.
engg (15)
(a)
(b)
150
(c)
25
econ (12)
125
DB-Real
DB-Synth
20
blo (16)
frequency
100
frequency
100
science (100)
75
15
5
humanities
10
50
50
sociology (58)
25
5
ctx
rel
var
m
1 2 3+
Figure 5: Best non-oracle agent's performance (HMS) (a) across domains, (b) for goal types (dimen- sion to be discovered), and (c) for different workflow lengths. In (c) workflow length categories for DB-REAL are s: < 10, m: >> 10, < 20. >> 20. For OB-SYNTH is the semantic tree height.
Human alignment with HMS. To assess the robustness of the HMS metric, we ask 3 annotators provide preference ranking over predicted hypotheses for 100 such random pairs (from DB- REALtest set) when compared to their respective gold hypotheses of each task. We then compute the corresponding preference ranking using our HMS metric and compare the two lists of preference rankings. We find human annotators agree with the HMS-based ranking 95% of the time. Each pair evaluated by three annotators (and majority alignment verdict is taken), with a Fleiss kappa of 0.91, showing a very good strength of agreement among annotators. For the null hypothesis, "there difference between human and HMS rankings," the power of the with 100 sample pairs is reject the null hypothesís, i.e., we find the rankings obtained by HMS are
< 0.05. Upon testing, statistically the same (p 0.001) as the rankings provided by human annotators. Our finding here supports the evidence in literature human alignment in LM-based evaluation for scientific claim verification (Wadden et al. 2022 Schlichtkrull et al.).
EXPERIMENTS
5.1 DISCOVERY AGENTS
We benchmark state-of-the-art LLM-based few-shot reasoning methods discovery agents with two closed models, GPT-4o and GPT-4-0125-previee (GPT-4p), and one open, Llama-3-70B, model powering the reasoning methods. A discovery agent takes the task description, paths to the task dataset(s) D. metadata about the datasets (description, column descriptions), and the goal, G. to produce natural language (NL) hypothesis specified by context, variables, and relationship.
go to solve the task, where provide demonstration solution code in the context. After code execution and based on the result, it generates the NL hypothesis and summarizes the workflow.
CodeGen generates the entire code
8

---

Published as a conference paper at ICLR 2025
HMS
60
Frequency
100
50
20
data selection
feature enginearing
summary statisties
meta regression
data sta saaaddrrzzaaion
data cleaning
data trann transformatioo
regression
data integratton
comples data join data matchii hing data peduplication
stalistic al naslsi
data explor atooo enalysis statistical samp campling pling
rassifiication
interaction analysis
mixture models
ecomometrics
quantle regression
ecological modelino
dimorsioon roduction
pollen modeling
correlation analysis
gitii confficiinn
gim
spa6 al analysis
me mediation analysis
Figure 6: Breakdown of performance (HMS) of the best non-oracle agent in several workflow categories their decreasing order of appearance the benchmark.
ReAct (Yao 2023) solves the task by generating thought and subsequent codes in multi-turn fashion based on the past thoughi-action-obberraiii trace.
Data Voyager coreeoosooommmmmmnnn data-driven discovery agent from (Majumder et al. 2024). It has four components, planner, code generator, data analyzer, and critic, that orchestrate the discovery process.
Reflexion (Oracle) (Shinn et al. 2023 is an extension of CodeGen agent, where at the end of one trial, we provide the "oracle" HMS score as an evaluation signal, and generates a reflection to improve (when HMS 1) in the next trial till it solves the task, maximum trials (3) are reached. NoDataGuess guesses the hypothesis (in OB-REAL) just from the dataset description and the goal without accessing the datasets where measure LLM's memorization of already published works.
5.2
MAIN RESULTS
Fig 4 left) shows that overall performance for all framework-LL.M pairs is low for both DB-REAL and DB-S ONNSY,,H, highlighting the challenging nature of the task and the benchmark. Most importantly, effective reasoning prompts such as React and planning with a self-critic
Voyager) do not help improve the simple CodeGen agent. But with oracle feedback, Reflexion (Oracle) significantly improves over CodeGen (base) performance. Analysis reveals that almost all non-reflexion agents solve the easiest (in terms of workflow category and length) instances from the benchmark. GPT-4o refuses to hallucinate in the NoDataGuess baseline, whereas surprisingly, Llama-3 performs similarly in both data and no-data modes. We additionally observe that agents that perform well DB-REAL may not perform equally (well) in 1D--YNNH possibly due to the differentiating nature of underlying workflows. For e.g., while it is easier perform linear regression for discovery agents, performing non-linear regression embedded in DB-SYNI is hard.
5.3 DISCUSSION
Context important. Fig rriht) shows score trends in the three-pronged discovery process. positive trend between ctxalign and varFı signifies that to predict variables accurately, accurate context prediction is necessary. However, correct identification of context is an important first step, although it does not guarantee success. Similarly, predicting the variables correctly increases the chance of predicting relationships among them. Additionally, we find clusters (e.g.. Llama-3 has consistently lower scores) in terms of dimension-level performance around I.I ..MMs-agent frameworks. Also, across LLMss a very similar trend is observed as they operate from different agent frameworks.
Performance across domains and goal types. Fig a) depicts that biology (0%) and engineering (7%) perform the worst due to their higher dependence on advanced statistical methods, while economics (25%) and sociology (23%) perform better. Additionally, Fig 5(b) shows goals related to discovering a relationship given context and variables are more easily solved than the other two types of goals, finding context and variables. This is explained by the complexity of the hypothesis search, which is broader for finding the right context or a set of variables given a fixed relationship, whereas finding the relationship given context and variables is easier.

---

Published as a conference paper at ICLR 2025
Goal: In which century did house sizes and daggers significantly decrease simuitancourly for the second time since the start of the observational data?
Goal: Which documentation format the most frequently used requirements in enabled system projects after bootstrapping?
Goal:What the
the relationship
between the rate of body elongation
evolution and speciation rates?
Notchooks (46.504% subjects, 95% 166, 46,879) for significance with bootstrapoingl most frequently used documentation format requirements ML-enabled system projects.
Gold: Rate of body elongation evolution has no signifcantt relation with speciation rates.
Gold: Around 1000 BCE, house sizes and daggers significantly decreased simultancously the second time.
There statistically significant ppositive relationship between rate of body elongatioo evolution and speciation with prvalue of approvimately 3.09e-21
Prediited::
occurrence of significant simultancous decrease in house sizes and daggers happened in the 10th century BCE.
Predicted:
Predicted: Notebooks are the most frequently used format documenting
rate of body elongation evolution expleins roughly 17 17.8% the variability speciation rates.
requirements in MI-enabled system projects.
Workflow categories:
Workflow categories:
Data Semantics: Ø
Data selection: Y
Workflow categorles:
cleaning:
Summary statistics: V
Semantics:
Time series segmentation:
Bootstrapping for statistical test:
X
Multivariate regression:
X
HMS: 100
HMS: 50
1% with significonce not reported)
HMS::
(wrong relationship relationnsiprpprreedd reported)
Figure 7: Representative examples of tasks where we qualitatively evaluate non-oracle best agent's trajectories along with their performance indicated by HMS.
Domain knowledge dependency. To check if additional domain knowledge helps agents perform better, we collect targeted domain knowledge for the archaeology-relatee tasks that needed significant domain knowledge during data collection. When added as additional hints, we find that Data Voyager's (GPT-4p) performance jumps from 9.9% (w/o domain knowledge) to 25.4% (w domain knowledge) for archacology tasks.
Workflow complexity barrier. Almost all agents struggle more with tasks involving complex statisti- cal techniques, complex data preparation methods, or dopain-specific models. The top three workflow categories where the best non-oracle model was highly performant are correlation analysis, time series analysis, and social-science-relttee analysis, whereas the lowest three workflow categories are spatial analysis, pollen modeling, and ecological modeling. Fig. Flsoos overlayed HMS on several workflow categories with several informative spikes and drops. Simpler, common workflow categories (c.g., correlation, regression, data cleaning) see performance spikes, whereas long-tail workflow categories pose a great challenge even to the best discovery agent. The richness and the requirement of long-tail analysis distinguish our benchmark from the traditional data-analysis benchmarks, providing systematic way evaluate LLMs' capability in realistic data-driven discovery.
Impact of workflow length. Inherently, the difficulty of the tasks is measured by the gold workflow length
the height of the semantic tree (DB-SYNTH). Figure shows decreasing trend in performance as workflow length (hence, complexity) increases. The performance drops significantly even for medium-lengle workflows, highlighting current agents' limitations.
Qualitative (error) analysis. Fig. 7 depicts three examples where the best non-oracle agent shows varied performance. While the common tendency of LLMs is arrive an answer goal as soon as possible, this behavior appears to be costly for our benchmark. For example, when asked to perform additional significance tests to verify the importance of summary statistics, the agent does not pursue it consistently, resulting in low scores. Very often, most discovery agents will not perform thorough reasoning while solving a task from the benchmark, For example, when several predictive factors coexist, the relation between two variables should be validated by taking all of them into account (c.g., via a multivariate regression). However, the impatient discovery agents register simpler analysis, which (in Fig. eexample) yield either underspecified or wrong answers.
6 CONCLUSION
We present DISCOVERYBENCH. the first data-driven discovery benchmark consisting of 264 discovery tasks that capture real scientific workflows extracted from published works. We supplement this with 903 structurally generated synthetic tasks, tailored to evaluate discovery agents for complimentary workflows. We benchmark state-of-the-art art reasoning frameworks with the most advanced LLMs, but the best agent's performance only peaks at 25% underscoring the challenging nature of the task and the benchmark. We hope our timely contribution can increase interest and efforts in making progress on reliable and reproducible autonomous scientific discovery using large generative models.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Karl I. Alexander, Cornelius Riordan, James Fennessey, and Aaron M Pallas. Social background, academic resources, and college graduation: Recent evidence from the national longitudinal survey. American Journal of Education, 90(4):315-333 1982.
Antonio Pedro Santos Alves, Marcos Kalinowski, Görkem Giray, Daniel Mendez, Niklas Lavesson, Kelly Azevedo, Hugo Villamizar, Tatiana Escovedo, Helio Lopes, Stefan Biffl, et al. Status quo and problems of requirements engineering for machine learning: Results from an international survey. In International Conference on Poduuct-Focued Software Process Improvement, pp. 159-174. Springer, 2023.
Antonio Pedro Santos Alves, Marcos Kalinowski, Daniel Mendez, Hugo Villamizar, Kelly Azevedo, Tatiana Escovedo, and Helio Lopes. Industrial practices of requirements engineering for ml-enabled systems in brazil. arXiv preprint arXivv2407.18890 2024.
Robert Apel and Gary Sweeten. The impact of incarceration on employment during the transition to adulthood. Social problems, 57(3):448-477 2010.
Elizabeth N Appiah. The effect education expenditure on per capita gdp in developing countries. International Journal of Economics and Finance, 9(1)):136-1) 2017.
Johanna Brinkmann. Copper output, demand for wood and energy expenditure-evaluttie economic aspects of bronze age metallurgy. How's life, pp. 11-34, 2019.
Jan Piet Brozio, Johannes Müller, Martin Furholt, Wiebke Kirleis, Stefan Dreibrodt, Ingo Feeser, Walter Dörfler, Mara Weinelt, Hendrik Raese, and Annalena Bock. Monuments and economies: What drove their variability in the middle-holocene neolithic? The Holocene, 29(10):1588-1511 2019.
Jan Piet Brozio, Jutta Kneisel, Stefanie Schaefer-Di Maida, Julian Laabs, Ingo Feeser, Artur Ribeiro, and Sebastian Schultrich. Patterns socio-cconomic cultural transformations in neolithic and bronze age societies in the central northern curopean plain: Human-envirommental interaction concerning bourdieu's 's forms of capital. In Perspectives on Socio-environmental Transformations in Ancient Europe, pp. 105-142. Springer, 2024.
Felipe Cerezer, Cristian Dambros, Marco TP Coelho, Fernanda AS Cassemiro, Elisa Barreto, James S Albert, Rafael O Wüest, and Catherine H Graham. Accelerated body size evolution in upland environments correlated with recent speciation in south american freshwater fishes. Nature Communications, 14(1):6070, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333 03374, 2021.
Marta Dal Corso, Wiebke Kirleis, Jutta Kneisel, Nicole Taylor, Magdalena Wieckowska-Lüth and Marco Zanon. How's Life? Living Conditions the 2nd and Millennia BCE. Sidestone Press, 2019.
Christopher Dougherty. Numeracy, literacy and earnings: Evidence from the national longitudinal survey of youth. Economics of education review, 22(5):511-521, 2003.
Ingo Feeser, Walter Dörfter, Jutta Kneisel, Martin Hinz, and Stefan Dreibrodt. Human impact and population dynamics in the neolithic and bronze age: Multi-proxy evidence from north-wester central europe. The Holocene, 29(10):1596-1660 2019.
Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. NeurlPS, 2015.
Jinlan Fu, See- See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate you desire. ArXiv, 00122322041166 2023. URL https: /api. semanticscholar org/CorpusiD? 2.6662
11

---

Published as a conference paper at ICLR 2025
Sally Galbraith, James A Daniel, and Bryce Vissel. A study of clustered data and approaches to its analysis. Journal of Neuroscience, 30632):0001111008,, 2010.
Peter Gijsbers, Marcos L. P. Bueno, Stefan Coors, Erin LeDell, Sébastien Poirier, Janek Thomas, B. Bischl, and Joaquin Vanschoren. Amlb: an automl benchmark. ArXiv, abs/2207.12560) 2022. URL http::tpt:///mm //ppiappp...//// www.app...aaaaaam.mmm..mmmm....e semanticschol ar org/ CorpusID: 251 066648
David Glass and Ned Hall. A brief history of the hypothesis. Cell, 134(3)378-331, 2008.
Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, et al. Blade: Benchmarking language model agents for data-driven science. arXiv preprint arXiv:2408.09660 2024.
Rachel Heyard and Leonhard Held. Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects. Technical report, Center for Open Science, 2024.
Haifeng Jin, François Chollet, Qingquan Song, and Xia Hu. Autokeras: An automl library for deep learning. J. Mach. Learn. Res., 24:6:1 2023.
Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. Dsbench: How far are data science agents to becoming data science experts?, 2024. URL aattps://ariiv. aawaaaiiiiiv...... www.iivoorg/aaao abs// http://..rr/bb/////////
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool. Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-58)) 2021.
Elke Kaiser and Wolfram Schier. Time and Materiality: Periodization and Regional Chronologies at the Transition from Bronze to Iron Age in Eurasia (1200-600 BCE). Edited by Elke Kaiser and Wolfram Schier. Verlag Marie Leidorf GmbH, 2021.
Hiroaki Kitano. Artificial intelligence to win the nobel prize and beyond: Creating the engine for scientific discovery. Al magazine, 37(1):39 49, 2016.
J Kneisel. Chronology and transformation. the transition from bronze to iron age in northern europe. Time and materiality: Periodization and regional chronologies at the transition from Bronze to Iron Age in Eurasia (1200-600 BCE), pp. 237-263, 2021.
Pat Langley. Data-driven discovery of physical laws. Cogn. Sci., 5:31-54, 1981. URL https 396942511
Erin LeDell and S. Poirier. H20 AutoML: Scalable automatic machine learning. In Proceedings of the AutoMI. Workshop ICML., 2020.
Michael Y. Li, Emily B. Fox, and Noah D. Goodman. Automated statistical model discovery with ies.ntissscoo
language models. ArXiv, abs/2402. 17879, 2024.
org/CorpusID: 02:021866116
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction followiin models.
eval 2023.
Bill Yuchen Lin, Khyathi Chandu, Faeze Brahman, Yuntian Deng, Abhilasha Ravichander, Valentina Pyatkin, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking language models with challenging tasks from real users in the wild, 2024. URI.
co/
praces/allenale
Xiao Liu. Zirui Wu, Xueqing Wu, Pan Lu, Kai-We Wei Chang, and Yansong Feng. Are 1lms capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data. arXiv preprint arXiv:2402. 17644, 2024.
Luise Lorenz. Kommunikationsstrrtturer mittelneolithiscer Gesellschaften im nordmitteleuropäts chen Tiefland. Verlag Dr. Rudolf Habelt GmbH, in Kommission, 2018.
12

---

Published as a conference paper at ICLR 2025
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL htttpp:://ppemeeeeeeenntttttteee wwt.:n///oooooooo d= d=
Schaefer-Di Maida et al. Unter hügeln: Bronzezeitliche transformationsprooess in schleswig-holster am beispiel des fundplatzes von mang de bargen (bornhöved, kr. segeberg) band 1, 2023.
Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Chris Dallison-Burchh and Peter Clark. CLIN: A continually learning language agent for rapid task adaptation and generalization. arXiv preprint arXhv:2310.10134 2023.
Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Sanchaita Hazra, Ashish Sabharwal, and Peter Clark. Data-driven discovery with large generative models. ICML, 2024.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference Empirical Methwds in Naturul Language Processing, EMNLP. Association for Computational Linguisties, URL https:
2023, doi: 10. 1/2333//1222..MMMLLL v1/2023.
741
Whitney Moore, Steven Pedlow, Parvati Krishnamurty, Kirk Wolter, and II. Chicago. National longitudinal survey of youth 1997 (nlsy97). National Opinion Research Center, Chicago, IL, 254: 22, 2000.
Gianmarco I P Ottaviano, Giovanni Peri, and Greg C Wright. Immigration, offshoring, and american jobs. American Economic Review, 10355);1955-1959, 2013.
Lakshman Chandra Pal. Impact of education economic development. Khazanah Pendidikan Islam, 5(1):10-19,2023
Alessio Palmisano, Andrew Bevan, Alexander Kabelindde, Neil Roberts, and Stephen Shennan. Long- term demographic trends in prehistoric italy: Climate impacts and regionalised socio-ecological trajectories. Journal of world prehistory, 34(3):381 432,, 2021.
Eóin W Parkinson, T Rowan McLaughlin, Carmen Esposito, Simon Stoddart, and Caroline Malone. Radiocarbon dated trends and central mediterranean prehistory. Journal of world prehistory, 34: 317-379, 2021.
Norimah Rambeli, Dayang Affizzah Awang Marikan, Jan M Podivinsky, Rosilawati Amiruddin, and Ismadi Ismail. The dynamic impact of government expenditure in education economic growth. International Journal of Business and Society, 22(3):1487-1507 2021.
Mare Riera, Joan Pino, Llorenç Sáez, Pere Aymerich, and Yolanda Melero. Effect of introduction pathways on the invasion success of non-native plants along environmental gradients. Biological Invasions, pp. 1-20, 2024.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Soumya Sanyal, Tianyi Xiao, Jiacheng Liu, Wenya Wang, and Xiang Ren. Are machines better inference gaps in entailment verification. In Findings of the Association for Computational Linguistics, ACL, 2024. https:///d 111.111111133111//1 -ccl aal.aacl.6..
at complex reasoning? unveiling
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
Michael Schlichtkrull, Zhijiang Guo, and Andreas Vlachos. Averitec: A dataset for real-world claim verification with evidence from the web. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems, NeurlPS 2023. URL
ppappr_files/papeee___pe ni ns cttps/ccppppppppe wttpppss.cppppppppaaaa
of
and Benchearks..ee. html
Zezhi Shao, Fei Wang. Yongjun Xu, Wei Wei, Chengqing Yu,, Zhao Zhang. Di Yao, Guangyin Jin, Xin Cao, Gao Cong, Christian S. Jensen, and Xueqi Cheng. Exploring progress in multivariate time se- ries forecasting: Comprehensive benchmarking and heterogeneity analysis. ArXiv, abs/2310.061 19, 2023.
Sumuk Shashidhar, Abhinav Chinta, Vaibhav Sahai, Zhenhailong Wang, and Heng Ji. Democra- tizing Ilms: An exploration of cost-performance trade-offs in self-refined open-source models. ArXiv, abs/2310.07611, 2023. URL htthhts:://ppp www.atiissccioo...ee.. oot..eecccccccoo.oo...o.oooooooo wwt.cccho.....ooooooooo......... 263834891
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu
Yao. Reflexion: Language agents with verbal reinforcement learning. In NeurlPS, 2023. URL
https://agg httpp:///pp..eennnncccch.... www.steiinnnnoo........omcmm
wtt/CorpusIII........... 225833333 258833055
Patricia K Smith, Barry Bogin, and David Bishai. Are time preference and body mass index associated?: Evidence from the national longitudinal survey of youth. Economics & Human Biology, 3(2):259-270) 2005.
Christoph Sommerfeld.
Gerütegeld Sichel: studien zur monetüren Struktur broncezeitlicher Horte im nördlichen Mitteleuropa, volume 19. Walter de Gruyter, 2013.
William Hedley Thompson and Simon Skau. On the scope of scientific hypotheses. Royal Society Open Science, 10(8):230607, 2023.
David Wadden, Kyle Lo, Bailey Kuchl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Han- nanch Hajishirzi. Scifact-open: Towards open-domain scientitic claim verification. In Find- ings of the Association for Computational Linguistics: EMNLP. Association for Computa- -WW.DIINNGG-EELP.347.. EMNLP.347. URL https 347
tional Linguistics, 2022. doi: (0.18665/1/2002.FINIIIC
//doi.org/10... 000pr//1..16653/112220002 ftno22.finninn findiiiiigggno
Helena Weatherly, Kendy Lopez, and Carol Tierra. The impact of education on gdp per capita. 2022.
Wes McKinney. Data Structures for Statistical Computing Python. In Stéfan van der Walt and Jarrod Millman (eds.), Proceedings of the 9th Python in Science Conference, pp. 56 - 61, 2010. doi: 10.25080/Majora wwweeat2lfll-eeee -00a.
Zhen Yang, Xiaodong Liu, Tong Li, Di Wu, Jinjiang Wang, Yunwei Zhao, and Han Han. A systematic literature review methods and datasets for anomaly-bases network intrusion detection. Comput. Secur. 116: 102675, 2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In ICLR, 2023. URL
https:://peerev/e// .nt. www.:orummiidiiie.......
vluYUL-X.
Zhiqiang Yuan, Junwei Liu, Qiancheng Zi, Mingwei Liu, Xin Peng, and Yiling Lou. Eval- uating instruction-tuned large language models code comprehension and generation. ArXiv, abs/2308.0133222 004200 2023. httpss/:://appi. www.nticscho..oooooo.. oor.:cccccc.aaaoooooooruoo. 26C379087
Khaing 7aw, Darrick Hamilton, and William Darity. Race, wealth and incarceration: Results from the national longitudinal survey of youth. Race and Social Problems, 8:103-115, 2016.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng. Tanya Goyal, and Dangi Chen. Evaluating large language models at evaluating instruction following. ArXiv,
2023. URL
/ inseeantiscchi a www. Copus 263834882
Shujian Zhang, Chengyue Gong, Lemeng Wu, Xingchao Liu, and Mi Zhou. AutoML-GPT: Automatic machine learning with GPT. ArXiv, abs/2305.02499, 2023.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A LIMITATIONS
We currently filtered domains and tasks that required forecasting, simulation, or very specific modeling (species distribution, infection spread, astrophysics equations for exoplanets) in the DB-REAL benchmark as they were very time-consuming to replicate as well as discover hypotheses. As result, we discarded more papers focused on natural and physical sciences compared to social sciences, which we plan to include in future benchmarks.
We currently do not tackle the challenge of understanding and processing massive datasets, such as the 8.92 petabytes data from the Cancer Genome Atlas (https:/ While the potential discover new insights from such vast data volumes is significant, ensuring these findings are robust and not subject to p-hacking remains unaddressed by our current methods.
gdc. cancer. gov).
We currently do not handle multi-modal data and complex pipelines, such those needed for analyzing satellite and other geospatial data relevant to climate science and astronomy data. This would involve multiple stages of data processing, the use of various tools, and managing workflow complexities, for example, analyzing thousands of species patterns combined with satellite data to study habitats.
There could be many potential societal consequences of systems tuned on our proposed benchmark since involves using LLMs, such as policy misuse, legal ramifications, and false discovery. On the positive side, our proposed benchmark can advance the rate of discovery, leading to improved standard of living and social well-being.
Ethical Considerations
B DATA COLLECTION FOR DB-REAL
For data-first approach, replication took 15 to 40 person-hour for each rel-rel paper and up to 90 person-hours for the GBIF dataset, where specialized domain knowledge and tools led to higher complexity, All papers replicated the NLS dataset were included, while less than half the papers in specialized datasets like GBIF and WBOD were added BESCOHERYBENCH.
Citation/Repositories for DB-REAL: List of scientific works from where we have replicated our gold workflows and hypotheses:
1. Sociology: Zaw et al.
2016
Apel & Sweeten 2010
Alexander et al. 1982 Smith et al.
2005 Dougherty
2003)
2. Biology: (Cerezer
2023
Riera et
2024
3. Economics: (Pal 2023
Appiah
2017
Weatherly etal.
2022
Rambeli et al.
2021 Ottaviano
ctt al. 2013)
4. Engineering: Alves et al.
2023
5. Meta-science: (Heyard Held
2024
6. Humanities: (Brozio et al.
2024 2019 Lorenz 2018
Maida et al. 2023 Sommerfeld
2013; Feeser et al. 2019 Parkinson et al. 2021 Kneisel 2021
Kaiser & Schier 202 I
Brinkmann 2019 Dal Corso et al. 2019 Palmisano al.
2021 Parkinson et al. 2021)
Distribution: All assets come under CC license or open licenses (Table All NI-S-rlltte datasets are licensed under CCZero, and the rest of the datasets are licensed under CC by 4.0, which allows them to be freely used/distributed as long as they are properly cited. In NLS and ML engineering datasets, where there are human subjects, datasets are anonymized to protect respondent confidentiality.
C DATA GENERATION FOR DB-SYNTH
For leaves, we use different sampling strategics based on the data type. Specifically, for categorical nodes, sample instances with replacement from the range of allowed values, whereas for numeric,
15

---

Published as a conference paper at ICLR 2025
Dataset
Paper/Citation
License
Source
nls_raw, n sis_mm__raa__ pls_bmi, ais_ses, nls_ incarceration
National Longitudinal Surveys MMoore et al. 2000)
Creative
Link
Commons
CCZero
introduction
Effect of introduction pathways on the invasion success of non-native plants along environmental gradients (Riera et al. 2024)
CC by 4.0
Link
plants
worldbank education gdp, worldbank education_gdp. indicators
vorldbank_educationn
WorldBank Data
CC by 4.0
Link
https: http:///aaaaaaan. worddannk.
org/source/
world <02:0--@ree.mmmmmn-nnnccccc...
archaeology
Patterns of Socio-economic and Cultural Transformations Neolithic and Bronze Age Societies in the
CC by 4.0
Link
Central Northern European Plain
(Brozio et al.
2024)
meta srta_regrresion__2 raw, meta_regression
Meta-regression explain shrinkage and heterogeneity large-scale replication projects (Heyard & Held 2024)
CC by 4.0
Link
immigration
Immigration, Offshoring, and American Jobs (Ottaviano et al. 2013)
CC by 4.0
Link
offshoriiggeeffect.on..
employment
evolution_freshwatee
Accelerated body size evolution in upland environments is correlated with recent speciation in South
CC by 4.0
Link
fish
American freshwater fishes (Cerezer
et al. 2023)
requirements_
Industrial Practices of Requirements Engineering for ML-Enabledd
CC by 4.0
Link
engineering_fow for_Mi enabled_systems
Systems in Brazil Alves et al. 2024)
Table 3: Licenses for original data sources. The Creative Commons Attribution license allows the re-distribution and re-use of a licensed work on the condition that the creator appropriately credited.
first select distribution (e.g., normal) and its parameters based the specified range and then perform sampling. For each subsequent level in FF, we create new columns for nodes by simply executing their pandas expression To recover from any execution errors, we additionally use self-refine (Madaan et al. (223) approach to generate new pandas expressions guided by the execution error logs. Finally, to mimic real-wor weal-world challenges in data collection, we probabilistically perturb each instance æ E by adding noise or dropping values to create missing date After generation, D contains a column for each node in J.
The expressions associated with the hypothesis themselves are generated by prompting the model with natural-language workflow steps, e.g., "within-cluster analysis" and "temporal analysis." We first curate pool of real workflow steps from DB-Real workflows. Then encourage diversity in plausible (according to the LLM) workflows and to increase coverage (by accounting for workflows that could not cover in DB-Real for practiiallepproduuciilttyy reasons), allow LLMss combine several foak-workflow steps (type or variable derivation, type of relationship derivation, etc) plausibly through drawing from the real workflow pool. This significantly improves the likelihood of testing
*The expression is guarantced to only have variables already generated due to the bottom-up construction. Fach value is noised independently; therefore, each row has sufficient data useful for discovery.
16

---

Published as a conference paper at ICLR 2025
synthetic workflows that connect well with the real ones. Moreover, also self-refine and human filtering to monitor the quality of the expressions and the variable metadata therein after tree and data generation. E.g.. values from a specified range rather than a random range.
substantial improvement in quality when generating variable
D COMPOSITION OF DISCOVERYBENCH
D.1 METADATA STRUCTURE
id: An identifier for the metadata.
domain: The broad field of study or area of research.
workflow Laallowwtags:: A set of keywords summarizing the main processes techniques used in the replication implementation. They provide an overview of the methodological approach and facilitating the identification of relevant analytical techniques.
domain knowledge:
Contextual information or insights related to the dataset, explaining how certain behav- iors variables can be interpreted within the field of study.
It helps open avenues to think in directions that LLM might not have considered otherwise, broadening the understanding of the field.
datasets: Contains detailed information about the datasets used, including:
name:
The name or filename of the dataset.
description: A summary of the dataset's contents and the type of data includes.
max__depth The maximum hierarchical level of nested data structures within the dataset, indicating the complexity of the data.
columns: Detailed descriptions of each column in the dataset, including:
* name:
The column's name or header.
description: Explanation of the data contained in the column and its significance.
depth: The hierarchical level of the column within the dataset, indicating its structural position.
hypotheses: Statements or predictions being tested, divided into:
-main: Primary hypotheses that are central to the discovery task.
workflow: A step-by-step description of the replication process followed to validate the hypotheses, outlining the methods and procedures used from data preparation to final analysis. Some of the workflows and sub-workflows are high-level and thus the same for different queries as they follow the same implementation leading range of hypotheses.
queries: Goals related to each hypothesis, each including:
qid: A unique identifier for the goal for a given true/gold hypothesis.
difficulty: Categorization of the difficulty. Structurally defined for DB-SYNTH using the semantic tree definition.
true hypothesis: The hypothesis being tested through the goal. This defines the primary statement or prediction under investigation.
relevant cols: Columns from the dataset that are relevant to answering the query,
indicating the specific data points that be used in the analysis. Only appears for DB-SYNTHI.
The column being predicted or the dependent variable the analysis. Only appears for DB-SYNTH.
target_cerr col:
question type: The type of question being asked categorizing the nature the inquiry.
question: The discovery goal.
17

---

Published as a conference paper at ICLR 2025
D.2 DIRECTORY STRUCTURE FOR DB-REAL
There may be more than one query per metadata. The train split contains 14 metadata files and 25 queries. The test split contains 144 metadata files and 239 queries. Metadata folders with the same prefixes use the same underlying dataset with either a subset a preprocessed version. When dealing with full dataset (i.e., nls nls_raw) raw), the task becomes substantially harder due the data preparation required.
test
archaeology
meta_regression
meta__regression
iols_incarceration -nls_raw
-nls_ses
wo:1dbban__educaaiioo_oo
trainn
evoluttioo__freshveterrr..
i hww.grag......ooffooooggggggeeed
n _ns__mi
-ns_n_s__mii raw
D.3 DIRECTORY STRUCTURE FOR DB-SYNTH
There is one query per metadata. The train split contains 551 metadata files (queries), the dev split contains 153 metadata files (queries), and the test split contains 200 metadata files (queries).
test amcient-languugees__
attiFFicial-ecccysteemss
astrooomy__=_*
b
calinng-cometittionss...
cigiital-artiistry_..+
ficurristic-cechhooooyy....
imppzess1.011st-artt..
machine--leamiingg_..
www.lecllar-ggsstrooon....
neurosciencee_-_*
philosophical--@eeree.....
robotics_*_*
t trainn
adventure-traaeel...
ancleni--architeccure_...
anciient-astronomy_..
aviatioo_=__+
bacalversstyycooseevaioo...
cxyptic-puzzles____
cryptocurrenay_**.
culinary-arta_=__
cybersecurityy
anw/rrommentallacciivvsssuuu
fashion-desiinn_#__
fine-arts_
literary-ciassicss....
marine--boology_...
18

---

Published as a conference paper at ICLR 2025
marine-conservatioon
meddeval-=iterrttueee
musical--thrrppy_._
phot.ography___*
rebotic-exxpooees__..
sclar-powe:_-_*
space-tourism__+44
steempynnk-cultturre+++++
cleattee--rrocsuttiinnn.____
underraater--prchheeoloo....
urbai-garrdening____
vintagge-auttommbbleess
virtual-realityy
EXPERIMENTS
For GPT-based models, we use OpenAl API hatte::/pllaffomm.eeaa...oo//// models), and for Llama3, used Together API fttpsss//dos..ttgtteettt.t liference-modelle
E.1 DISCOVERY AGENT
The command di discovery_ageem Py is used with various options to customize its behavior for discovery tasks. Below are the options explained:
Usage: ciscovery_agent.pp [CPTIONS] QUERY Executes the discovery agent with specified options.
Options:
-agent_tyye [coder react]: Specifies the type of agent use for discovery. The default type is coder. Options include coder for code-related tasks and react for reactive tasks.
model__nme TEXT: Sets the model to be used. The default is spt-4o- Available models ---00- chat, claude 3-opus, and gemini remini-pro. An exhaustive list is available in
include 9pt-4-turbo
json.
The default path is -app__confii TEXT: Path the API configuration file.
config/api_confiig json.
-loo__ile TEXT: Specifies the path to the log file where operations details are stored.
-metadata_path TEXT: Path the metadata file. This option is required.
-metadata_typa [real |sytt]]: Specifies the type of metadata, where real stands for actual metadata and synth for synthetic. This option is required.
-add__dmmain_kkkowwlddeg Includes domain-specific knowledge in the query processing.
aadd_workflow_tage Includes workflow tags in the query to enhance context.
-heip: Displays the help message and exits, showing all available command options.
F.2 EVALUATION
Explain about evaluation line and then explain the CLI usage here.
The command di discovery_evad seaal.pee.ppp py is used to evaluate the outputs generated by the discovery agent. Below the detailed descriptions of the command options:
Usage: seroovery_vval_. py [OPTIONS] QUERY Executes the evaluation agent with specified options and query.
Options:
19

---

Published as a conference paper at ICLR 2025
--gold_byye hypo TEXT: Specifies the gold standard hypothesis for comparison. This field is required.
ggold_workflee TEXT: Specifies the gold standard workflow to be used as a reference during evaluation.
-pred_hypo TEXT: Specifies the predicted hypothesis generated by the discovery agent. This field is required.
-pred_workflow TEXT: Specifies the predicted workflow generated by the discovery agent.
mmetadata_patte TEXT: Specifies the path to the metadata file that is utilized during evaluation. This field is required.
-metadata_type [real |sytth] Determines the type of metadata used in the evalua- tion, where real indicates actual metadata and synth indicates synthetic metadata. This field is required.
-eval_octppat_pttt TEXT: Specifies where the evaluation results should be saved. -help: Displays the help message and exits, detailing all available command options.
20

---

Published as a conference paper at ICLR 2025
F EVALUATOR PROMPTS
We provide below the exact prompts used for our GPT-4 based evaluation of the generated hypothesis against the gold hypothesis.
Listing Decomposition Prompt to obtain context from hypothesis.
decomposition_pren prompt prompt E""")
Given set of dataset columns, ground-truth hypothesis, and the analysis workflow used, your task is to extract the context present in the hypothesis.
Here are the definitions for context:
Contexts: Roundary conditions that 1imit the scope of sub-bypothesis. E.g., "for men over the age 30", "in Asia and Europe", or "None" if there is no boundary condition specified.
Make sure only use the information present in the hypothesis and the workflow. Do not add any new information. If no context can be extracted, return an empty list.
Here is the metadata for the task:
json
{{
"datasets"; (dataset_metaddae "hypothesis"; "(hypothesis)", "workflox": "(workilow)"
}}
Return your answer as JSON object in the following format:
""jjson
{{
"hypothosis";
{{
"text"; the hypothesis in natural language, "context": short text description of the context of the hypothesis,
"explanation": a short text explanation for the breakdown of the sub-hypothesis
21

---

Published as a conference paper at ICLR 2025
Listing Matching prompt to match contexts two sub-hypotheses.
matooinggpprompt f*nm
Given gold hypothesis, gold context, predicted hypothesis, and a predicted context, your task is to determine if the predicted context semantically matches the ground-truth context.
Here is the definition for Context: Boundary conditions that limit the scope of a sub-hypothosis. E.g., "for men over the age of 30", "in Asia and Europe", or "None" if there is boundary condition speci fied.
If the predicted cted context matches the gold context, return true, otherwise return false.
Here is the metadata for the task: jsjoon {{
"gold_hyyothesis": "(gola_hypotheis),,
ggold_context'" [ggold_context]]
"(pred hypothespptt ppredicted_context "(pred_context)
}
Return your answer as JSON object the fol lowing format:
s*json
{{
"match": true false
}}
22

---

Published as a conference paper at ICLR 2025
Listing 3 Prompt for variable alignment between two sub-hypotheses.
main context = -E" E"n"
You are going compare natural-language hypotheses нурод and Нyроß accompanied with optional workflows: WorkflowA for Нурод and WorkflowB for НуроВ.
Both the hypotheses anawer the natural language query "QUERY" over the dataset (s) described by dataset description (s) and column description (s) below.
Compare Hypoλ and нуров in terms three aspects: Contexts, Variables, and Relations.
E.g., for the hypothesis "From 1995 to 2009, the number of sandhill eranes around the tundra (Indigilka River) surged by astounding ~10x":
Contexts refer to the stratification of the data under which the given hypothesis is True. E.g., "For all women", "From 1995 to 2009".
variables (either dependent independent) that are mentioned the hypothesis. E.g., number of sandhill cranes, location.
Variables refer the
Relations refer to the form of relation between the variables. E.g., "surged by -10x".
Answer the following questions for a given pair of hypotheses, Hypoλ and Hypoß, along with an explanation grounded on the QUERY and the (STA
Here is the metadata for the task: ""json
{(
"datasets"; (datasets_json),
"query": (query),
"HypoA": (goid_hypo),
"KorkflowA": (gold_workflow/
"HypoB": (gen_hypo),
"WorkflowB": (gen_workflow)
)
(variable_question)]
variable question "**\
Question: For both Hypоλ and HypоB, what are the different
variables found in the hypotheses? \
Return your answer as JSON object in the following format: ""json
[1
"sizeA": num variables used in Hypoλ
"sizeB": num of variables used in Нyров
"intersection"; num of variables common in Hypоλ and HypoB. Use *fuzzy matchings to determine intersection, accounting for paraphrases or slightly different surface forms "explanation"; a short text explanation about the variables H
Answer: unn
23

---

Published as a conference paper at ICLR 2025
Listing 4 Prompt for relationship alignment between sub-hypotheses.
ma n_context fann
You are going compare two natural-languag hypotheses НуроА and НуроВ accompanies led with optional workflows: WorkflowA for Hypoλ and WorkflowB for НypоB.
Both the hypotheses anawer the natural language query "QJERY" the dataset (s) described by dataset description and column description (s) below.
Compare Нyрод and НуроВ in terms of three aspects: Contexts, Variables, and Relations.
R.g., for the hypothesis "From 1995 to 2009, the number of sandhill crares around the tundra (Indigi River) surged by astounding =10x";
Contexts refer to the stratification of the data under which the given hypothesis is True. E.g., "For all women", "From 1995 to 2009".
Variables refer the set variables (either dependent or independent) that are mentioned in the hypothesis. E.g., number of sandhill cranes, location.
Relations refer to the form of relation between the variables. E.g., "surged by ~10x".
Answer the following questions for a given pair of hypotheses, Hyроλ and Eypоß, along with an explanation grounded on the QUERY and the DATASET(S).
Here is the metadata for the task: j'joon
[(
"datasets"; (datasets_jsone
"query": (query),
"HypoA": (gold_hypo),
"WorkflowA": (gold_workflow)
"Hypoв": (gen_hypo), "Workflow8": (gen_workflow) }}
[variable_questionn] question)"""
citeension_questiiie ***
Question: Does Hypoß exhibit the same relation as HypoA? Compare using the following example hierarchy of relationships (based on specificity); "there exists relationship" "positive relationship" "positive AND (linear OR quadratio)* "positive AND linear." Options: A) very similar B) similar but general than HypoA C) different
Return your ""json
as JSON object in the following format:
[(
"anawer"; one of the options from A) very similar B) similar but general than Hypoλ C) fferent
"explanation"; short text explanation about the relationship comparison 11"
Answer; nn8
24


Reviews:
[{"review_id": "uDXqCxK3oO", "reviewer": "Reviewer_8cxn", "summary": "In this paper, the authors consider the question of how capable\nstate-of-the-art LLMs are at automated data-driven discovery. More\nprecisely, the authors present a benchmark for this task, designed not\nonly to assess LLMs’ capability in discovery tasks but also to provide\ninformation for improving these capabilities.\n\nThe benchmark presented in this paper consists of 264 manually\ncollected tasks, along with 903 tasks that were synthetically\ngenerated. This benchmark is used to evaluate several LLM-based\nreasoning frameworks, providing useful information on the current\ncapabilities of such systems in automated data-driven discovery.", "questions": "The generation DB-SYNTH consists of four steps intended to capture\nsynthetic task examples from workflows in published works. In\nparticular, a task dataset is constructed in the third step by\ngenerating synthetic data using various sampling strategies. The\nauthors do not explain how these steps, and particularly the data\ngeneration step, provide good representations of the task examples from\nworkflows in published work (notice that there is a missing reference\nto a section in the data generation paragraph). This justification\nshould be part of this work, as if the synthetic tasks are not good\nrepresentatives of the real examples, then we could be asking the LLMs\nto solve tasks that may be too simple or too complicated, which could\nreduce the quality of the tasks in the benchmark.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "S1) The authors introduce a simple yet expressive notion of\ndata-driven hypothesis. Discovering and validating such hypotheses\nfrom a dataset is a challenging problem for LLMs.\n\nS2) The authors develop a comprehensive benchmark to test the\ncapabilities of LLMs in discovering data-driven hypotheses.\n\nS3) The authors use the benchmark to test some popular LLM-based\nreasoning frameworks, drawing useful conclusions about the\ncapabilities of these systems in discovering data-driven hypotheses.", "weaknesses": "W1) As part of the benchmark, the authors developed some synthetic\ntests. These tests are supposed to capture synthetic task examples\nconstructed from workflows in published works. However, the authors do\nnot clearly explain in what sense these synthetic tests properly\nrepresent these workflows.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b32d5zvMmY", "reviewer": "Reviewer_9u8G", "summary": "The advances in data science and AI seem to be evolving the process of scientific discovery (the scientific method) by (drastically) reducing the time needed to formulate hypotheses, conduct experiments, and evaluate results. This paper proposes a benchmark to evaluate the potential role of LLMs in the automatic formulation of hypotheses, starting from a dataset and a natural language question about that dataset.\n\nTypically, for a classification or regression task, a human will identify certain attributes of interest (A_1, …, A_n), a target attribute Y, and attempt to build a relationship between these elements (A_1, …, A_n → Y). This is the current process for generating new knowledge. In this work, the authors aim to determine whether using LLMs allows for the production of valid knowledge in a more general setting: providing a dataset, posing a general question in natural language to the system using an LLM, and obtaining a response in the form of a scientific discovery. The main question the authors seek to answer is how well this approach performs using state-of-the-art LLMs.\n\nThere are some recent tools that evaluate the performance of LLMs in testing well-defined hypotheses (clearly formulated questions on a dataset). These methods utilize a constrained search space to look for hypotheses. DiscoveryBench expands the search space by finding hypotheses in a broader context, closer to a human approach. As expected, the performance of LLMs decreases in this setting. Indeed, the paper shows that the best results achieve 25% performance (which I interpret as 25% of responses being valid knowledge, even if it is a bit more complex to interpret).", "questions": "How can it be ensured that the data alterations in the proposed datasets (page 6) do not lead to contradictions with the discovery goal, resulting in only one answer—the target hypothesis?\n\nMinor remarks:\n - Pg 3 : discopagevery --> discovery\n - Pg 6: Sec ?? --> update the number of the section", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The role of LLMs in the scientific method is still unknown (if one exists), and evaluating their capacity to accelerate the process of knowledge discovery is a topic of significant interest. This work represents an incremental advancement in this domain, providing a formal definition of data-driven hypotheses and expanding the search space for these hypotheses. The paper also proposes evaluating proposed hypotheses against a gold standard using semantic similarity measures. In my opinion, this work is of genuine interest to the community.\n\n- The proposition is well described, and the code is available on GitHub. The authors have done impressive work collecting datasets, defining objectives for these datasets, and constructing gold standard hypotheses.\n\n- The results are clearly presented and discussed, demonstrating the limitations of current LLMs. This methodology can be utilized in the future to observe potential progress in the field of LLMs.", "weaknesses": "- The process of finding a data-driven hypothesis can be time/energy-consuming. This aspect is not discussed in the paper, and I understand that the page limit might not allow space for such discussions. Although this is not the main focus of the paper, it could be useful to explore whether there is a relationship between the size of the search space and the performance of the results.\n\n- The proposed formalism for defining data-driven hypotheses, discovery goals, and task difficulty is somewhat limited. At the same time, I understand that human expertise is essential in this process, and complete automation is difficult, if not impossible, to achieve.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DdeGMPKGdQ", "reviewer": "Reviewer_rQhh", "summary": "The work deals with the problem of data-driven discovery using large language models.", "questions": "Please read my comments/questions from the above field. In addition: \n\n- One suggestion that I have for the authors is to start with the example given in lines 247 and show how G, h, \\psi, c, u, r, and \\mathcal{V}_D look like for this example. \n- Please clarify whether a hypothesis is represented as a sentence, a semantic tree, or a ternary tuple, and to explain how these different representations relate to each other if multiple are used.\n- Please define the space of valid verification procedures V_D and clarify whether finding V_D itself is part of the task or if it's given. \n- Another suggestion that I have for the authors is to look into automated theorem proving, (e.g., “Handbook of Automated Reasoning”, Volume 1, 2001), where the notions of hypotheses, theorem proving, proof trees, are formally defined. I would suggest using this notation to formally define the data-driven discovery task. I believe that this work will benefit a lot from a more rigorous formulation, e.g., to my understanding, the elements in \\mathcal{V}_D map to proof trees.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The authors attempt to formalize the problem of data-driven discovery, which, in my opinion, is a crucial step towards solving this task. In addition, a benchmark is introduced -- unfortunately, I did not have the chance to look at it in more detail, but the authors claim that they will publicly release it.", "weaknesses": "I am confused with the formulation of the problem of data-driven discovery as in Section 3. In line 049, the authors give an example of a data-driven discovery task: “How did urban land use affect the invasion of introduced plants in Catalonia?” The answer to this task is the ways urban land used affected the invasion of introduced plants in Catalonia (if this is the case). However, in Section 3, line 152, you define a hypothesis as a tuple of three elements and in line 141, you say that a hypothesis maps to supported or unsupported. So, if a hypothesis maps to true or false (supported or unsupported), then the example task that you have in line 049 cannot map to your formulation. Please clarify what is the case and, please specify concretely how the task in line 049 translates to your formulation. \n\nI have a few other comments/questions regarding the formulation in Section 3:\n- In line 139, the authors say that a hypothesis is a sentence/semantic tree, and then in line 152 the authors define a hypothesis as a ternary tuple. Please fix this inconsistency, as it is confusing to the readers. \n- In line 141, the authors talk about a verification procedure \\mathcal{V}_D mapping a hypothesis to supported and unsupported. Is my understanding correct, that the goal of the data-driven task is both to find whether a hypothesis is supported and unsupported and to return the exact \\mathcal{V}_D? If my understanding is correct, then how do you define the space of valid verification procedures \\mathcal{V}_D? I would kindly ask the authors to clarify this part, as it is missing. \n- In line 152, the authors write h := \\psi(c,u,r). That formulation essentially means that h (the hypothesis) is defined as its answer, \\psi(c,u,r). I believe what the authors want to say is that \\mathcal{V}_D(h) = \\psi(c,u,r), i.e., that the answer to h is defined as \\psi(c,u,r). Please fix the notation accordingly or clarify what you meant.  \n- Continuing with my previous comment, I would kindly ask the authors clarify what is the relationship between \\mathcal{V}_D and \\psi.    \n- In line 161, the authors give another definition of the data-discovery task: that of given a goal G, to find a hypothesis h. From Section 3, I understood that the objective is: given a hypothesis h = (c,u,r) to derive \\psi (or \\mathcal{V}_D). I would kindly ask the authors to define concretely the problem they are dealing with, addressing all my comments from above, as now it is difficult for the reader to understand what is happening. \n- In line 293, the authors talk about hypothesis semantic trees, giving an example in Figure 1. However, in line 152, the hypothesis is a ternary tuple. \n- In line 240, the authors talk about the implementation workflow, however, they give no definition/specification of them in Section 3. \n\nThe above issues, make it difficult for the reader to understand this work and the proposed benchmark. I am willing, however, to increase my score if the authors address my comments/questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)