Title: Data-Driven Discovery of Dynamical Systems in Pharmacology using Large Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: ODE Discovery

LLM Discovery

LLM
Abstract: The discovery of dynamical systems is crucial across a range of fields, including pharmacology, epidemiology, and physical sciences. *Accurate* and *interpretable* modeling of these systems is essential for understanding complex temporal processes, optimizing interventions, and minimizing adverse effects. In pharmacology, for example, precise modeling of drug dynamics is vital to maximize therapeutic efficacy while minimizing patient harm, as in chemotherapy. However, current models, often developed by human experts, are limited by high cost, lack of scalability, and restriction to existing human knowledge. In this paper, we present the **Data-Driven Discovery (D3)** framework, a novel approach leveraging Large Language Models (LLMs) to iteratively discover and refine interpretable models of dynamical systems, demonstrated here with pharmacological applications. Unlike traditional methods, D3 enables the LLM to propose, acquire, and integrate new features, validate, and compare dynamical systems models, uncovering new insights into pharmacokinetics. Experiments on a pharmacokinetic Warfarin dataset reveal that D3 identifies a new plausible model that is well-fitting, highlighting its potential for precision dosing in clinical applications.

Data-Driven Discovery of Dynamical Systems in Pharmacology using Large Language Models

Abstract

The discovery of dynamical systems is crucial across a range of fields, including pharmacology, epidemiology, and physical sciences. Accurate and interpretable modeling of these systems is essential for understanding complex temporal processes, optimizing interventions, and minimizing adverse effects. In pharmacology, for example, precise modeling of drug dynamics is vital to maximize therapeutic efficacy while minimizing patient harm, as in chemotherapy. However, current models, often developed by human experts, are limited by high cost, lack of scalability, and restriction to existing human knowledge. In this paper, we present the Data-Driven Discovery (D3) framework, a novel approach leveraging Large Language Models (LLMs) to iteratively discover and refine interpretable models of dynamical systems, demonstrated here with pharmacological applications. Unlike traditional methods, D3 enables the LLM to propose, acquire, and integrate new features, validate, and compare dynamical systems models, uncovering new insights into pharmacokinetics. Experiments on a pharmacokinetic Warfarin dataset reveal that D3 identifies a new plausible model that is well-fitting, highlighting its potential for precision dosing in clinical applications.

1 Introduction

The discovery of dynamical systems models plays a fundamental role across various domains, including pharmacology, epidemiology, and physical systems. In pharmacology, pharmacokinetic (PK) models are essential for understanding and predicting the time course of drug absorption, distribution, metabolism, and excretion in the body, which is crucial for optimizing therapeutic efficacy, minimizing toxicity, and personalized treatment regimens in diseases such as cancer, cardiovascular disorders, and infectious diseases [Gabrielsson and Weiner, 2001, Hedaya, 2012]. For example, cardiovascular disorders requiring Warfarin therapy affect tens of millions of individuals globally [Lee and Klein, 2013], highlighting the critical need for superior models to understand the dynamic impact of drugs and precision dosing, thereby assisting both doctors and patients.

The goal of PK modeling is to discover an underlying closed-form ordinary differential equation (ODE) f from a dataset of observed patient trajectories. This problem is of significant interest to the machine learning (ML) community, as evidenced by previous non-interpretable ML modeling approaches aimed at developing better PK models [Chan and van der Schaar, 2022]. Such datasets are collected from expensive human clinical trials, necessitating the careful selection of which features

to acquire and analyze during these trials [Guidance, 2010]. Pharmacometricians then leverage this data, using their existing knowledge to select appropriate pharmacokinetic models and employ standard statistical tools to infer the parameters of these models. For instance, they often fit a single compartmental PK model, a closed-form ODE model with 3-5 parameters, to the data [Chen and Abuassba, 2021]. This process is iterative, refining the models until the most accurate one that generalizes well to new patient trajectories is identified. However, this traditional model discovery approach is often ad-hoc and fundamentally limited by the human expert's time, experience, and implicit assumptions about the underlying pharmacokinetic processes.

Discovering interpretable pharmacokinetic models is traditionally performed by human experts through a scientific process of proposing, collecting, and validating models. Recently, there has been a growing call for artificial intelligence-driven methods to enhance pharmacological modeling [Ryan et al., 2024, Singh et al., 2023, Cheng et al., 2022]. This established process involves iterative steps, each presenting distinct challenges in constructing accurate pharmacokinetic dynamics models.

Selecting appropriate temporal models involves complex decisions on model class, state and feature variables, parameters, and their relationships, far exceeding the simplicity of standard pharmacometrics models like single, double, or triple compartmental models [Chen and Abuassba, 2021]. This complexity demands aligning model intricacy with data availability, where simple models suit small datasets and complex models fit larger ones. Acquiring the right features is crucial, as it must be done with limited prior information, unlike feature selection, which evaluates pre-existing features [Li et al., 2017]. The evaluation must identify inadequate models and explain why, requiring diverse and timeintensive tools beyond a single metric like validation MSE. These challenges are interrelated: strategic data acquisition and thorough evaluation are essential for effective modeling. Despite advancements in tackling these challenges independently, temporal modeling remains manual, limiting speed and scalability, underscoring the need for integrated and automated systems to enhance efficiency.

To address these challenges, we develop the Data-Driven Discovery (D3) framework, leveraging Large Language Models (LLMs) to iteratively discover and refine interpretable dynamics models, directly relevant to pharmacology, epidemiology and ecology applications. D3 features three agents: Modeling, Feature Acquisition, and Evaluation, which collaborate iteratively. The Modeling Agent uses LLMs' code generation and natural language understanding to explore the model space by generating hypothetical models. The Data Acquisition Agent utilizes LLMs' zero-shot and few-shot learning to optimize data acquisition based on summary statistics, text descriptions, and prior knowledge. The Evaluation Agent conducts comprehensive evaluations by computing refined validation MSE metrics, and alignment with prior process understanding, providing precise feedback to improve the other agents' performance.

Contributions: 1 ⃝ We propose the Data-Driven Discovery (D3) framework, a novel approach leveraging Large Language Models (LLMs) to iteratively discover and refine interpretable dynamics models, advancing pharmacokinetic modeling (Section 3). 2 ⃝ D3 overcomes the challenges of uncovering interpretable dynamical systems by using LLMs to explore vast model spaces and integrate unstructured data, producing models with few parameters that rival black box neural networks. D3 accurately discovers dynamics models across pharmacology, epidemiology, and ecology, matching the accuracy of existing methods while enhancing interpretability. It discovers a new, more accurate PK model for Warfarin, validated by expert pharmacometricians. We also gain insight into D3's ability to iteratively improve its models, selectively acquire features, and discover precise dynamics models.

2 The vast model space for temporal modeling

In this section, we introduce the model space that D3 searches through, which contains a diverse range of temporal models with varying degrees of refinement. Our focus is on Ordinary Differential Equation (ODE) models, which is one of the most widely used methods for modeling temporal dynamics, which include pharmacokinetic, physiological and epidemiological dynamics [Auger et al., 2008]. We are interested in modeling the evolution of a set of state variables x n ( t ) ∈ R D for individuals n = 1 , . . . , N over the time horizon t ∈ [0 , T ] . The state variables represent the target variables that the human experts are interested in modeling; and there may exist other feature variables that can help predict the evolution of state variables. We assume we are provided with a dataset of individuals trajectories such that D = { ( x n ( t ) , a n ( t )) | n = 1 , . . . , N, t ∈ [0 , T ] } , where a n ( t ) ∈ R K denotes the observed individual-level features for individual. We can categorize ODE models into three levels (R1 - R3) based on their capacity for refinement.

R1: Refinement through initial conditions x n (0) . This level involves the simplest form of ODEs, commonly referenced in scientific literature [Schiesser, 2014]. These equations are defined over state variables x ( t ) ∈ R D , incorporate global parameters θ , and involve a function f : R D +1 → R D . The time t spans the interval [0 , T ] :

Considering N individuals, each represented by the trajectory x n ( t ) for n = 1 , . . . , N , all follow the equation above. According to the uniqueness theorem for initial value problems in ODEs [Lindelöf, 1894], any differences among individuals' trajectories can be attributed solely to variations in the initial conditions x n (0) , under common regularity conditions on f .

R2: Refinement through observed features a n . Suppose we have access to a set of individual-level feature variables a n . We can enhance the model's refinement by incorporating these features into the ODE to enable unique dynamics for different individuals:

R3: Refinement through acquired features h n . Often, not all relevant variables are initially measured and available for analysis. In such cases, one may consider acquiring additional features h n ( t ) ∈ R J to be integrated into the ODE. Here h n ( t ) is a collection of J features from the set of all acquirable features H , where J ≤ |H| . This third level of refinement reflects one data acquisition challenge, i.e., identifying and collecting informative features h n ( t ) :

After selecting the level of refinement (R1-R3), it is necessary to determine the functional form of the ODE f , which defines the interactions among various variables and parameters that govern the dynamics. There are two dominant approaches to parameterize f : using a concise closed-form white-box equation f ∈ M C or employing a neural network f ∈ M N . The former approach is prevalent in the field of symbolic regression [Billard and Diday, 2002], while the latter is utilized in Neural ODEs [Chen et al., 2018]. Additionally, f can incorporate both a closed-form component and a neural component, resulting in a hybrid ODE model. Nevertheless, all three approaches involve searching within large combinatorial spaces-closed-form equations, neural architectures, and both-presenting significant computational challenges.

3 Data-Driven Discovery (D3)

We aim to uncover the true underlying pharmacokinetic dynamical system, denoted as f ∗ . Our goal is to continually approximate f ∗ as accurately as possible using proposed models f θ , with parameters θ . For simplicity, we will refer to these models as f in all subsequent references, omitting the θ term. To appropriately identify the refinement model level needed from the vast model space discussed above, we now introduce the Data-Driven Discovery (D3) framework, as depicted in Figure 1. This consists of three LLM agents: the Modeling Agent G , the Feature Acquisition Agent A , and the Evaluation Agent E . These three agents work together to form the model improvement loop and the data acquisition loop. The implementation details of the Agents are provided in Appendix F.

3.1 Inputs to Data-Driven Discovery (D3)

To begin the process, D3 requires a clearly defined description of the system in natural language, that details the modeling task, of creating a well-fitting model of either a white-box model or a hybrid model; that is a white-box model with a black-box neural network fitted to the residuals of the white-box model. D3 supports both discovering only white-box models and hybrid models, and we leave this choice up to the end user to decide for their application. Specifically, a user must provide a dataset of individual's trajectories D , and a system description and the names and descriptions of any features that exist within the dataset, which can include the ranges of those features. Prior information can be added here by the practitioner into the system description if it is available. We now discuss how the three agents of Modeling, Feature Acquisition and Evaluation interact to discover iteratively better-fitting f models. We provide full implementation details for the framework in Appendix F.

3.2 Modeling Agent G

D3's Modeling Agent G is tasked to iteratively propose and refine the temporal model f i , where i = 1 , . . . , I indicates the iteration. We design G following the three principles below.

<latexi sh

1_b

64="

ZLRp+

OyKzMWU

8

v

Ng

/HA

>

B

c

V

S

EJ3

r

q

9

0

CF

2

oQ

u

D

m

G

j

Y

w

I

f

d

X

7

k

P

5

n

T

Reflection

1.

Incorporate Drug Efficacy and

Tumor Resistance

2.

Non-Linear Dynamics

3.

Model Drug Synergy …

System Description To

Model

Predict Treatment Response for

NSCLC with Bio-Mathematical

Models.

Variables:

•

tumor_volume (cm^3)

•

chemotherapy_drug_concentrat

ion (mg/m^3)

Time: Days.

Description

Modeling

Code -

Features 2

def __init__(self):

def forward(self,tumor_volume,

chemotherapy_drug_concentration)

Val MSE Loss: 16.3

Evaluator

Can acquire features

of

[

radiotherapy_dosage,

patient_age, ...]

LLM Resp.

Acquire:

radiotherapy_dosage

;

predicted improvement 15%

Acquired

radiotherapy_dosage

Val MSE Loss: 8.9

Acquired

chemotherapy_dosage

Previous top-k

models.

1.

{Code}. Loss.16.3

Evaluator

Evaluator

def __init__(self):

super(StateDifferential, self).__init__()

# Define the parameters for the model

self.k_growth = nn.Parameter(torch.tensor(0.1))

self.k_decay = nn.Parameter(torch.tensor(0.1))

self.k_radio_sensitivity = nn.Parameter(torch.tensor(-0.1))

def forward(

self,

tumor_volume: torch.Tensor,

chemotherapy_drug_concentration: torch.Tensor,

chemotherapy_dosage: torch.Tensor,

radiotherapy_dosage: torch.Tensor,

) -> Tuple[torch.Tensor, torch.Tensor]:

# Tumor growth model

d_tumor_volume__dt = (

self.k_growth * tumor_volume

- self.k_chemo_effect * chemotherapy_drug_concentration * tumor_volume

- self.k_radio_sensitivity * radiotherapy_dosage * tumor_volume

)

# Chemotherapy drug concentration change

d_chemotherapy_drug_concentration__dt = (

chemotherapy_dosage - self.k_decay * chemotherapy_drug_concentration

)

return (d_tumor_volume__dt, d_chemotherapy_drug_concentration__dt)

Best discovered model f is returned

System Description To

Model

...

Variables:

•

tumor_volume (cm^3)

•

chemotherapy_drug_concentrat

ion (mg/m^3)

•

radiotherapy_dosage (Gy)

Time: Days.

Code -

Features 3

def __init__(self):

def forward(self,tumor_volume,

chemotherapy_drug_concentration)

Can acquire features

of

[

patient_age,

chemotherapy_dosage, ...]

Acquire:

chemotherapy_dosage

;

predicted improvement 12%

LLM Resp.

System Description To

Model

...

Variables:

•

tumor_volume (cm^3)

•

chemotherapy_drug_concentrat

ion (mg/m^3)

•

radiotherapy_dosage (Gy)

•

chemotherapy_dosage (mg/m^3)

Previous top-k

models.

1.

{Code}. Loss.16.3

2.

{Code}. Loss.8.9

Reflection

1.

Incorporate Carrying Capacity

2.

3.

Regularization …

Val MSE Loss: 1.2

Code -

Features 4

def __init__(self):

def forward(self,tumor_volume,

chemotherapy_drug_concentration)

Can acquire features

of

[

patient_age,

genetic_markers, ...]

No acquisition

LLM Resp.

System Description To

Model

...

Variables:

•

tumor_volume (cm^3)

•

chemotherapy_drug_concentrat

ion (mg/m^3)

•

radiotherapy_dosage (Gy)

•

chemotherapy_dosage (mg/m^3)

...

Previous top-k

models.

1.

{Code}. Loss.16.3

2.

{Code}. Loss.8.9

3.

{Code}. Loss.1.2

4.

{Code}. Loss.0.1

5.

{Code}. Loss.0.05

...

f

Iteration 1

...

Iteration 2

Iteration 3

Iteration 20

Representing the model f i as code . As we discussed in Section 1, one key challenge in evidencebased refinement is the vast space of possible ODE models, ranging from closed-form ODEs to complex neural Neural ODEs and hybrid models. As a prerequisite to finding the most suitable model, we need a consistent and flexible way to represent these diverse models such that they can be trained and validated on data. As such, D3 represents the model f i as code in Python language. Specifically, each model is defined as a class derived from 'nn.Module' in PyTorch [Paszke et al., 2017] that specifies the free parameters and the computation logic. The Modeling Agent G leverages an LLM to generate the model code f i in each step.

Informing generation with context c . Effectively searching through the vast model space presents another challenge. Thankfully, in many applications there exists rich context c about prior works and domain knowledge. However, this rich source of information was previously locked in unstructured documents and texts, making it hard for automated algorithms to make use of it. By leveraging the LLM's capability to understand unstructured documents, the Modeling Agent G can now access and leverage this information, therefore proposing more informed models and searching more effectively. In this work, we consider the provision of the following categories of problem context of {system description} and {feature description} (Appendix F).

Iterative model improvements based on rich verbal feedback r i -1 and memory s i -1 . To enhance the quality of the generated model f i , we have designed the agent G to iteratively improve the model based on the presence of any evaluation verbal feedback and its knowledge of the previously generated models and how they performed on a validation subset of given dataset D of trajectories. At each iteration i , the Evaluation Agent E evaluates the previous model f i -1 using the validation data subset of D i and provides rich verbal feedback r i -1 (c.f. Figure 1), which forms the foundation for the next generated model f i . The Modelling Agent G utilizes memory s i -1 to track the top-k best-performing models so far, along with their associated evaluation feedback, here their respective losses. Further details on evaluation and data acquisition are discussed in the following section. This iterative process allows for continuous refinement and optimization of models based on their past performance and targeted feedback.

In summary, the Modeling Agent G encapsulates a comprehensive range of model structures through computer code f i and efficiently explores the vast model space by integrating unstructured prior knowledge c , and utilizing data-informed feedback r i -1 along with memory s i -1 . The operation of G can be mathematically represented as follows:

3.3 Feature Acquisition Agent A

D3's Feature Acquisition Agent A is tasked with iteratively proposing new features for acquisition. These features are selected based on their potential to improve the next model's performance in modeling the target state x n ( t ) .

Where h i is the next feature to acquire. Specifically, when a feature is acquired we make the assumption, that it is acquired for all individuals, therefore at the next iteration the dimension of the individual-level features a n ( t ) ∈ R K , n = 1 , . . . , N increases by one, to a n ( t ) ∈ R K +1 , n = 1 , . . . , N . It is possible to do this in clinical trials, where pharmacokinetic datasets of patients are collected [Dziura et al., 2013], for example acquiring the individual's ages, or other lab biomarker measurements or tests [Derraik et al., 2021].

Estimating the value of the feature . We leverage the value of information [Feltham, 1968] framework to provide a principled way to determine which feature, if any to acquire next. This follows as:

where the first two terms capture the improvements in validation loss L when model f i is trained on the new dataset that includes the new feature, that is D = { ( x n ( t ) , a n ( t ) , h n ( t )) |∀ n ∈ [ N ] } and l ( h i ) represents the cost for acquiring h i . Estimating V ( h i ) presents a statistical challenge and a computational challenge. Firstly, in practice, we often do not have access to h i to begin with (as the goal is to acquire new features)-hence we need a way to estimate E [ L ( f i , h i , D i -1 )] without fully accessing h i . Secondly, even when h i is available, computing the two loss terms involves fitting models twice which can be computationally challenging.

To address these challenges, the Feature Acquisition Agent A predicts the value of feature V ( h i ) from the available information about h i , as follows:

where c h i represents the unstructured prior information about the feature h i (e.g. data descriptions and metadata), T ( h i ) represents the available summary statistics (e.g. range and type of values from a small subset of data h i ), D i -1 represents the existing data, and g is the prediction function.

The prediction challenge above corresponds to zero-shot or few-shot learning in ML. In the zero-shot case, we need to predict ˆ V ( h i ) without having access to any 'label' V ( h ) , for some feature h . In the few-shot case, we have access to a small number of feature-label pairs, { c h i , T ( h i ) , h i -1 , V ( h i -1 ) } to learn the prediction function g . The Feature Acquisition Agent A leverages LLM's capability in zero-shot and few-shot learning to address this estimation challenge. Specifically, we inform the LLM by providing high-level statistics T ( h i ) and a description of the feature c h i in the form of the feature name to warm-start the zero-shot learning. When the h i has been included in the data, we inform the LLM about the validation loss associated with those data v ( h i ) and add it to the few-shot examples to guide the estimation.

Context, feedback and memory . Similar to G , A is also able to leverage unstructured problem context c , evaluation feedback r i -1 , and memory s i -1 to guide the data acquisition proposal.

3.4 The Evaluation Agent E

The Evaluation Agent E is vital for iteratively improving the Modeling Agent G and Feature Acquisition Agent A . It provides feedback r i :

Feedback r i can be numeric, such as validation loss or mean squared error (MSE) of the model f i on the validation subset D i , or more refined metrics like validation loss per target dimension in D . It can

also be textual, offering detailed suggestions for model improvements, including clinically plausible modifications generated by an LLM [Shinn et al., 2024].

Evaluating dynamical systems is complex due to intricate interactions and temporal dependencies. The Evaluation Agent dynamically assesses both model performance and plausibility , ensuring models are both accurate and interpretable. This comprehensive evaluation is crucial for developing sophisticated, clinically relevant pharmacokinetic models that address real-world complexities.

4 Related Works

Table 1: Comparison with related works in addressing the Modeling and Data Acquisition Challenges. An empty field means not applicable. Refinement Level : the level of refinement (R1 - R3) the method can capture. Class of f : the class of ODE model f (closed-form M C , neural networks M N ). Context c : whether the method leverages unstructured contextual information to guide search. Sample N : whether the method is able to acquire new samples. Feature h : whether the method can acquire new feature variables. Goal : whether the goal of data acquisition is to improve training, evaluation, or both.

M

∪ M

Our work focuses on autonomously learning temporal models while acquiring data, with several relevant research strands, as summarized in Table 1, which is expanded in Appendix A.

ODE learning methods. Symbolic regression methods like SINDy and D-CODE [Koza, 1994, Brunton et al., 2016, Qian et al., 2022] can discover closed-form ODEs f ∈ M C using genetic algorithms. They can incorporate features a n (R2 refinement) but struggle with many variables (e.g., >20) due to computational complexity. Neural ODEs use neural networks f ∈ M N to handle many variables [Chen et al., 2018, Dupont et al., 2019, Zaytar and El Amrani, 2016, Devlin et al., 2018, Sehovac and Grolinger, 2020], but have numerous free parameters, risking overfitting with insufficient data. D3 captures f ∈ M C ∪ M N and supports R1 - R3 refinement, adapting based on data availability. Both Symbolic regression and Neural ODEs require manual context incorporation c via hyperparameters and do not address Data Acquisition Challenges.

AI for automated modeling. AI Feynman [Udrescu and Tegmark, 2020] enhances symbolic regression by using the physical units of variables, which provide additional constraints to narrow the search space. However, its utility outside physics is limited, as units in other fields carry less information. Eureqa [Ma et al., 2023] applies LLMs to model the reward function in reinforcement learning, integrating unstructured context and representing models as code. Unlike D3, Eureqa cannot automatically refine models, acquire new data, or easily apply to learning temporal dynamics like ODEs.

Data Acquisition. Methods like Active Feature Acquisition (AFA) [Ma et al., 2018, Gong et al., 2019] measure additional features on existing samples to improve performance. Active Learning [Sebastiani and Wynn, 2000, Settles, 2009, Sener and Savarese, 2017, Imberg et al., 2020] and Active Testing [Lowell et al., 2018, Kossen et al., 2021] acquire new samples to increase training or evaluation data size while keeping features constant. These methods do not address the zero-shot setting, where data acquisition decisions must be made before observing any target data (Equation 7). Additionally, most Active Learning methods focus on acquiring labels for supervised learning, which is not directly applicable to temporal modeling tasks.

5 Experiments and Evaluation

In this section, we demonstrate that D3 can discover well-fitting pharmacokinetic dynamical system models for a range of diverse PK datasets, including an epidemiological and ecological dataset.

Benchmark Datasets . Our evaluation encompasses six real-world datasets with clinical relevance, each originating from either real-world data or highly accurate simulators developed by human experts.

Three datasets are based on a state-of-the-art biomedical Pharmacokinetic-Pharmacodynamic (PKPD) model of lung cancer tumor growth, which simulates the combined effects of chemotherapy and radiotherapy in lung cancer [Geng et al., 2017] (Equation (11)). This model has been widely utilized in previous research [Bica et al., 2020, Seedat et al., 2022, Melnychuk et al., 2022]. Specifically, we employ this bio-mathematical model to generate three variations: lung cancer without treatment ( Lung Cancer ), lung cancer treated with chemotherapy ( Lung Cancer (with Chemo.) ), and lung cancer treated with both chemotherapy and radiotherapy ( Lung Cancer (with Chemo. & Radio.) ). Additionally, we utilize an intricate COVID-19 epidemic agent-based simulator ( COVID-19 ) [Kerr et al., 2021], to provide an epidemiological dataset. Another dataset comes from an ecological model simulating a microcosm of algae, flagellate, and rotifer populations ( Plankton Microcosm ), replicating a three-species prey-predator experimental system [Hiltunen et al., 2013]. Finally, we include a real Pharmacokinetic (PK) dataset of Warfarin patients ( Warfarin ) [Janssen et al., 2022]. Detailed information about all benchmark datasets is provided in Appendix B.

Benchmark Methods . To evaluate the performance of D3 and establish its competitive performance, we conduct comparisons with leading modeling methods for ODEs. Specifically, we benchmark against advanced black-box models that have many parameters, such as neural ODEs with action inputs, known as DyNODE [Chen et al., 2018, Alvarez et al., 2020]. Also in this class are recurrent neural networks ( RNN ) and a state-of-the-art transformer model ( Transformer ). In addition, we include white-box transparent dynamical systems models identified through equation discovery techniques, such as Sparse Identification of Nonlinear Dynamics ( SINDy ) [Brunton et al., 2016]. Moreover, D3 supports two discovery modes, of which we compare against both, discovering only white-box models ( D3-white-box ) and discovering hybrid models ( D3-hybrid ) 3 . We also perform ablations of D3, of zero-shot generated model from D3 as ( ZeroShot ) and the same model with optimized parameters ( ZeroOptim ). Detailed descriptions of the implementations, hyperparameters, and experimental procedures for these benchmarks can be found in Appendix E.

Evaluation Metrics . To assess the performance of our benchmark methods, we use the mean squared error (MSE) on a held-out test dataset of state-action trajectories. This evaluation is conducted over ten runs, each initialized with different random seeds. We report the average MSE from these runs along with their 95% confidence intervals. Further details can be found in Appendix C.

6 Main Results

We conducted a comprehensive evaluation of our benchmark methods across all datasets, as tabulated in Table 2. We observe that D3 can discover well-fitting dynamical system models, achieving low mean squared error in test predictions on the held-out test dataset of individual trajectories. Crucially, it can discover concise closed-form equation white-box models, of 5-15 parameters that can outperform some of the standard white-box and black-box modeling methods, indicating it is discovering well-fitting underlying equations for the respective systems of interest.

6.1 Case Study: Discovering Novel PK Models for Warfarin

PK models of warfarin, a widely used anticoagulant, are crucial due to its extensive use in treating deep vein thrombosis, pulmonary embolism, and stroke prevention in atrial fibrillation and patients with mechanical heart valves, with millions of prescriptions globally [Lee and Klein, 2013]. We applied D3 to a public dataset of 33 patient trajectories [Janssen et al., 2022], discovering a novel PK model for warfarin that outperforms existing literature (Table 3). We compare the test MSE of the

discovered model against existing models and provide pharmacologist feedback. Full model details and results from running D3-hybrid on the same dataset are in Appendix G.

Existing Wafarin PK Model . The standard pharmacokinetic (PK) warfarin model from the literature [Lv et al., 2017, Hamberg, 2013] is the following:

where C is the concentration of warfarin, D is the dosage administered, A represents the age of the patient, S denotes the sex of the patient (1 for male, 0 for female), k a is the absorption rate, k e, base is the base elimination rate, k e, age represents the decrease in elimination rate per year increase in age, and k e, sex is the difference in elimination rate between sexes. This standard model achieves a test loss of 0.646. Whereas D3 can discover both a white-box model with a test loss of 0.39 and a hybrid model with a test loss of 0.271, which both outperform the standard model. Of particular relevance is the white-box model as it is fully interpretable by pharmacometricians, whilst still being a precise model.

New Discovered PK Warfarin Model . D3-white-box discovered a new warfarin PK white-box model with a test loss of 0.39, of the following:

where the additional parameters k decay represent the natural decay rate of warfarin concentration and K m is the Michaelis constant indicating the warfarin concentration at which the metabolism rate is half its maximum. The parameters k ds , k as, and k da are the interaction terms for dosage-sex, age-sex, and dosage-age, respectively, and A and S are the sample population means for the covariates of age and sex. This new PK model includes the original parameters along with additional interaction terms and transformations, enhancing the model's complexity and accuracy.

The model discovered by D3 introduces key innovations in warfarin pharmacokinetics. Square Root Transformation for Dosage Effect : This non-linear transformation moderates high doses, enhancing sensitivity to dosage variations, unlike standard models. Natural Decay Term : Proportional to the current warfarin concentration, it better reflects elimination kinetics and improves time-based predictions. Michaelis-Menten Saturation Kinetics : Addresses metabolic pathway saturation at higher concentrations, crucial for warfarin. Interaction Terms (dosage-sex, age-sex, dosage-age) : Capture complex interactions between patient-specific factors, providing a nuanced understanding of drug dynamics. These advancements make D3 a powerful tool for enhancing pharmacokinetic predictions in clinical settings. The discovered hybrid model is detailed in Appendix G.

Expert Clinical Commentary. We sought feedback from expert pharmacologists on the discovered model 4 : Prof. Eoin McKinney, Clinician . 'This model is significant, as consortiums are dedicated to improving Warfarin modeling [Consortium, 2009]. The model adds novel components, such as the Michaelis component for time-varying changes and novel interaction terms like age-sex.' Jean-Baptiste Woillard, Pharmacologist . 'The model is promising and pharmacokinetically plausible. The next step is to apply D3 to other clinically relevant PK drug datasets.' Richard Peck, Clinical Pharmacologist . 'This model is reasonable and potentially superior. It represents a significant advance in clinical pharmacology by automatically identifying robust PK models.'

6.2 Insight Experiments

This section provides an in-depth analysis of D3's effectiveness related to its benchmark counterparts.

Can D3 perform feature acquisition and leverage the LLM prior information to perform this better? To explore the feature acquisition performance of D3, we showcase this component working

Validatio

10 0

10

-

1

0

3

6

9

12

Number of Generations

0

3

6

9

12

15

18

Number of Generations

10

-

1

10 0

10 1

10 2

Validation MSE

D3-random-acquisition

D3-feature-selective-acquisition

D3-no-feature-acquisition

1

2

3

4

5

6

Number of Generations

3

×

10

-

1

4

×

10

-

1

Validation MSE

18

7

DI3-random-acquisition

DI3-feature-selective-acquisition

DI3-no-feature-acquisition

1

2

3

4

5

6

7

8

9

10

10

-

3

10

-

1

10 1

Validation MSE

Description:

Initial white box

linear model.

Features selected: 2

(tumor_volume,

chemotherapy_drug_concentration)

Description:

White box model:

Gompertzian growth, chemotherapy

decay, radiotherapy effects;

parameters: growth rate, drug and

radiotherapy efficacy.

Features selected: 3

(tumor_volume,

chemotherapy_drug_concentration,

radiotherapy_dosage)

Description:

Hybrid model: white box

with Gompertzian growth, chemo

decay, radiotherapy, and genetic

markers; black box MLP for complex

interactions and residuals.

Features selected: 5

(tumor_volume,

chemotherapy_drug_concentration,

radiotherapy_dosage, genetic_markers,

chemotherapy_dosage)

Description:

Hybrid model: white box

with Gompertzian growth, chemo

decay, radiotherapy, chemo dosage,

genetic markers; black box MLP for

complex interactions and residuals.

Parameters optimized for growth rate,

drug and radiotherapy efficacy, and

genetic effects.

Features selected: 5

(tumor_volume,

chemotherapy_drug_concentration,

radiotherapy_dosage, genetic_markers,

chemotherapy_dosage)

Number of Generations

Top-1 DI3

in full, by comparing it against the baseline of a random feature acquirer policy, a null policy that just uses the existing features and our approach of D3 that leverages the LLM to quantify the value of information for features, where no training data exists for them in the dataset. We observe, as in Figure 2, that D3 converges the fastest, and achieves the overall highest performance, whereas the other feature acquisition methods fall short. Experimental details are in Appendix G.2.

Can D3 evolve its modular model to fit the system best? Weanalyze this from an empirical point of view to determine if D3 can correctly evolve the generated model and reduce its prediction error over subsequent generations. We observe that D3 can indeed understand, reason, and iterate the generated code representation of the model to incorporate a better fitting model, as observed in Figure 3.

7 Conclusion

Summary . In this paper, we proposed Data-Driven Discovery (D3) framework, to iteratively discover and refine interpretable models of pharmacological dynamics, where it has the ability to acquire and integrate new features, validate, and compare pharmacological dynamical systems models, thereby uncovering new insights into pharmacokinetic and physiological processes. Specifically, applied to a real Warfarin PK dataset we were able to discover a better performing Warfarin PK model and provide new insights into what an optimal PK model for Warfarin may need to possess, such as additional Michaelis components.

Limitations & Future work . There exist limitations to the current approach. First, the LLM discovery framework is an initial framework, and the utility of the LLM to generate better models could be enhanced, tree-based generation strategies such as the tree of thought [Yao et al., 2024], or graph of thought [Besta et al., 2024]. Specifically, it relies on a capable enough LLM, that can use tools, where we specifically used GPT4 as the underlying LLM. Moreover, we make the assumption that when D3 acquires a new feature it acquires that feature for all the individuals within the existing dataset, we leave for future work to consider applications where this assumption may not always be true. Furthermore, we provide to the LLM a system description from a user, future works could explore avoiding the user to provide this, and instead leveraging a form of automatic retrieval augmented generation [Lewis et al., 2020].

Broader Impact and Ethical Considerations . Principally D3 aims to discover interpretable models of pharmacological dynamics. However, the final discovered models should always be checked

15

8

9

10

by appropriate human experts and validated in additional held-out datasets before any clinical use. Furthermore, D3 is a tool, which could be misused by a malicious user with unethical system descriptions as input to discover a potentially biased model. Moreover, D3 leverages LLMs and their feedback as an integral component, however, LLMs are prone to hallucinations, thereby motivating any LLM-generated outputs that are shown to the user should have a content filter applied to them.

Acknowledgments and Disclosure of Funding

We extend our gratitude to the anonymous reviewers, area and program chairs, members of the van der Schaar lab, and Andrew Rashbass for their valuable feedback and suggestions. SH gratefully acknowledges the sponsorship and support of AstraZeneca. ZQ did this work whilst a postdoc in the van der Schaar lab. This work was funded by Microsoft's Accelerate Foundation Models Academic Research initiative.

References

Victor M Martinez Alvarez, Rare¸ s Ro¸ sca, and Cristian G F˘ alcu¸ tescu. Dynode: Neural ordinary differential equations for dynamics modeling in continuous control. arXiv preprint arXiv:2009.04278 , 2020.

Pierre Auger, Pierre Magal, and Shigui Ruan. Structured population models in biology and epidemiology , volume 1936. Springer, 2008.

Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 17682-17690, 2024.

Ioana Bica, Ahmed M. Alaa, James Jordon, and Mihaela van der Schaar. Estimating counterfactual treatment outcomes over time through adversarially balanced representations. In International Conference on Learning Representations , 2020.

Lynne Billard and Edwin Diday. Symbolic regression analysis. In Classification, clustering, and data analysis: recent advances and applications , pages 281-288. Springer, 2002.

Willem Bonnaffé and Tim Coulson. Fast fitting of neural ordinary differential equations by bayesian neural gradient matching to infer ecological interactions from time-series data. Methods in Ecology and Evolution , 14(6):1543-1563, 2023.

Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences , 113(15):3932-3937, 2016.

Alex Chan and Mihaela van der Schaar. Synthetic model combination: An instance-wise approach to unsupervised ensemble learning. Advances in Neural Information Processing Systems , 35: 27797-27809, 2022.

Bing Chen and Adnan OM Abuassba. Compartmental models with application to pharmacokinetics. Procedia Computer Science , 187:60-70, 2021.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

Limei Cheng, Yuchi Qiu, Brian J Schmidt, and Guo-Wei Wei. Review of applications and challenges of quantitative systems pharmacology modeling and machine learning for heart failure. Journal of pharmacokinetics and pharmacodynamics , pages 1-12, 2022.

International Warfarin Pharmacogenetics Consortium. Estimation of the warfarin dose with clinical and pharmacogenetic data. New England Journal of Medicine , 360(8):753-764, 2009.

José GB Derraik, Wason Parklak, Benjamin B Albert, Kongsak Boonyapranai, and Kittipan Rerkasem. Fundamentals of data collection in clinical studies: simple steps to avoid 'garbage in, garbage out'. The International Journal of Lower Extremity Wounds , 20(3):183-187, 2021.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. Advances in neural information processing systems , 32, 2019.

James D Dziura, Lori A Post, Qing Zhao, Zhixuan Fu, and Peter Peduzzi. Strategies for dealing with missing data in clinical trials: from design to analysis. The Yale journal of biology and medicine , 86(3):343, 2013.

Gerald A Feltham. The value of information. The accounting review , 43(4):684-696, 1968.

Johan Gabrielsson and Daniel Weiner. Pharmacokinetic and pharmacodynamic data analysis: concepts and applications. 2001.

Changran Geng, Harald Paganetti, and Clemens Grassberger. Prediction of Treatment Response for Combined Chemo- and Radiation Therapy for Non-Small Cell Lung Cancer Patients Using a Bio-Mathematical Model. Scientific Reports , 7(1):13542, October 2017. ISSN 2045-2322. doi: 10.1038/s41598-017-13646-z.

Wenbo Gong, Sebastian Tschiatschek, Sebastian Nowozin, Richard E Turner, José Miguel HernándezLobato, and Cheng Zhang. Icebreaker: Element-wise efficient information acquisition with a bayesian deep latent gaussian model. Advances in neural information processing systems , 32, 2019.

Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. Multi-dimensional recurrent neural networks. In International conference on artificial neural networks , pages 549-558. Springer, 2007.

FDA Draft Guidance. Adaptive design clinical trials for drugs and biologics. Biotechnol Law Rep , 29 (2):173, 2010.

Anna-Karin Hamberg. Pharmacometric models for individualisation of warfarin in adults and children . PhD thesis, Acta Universitatis Upsaliensis, 2013.

Mohsen A Hedaya. Basic pharmacokinetics . CRC Press, 2012.

Teppo Hiltunen, LE Jones, SP Ellner, and Nelson G Hairston Jr. Temporal dynamics of a simple community with intraguild predation: an experimental test. Ecology , 94(4):773-779, 2013.

Samuel Holt, Alihan Hüyük, Zhaozhi Qian, Hao Sun, and Mihaela van der Schaar. Neural laplace control for continuous-time delayed systems. In International Conference on Artificial Intelligence and Statistics , pages 1747-1778. PMLR, 2023a.

Samuel Holt, Zhaozhi Qian, and Mihaela van der Schaar. Deep generative symbolic regression. In The Eleventh International Conference on Learning Representations , 2023b. URL https: //openreview.net/forum?id=o7koEEMA1bR .

Samuel Holt, Alihan Hüyük, and Mihaela van der Schaar. Active observing in continuous-time control. Advances in Neural Information Processing Systems , 36, 2024a.

Samuel Holt, Tennison Liu, and Mihaela van der Schaar. Automatically learning hybrid digital twins of dynamical systems. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024b. URL https://openreview.net/forum?id=SOsiObSdU2 .

Samuel Holt, Max Ruiz Luyten, and Mihaela van der Schaar. L2MAC: Large language model automatic computer for extensive code generation. In The Twelfth International Conference on Learning Representations , 2024c. URL https://openreview.net/forum?id=EhrzQwsV4K .

Samuel I Holt, Zhaozhi Qian, and Mihaela van der Schaar. Neural laplace: Learning diverse classes of differential equations in the laplace domain. In International Conference on Machine Learning , pages 8811-8832. PMLR, 2022.

Henrik Imberg, Johan Jonasson, and Marina Axelson-Fisk. Optimal sampling in unbiased active learning. In International Conference on Artificial Intelligence and Statistics , pages 559-569. PMLR, 2020.

Alexander Janssen, Frank C Bennis, and Ron AA Mathôt. Adoption of machine learning in pharmacometrics: An overview of recent implementations and their considerations. Pharmaceutics , 14(9): 1814, 2022.

Krzysztof Kacprzyk, Samuel Holt, Jeroen Berrevoets, Zhaozhi Qian, and Mihaela van der Schaar. ODE discovery for longitudinal heterogeneous treatment effects inference. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/ forum?id=pxI5IPeWgW .

Cliff C Kerr, Robyn M Stuart, Dina Mistry, Romesh G Abeysuriya, Katherine Rosenfeld, Gregory R Hart, Rafael C Núñez, Jamie A Cohen, Prashanth Selvaraj, Brittany Hagedorn, et al. Covasim: an agent-based model of covid-19 dynamics and interventions. PLOS Computational Biology , 17(7): e1009149, 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample-efficient model evaluation. In International Conference on Machine Learning , pages 5753-5763. PMLR, 2021.

John R Koza. Genetic programming as a means for programming computers by natural selection. Statistics and computing , 4:87-112, 1994.

Siddharth Krishna Kumar. On weight initialization in deep neural networks. arXiv preprint arXiv:1704.08863 , 2017.

Ming Ta Michael Lee and Teri E Klein. Pharmacogenetics of warfarin: challenges and opportunities. Journal of human genetics , 58(6):334-338, 2013.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems , 33: 9459-9474, 2020.

Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P Trevino, Jiliang Tang, and Huan Liu. Feature selection: A data perspective. ACM computing surveys (CSUR) , 50(6):1-45, 2017.

Ernest Lindelöf. Sur l'application de la méthode des approximations successives aux équations différentielles ordinaires du premier ordre. Comptes rendus hebdomadaires des séances de l'Académie des sciences , 116(3):454-457, 1894.

David Lowell, Zachary C Lipton, and Byron C Wallace. Practical obstacles to deploying active learning. arXiv preprint arXiv:1807.04801 , 2018.

Chunxiao Lv, Changxiao Liu, Zhuhua Yao, Xiumei Gao, Lanjun Sun, Jia Liu, Haibo Song, Ziqiang Li, Xi Du, Jinxia Sun, et al. The clinical pharmacokinetics and pharmacodynamics of warfarin when combined with compound danshen: a case study for combined treatment of coronary heart diseases with atrial fibrillation. Frontiers in Pharmacology , 8:826, 2017.

Chao Ma, Sebastian Tschiatschek, Konstantina Palla, José Miguel Hernández-Lobato, Sebastian Nowozin, and Cheng Zhang. Eddi: Efficient dynamic discovery of high-value information with partial vae. arXiv preprint arXiv:1809.11142 , 2018.

Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310.12931 , 2023.

Valentyn Melnychuk, Dennis Frauen, and Stefan Feuerriegel. Causal transformer for estimating counterfactual outcomes. In International Conference on Machine Learning , pages 15293-15329. PMLR, 2022.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.

Zhaozhi Qian, Krzysztof Kacprzyk, and Mihaela van der Schaar. D-CODE: Discovering closed-form ODEs from observed trajectories. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=wENMvIsxNN .

David K Ryan, Rory H Maclean, Alfred Balston, Andrew Scourfield, Anoop D Shah, and Jack Ross. Artificial intelligence and machine learning for clinical pharmacology. British Journal of Clinical Pharmacology , 90(3):629-639, 2024.

William E Schiesser. Computational mathematics in engineering and applied science: ODEs, DAEs, and PDEs . CRC press, 2014.

Paola Sebastiani and Henry P Wynn. Maximum entropy sampling and optimal bayesian experimental design. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 62(1):145-157, 2000.

Nabeel Seedat, Fergus Imrie, Alexis Bellot, Zhaozhi Qian, and Mihaela van der Schaar. Continuoustime modeling of counterfactual outcomes using neural controlled differential equations. arXiv preprint arXiv:2206.08311 , 2022.

Ljubisa Sehovac and Katarina Grolinger. Deep learning for load forecasting: Sequence to sequence recurrent neural networks with attention. Ieee Access , 8:36411-36426, 2020.

Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489 , 2017.

Burr Settles. Active learning literature survey. 2009.

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Shruti Singh, Rajesh Kumar, Shuvasree Payra, and Sunil K Singh. Artificial intelligence and machine learning in pharmacological research: Bridging the gap between data and drug discovery. Cureus , 15(8), 2023.

Silviu-Marian Udrescu and Max Tegmark. Ai feynman: A physics-inspired method for symbolic regression. Science Advances , 6(16):eaay2631, 2020.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems , 36, 2024.

Mohamed Akram Zaytar and Chaker El Amrani. Sequence to sequence weather forecasting with long short-term memory recurrent neural networks. International Journal of Computer Applications , 143(11):7-11, 2016.

Appendix

Table of Contents

A Additional Related Work

We expand on the related work contained within the main paper.

ODE learning methods. Symbolic regression methods [Koza, 1994, Kacprzyk et al., 2024, Holt et al., 2023b] such as SINDy and D-CODE [Brunton et al., 2016, Qian et al., 2022] can automatically discover closed form ODEs f ∈ M C through genetic algorithms. In principle, they can incorporate features a n , achieving refinement R2, but they struggle with many variables (e.g., more than 20) due to computational complexity. On the other hand, Neural ODE leverages neural networks f ∈ M N to capture a large number of variables [Chen et al., 2018, Dupont et al., 2019, Zaytar and El Amrani, 2016, Devlin et al., 2018, Sehovac and Grolinger, 2020, Holt et al., 2022, 2023a, 2024a]. However, Neural ODEs involve many free parameters and cannot automatically reduce the level of refinement when there is insufficient training data, leading to overfitting. In comparison, D3 can capture f ∈ M C ∪ M N and R1 - R3 levels of refinement; it can further adapt the level of refinement based on data availability. Furthermore, in both Symbolic regression and Neural ODE, human experts need to manually incorporate the problem context c through the specification of many hyperparameters. Finally, neither of these methods assumes fixed training data and does not address the Data Acquisition Challenges.

AI for automated modeling. AI Feynman [Udrescu and Tegmark, 2020] improves upon the symbolic regression methods by leveraging a specific type of information c , i.e. the physical unit of each variable (e.g., meter). In applications in physics, knowing the units provides additional constraints on the variable's relationship, thus helping narrow down the search space. However, its utility outside physical sciences has been limited. More recently, Eureqa [Ma et al., 2023] applies LLMs to modeling the reward function of reinforcement learning algorithms. Eureqa is similar to D3 in its ability to integrate unstructured context c and represent the model as code. However, Eureqa cannot automatically adjust the level of refinement or acquire new data and it does not easily apply to learning temporal dynamics, e.g. ODEs. Furthermore, other LLM code generation [Holt et al., 2024b,c] approaches cannot acquire new data.

Data Acquisition. Methods have been developed to address specific data acquisition challenges. Active Feature Acquisition (AFA) attempts to measure additional feature variables h on existing training samples to improve predictive performance [Ma et al., 2018, Gong et al., 2019]. On the other hand, Active Learning [Sebastiani and Wynn, 2000, Settles, 2009, Sener and Savarese, 2017, Imberg et al., 2020] and Active Testing [Lowell et al., 2018, Kossen et al., 2021] attempt to acquire new samples to increase the size of training or evaluation data N while keeping the features constant. Neither field has proposed a method to address the zero-shot setting, where the data acquisition decision must be made before observing any target data (Equation 7). Moreover, most existing methods in Active Learning focus on acquiring labels for supervised learning given unlabelled samples, which is not immediately applicable to temporal modeling tasks.

B Benchmark Dataset Environment Details

In the following, we present six clinically relevant datasets, each of which is either sourced from real-world data or generated from a high-fidelity simulator developed by domain experts.

B.1 Cancer PKPD

Three of our environments are derived from a state-of-the-art biomedical PharmacokineticPharmacodynamic (PKPD) model of lung cancer tumor growth, used to simulate the combined effects of chemotherapy and radiotherapy in lung cancer [Geng et al., 2017]. This model has been extensively used in other works [Seedat et al., 2022, Bica et al., 2020, Melnychuk et al., 2022]. We use this bio-mathematical lung cancer model to create three variations: no treatments ( Lung Cancer ), chemotherapy only ( Lung Cancer (with Chemo.) ), and both chemotherapy and radiotherapy ( Lung Cancer (with Chemo. & Radio.) ). For each model, we sample a respective dataset. Below, we detail the general case of Lung Cancer (with Chemo. & Radio.) , which originates from the general Cancer PKPD Model, and then describe the variations.

Cancer PKPD Model . This model simulates the combined effects of chemotherapy and radiotherapy on lung cancer tumor growth [Geng et al., 2017], as shown in Equation (11). The model predicts

tumor volume x ( t ) over time t in days post-diagnosis. The model includes two binary treatments: (1) radiotherapy u r t and (2) chemotherapy u c t .

The parameters K,ρ,β c , α r , β r for each simulated patient are specified in Geng et al. [2017] and summarized in Table 4. Additionally, the chemotherapy drug concentration C ( t ) follows an exponential

decay with a half-life of one day:

The chemotherapy binary action represents increasing the C ( t ) concentration by 5 . 0 mg / m 3 of Vinblastine given at time t . The radiotherapy concentration d ( t ) represents 2 . 0 Gy fractions of radiotherapy given at timestep t , where Gy is the Gray ionizing radiation dose.

Time-dependent confounding. We introduce time-varying confounding by modeling chemotherapy and radiotherapy assignment as Bernoulli random variables. The probabilities p c and p r depend on tumor diameter as follows:

where D max = 13 cm is the maximum tumor diameter, δ c = δ r = D max / 2 , and ¯ D ( t ) is the average tumor diameter. The parameters γ c and γ r control the extent of time-varying confounding, with γ c = γ r = 2 .

Sampling datasets. Using the Cancer PKPD model, we sample N = 1 , 000 trajectories, corresponding to N = 10 , 000 patients. Initial tumor volumes are sampled from a uniform distribution x (0) ∼ U (0 , 1149) , and patient trajectories are forward simulated for 60 days using the Cancer PKPD Equation (11) and the action policy of Equation (13), employing a Euler stepwise solver. This process generates one dataset sample. We repeat this with independent random seeds to create D train , D val , D test. For each benchmark method run with a random seed, we re-sample the datasets. Each variation includes either chemotherapy, both chemotherapy and radiotherapy, or neither. We provide further details of this dataset's system description and variable descriptions in Appendix F.5.

B.2 COVID-19

We utilize the sophisticated epidemic agent-based simulator COVASIM [Kerr et al., 2021] to model COVID-19 epidemics. This advanced simulator is capable of simulating both non-pharmaceutical interventions (such as lockdowns, social distancing, and school closures) and pharmaceutical interventions (such as vaccinations). In this agent-based simulator, each agent represents an individual within the population, and can be in one of several states: susceptible to COVID-19, exposed, infectious, or recovered (including deaths).

We use COVASIM with its default parameter settings provided by the open-source implementation 5 . COVASIM simulates a population of individuals, and to ensure accuracy, we simulate 24 countries,

collecting trajectories for each. For each simulation, we use a population size of 1 , 000 , 000 individuals, simulating each individual separately (disabling simulation rescaling). The simulation begins with a random number of individuals initially infected with COVID-19, I (0) = U (10 , 000 , 100 , 000) , and proceeds for 60 days.

We repeat this process with independent random seeds to generate D train , D val , D test . For each benchmark method run with a different random seed, we re-sample the datasets. Detailed descriptions of this dataset's system and variables are provided in the prompt template outlined in Appendix F.5.

B.3 Plankton Microcosm

This subsection describes an ecological model of a microcosm consisting of algae, flagellate, and rotifer populations, replicating an experimental three-species prey-predator system [Hiltunen et al., 2013]. We use the dataset provided by [Bonnaffé and Coulson, 2023] 6 . The dataset contains a single trajectory of 102 time steps. We split the data into training, validation, and test sets with proportions of 70%, 15%, and 15%, respectively, ensuring that the splits maintain the chronological order to preserve temporal causality.

Detailed descriptions of this dataset's system and variables are provided in the prompt template outlined in Appendix F.5.

B.4 PK Wafarin Dataset

Here we describe the real PK Wafarin dataset from a clinical trial that is publicly available [Janssen et al., 2022]. We use the NOMEN dataset available at https://github.com/Janssena/ SI-AIEP-paper . This publicly available dataset consists of 32 patients who received warfarin in a clinical trial, which was originally designed to determine how to predict drug concentrations in patients. The dataset includes a total of 251 warfarin concentration measurements, with a median of six measurements per patient. Each patient received a single dose of warfarin at t = 0 , and measurements were taken at t ∈ { 0 . 25 , 0 . 5 , 1 . 0 , 2 . 0 , 4 . 0 , 6 . 0 , 12 . 0 , 24 . 0 , 48 . 0 , 72 . 0 , 96 . 0 , 120 . 0 } . The available covariates in the dataset are patient weight, age, and sex. We follow the dataset's original pre-processing code, which is provided at https://github.com/Janssena/SI-AIEP-paper . We split the data into training, validation, and test sets with proportions of 70%, 15%, and 15%, respectively, ensuring that the splits maintain the chronological order to preserve temporal causality. This dataset is released under a GPL-3.0 license.

Detailed descriptions of this dataset's system and variables are provided in the prompt template outlined in Appendix F.5.

C Evaluation Metrics

We utilize mean squared error (MSE) to assess the benchmark methods on a separate test dataset comprising individual trajectories, denoted as D test. This assessment is based on the loss defined in Equation (14) and reported as MSE. The metrics are averaged over ten runs with different random seeds, and we provide these averages along with their 95% confidence intervals. Specifically we provide the mean confidence interval 7 . For each random seed, we independently generate new training, validation, and test datasets when a simulator is available. Specifically, for each simulation, we ensure that the validation and test datasets contain the same number of trajectories as the training set. Each baseline model is trained on the training dataset, with early stopping applied using the validation dataset when supported by the method. Subsequently, we evaluate the performance of each baseline on the test dataset. This entire procedure is repeated for each random seed. All experiments and training were conducted using a single Intel Core i9-12900K CPU @ 3.20GHz, 64GB RAM, and an Nvidia RTX3090 GPU with 24GB of memory.

C.1 Model Optimization Losses

We evaluate the optimization loss using mean squared error (MSE) on a dataset D , and also consider a component-wise MSE loss for more detailed analysis.

MSE Loss. Specifically, we optimize the following mean squared error objective:

where N × T represents the total number of state-action pairs in the dataset. The goal is to find the parameters θ ∗ that minimize this loss, i.e., θ ∗ = arg min θ L ( θ, D train ) . Weoptimize θ using stochastic gradient descent with the Adam optimizer [Kingma and Ba, 2014], although other optimization algorithms, such as black box optimizers, could also be employed.

MSELoss per component. To gather detailed quantitative statistics on the performance of the trained system model, we collect the validation loss per component. Using ( j ) to denote the predictions for the j th component, we define:

We aggregate these scalar validation losses per component into a vector ω = [ ω 1 , ω 2 , . . . , ω m ] , and compute the mean validation loss as v = 1 m ∑ m j =1 ω j ( θ ∗ , D val ) .

D Pharmacologists Feedback Statements

We sought feedback from three expert pharmacologists on the discovered model. They have validated and commented on the discovered model and the impact of such a tool as D3.

Prof. Eoin Mckinney, Clinician . 'This model is significant, as consortiums are dedicated to improving Warfarin modeling [Consortium, 2009]. The updated PK model (Equation (10)) adds complexity with interaction terms and introduces two new variables: the Michaelis constant and the natural decay rate of warfarin concentration. The Michaelis constant, which varies between individuals or over time due to factors like temperature and pH, is presumably estimated from observed data. The natural decay rate is likely derived from prior reports but also varies between individuals. The model has proposed minimal-cost new features (a constant and interaction terms), but it's worth noting the genotype of certain warfarin-metabolizing enzymes (CYP2C9 and VKORC1) dictates clearance rates and levels, though the model did not recommend measuring these, possibly due to the associated costs. The principal clinical relevance of the improved model is better prediction of individual warfarin doses. Numerous algorithms attempt this, supported by a consortium focused on dose-prediction models. The data used, including public datasets, supports demonstrating the clinical utility of the improved PK model, highlighting the relevance of this problem despite newer alternatives to warfarin.'

Jean-Baptiste Woillard, Pharmacologist . 'The model is promising and pharmacokinetically plausible. I have data for mycophenolic acid, which has a challenging, complex absorption profile that would be interesting to test. The challenge extends to PK/PD, linking exposure and effect (e.g., tumor growth). Monolix, based on the SAEM algorithm, includes relevant datasets such as the PK/PD Warfarin dataset and a PK dataset for tacrolimus in heart transplants, which could serve as valuable benchmarks for comparison. Additionally, TMDD models, where clearance changes with tumor size, are of particular interest. It would be beneficial to see performance comparisons with these datasets using your approach.'

Richard Peck, Clinical Pharmacologist . 'This model is reasonable and potentially superior. It represents a significant advance in clinical pharmacology by automatically identifying robust PK models. While I am not a modeler or an expert in warfarin modeling, the discovered model appears suitable for your purpose. The most interesting aspect of this work is that D3 identified an alternative model that could be superior. While improving PK models for warfarin might not have much clinical use since warfarin dosing is monitored by INR rather than concentration, the ability to automatically identify robust PK models is a tremendous advance. Publishing about DI3 in clinical pharmacology literature would be significant. I haven't seen anything like this before. However, to show superiority,

consider studying a drug with a more challenging PK. For such a publication, involving an expert pharmacometrician would be beneficial.'

E Benchmark Method Implementation Details

To evaluate the performance of D3 and establish its competitive performance, we conduct comparisons with leading modeling methods for ODEs. Specifically, we benchmark against advanced black-box models that have many parameters, such as neural ODEs with action inputs, known as DyNODE [Chen et al., 2018, Alvarez et al., 2020]. Also in this class are recurrent neural networks ( RNN ) and a state-of-the-art transformer model ( Transformer ). In addition, we include white-box transparent dynamical systems models identified through equation discovery techniques, such as Sparse Identification of Nonlinear Dynamics ( SINDy ) [Brunton et al., 2016]. Moreover, D3 supports two discovery modes, of which we compare against both, discovering only white-box models ( D3-white-box ) and discovering hybrid models ( D3-hybrid ). We also perform ablations of D3, of zero-shot generated model from D3 as ( ZeroShot ) and the same model with optimized parameters ( ZeroOptim ).

DyNODE DyNODE is a neural network-based model that integrates control inputs into the neural ordinary differential equation (ODE) framework [Chen et al., 2018], as outlined by Alvarez et al. [2020]. Our implementation of DyNODE features a 3-layer Multilayer Perceptron (MLP) with hidden layers consisting of 128 units each, utilizing tanh activation functions. We initialize weights using the Xavier method [Kumar, 2017]. For consistency and competitiveness, we apply the same objective, optimizer, and hyperparameters as used in D3. Specifically, we utilize the Adam optimizer [Kingma and Ba, 2014] with a learning rate of 0.01, a batch size of 1,000, and early stopping with a patience of 20. The model is trained for 2,000 epochs to ensure convergence.

RNN Recurrent Neural Networks (RNNs) [Graves et al., 2007] serve as a standard benchmark for autoregressive time series next-step prediction. In our implementation, the input dataset is normalized according to the training dataset. The model consists of a Gated Recurrent Unit (GRU) RNN that maps the state-action dimension to a hidden dimension of 250 across two layers. This hidden representation is then passed through a linear layer to convert it back to the state dimension, enabling next-step prediction. For a fair comparison, we employ the same objective, optimizer, and hyperparameters used in D3. Specifically, we utilize the Adam optimizer [Kingma and Ba, 2014] with a learning rate of 0.01, a batch size of 1,000, and early stopping with patience of 20 epochs. The model is trained for 2,000 epochs to ensure convergence.

Causal Transformer The Causal Transformer represents a cutting-edge model designed for estimating counterfactual outcomes, as detailed by [Melnychuk et al., 2022]. Given its intricate structure, which includes three distinct transformer networks for processing covariates, past treatments, and past outcomes, we opted for a streamlined approach more suitable for our datasets and task domains. Specifically, we employed a single transformer to model past outcomes. This implementation utilizes a standard transformer encoder, where the input data is normalized to the training dataset. The state-action input dimensions are encoded into a 250-dimensional embedding vector via a linear layer, followed by the application of a standard positional encoder [Melnychuk et al., 2022]. This data is then processed through a transformer encoder layer with a head size of 10 and a dropout rate of 0.1. The output from this layer is passed through a linear layer to reconstruct the next state, matching the state dimension size. Training this model involves the AdamW optimizer [Kingma and Ba, 2014] with a learning rate of 0.00005, complemented by a step learning rate scheduler with a step size of 1.0 and gamma of 0.95. Gradient clipping is applied at 0.7, and the model is trained with a batch size of 1,000. Early stopping is used with patience of 20 epochs, and training is conducted for up to 2,000 epochs to ensure convergence.

SINDy Sparse Identification of Nonlinear Dynamics (SINDy) [Brunton et al., 2016] is a data-driven methodology designed to uncover the governing equations of a dynamical system directly from time-series data, resulting in a transparent, closed-form mathematical model. The SINDy algorithm operates by performing sparse regression iteratively on a library of candidate functions to find the most parsimonious and accurate representation of the system's dynamics.

In our approach, we employ a second-order polynomial library, L = { 1 , x 0 , x 1 , x 0 x 1 } , as the feature library. To calculate the time derivatives from the input time-series data, we use finite difference

approximations of first order. We maintain the alpha parameter at 0.5 for all experiments, with a sparsity threshold of 0.02, except for the COVID-19 dataset where it is set to 1 × 10 -5 .

Data-Driven Discovery (D3) We defer to Appendix F for implementation-specific details. Specifically, ZeroShot and ZeroOptim are ablations of D3, where these use the exact same setup, hyperparameters, and prompts. First, ZeroShot generates one model without optimizing it's parameters θ to the training dataset split, thereby assessing the model's output loss directly from the LLM. Second, ZeroOptim re-uses the same setup as ZeroShot , however now optimizes the parameters to the data, using the training setup described in Appendix F.2.

F D3 Implementation Details

The data-driven discovery method follows the framework as outlined in Section 3. We present pseudocode in Appendix F.1, how the code-generated models f θ are trained in Appendix F.2, prompt templates in Appendix F.4, system description prompts in Appendix F.5 for each dataset. Specifically, we find a top-K, where K = 16 is sufficient. Additionally, we use the LLM of GPT4-1106-Preview, with a temperature of 0.7.

F.1 D3 Pseudocode

Algorithm 1 Pseudocode for D3 Framework

F.2 Training Models

Upon generation by the Modelling LLM Agent, the model f is output as code, specifically a PyTorch [Paszke et al., 2017] neural network module. This code is then executed and the module is trained on the provided training dataset. The Modelling LLM Agent observes a code skeleton within the system description context c , examples of which are given in Appendix F.5. The code skeleton must be a 'torch.nn.Module' named 'StateDifferential', with initialized parameters and a forward function that computes the state differential using the state and action as input variables. The LLM completes, but does not alter, the skeleton, simplifying text processing, module execution, and model training.

The model is trained using the standard MSE loss function Equation (14), optimized with the Adam optimizer [Kingma and Ba, 2014], using a learning rate of 0.01, a batch size of 1,000, and early stopping with a patience of 20 epochs. Training proceeds for up to 2,000 epochs to ensure convergence and fair comparison.

After training, the validation MSE and per-component MSE (validation loss per state output dimension) are computed Equation (15). The trained model is appended back into s i with a string representation including initialized parameter values. Providing these optimized parameters aids the LLM in suggesting effective initial values in subsequent generations, which are then refined through further optimization.

Computational Efficiency . The computational efficiency of D3 overall arises from two sources: Using a Large Language Model (LLM) to generate the completions of the respective agents, and fitting the model, we now discuss each of these separately. First, using a capable enough LLM such as GPT4, for performing the completions for the agents can be computationally demanding, however, this computer costs with the computational cost per token of the underlying LLM that is used, and scales in the aspect of data with the number of tokens required to process, which we could envisage arises from having the LLM utilize a dataset with many features, each with their own textual description scaling the number of tokens in the input that the LLM has to process. One mitigation for this is to use LLMs that can handle larger context window sizes, and or be more selective about the input tokens fed into the LLM, such as restricting the number of in-context examples in the top-k examples, by reducing k, a hyperparameter. Second, we represent the models with parameters as a PyTorch module and use Pytorch to optimize the parameters, using a standard ML pipeline to train neural networks using stochastic gradient descent. Such approaches should scale with the number of input features, and parameters, however as with training any large eventual parameter model will scale as well as any other neural network-based approach in the same data pipeline. We note that more complicated distributed neural network training paradigms exist, however emphasize the focus of this work was on an initial framework, and leave the specific implementation for scale for future work.

F.3 System Description

Our method is initiated by an expert who provides a structured prompt that provides the system {system description} and {feature description} .

The system description provides a holistic description of the system, specifying the state variables x ( t ) and actions a ( t ) that are used to model the system. The variables are described semantically.

The feature description define features, names, units, and their ranges.

In addition to these components, the structured prompt includes a skeleton code program {skeleton program} to instruct the LLM to synthesize executable code in a pre-determined format. We refer to this prompt as the context c .

F.4 D3 Prompt Templates

For all prompt templates used, please see the code https://github.com/samholt/ DataDrivenDiscovery .

F.5 D3 System Description Prompts

By following our proposed system requirements format Appendix F.3 we constructed prompts for each of the datasets that we evaluated against, which are provided in the code, https://github. com/samholt/DataDrivenDiscovery .

G Additional Experiments

G.1 Warfarin Case Study Additional Results

Hybrid PKPD Warfarin Model with Neural Network Integration . This model incorporates a simplified pharmacokinetic-pharmacodynamic (PKPD) structure with additional neural network complexity, achieving a validation loss of 0.276. The model is defined as:

where F represents the bioavailability of the drug, V denotes the volume of distribution, k c is the clearance rate of warfarin, β a is the slope of the age effect on drug clearance, β s is the offset for the sex effect on drug clearance, MLP ( C, D, A, S ) is a multi-layer perceptron output, representing complex relationships in the data captured by the neural network, and R is a regularization term to prevent overfitting, with λ being the regularization weight. The addition of a neural network

allows the model to learn nonlinear relationships directly from data, providing improved predictive performance.

Existing Wafarin PK Model. Test MSE Loss: 0.6461 .

D3-white-box. Test MSE Loss: 0.39

D3-hybrid. Test MSE Loss: 0.271

G.2 Insight Experiment: Feature Acquisition Setup

For this experiment, we adapted the Lung Cancer (with Chemo. & Radio.) dataset simulator, to form a new variation of this dataset, where D3 starts with two features, which are the main state x ( t ) features that it intends to model, and can optionally acquire new features a ( t ) from a set of possible 22 features, which are listed below. Of importance, two features directly affect the underlying behaviour of this dynamical system, these being radiotherapy_dosage and chemotherapy_dosage , and the rest of the features we model as random white noise features, hence have no effect on the underling system, however D3 is unaware of this, and must use the feature description, from the features name to determine relative value of improved performance that it can attain by acquiring that feature, and propose new features to acquire along with their expected improvement.

Specifically, in this setup, we allow the feature acquistion agent to select from the following set of features at each iteration:

We also make the assumption that each feature that can be acquired has equal cost l of acquisition, and that the agent has an unlimited feature acquisition budget, however these settings can be readily changed depending on the specific application.

To provide comparable baselines, we compare D3 against ablations of itself, which is a random feature acquirer Feature Acquisition Agent, that selects a feature randomly for acquisition, and a null, or no Feature Acquisition Agent, that only observes at all iterations the target states x ( t ) to model. We observe experimentally that D3 that can leverage the LLM to quantify the value of information for features, where no training data exists for them in the dataset. We observe, as in Figure 2, that D3 converges the fastest, and achieves the overall highest performance, whereas the other feature acquisition methods fall short.

We provide part of the logs from one of the random seed runs, including part of it due to the logs being extensive.

G.3 Other Datasets Discovered White-Box Models

In the main paper we presented the best Warfarin discovered white-box model, however after running D3-white-box across all the other datasets, we include a few of the white-box models that it discovered and generated for each other respective dataset used that we compared against. Of particular interest, is that these white-box models fit the dataset well, and can even outperform some of the other neural network based baselines-we highlight that with the right functional form of the white-box model, we can achieve better performance than a largely over-parameterized neural network model, that has signficiantly more parameters (such as an RNN with 18,000 parameters), than a white-box model with 5-15 parameters.

Lung Cancer

H Future Work

The Data-Driven Discovery (D3) framework demonstrates initial success in generating interpretable pharmacokinetic (PK) models using an LLM-based iterative process. However, future work can expand and deepen the D3 approach in multiple directions:

Tree-Based and Graph-Based Generation Strategies: Extending D3 with tree-based generation strategies, such as Tree of Thought [Yao et al., 2024] or Graph of Thought [Besta et al., 2024], could provide a more structured exploration of model space. Such approaches could enhance D3's capacity to generate increasingly complex models over time, aligning with a growing need for adaptive modeling frameworks that account for intricate biological dynamics and drug interactions.

Robustness Through Retrieval-Augmented Generation: Incorporating RetrievalAugmented Generation (RAG) techniques could bolster D3's robustness by enabling ondemand access to external domain knowledge. RAG techniques [Lewis et al., 2020] can

serve as a mechanism to manage LLM hallucinations and ensure that model suggestions reflect state-of-the-art pharmacological principles. This capability would minimize reliance on the initial system descriptions and provide more flexible, context-driven model adjustments.

Adaptive Feature Acquisition: The current D3 implementation assumes that newly acquired features apply uniformly to all individuals in a dataset, which may not be feasible in practice due to data variability and differing acquisition costs across individuals. Future iterations could focus on more nuanced feature acquisition strategies that adapt to individual data constraints, potentially exploring patient-specific feature prioritization algorithms to reduce acquisition costs in clinical settings.

Enhanced Tool Use by LLM: While D3 utilizes GPT-4 for model discovery, integrating more advanced tool usage capabilities could further enhance its performance. Developing mechanisms for LLMs to independently call and evaluate external computational tools could extend D3's evaluation capacity, leading to richer and more rigorous model feedback, particularly for complex, multi-stage PK processes.

Extending Applications to Multi-Phase PK/PD Studies: Expanding D3's applicability to multi-phase PK/PD studies, such as those involving dynamic drug interaction effects or multi-compartmental absorption models, could broaden the impact of the framework. These applications would push D3 toward modeling scenarios with temporal and interventional complexity, where capturing delayed and interactive effects becomes paramount.

These advancements would not only strengthen D3's theoretical underpinnings but also enhance its practical applicability in clinical pharmacology, potentially contributing to improved drug efficacy and safety.

I Additional Sections

I.1 Cost Function Considerations

In our feature acquisition process, the cost function l ( h i ) plays a central role in determining the feasibility of acquiring new features. This function captures the computational, logistical, and practical costs associated with acquiring additional data for model refinement, thereby guiding the D3 framework to make efficient, data-driven decisions.

While our primary focus has been on computational costs and feature utility, it is important to recognize that feature acquisition often involves tradeoffs, particularly when biomarkers or other data types are costly, time-consuming, or invasive to obtain. We outline several key considerations and potential tradeoffs below:

Cost-Benefit Analysis of Biomarker Acquisition : In pharmacokinetic (PK) modeling, biomarkers can provide high-impact data that significantly improve model accuracy. However, some biomarkers (e.g., genetic markers, certain plasma proteins) are costly or require invasive procedures. Thus, the acquisition cost l ( h i ) should reflect not only monetary or computational expenses but also potential patient discomfort and clinical feasibility. This factor can be critical when evaluating the value of acquiring such features in real-world applications.

Marginal Utility and Redundancy Avoidance : Including a cost function allows D3 to weigh the marginal utility of new features against their acquisition cost, thereby avoiding redundancy. For instance, if a biomarker's expected contribution to model improvement is low, D3 can prioritize less costly or more readily available features, optimizing overall efficiency. This approach minimizes unnecessary data collection, thus reducing the risk of overfitting and keeping model complexity manageable.

Scalability of Feature Acquisition : In large-scale clinical studies, acquiring high-cost features across a population sample may be prohibitive. D3's cost function can be adjusted to prioritize scalable data collection strategies by favoring features with lower per-patient costs or those that require fewer resources to standardize across study participants. This balance enables broader, real-world applicability while still allowing D3 to leverage highvalue data where feasible.

Dynamic Cost Function Adaptation : As D3 iterates and learns from acquired features, the cost function l ( h i ) can dynamically adapt based on the observed impact of previous acquisitions. For example, if certain features consistently yield low model improvement relative to cost, the function can adjust to deprioritize similar features. Such dynamic adaptation helps D3 achieve a more nuanced balance between cost and utility as it iteratively refines the model.

These cost function considerations are integral to D3's feature acquisition approach, supporting a more balanced model refinement process that aligns predictive accuracy with practical feasibility. Future iterations of D3 could further incorporate adaptive mechanisms to fine-tune these tradeoffs, potentially optimizing feature acquisition across varying clinical and resource settings.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, in this paper we introduce the data-driven discovery (D3) framework, verify it experimentally against existing modeling ML methods, and verify the new novel discovered PK model on Wafarin with pharmacometrician experts. We detail the framework in Section 3, provide full implementation details in Appendix F, provide the experimental results to verify our core claims of discovering well-fitting models in Section 6, and our discovered Wafarin PK model as a case study in Section 6.1. We further discuss key assumptions of the method in Section 3 and outline important limitations in Section 7, with the 'limitations & future work section'.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have a dedicated limitations section in Section 7, titled 'limitations & future work section'. We also discuss the key assumptions of the method when we introduce it, such as in Section 3, for example in Section 3.3 we state the assumptions made, and how relevant they are in clinical practice, where D3 is designed to work. We also discuss the computational efficiency of D3 in Appendix F.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include any theoretical results, therefore this question is not applicable.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: To ensure thorough reproducibility, in addition to D3's description within the main paper (Section 3), we include full implementation details of D3 in Appendix F. To fully reproduce the experiments we detail the exact datasets, and how they were preprocessed or generated in Appendix B, how the respective evaluation metrics were computed in Appendix C, along with the corresponding full implementation details of the implemented benchmark baseline methods in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code is available at https://github.com/samholt/ DataDrivenDiscovery . Regarding each dataset, we provide full implementation details to either re-recreate the dataset or links to exactly where the dataset comes from, this is all detailed in the extensive Appendix B. To reproduce the experimental results we provide detailed implementations of D3 in Appendix F, the evaluation metrics in Appendix C and the baseline implementation details in Appendix E.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide this full information within the dataset implementation section in Appendix B, evaluation section Appendix C, and the training details for each method in the respective implementation section of Appendices E and F.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We accompany results with error bars in the main results, as shown in Section 6. Weexplain how these are computed, and that they performed over random seed runs, which is provided in the main experimental results section, Section 6, and expanded in the evaluation metrics section, Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide full details of the compute used in Appendix C, and further detail the LLM used in Appendix F and for the respective baselines in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, and we provide an explicit ethical consideration discussion section in Section 7.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Yes, and we provide an explicit broader impact discussion section in Section 7.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: It is not applicable, however, we attempt to cover such discussions in the 'Broader Impact and Ethical Considerations' section in Section 7.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes we provide full creditiation to the original creators of the synthetic datasets, and or provide the links directly alongside licenses for the public datasets used within the paper, and detail all of this in Appendix B.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release any new assets, therefore this is not applicable.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not perform any crowdsourcing, or research directly with human subjects, therefore this is not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This is not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper  presents the data discovery framework. It uses a LLM to iteratively discover and refine interpretable models of pharmacological dynamics. The D3 framework consists of three agents, which work collaboratively to propose, acquire, and integrate new features, validate models, and uncover new insights into pharmacokinetic processes. The framework is designed to address the limitations of traditional pharmacokinetic models that are often constrained by human expertise and scalability issues. The D3 framework was validated using a real pharmacokinetic dataset of Warfarin, demonstrating its ability to uncover a new, plausible pharmacokinetic model that outperforms existing models.\n\nSTRENGTHS:\nThis paper is interesting. It innovatively applies a LLM to generate an interpretable the skeleton of a dynamic system and optimize the system via tranining dataset. By progressively adding relevant variables, the authors ultimately obtaining a precise and interpretable closed-form ODE. The paper leverages the extensive knowledge and self-reflection capabilities of LLMs, and the writing is clear and the experiments are comprehensive. I have learned a great deal from this paper, particularly admiring the authors' adept use of LLM agent capabilities.\n\nWEAKNESSES:\nAll simulated datasets (and the ODE parameters) used in this study were publicly available before GPT-4's knowledge cutoff date. It is likely that OpenAI trained GPT-4 on relevant literature, leading to a significant knowledge leakage issue. I appreciate the author's perspective that LLMs encompass a vast amount of potentially usable knowledge. However, I do not believe that allowing LLMs to see the standard answers during training is an appropriate experimental setting. It will be better If the authors could demonstrate the feasibility of this method using real complex high-dimensional datasets whose dynamic is really unknown. After all, this is the kind of task faced in real pharmacokinetic studies. Meanwhile, the difference between this paper and many previous pharmacokinetic studies is that this paper does not conduct experiments on causal discovery. Instead, the model performance is evaluated through the MSE. It seems that this paper does not actually need to rely on simulated data and could use real high-dimensional time-series observational data for experiments.\n\nThe Table 3 did not report the performance of baseline models. I believe that comparing only the so-called \"standard model\" in Table 3 is inappropriate. Researchers in many other disciplines do not prioritize performance to the same extent as those in the machine learning community. They often trade off performance for model simplicity, valuing clear and concise formulas over marginal gains in performance. Thus, it is not very convincing that the authors achieved a performance advantage by using a method that resulted in a much more complex system compared to the standard model.\n\nThe baselines used in this paper are somewhat outdated. More recent studies published in the last two years, such as D-CODE and PGM (arxiv 2105.02522), could perform similar tasks as this paper. It is unclear why these were not included in the comparisons.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

comments: SUMMARY:
This paper presents the D3 Data Driven Discovery framework which uses GPT4-1106-Preview in a framework to iteratively consider modifying the features used in the ODE.  The dynamical systems are evaluated on MSEs of held-out state-action trajectories.

STRENGTHS:
The paper presents an innovative general strategy for searching the space of pharmacokinetic models in terms of which features to use.  The high level approach is accessible.  In their experiments, the D3-Hybrid (proposed) approach has the lowest MSE on 4 of 6 datasets.

WEAKNESSES:
The model space for the Modeling Agent could be better scoped---it appears to be whatever code the GPT-4 response is to the query.  What happens in event of failure?  Is this an automated framework?

Typically you want interpretable and performance pharmacokinetic models, so there is a tradeoff between validation performance (e.g. MSE) and simplicity (measure in some way, e.g. MDL, number of features, etc), which creates a frontier for each model class.  Could this be another way to compare the performance of these models?  An analogous question would be related to acquisition costs $l(h_i)$ as well.

QUESTIONS:
What happens when the Modeling Agent produces an invalid model? What models are possible?

LIMITATIONS:
There is a limitation section, though the limitations appear more to be feature requests, rather than considerations that users would need to be aware of in using or building upon the work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents the Data-Driven Discovery (D3) framework, a novel approach that iteratively discovers and refines interpretable pharmacological dynamical models using LLMs. This framework is novel and innovative in its domain, it is designed to address limitations in traditional pharmacokinetic (PK) modelling, which often relies on human expertise, and prior data in various formats. It is also constrained by existing knowledge. D3 leverages LLMs to propose and integrate new features, validate suggested models, and uncover new insights in PK. The framework demonstrates its effectiveness through experiments on various PK datasets, including a real-world Warfarin dataset, where it identified a new, well-fitting PK model outperforming existing literature.  The D#  framework utilises the following agents:\nModelling agent: Writes Python code for an AI model with the information the LLM has acquired    \nEvaluator agent: Evaluates the performance of the model by the modelling agent. The metric used is MSE (Mean Squared Error) \nFeature acquisition agent: Based on the evaluation of the model, more features are added to the model to improve accuracy (reduce MSE). A feature is acquired based on an estimation of the feature’s value using existing frameworks and available information such as text-based description of the feature, summary statistics and the feature’s existing data.\n\nSTRENGTHS:\nThe Data-Driven Discovery (D3) framework introduces a novel approach to pharmacokinetic modelling by leveraging Large Language Models (LLMs). This method appears to be highly original within its domain, representing a unique application of well-known techniques to pharmacology-specific challenges. The framework's ability to explore multiple models and incorporate unstructured data distinguishes it from traditional methods that rely heavily on human expertise and predefined knowledge bases.\n\nI would judge the quality of the work as high, with claims well supported by theory and experimentation. The paper provides detailed information on D3 implementation, prompts, metrics, and compares its performance against other methods. The results demonstrate that D3 is capable of identifying well-fitting models and providing valuable insights, such as in the Warfarin case study where it uncovered less-intuitive features, interactions and combinations of them.\n\nThe paper is clearly written and well-structured, with a logical flow of sections. It effectively uses tables, graphs, and equations to communicate information, although some text could be improved for readability. The methodology is explained with sufficient detail, covering how each component works.\n\nThe significance of the framework lies in its potential to accelerate pharmacological modelling by automating the discovery and refinement of interpretable models. While it still requires some human expertise, D3 reduces dependency on experts to a degree, assisting them in their work. Its ability to uncover new insights into pharmacokinetic processes could have important implications for optimising drug dosing and minimising adverse effects.\n\nIn terms of the novelty in AI, while agentic workflows are not new, the usability of D3 by non-AI specialists may be significant. The \"evolving\" nature of the system for non-AI domain specialists can be impactful, potentially bridging the gap between AI capabilities and pharmacological expertise.\n\nWEAKNESSES:\nIn general, the paper is inspiring and is solid in its experimentations and validations. There are a few remarks, some of them for future consideration, while others might improve the clarity of the paper:\n\n1. Cost function considerations: the current description of the cost function l(h_i)  might be too simplistic and not fully capture the complexity of acquiring new features. Cost functions are often difficult to quantify accurately. Adding a few sentences or a small subsection to discuss the aspects of the cost function and its potential impact and tradeoffs will provide a more comprehensive understanding. \n2. The D3 framework assumes that once a feature is selected, it is available for all individuals in the dataset. For example, certain biomarkers can be measured only in a small sample of patients. It would be interesting to discuss potential performance and limitations in light of missing data. Note: the authors briefly outlined it but elaboration on how to address it would be good given the commonality of this scenario. \n3. While authors called out the scalability and computational complexity of the framework, it would be interesting to see specific numbers around time to completion wrt the numbers of features. \n4. In line 212 it is mentioned that ‘The Evaluation Agent dynamically assesses both model performance and plausibility <…>’. Not sure if understood how plausibility is measured (couldn’t find it in the prompt in the Appendix either). Would be nice to see a bit more elaboration.\n5. Table 2 has mixed formatting: mixing scientific and regular notation: like 0.000245 and 2.47e+03 are both present. Unifying the notation will be more consistent. In addition, the variability of the results in terms of MSE is concerning. Might be useful to think of a more coherent metric or to provide an explanation for such extreme variability (e.g. 5780 vs. 7.07 for Lung Cancer or 719 vs. 0.3 for Lung Cancer with Chemo ).   \n6. The choice of LLM is quite significant and it is stated for the first time in Limitations on page 9. This could be brought earlier.\n\nQUESTIONS:\n* Why was MSE used as the metric for accuracy?  And on the same topic, in Table 2, what would be an interpretation of units in MSE? \n* Do you have ideas as to how to combat the limitations mentioned such as hallucinations? \n* Do you have a view on how to address the cost/ability of the LLM to distinguish between “easier” and “harder” features?  Such as: “don’t include biomarker x as it’s difficult to acquire from patient”? \n* Have you considered the inverse bias problem when models such as GPT-4 are restricted not to use biased features but it could be informative in the pharmaceutical environment to a PK model, such as ethnicity? \n* I'm still not fully clear how the LLM acquires the unstructured context/metadata about each feature.\n\nLIMITATIONS:\nOutlined in the previous sections. They are outlined and very helpful but some of them spark interesting questions and would like to see some of them elaborated on a bit more like missing data.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the Data-Driven Discovery (D3) framework, which leverages Large Language Models (LLMs) to iteratively discover and refine interpretable models of pharmacological dynamics. This approach enables the LLM to propose, acquire, and integrate new features, validate, and compare pharmacological dynamical systems models. The framework is demonstrated on a real pharmacokinetic datasets, highlighting its potential for clinical applications.\n\nSTRENGTHS:\nThe D3 framework’s ability to iteratively improve models and acquire new features enhances its performance and robustness. The writing is clear and well-structured, and the methodology is well-explained. Figures and tables effectively convey key information. The problem setup, where LLM agents act together for clinically relevant tasks, is interesting and potentially useful for developing better models the current baselines. The authors demonstrate the framework’s performance on a clinically relevant warfarin dataset and show effective results. Additionally, they validate the model by obtaining feedback from clinicians, which improves the robustness of the evaluation.\n\nWEAKNESSES:\n1. Limited Novelty: The paper has very limited novelty in terms of machine learning and does not propose any new tools. It effectively uses existing toolboxes for a clinically relevant task. While applying an existing toolbox in the context of a new task is perfectly fine, similar methods have been published before, particularly those related to automated science labs(self driving labs).\n2. Lack of Guardrails for Hallucinations: Models like GPT-4 are prone to hallucinations and can generate incorrect information. The framework does not appear to have guardrails to mitigate this risk, which is a serious concern given the clinical relevance of the task. While the authors have acknowledged this as a limitation, it remains a critical issue.\n3. Prompt Dependency: The model’s performance is highly dependent on the prompts provided. It is crucial to formulate the prompts correctly for the LLM to generate reasonable responses. This dependency suggests that the prompting framework might need adjustments for each dataset, leaving users vulnerable to the LLM’s unpredictability.\n4. Scalability and Computational Efficiency: The framework’s scalability and computational efficiency are not thoroughly discussed. Given the iterative nature of the model refinement process, it could be computationally intensive compared to other baselines and may not be practical for larger datasets without significant computational resources.\n5. Better performance of other baselines : The performance of the D3 framework might be dependent on the data. The paper shows better performance of models like RNN and transformers on tasks such as COVID-19 and Warfarin.  The paper does not any comment on why that might be the case.\n\nQUESTIONS:\n1. Given that models like GPT-4 can generate incorrect information, what measures have you considered or could implement to mitigate this risk, especially for clinically relevant tasks? I think implementing a RAG like model to address hallucinations will be highly relevant in this case and make the paper better. \n2. How do you ensure consistency and reliability in the prompts across different datasets? Have you considered strategies to standardize the prompting process to minimize variability and improve predictability? Did you experiment with prompts multiple times to arrive at a reasonable framework?\n3. Can you provide more information on the scalability and computational efficiency of your framework? How does the iterative model refinement process impact computational resources compared to other baselines, and what are the practical implications for larger datasets ?\n4. The paper shows that models like RNN and transformers perform better on tasks such as COVID-19 and Warfarin. Can you explain why this might be the case and what limitations the D3 framework has in these scenarios? How might the performance be improved?\n\nIf the authors address some of the concerns and reevaluate the frameworks with the clinicians, I will be happy to change the score. I get a sense from the expert comments that the datasets evaluated upon might not be challenging enough.\n\nLIMITATIONS:\nThe authors have acknowledged the limitations of their model. However, I think some of the points mentioned, such as retrieval-augmented generation, must be implemented in the framework to make the model more robust.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops an LLM-assisted equation discovery framework especially for pharmacokinetic process. Three agents of Modeling Agent, Evaluator Agent and Feature Acquisition Agent are built to explore, refine and iterate vast model space, including three levels of initial conditions, observed features and possible acquired features. Several experiments mainly on simulated benchmarks are conducted to evaluate the performance of the proposed framework. Overall, this paper investigated whether LLM agents can help effectively search and determine the model space for dynamics modeling and process discovery in pharmacology.\n\nSTRENGTHS:\n1. This paper investigates an interesting problem of LLM agents for pharmacological dynamical system discovery, which show some promising results of LLM for pharmacokinetic modeling. \n2. The feature acquisition could be a novel part, which leverages the knowledge capability of LLM and well matches with the application demand in pharmacokinetic modeling.\n3. The paper is well-organized, with clear sections and detailed explainations. The use of diagrams and examples, such as the iterative process involving the three agents, aids in understanding the complex interactions within the D3 framework.\n\nWEAKNESSES:\n1. The benchmark methods are not recent works, which might not well identify the effectiveness of the proposed framework. For example, the symbolic regression methods like D-CODE [ICLR'22] are not compared. \n2. To my understanding, the D3 model as well as other symbolic regression models discover equations from data-driven training, and then evaluate the performance based on the discovered equation. If I understand correctly, to me the experiments are not enough to prove the model effectiveness because of only five datasets with five equations. I would like to set several simulated datasets with different simulation equations for further investigation.\n3. There is no ablation study on model design or case study on different module choice, which makes it hard to evaluate the robustness of the proposed framework. Also, can LLM agents provide explanations on their output results to make a more transparent solution.\n\nQUESTIONS:\n1. How is the memory module implemented in this paper, and what exactly is the memory $s_i$? Are there several choices for memory and how would they affect the performance?\n2. How much is the cost for training a D3 framework on certain datasets, especially how much would it cost for calling GPT API? Can other open-sourced LLM be used for same tasks and how is their performance?\n3. RNN and Transformer models usually perform similarly in prediction tasks. However, in Table 2, RNN and Transformer perform differently on several datasets. For example, on Lung Cancer, RNN's MSE is 1.16e6, which I am wondering if the training converges successfully.\n\nLIMITATIONS:\nThe pharmacologists feedback statements highly depend on the personal knowledge and scope, as well as their understanding to the dataset or modeling. In my opinion, this part especially with only three experts are not worth for evaluation and should be excluded to avoid bias.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}