Data-Driven Discovery of Dynamical Systems in Pharmacology using Large Language Models
Authors: Samuel Holt
Zhaozhi Qian
Tennison Liu
Jim Weatherall
Mihaela van der Schaar
Venue: NeurIPS 2024 poster (2024)
Keywords: ODE Discovery
LLM Discovery
LLM
Primary area: machine_learning_for_healthcare

Abstract: The discovery of dynamical systems is crucial across a range of fields, including pharmacology, epidemiology, and physical sciences. *Accurate* and *interpretable* modeling of these systems is essential for understanding complex temporal processes, optimizing interventions, and minimizing adverse effects. In pharmacology, for example, precise modeling of drug dynamics is vital to maximize therapeutic efficacy while minimizing patient harm, as in chemotherapy. However, current models, often developed by human experts, are limited by high cost, lack of scalability, and restriction to existing human knowledge. In this paper, we present the **Data-Driven Discovery (D3)** framework, a novel approach leveraging Large Language Models (LLMs) to iteratively discover and refine interpretable models of dynamical systems, demonstrated here with pharmacological applications. Unlike traditional methods, D3 enables the LLM to propose, acquire, and integrate new features, validate, and compare dynamical systems models, uncovering new insights into pharmacokinetics. Experiments on a pharmacokinetic Warfarin dataset reveal that D3 identifies a new plausible model that is well-fitting, highlighting its potential for precision dosing in clinical applications.

Data-Driven Discovery of Dynamical Systems in Pharmacology using Large Language Models
Samuel Holt" University of Cambridge sih31 @ cam.acc.k
Zhaozhi Qian Elm UK zqian rqian@elm.sa
Tennison Liu University of Cambridge tl522 @ cam.ac.uk
James Weatherall AstraZeneca
Mihaela van der Schaar
University of Cambridge
mv472@cam.ac.uum
Abstract
The discovery of dynamical systems crucial across a range of fields, including pharmacology, epidemiology, and physical sciences. Accurate and interpretable modeling of these systems is essential for understanding complex temporal pro- cesses, optimizing interventions, and minimizing adverse effects. In pharmacology, for example, precise modeling of drug dynamics is vital to maximize therapeutic efficacy while minimizing patient harm, in chemotherapy. However, current models, often developed by human experts, are limited by high cost, lack of scal- ability, and restriction existing human knowledge. In this paper, present the Dala-Driven Discovery (D3) framework, a novel approach leveraging Large Language Models (LLMs) iteratively discover and refine interpretable models of dynamical systems, demonstrated here with pharmacological applications. Unlike traditional methods, 1D3 enables the I.LM propose, acquire, and integrate new features, validate, and compare dynamical systems models, uncovering new in- sights into pharmacokinetics. Experiments on a pharmacokinetic Warfarin dataset reveal that D3 identifies a new plausible model that is well-fitting, highlighting its potential for precision dosing in clinical applications.
1 Introduction
The discovery of dynamical systems models plays fundamental role across various domains, in- cluding pharmacology, epidemiology, and physical systems. In pharmacology. pharmacokinetic (PK) models are essential for understanding and predicting the time course of drug absorption, distribution, metabolism, and excretion in the body, which is crucial for optimizing therapeutic efficacy, minimiz- ing toxicity, and personalized treatment regimens diseases such cancer, cardiovascular disorders, and infectious diseases [Gabrielsson and Weiner, 2001, Hedaya, 2012]. For example, cardiovascular disorders requiring Warfarin therapy affect tens millions of individuals globally [Lee and Klein, 2013), highlighting the critical need for superior models to understand the dynamic impact of drugs and precision dosing, thereby assisting both doctors and patients.
The goal of PK modeling is to discover an underlying closed-form ordinary differential equation (ODE) f from a dataset of observed patient trajectories. This problem is of significant interest to the machine learning (ML) community, as evidenced by previous non-interpretable ML modeling approaches aimed at developing better PK models [Chan and van der Schaar, 2022]. Such datasets collected from expensive human clinical trials, necessitating the careful selection of which features
"qqual Contribution.
Work done Postdoc the University of Cambridge.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

to acquire and analyze during these trials [Guidance, 2010]. Pharmacomeetricans then leverage this data, using their existing knowledge to select appropriate pharmacokinetic models and employ standard statistical tools to infer the parameters of these models. For instance, they often fit single compartmental PK model, a closed-form ODE model with 3-5 parameters, to the data [Chen and Abuassba, 2021]. This process is iterative, refining the models until the most accurate one that generalizes well to new patient trajectories is identified. However, this traditional model discovery approach is often ad-hoc and fundamentally limited by the human expert's 's time, experience, and implicit assumptions about the underlying pharmacokinetic processes.
Discovering interpretable pharmacokinetic models is traditionally performed by human experts through a scientific process of proposing, collecting, and validating models. Recently, there has been a growing call for artificial intelligence-driveee
methods to enhance pharmacological modeling [Ryan et al., 2024, Singh et al., 2023, Cheng et al., 2022]. This established process involves iterative steps, each presenting distinct challenges in constructing accurate pharmacokinetic dynamics models.
Selecting appropriate temporal models involves complex decisions on model class, state and feature variables, parameters, and their relationships, far exceeding the simplicity of standard pharmacomet- rics models like single, double, triple compartmental models [Chen and Abuassba, 2021J. This complexity demands aligning model intricacy with data availability, where simple models suit small datasets and complex models fit larger ones. Acquiring the right features is crucial, as it must be done with limited prior information, unlike feature selection, which evaluates pre- pre-existing features [Li et al., 2017]. The evaluation must identify inadequate models and explain why, requiring diverse and time- intensive tools beyond a single metric like validation MSE. These challenges are interrelated: strategic data acquisition and thorough evaluation are essential for effective modeling. Despite advancements in tackling these challenges independently, temporal modeling remains manual, limiting speed and scalability, underscoring the need for integrated and automated systems to enhance efficiency.
To address these challenges, we develop the Data-Driven Discovery (D3) framework, leveraging Large Language Models (LLMs) to iteratively discover and refine interpretable dynamics models, directly relevant to pharmacology, epidemiology and ecology applications. D3 features three agents: Modeling. Feature Acquisition, and Evaluation, which collaborate iteratively. The Modeling Agent uses LLMs' code generation and natural language understanding to explore the model space by generating hypothetical models. The Data Acquisition Agent utilizes L.MS' zero-shot and few-shot learning to optimize data acquisition based on summary statistics, text descriptions, and prior knowl- edge. The Evaluation Agent conducts comprehensive evaluations by computing refined validation MSE metrics, and alignment with prior process understanding, providing precise feedback to improve the other agents' performance.
Contributions: ! We propose the Data-Driven Discovery (D3) framework, a novel approach leveraging Large Language Models (LLMs) to iteratively discover and refine interpretable dynamics models, advancing pharmacoknnetic modeling (Section 3). ② D3 overcomes the challenges of uncovering interpretable dynamical systems by using LLMs to explore vast model spaces and integrate unstructured data, producing models with few parameters that rival black box neural networks. D3 accurately discovers dynamics models across pharmacology, epidemiology, and ecology, matching the accuracy of existing methods while enhancing interpretability. It discovers new, more accurate PK model for Warfarin, validated by expert pharmacomeetricans. We also gain insight into D3's ability to iteratively improve its models, selectively acquire features, and discover precise dynamics models.
2 The vast model space for temporal modeling
In this section, we introduce the model space that D3 searches through, which contains a diverse range of temporal models with varying degrees of refinement. Our focus is on Ordinary Differential Equation (ODE) models, which is one of the most widely used methods for modeling temporal dynamics, which include pharmacokinetic, physiological and epidemiological dynamics [Auger et al., 2008]. We are interested in modeling the evolution of a set of state variables (t) E R for individuals n 1....,NN N over the time horizon E [O. The state variables represent the target variables that the human experts are interested in modeling: and there may exist other feature variables that can help predict the evolution of state variables. We assume we are provided with a
dataset of individuals trajectories such that D = ((x,,(), (!)) n I,....N N, L E [0, T)). where a, E R denotes the observed individual- models into three levels (R1- R3) based on their capacity for refinement.
features for individual. We can categorize ODE

---

R1: Refinement through initial conditions (0). This level involves the simplest form of ODEs, commonly referenced in scientific literature [Schiesser, 2014]. These equations are defined over state variables x(I) E RRD, incorporate global parameters 0, and involve function f' IR D+1 IRD The time spans the interval [0, T):
= ((x,),)))) Vn E [N]
(1)
Considering N individuals, each represented by the trajectory x, (1) equation above. According to the uniqueness theorem for initial value problems in ODEs [Lindelöf, 1894], any differences among individuals' trajectories can be attributed solely to variations in the initial conditions x,, (0), under common regularity conditions on f.
N, all follow the
Refinement through observed features a.. Suppose we have access to set of individual-level feature variables an. We can enhance the model's 's refinmment by incorporating these features into the ODE to enable unique dynamics for different individuals:
R2:
(1) = [xxx (1), 1, a., (1),
R3: Refinement through acquired features h Often, not all relevant variables are initially measured and available for analysis. In such cases, one may consider acquiring additional features (t) E R' to be integrated into the ODE. Here (t) is collection of features from the of all acquirable features H. where J < [7]. This third level of refinement reflects one data acquisition challenge, i.e., identifying and collecting informative features h, (t):
xx (t) (t), t, a. (f), (t),
After selecting the level of refinement (R1-R3), it is necessary to determine the functional form of the ODE f. which defines the interactions among various variables and parameters that govern the dynamics. There are two dominant approaches parameterize f: using concise closed-form white-box equation f E MM or employing a neural network ʃ E _MN. The former approach is prevalent in the field of symbolic regression [Billard and Diday, 2002], while the latter is utilized in Neural ODEs [Chen et al., 2018]. Additionally, f can incorporate both closed-form component and a neural component, resulting in a hybrid ODE model. Nevertheless, all three approaches involve searching within large combinatorial spaces close---ood---oom equations, neural architectures, and both presentrenn significant computational challenges.
3 Data-Driven Discovery (D3)
We aim to uncover the true underlying pharmacokinetic dynamical system, denoted as f*, Our goal to continually approximate f* as accurately as possible using proposed models fo. with parameters 0. For simplicity, we will refer to these models as ff in all subsequent references, omitting the 0 term. To appropriately identify the refinement model level needed from the vast model space discussed above, we now introduce the Data-Driven Discovery (D3) framework, as depicted in Figure 1. This consists of three LLM agents: the Modeling Agent G, the Feature Acquisition Agent A, and the Evaluation Agent E. These three agents work together to form the model improvement loop and the data acquisition loop. The implementation details of the Agents are provided in Appendix F.
3.1 Inputs to Data-Driven Discovery (D3)
To begin the process, D3 requires clearly defined description of the system in natural language, that details the modeling task, of creating a well-fitting model of either a white-box model hybrid model; that is a white-box model with a black-box neural network fitted to the residuals of the white-box model. D3 supports both discovering only white-box models and hybrid models, and we leave this choice up to the end user to decide for their application. Specifically, a user must provide a dataset of individual's trajectories D, and system deseription and the names and descriptions of any features that exist within the dataset, which can include the ranges of those features. Prior information can be added here by the practitioner into the system description if it is available. We now discuss how the three agents of Modeling, Feature Acquisition and Evaluation interact to discover iteratively better-fitting/ f models. We provide full implementation details for the framework in Appendix F,
3.2 Modeling Agent G
D3's Modeling Agent G is tasked to iteratively propose and refine the temporal model Ses where i I,..., I indicates the iteration. We design G following the three principles below.
3

---

Iteration
Iteration
Iteraiion 3
Ieration 20
Description
Mode2
Sescrigtice
Sypiium Oescraption Te
Sporiam Duscription To
System Description
Mode))
Memeoniit by
Evaluator
Evaluator
Mesicas Eape
Presioa top-k
(Cote)
25.3
Modeling
(Gane) 15..
sedads,
3: (Co0)): Less 8.55
(G088)
Lons
Now lerfact Verbilly
Code Peatures
LLM Resp
Now Poffact seibally
Le flacttoe
Resp.
Best discovered model is returned
Evaluator
Reffaction
Us: AGE Loss: 26.3
Papenerate Model
cature Acq.
Regororate Focea
E3X copuino foatares
Code Featurea
de dy
Val. NEE
Val MSE Look: 1.3
LLM Resp.
of Jusiie
of Butheiae
Acqudre:
Angutret
aequtctiionn
Aequsired
Acsidred
Figure I: Data- Meta-Driven Discovery (D3) Overview. Given a dataset of trajectories of individuals D, D3 can discover a well-fitting model f. that is either white-box model or a hybrid model, combining a white-box model with a neural network component fit to the residuals. First, a user provides a system description c for the model, which includes the feature names, their units and ranges. The Modeling agent uses the textual description and generates a model, represented as code. Next, the Evaluator agent evaluates the generated model on held-out validation dataset and provides a loss metric. Next, the Feature Acquisition agent decides if it wants to acquire new feature, that could aid in generating better models. This iterative process repeats, when there exists one or more previously generated models, the Evaluator agent provides a verbal reflection on how the model can be improved, which is used by the Modeling agent at subsequent iterations. This process repeats, discovering better fitting models, and after a set number of iterations, the best fitting model is returned, as code with its parameters optimized to the given training subset of the given dataset D.
Representing the model f. as code. As we discussed in Section I, one key challenge in evidence- based refinement is the vast space of possible ODE models, ranging from closed-form ODEs to complex neural Neural ODEs and hybrid models. As prerequisite to finding the most suitable model, we need a consistent and flexible way to represent these diverse models such that they can be trained and validated on data. As such, D3 represents the model f. as code in Python language. Specifically, each model defined as a class derived from 'nn.Module' PyTorch [Paszke et al., 2017] that specifies the free parameters and the computation logic. The Modeling Agent G leverages an LLM to generate the model code Se in each step.
Informing generation with context Effectively searching through the vast model space presents another challenge. Thankfully, in many applications there exists rich context e about prior works and domain knowledge. However, this rich source of information was previously locked in unstructured documents and texts, making it hard for automated algorithms to make use of it. By leveraging the LLM's capability to understand unstructured documents, the Modeling Agent G' can now access and leverage this information, therefore proposing more informed models and searching more effectively. In this work, we consider the provision of the following categories of problem context of {system description: and {feature description) (Appendix F).
Iterative model improvements based on rich verbal feedback r,-1 and memory 8;-1- To enhance the quality of the generated model fi, we have designed the agent G to iteratively improve the model based on the presence of any evaluation verbal feedback and its knowledge of the previously generated models and how they performed on validation subset of given dataset D of trajectories. At each iteration i, the Evaluation Agent E evaluates the previous model fi-1 using the validation data subset of D, and provides rich verbal feedback r;-1 (c.f. Figure 1), which forms the foundation for the next generated model f.. The Modelling Agent G utilizes memory 8;-1 to track the top-k best-performing models so far, along with their associated evaluation feedback, here their respective losses. Further details on evaluation and data acquisition are discussed in the following section. This iterative process allows for continuous refinement and optimization of models based on their past performance and targeted feedback.

---

In summary, the Modeling Agent G encapsulates a comprehensive range of model structures through computer code f; and efficiently explores the vast model space by integrating unstructured prior knowledge and utilizing data-informed feedback r along with memory 8;-1- The operation of G can be mathematically represented follows:
fi = G(c, P
3.3 Feature Acquisition Agent A
D3's Feature Acquisition Agent A is tasked with iteratively proposing new features for acquisition. These features are selected based on their potential to improve the next model's performance in modeling the target state (t).
(5) Where h; is the next feature acquire. Specifically, when a feature is acquired we make the assumption, that it is acquired for all individuals, therefore at the next iteration the dimension of the individual-level features a (t) R
h4 = A(c, fi. Dj, Fi-1
I....,N N. increases by one, to (t) RK+1 n N. It is possible to do this in clinical trials, where pharmacokinetic datasets of patients are 1,...,N. collected [Dziura et al., 2013], for example acquiring the individual's ages, other lab biomarker measurements or tests [Derraik et al., 2021].
Estimating the value of the feature. We leverage the value of information [Feltham, 1968] frame- work to provide a principled way to determine which feature, if any to acquire next. This follows
V(h.) = E[L(f, D;-1)] - [[ D,-1)] -((h),
where the first two terms capture the improvements in validation loss L when model fi is trained on the new dataset that includes the new feature, that is D = {(x,, (t), at, h, (t)) E [N]] and I(h,) represents the cost for acquiring h; Estimating V(h,)) presents statistical challenge and computational challenge. Firstly, in practice, we often do not have access to h2 to begin with (as the goal is to acquire new features)- henne we need way to estimate E[L h2, D;-1)] without fully accessing ht. Secondly, even when h; is available, computing the two loss terms involves fitting models twice which can be computationally challenging.
To address these challenges, the Feature Acquisition Agent A predicts the value of feature V(h,) from the available information about h, follows:
(h.) = g(cn, ((h),,
where Ch. represents the unstructured prior information about the feature hę (e.g. data descriptions and metadata), Thhh)) represents the available summary statistics (e.g. range and type of values from a small subset of data h;)) D represents the existing data, and g is the prediction function.
The prediction challenge above corresponds to zero-shot or few-shot learning in ML. In the zero-shot case, need to predict V (h,,) without having access any "label" V V(h), for some feature h. In the few-shot case, we have access to a small number of feature-label pairs, {Ch,: T(h;), h (hh-)))) to learn the prediction function g. The Feature Acquisition Agent A leverages LLM''s capability in zero-shot and few-shot learning to address this estimation challenge. Specifically, we inform the LLM by providing high-level statisties T(h:) and a description of the feature Cs,, in the form of the feature name to warm-start the zero-shot learning. When the hú has been included in the data, we inform the LLM about the validation loss associated with those data vh::) and add it the few-shot examples to guide the estimation.
Context, feedback and memory. Similar to G. A is also able to leverage unstructured problem context c, evaluation feedback r;-1. and memory 8;-1 to guide the data acquisition proposal.
3.4
The Evaluation Agent E
The Evaluation Agent E is vital for iteratively improving the Modeling Agent G and Feature Acquisition Agent A. It provides feedback r;:
(8) Feedback r; can be numeric, such as validation loss or mean squared error (MSE) of the model fa on the validation subset D., more refined metrics like validation loss per target dimension in D. It can
Ti = Elc, D.)
5

---

also be textual, offering detailed suggestions for model improvements, including clinically plausible modifications generated by LLM [Shinn et al., 2024].
Evaluating dynamical systems is complex due to intricate interactions and temporal dependencies. The Evaluation Agent dynamically assesses both model performance and plausibility, ensuring models are both accurate and interpretable. This comprehensive evaluation is crucial for developing sophisticated, clinically relevant pharmacokinetic models that address real-world complexities.
4 Related Works
Table 1:
Comparison with related works in addressing the Modeling and Data Acquisition Challenges. An empty field means not applicable. Refinement Level: the level of refinement (RI RR3) the method can capture. Class of f: the class of ODE model f (closed-form Mc, neural networks M,). Context whether the method leverages unstructured contextual information guide search. Sample N: whether the method is able to acquire new samples. Feature h: whether the method can acquire new feature variables. Goal: whether the goal of data acquisition is to improve training, evaluation, or both.
Modeling Challenge Class f Context c
Data Acquisition Challenge
Method
Refinement Level
Zeroshot
Feature h Goal
Symbolic Reg. Neural ODE
RI, R2
Mc
X
R1, R2
MN
AI Feynman
R1
Mc
Eureka
RI, R2
Mc
V
AFA
Train
Active Learning
X
Train
LXX
Active Testing
Eval
D3
RI R3
Mc U.
Both
Our work focuses autonomously learning temporal models while acquiring data, with several relevant research strands, as summarized in Table 1, which is expanded in Appendix A.
ODE learning methods. Symbolic regression methods like SINDY and D-CODE [Koza, 1994, Brunton et al., 2016, Qian et al., 2022] can discover closed-form ODEs f MMc using genetic algorithms. They can incorporate features a, (R2 refinement) but struggle with many variables (e.g.. >20) due to computational complexity. Neural ODEs use neural networks f E MN to handle many variables [Chen et al., 2018, Dupont et al., 2019, Zaytar and El Amrani, 2016, Devlin et al., 2018, Schovac and Grolinger, 2020], but have numerous free parameters, risking overfitting with insufficient data. D3 captures f E MMc U My and supports R1 R3 refinement, adapting based on data availability. Both Symbolic regression and Neural ODEs require manual context incorporation c via hyperparameters and do not address Data Acquisition Challenges.
AT for automated modeling. AI Feynman [Udrescu and Tegmark, 2020] enhances symbolic regres- sion by using the physical units of variables, which provide additional constraints to narrow the search space. However, its utility outside physics is limited, as units in other fields carry less information. Eureqa [Ma et al., 2023] applies LLMs to model the reward function in reinforcement learning, integrating unstructured context and representing models as code. Unlike D3, Eureqa canmot auto- matically refine models, acquire new data, casily apply to learning temporal dynamics like ODEs.
Data Acquisition. Methods like Active Feature Acquisition (AFA) [Ma et al., 2018, Gong et al., 2019] measure additional features on existing samples to improve performance. Active Learning (Sebastiani and Wynn, 2000, Settles, 2009, Sener and Savarese, 2017, Imberg al., 2020] and Active Testing [Lowell et al., 2018, Kossen et al., 2021] acquire new samples to increase training or evaluation data size while keeping features constant. These methods do not address the zero-shot setting, where data acquisition decisions must be made before observing any target data (Equation 7). Additionally, most Active Learning methods focus on acquiring labels for supervised learning, which is not directly applicable to temporal modeling tasks.
5 Experiments and Evaluation
In this section, we demonstrate that D3 can discover well-fitting pharmacokinetic dynamical system models for range of diverse PK datasets, including an epidemiological and ecological dataset.
Benchmark Datasets. Our evaluation encompasses six real-world datasets with clinical relevance, each originating from either real-world data or highly accurate simulators developed by human experts.

---

Three datasets are based on state-of-the-ee biomedical Satpaaccckineii--Paaacooooo model of lung cancer tumor growth, which simulates the combined effects of chemotherapy and radiotherapy in lung cancer [Geng et al., 2017] (Equation (11)). This model has been widely utilized in previous research [Bica et al., 2020, Seedat al., 2022, Melnychuk et al., 2022]. Specifically, we employ this tio-mathematical model to generate three variations: lung cancer without treatment (Lung Cancer), lung cancer treated with chemotherapy (Lung Cancer (with Chemo.)), and lung cancer treated with both chemotherapy and radiotherapy (Lung Cancer (with Chemo. & Radio.)). Additionally, we utilize an intricate COVID-19 epidemic agent-based simulator (COVID-19) [Kerr et al., 2021]. to provide an epidemiological dataset. Another dataset comes from an ecological model simulating a microcosm of algae, flagellate, and rotifer populations (Plankton Microcosm), replicating a three-species prey-pus mey--redator experimental system [Hiltumen et al., 2013]. Finally, include a real Pharmacokinetic (PK) dataset of Warfarin patients (Warfarin) [Janssen et al., 2022]. Detailed information about all benchmark datasets is provided in Appendix B.
(PKPD)
Benchmark Methods. To evaluate the performance of D3 and establish its competitive performance, we conduct comparisons with leading modeling methods for ODEs. Specifically, we benchmark against advanced black-box models that have many parameters, such as neural ODEs with action inputs, known as DyNODE [Chen et al., 2018, Alvarez et al., 2020J. Also in this class are recurrent neural networks (RNN) and a state-of-the include white-box transparent dynamical systems models identified through equation discovery techniques, such as Sparse Identification of Nonlinear Dynamics (SINDy) [Brunton et al., 2016]. Moreover, D3 supports two discovery modes, of which we compare against both, discovering only white-box models (D3-white-box) and discovering hybrid models (D3-hybrid)'. We also perform ablations of D3, of zero-shot generated model from D3 (ZeroShot) and the same model with optimized parameters (ZeroOptim). Detailed descriptions of the implementations, hyperparameters, and experimental procedures for these benchmarks can be found in Appendix E.
transformer model (Transformer). In addition, we
Evaluation Metrics. To assess the performance of our benchmark methods, we use the mean squared error (MSE) on a held-out test dataset of state-action trajectories. This evaluation is conducted over ten runs, each initialized with different random seeds. We report the average MSE from these runs along with their 95% confidence intervals. Further details can be found in Appendix C.
6 Main Results
We conducted a comprehensive evaluation of our benchmark methods across all datasets, as tabulated in Table 2. We observe that D3 can discover well-fitting dynamical system models, achieving low mean squared error in test predictions on the held-out test dataset of individual trajectories. Crucially, it can discover concise closed-form equation white-box models, of 5-15 parameters that can outperform some of the standard white-box and black-box modeling methods, indicating it is discovering well-fitting underlying equations for the respective systems of interest.
Table 2: Evaluating Method Performance. We report the test prediction mean squared error (MSE) on held-out datasets across benchmarks. D3 consistently has the lowest error. Results are averaged over ten random seeds with 95% confidence intervals.
Lung Cuncer MSE
Ling Cancer (wis Chemo.) MSE
Cancer (with Chemo, Rudio.)
Plankcon Microcoom
COVID-
Wartarin PK MSE
Method
DyNone
326- 95
$5. 2--2288
16:2-6.35
0000997-00000 000883
24-2,69
0.716-0.17
SINDy
$25 95
11 11,310,442
13.7:00.35
0U60135100
03.5200500
6,542:
ZeroShoe
3.786+113.7766 6e+00
2042.8611
0,333110227 274
2.47p-03::2222
ZeroOppm
225
31881508
6,28 6,28
001131100013 00113
7,8 7,58
390 5 (0be+1)
KNN
1. 18e-0hh2 5.2 1e-04
719.1943
137.555 89
0003050000 0459
1. 47
I10499 11995-10000 9.8406
Transfomer
2.07:0.555
0.346:0.0700
11007-000318
3.42-0551117700
0.261:000915
1.33
D5-white-bon box
59.4-1 101
488111
2.42-2.02
0000005500000022
5921:117
196624405
1.8-hyhrid
472 9.16
0..073.0.0463
0.138 00222
1.80e-06111887--
1.8813.57
0647200.167
6.1
Case Study: Discovering Novel PK Models for Warfarin
PK models of warfarin, a widely used anticoagulant, are crucial due to its extensive use in treating deep vein thrombosis, pulmonary embolism, and stroke prevention in atrial fibrillation and patients with mechanical heart valves, with millions of prescriptions globally [Lee and Klein, 2013]. We applied D3 to public dataset of 33 patient trajectories [Janssen al., 2022], discovering novel PK model for warfarin that outperforms existing literature (Table 3). We compare the test MSE of the
*Code is available at https: //ggthht..bthttt//e/ cacchuu..omm/seee hawm//ammott//eiei Satmmooot///ataarrrooiiiiserrrr and we provide broader research group code base https: :// htt///gttttu/tt cotp//aaddeeehaarraaaaaaaaarrvaa
7

---

discovered model against existing models and provide pharmacologist feedback. Full model details and results from running D3-hybrid on the same dataset are in Appendix G.
Existing Wafarin PK Model. The standard pharmacokinetic (PK) warfarin model from the literature [Lv et al., 2017, Hamberg, 2013] is the following:
Table 3: Warfarin Modeling Comparison
Method
Warfarin Best Model Test MSE
Existing Warfarin PK
0.646
D3-white-box
0.39
dC dt
kin D lig C₁
D3-hybrid
0.271
(9)
he base + kic.age" A + ke,sox 5
where C the concentration of warfarin, D is the dosage administered, A represents the age of the patient, S denotes the sex of the patient (1 for male, 0 for female), ka is the absorption rate, ke, base is the base elimination rate, Ke.age represents the decrease in elimination rate per year increase in age. and ke,sex is the difference in elimination rate between sexes. This standard model achieves a test loss of 0.646. Whereas D3 can discover both a white-box model with a test loss of 0.39 and hybrid model with test loss of 0.271, which both outperform the standard model. Of particular relevance is the white-box model as it is fully interpretable by pharmacomettrcians, whilst still being a precise model.
New Discovered PK Warfarin Model. D3-white-box discovered a new warfarin PK white-box model with test loss of 0.39, of the following:
dC VD kioft
dt
Km + +C'
hiel = ke,base + ke,uge (A - A) + keesex -5
(10)
+ Kideaay C'+ kás D 155 5)
+ kias A)- (S (0S-5))
5) + kind D D-(A- (A-A)
where the additional parameters Addeay represent the natural decay rate of warfarin concentration and Km, is the Michaelis constant indicating the warfarin concentration at which the metabolism rate is half its maximum. The parameters kas, Kas, and kda are the interaction terms for dosage-sex, age-sex, and dosage-age, respectively, and A and are the sample population means for the covariates of age and sex. This new PK model includes the original parameters along with additional interaction terms and transformations, enhancing the model's 's complexity and accuracy.
The model discovered by D3 introduces key innovations in warfarin pharmacokinetics. Square Root Transformation for Dosage Effect: This non-linear transformation moderates high doses, enhancing sensitivity to dosage variations. unlike standard models. Natural Decay Term: Proportional to the current warfarin concentration, it better reflects elimination kinetics and improves time-based predictions. Michaelis-Menten Saturation Kinetics: Addresses metabolic pathway saturation at higher concentrations, crucial for warfarin. Interaction Terms (dosage-sex, age-sex, dosage-age); Capture complex interactions between patient-specifie factors, providing nuanced understanding of drug dynamics. These advancements make D3 a powerful tool for enhancing pharmacokinetic predictions in clinical settings. The discovered hybrid model is detailed in Appendix G.
Expert Clinical Commentary. We sought feedback from expert pharmacologists on the discovered model Prof. Eoin McKinney, Clinician. "This model is significant, as consortiums are dedicated to improving Warfarin modeling [Consortium, 2009]. The model adds novel components, such as the Michaelis component for time-varying changes and novel interaction terms like age-sex." Jean- Baptiste Woillard, Pharmacologist. "The model promising and pharmacokinetically plausible. The next step is to apply D3 to other clinically relevant PK drug datasets." Richard Peck, Clinical Pharmacologist. "This model is reasonable and potentially superior. represents a significant advance in clinical pharmacology by automatically identifying robust PK models."
6.2 Insight Experiments
This section provides in-depth analysis of D3's effectiveness related to its benchmark counterparts.
Can D3 perform feature acquisition and leverage the LLM prior information to perform this better? To explore the feature acquisition performance of D3, we showcase this component working
WWe provide full commentary in Appendix D.
8

---

10
Validation MSE
101
D3-andou--squusiie
D3 wwwffrrttrroooooro nntorn--ooeiiveaariiiiiii
100
10-1
0
3
6
9
12
15
18
Number Generations
Figure 2: Feature Acquisition. D3 performing adaptive feature acquisition in the Lung Cancer (with
Chemo. & Radio.) dataset. We observe that D3 still achieves the lowest test prediction error.
Top-1 DJ Model
Validation MSE
10
Decrlekee lybrid model
Genpura good
usi parti an grodch chene
materre trac hes MLF comple
greeis mahkrn Meick hockl
10
Docrigdax:
Corgeraii preats dhveehhogg
Frstars ulisted tn vallass
Penie net iibit growth giwal allsss
senaaae grond Fig and mery
voliat
Fratere whenth 3 tamer achiteryy
checaetoees precminntin
10-3
raceduuney, Arags poucic nartico.
1
2
3
4
5
6
7
8
10
Number of Generations
Figure 3: D3 iteratively discovers better models fi. Validation MSE of the model generated in each iteration, showing the best-generated model (Top-1 additioalll with few of the models labeled with their model descriptions, and features acquired the generation. D3 can effectively acquire and integrate new features, validate, and compare models to achieve a better-fitting model.
in full, by comparing it against the baseline of a random feature acquirer policy, a null policy that just uses the existing features and our approach of D3 that leverages the LLM to quantify the value of information for features, where no training data exists for them in the dataset. We observe, as in Figure 2, that D3 converges the fastest, and achieves the overall highest performance, whereas the other feature acquisition methods fall short. Experimental details are in Appendix G.2.
Can D3 evolve its modular model fit the system best? We analyze this from empirical point of view to determine if D3 can correctly evolve the generated model and reduce its prediction error over subsequent generations. We observe that D3 can indeed understand, reason, and iterate the generated code representation of the model to incorporate a better fitting model, as observed in Figure 3.
7 Conclusion
Summary. In this paper, proposed Maia--riven Discovery (D3) framework, iteratively discover and refine interpretable models of pharmacological dynamics, where it has the ability to acquire and integrate new features, validate, and compare pharmacological dynamical systems models, thereby uncovering new insights into pharmacokinetic and physiological processes. Specifically, applied to real Warfarin PK dataset provide new insights into what an optimal PK model for Warfarin may need to possess, such as additional Michaelis components.
able to discover a better performing Warfarin PK model and
Limitations & Future work. There exist limitations to the current approach. First, the LLM discovery framework is initial framework, and the utility of the L.M to generate better models could be enhanced, tree-based generation strategies such as the tree of thought [Yao et al., 2024], graph of thought [Besta et al., 2024]. Specifically, it relies on capable enough LLM, that can use tools, where we specifically used GPT4 as the underlying LLM. Moreover, we make the assumption that when D3 acquires new feature it acquires that feature for all the individuals within the existing dataset, we leave for future work to consider applications where this assumption may not always be true. Furthermore, could explore avoiding the user to provide this, and instead leveraging a form of automatic retrieval augmented generation [Lewis al., 2020].
provide to the LLM system description from user, future works
Broader Impact and Ethical Considerations. Principally D3 aims to discover interpretable models of pharmacological dynamics. However, the final discovered models should always be checked

---

by appropriate human experts and validated in additional held-out datasets before any clinical use. Furthermore, D3 is tool, which could be misused by malicious user with unethical system descriptions as input to discover a potentially biased model. Moreover, D3 leverages L.MSs and their feedback as an integral component, however, LLMs are prone to hallucinations, thereby motivating any L.L.-geeerated outputs that are shown to the user should have a content filter applied to them.
Acknowledgments and Disclosure of Funding
We extend our gratitude the anonymous reviewers, area and program chairs, members of the van der Schaar lab, and Andrew Rashbass for their valuable feedback and suggestions. SH gratefully acknowledges the sponsorship and support of AstraZeneca. ZQ did this work whilst a postdoc in the van der Schaar lab. This work was funded by Microsoft's ' Accelerate Foundation Models Academic Research initiative.
References
Victor M Martinez Alvarez, Rareş Rosca, and Cristian G Fälcujescu. Dynode: Neural ordinary differ- ential equations for dynamics modeling in continuous control. arXiv preprint arXiv:2009.042: 042788 2020.
Pierre Auger, Pierre Magal, and Shigui Ruan. Structured population models in biology and epidemi- ology, volume 1936. Springer, 2008
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17682-17690, 2024.
Ioana Bica, Ahmed M. Alaa, James Jordon, and Mihacla van der Schaar. Estimating counterfactual treatment outcomes over time through adversarially balanced representations. In International Conference on Learning Representations, 2020.
Lynne Billard and Edwin Diday. Symbolic regression analysis. Classification, clustering, and data analysis: recent advances and applications, pages 281-288. Springer, 2002.
Willem Bonnaffé and Tim Coulson. Fast fitting of neural ordinary differential equations by bayesian neural gradient matching to infer ecological interactions from time-series data. Methods in Ecology and Evolution, (4(6))1543-155 2023.
Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3932-3337 2016.
Alex Chan and Mihaela van der Schaar. Synthetic model combination: An instance-wist approach to unsupervised ensemble learning. Advances in Neural Information Processing Systems, 35: 27797-27809, 2022.
Bing Chen and Adnan OM Abuassba. Compartmental models with application pharmacokinetics. Procedia Computer Science, 187:60-70, 2021.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Limei Cheng, Yuchi Qiu, Brian Schmidt, and Guo-Wer Wei. Review of applications and challenges of quantitative systems pharmacology modeling and machine learning for heart failure. Journal of pharmacokinetics and pharmacodynamics, pages 1-12, 2022.
International Warfarin Pharmacogenetics Consortium. Estimation of the warfarin dose with clinical and pharmacogenetic data. New England Journal of Medicine, 360(8):753-764 2009.
José GB Derraik, Wason Parklak, Benjamin B Albert, Kongsak Boonyapranai, and Kittipan Rerkasem. Fundamentals of data collection in clinical studies: simple steps to avoid "garbage in, garbage out". The International Journal of Lower Extremity Wounds, 108 2021.
10

---

Jacob Devlin, Ming-
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. Advances in neural information processing systems, 32, 2019.
James D Dziura, Lori A Post, Qing Zhao, Zhixuan Fu, and Peter Peduzzi. Strategies for dealing with missing data in clinical trials: from design to analysis. The Yale journal of biology and medicine, 86(3):343, 2013.
Gerald A Feltham. The value information. The accounting review, 43(4):684 696, 1968.
Johan Gabrielsson and Daniel Weiner. Pharmacokinetic and pharmacodynamic data analysis: con- cepts and applications. 2001.
Changran Geng, Harald Paganetti, and Clemens Grassberger. Prediction of Treatment Response for Combined Chemo- and Radiation Therapy for Non-Small Cell Lung Cancer Patients Using a Bio-Mathematical Model. Scientific Reports, 7(1):13542, October 2017. ISSN 2045-2322. doi:
10.1038/s41598-01 017-1666-7-6
Wenbo Gong, Sehastian Tschiatschek, Sehastian Nowozin, Richard E Turner, José Miguel Hernández- Lobato, and Cheng Zhang. Icebreaker: Element- wise efficient information acquisition with bayesian deep latent gaussian model. Advances neural information processing systems, 32, 2019.
Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. Multi-dimensional recurrent neural networks. In International conference on artificial neural networks, pages 549-558. Springer, 2007.
FDA Draft Guidance. Adaptive design clinical trials for drugs and biologics. Biotechnol Law Rep, 29 (2):173, 2010.
Anna-Karin Hamberg. Pharmacometric models for individualisation of warfarin in adults and children. PhD thesis, Acta Universitatis Upsaliensis, 2013.
Mohsen A Hedaya. Basic pharmacokinetics. CRC Press, 2012.
Teppo Hiltunen, LE Jones, SP Ellner, and Nelson G Hairston Jr. Temporal dynamics of a simple community with intraguild predation: experimental test. Ecology, 94(4):773-779, 2013.
Samuel Holt, Alihan Hüyük, Zhaozhi Qian, Hao Sun, and Mihaela van der Schaar. Neural laplace control for continuous- delayed systems. In International Conference Artificial Intelligence and Statistics, pages 1747-1778. PMLR, 2023a.
Samuel Holt, Zhaozhi Qian, and Mihaela van der Schaar. Deep generative symbolic regression. In The Eleventh International Conference on Learning Representations, 2023b. URL https: wwwppreewww.eeeee/eeeeeeeee
Samuel Holt, Alihan Hüyük, and Mihaela van der Schaar. Active observing in continuous-time control. Advances in Neural Information Processing Systems, 36. 2024a.
Samuel Holt, Tennison Liu, and Mihaela van der Schaar. Automatically learning hybrid digital twins of dynamical systems. The Thirty-eightl Annual Conference on Neural Information Processing Systems, 2024b. URL
Samuel Holt, Max Ruiz Luyten, and Mihaela van der Schaar. L2MAC: Large language model automatic computer for extensive code generation. In The Twelfth International Conference on Learning Representations, 2024c. URL
net/ffoum/ Whh.hhsssWWWWK
Samuel Holt, Zhaozhi Qian, and Mihaela van der Schaar. Neural laplace: Learning diverse classes of differential equations in the laplace domain. International Conference on Machine Learning, pages 8-11-8832 PMLR, 2022.

---

Henrik Imberg, Johan Jonasson, and Marina Axelson-Fisk Optimal sampling in unbiased active learning. In Interaational Conference on Artificial Intelligence and Statistics, pages 559-569.
PMLR, R, 2020.
Alexander Janssen, Frank C Bennis, and Ron AA Mathôt. Adoption of machine learning in pharma- cometrics: An overview of recent implementations and their considerations. Pharmaceutics, 14(9): 1814, 2022.
Krzysztof Kaeprzyk, Samuel Holt, Jeroen Berrevoets, Zhaozhi Qian, and Mihaela van der Schaar. ODE discovery for longitudinal heterogeneous treatment effects inference. In The Twelfth In- ternational Conference on Learning Representations, 2024. URL /ttps://openreview net/ forumZiid-pIISIPeeig.
Cliff C Kerr, Robyn M Stuart, Dina Mistry, Romesh G Abeysuriya, Katherine Rosenfeld, Gregory R Hart, Rafacl C Núñez, Jamic A Cohen, Prashanth Selvaraj, Brittany Hagedorn, et al. Covasim: an agent-based model of covid-19 dynamics and interventions. PLOS Computational Biology, 17(7): e1009149, 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:/412.6980 2014.
Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample-efficient model evaluation. In International Conference on Machine Learning, pages 5753-5763. PMLR, 2021.
John R Koza. Genetic programming as a means for programming computers by natural selection. Statistics and computing, 4:87-112, 1994.
Siddharth Krishna Kumar. On weight initialization in deep neural networks. arXiv preprint
arXiv:1704.08888 2017.
Ming Ta Michael Lee and Teri E Klein. Pharmacogeneties of warfarin: challenges and opportunities. Journal of human genetics, 58(6):334-33) 2013.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. detrieval-auumeeter genera- tion for knowledge-intene iedge-intessive nlp tasks. Advances in Neural Information Processing Systems, 33: 9459-9474, 2020.
Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P Trevino, Jiliang Tang, and Huan Liu. Feature selection: A data perspective. ACM computing surveys (CSUR), 506) 2017.
Ernest Lindelöf. Sur l'application de méthode des approximations successives aux équations différentielles ordinaires du premier ordre. Comptes rendus hebdomadaires des séances de l'Académie des sciences, 116(3):454-455) 1894.
David Lowell, Zachary Lipton, and Byron C Wallace. Practical obstacles to deploying active learning. arXiv preprint arXiv: 1807.04801 2018.
Chunxiao Lv, Changxiao Liu, Zhuhua Yao, Xiumei Gao, Lanjun Sun, Jia Liu, Haibo Song, Ziqiang Li, Xi Du, Jinxia Sun, et al. The clinical pharmacokinetics and pharmacodynamics of warfarin when combined with compound danshen: a case study for combined treatment of coronary heart discases with atrial fibrillation. Frontiers Pharmacology, 8:826, 2017.
Chao Ma, Sebastian Tschiatschek, Konstantina Palla, José Miguel Hernández-Lobate Sebastian Nowozin, and Cheng Zhang. Eddi: Efficient dynamic discovery of high-value information with partial vae. arXiv preprint /809: 11 142, 2018.
Yecheng Jason Ma. William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. /2931, 2023.
Valentyn Melnychuk, Dennis Frauen, and Stefan Feuerriegel. Causal transformer for estimating counterfactual outcomes. In International Conference on Machine Learning, pages 15293-15329, PMLR, 2022.
12

---

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito,o, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Zhaozhi Qian, Krzysztof Kacprzyk, and Mihaela van der Schaar. D-CODE: Discovering closed-form ODEs from observed trajectories. In International Conference on Learning Representations, 2022.
URL httpp:://pperreeeeeee httpr/iewweeeeeeeeeee net//forum//.ddwWEtttt.....s
David K Ryan, Rory H Maclean, Alfred Balston, Andrew Scourfield, Anoop D Shah, and Jack Ross. Artificial intelligence and machine learning for clinical pharmacology. British Journal of Clinical Pharmacology, 90(3):629-639, 2024.
William Schiesser. Computational mathematics in engineering and applied science: ODEs, DAEs, and PDEs. CRC press, 2014.
Paola Sebastiani and Henry P Wynn. Maximum entropy sampling and optimal bayesian experimental design. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 62(1):145-11) 2000.
Nabeel Seedat, Fergus Imrie, Alexis Bellot, Zhaozhi Qian, and Mihaela van der Schaar. Continuous- time modeling of counterfactual outcomes using neural controlled differential equations. arXiv preprint ar arXiv:2206.088 083/ 2022.
Ljubisa Sehovac and Katarina Grolinger. Deep learning for load forecasting: Sequence sequence recurrent neural networks with attention. leee Access, 1--4426,,,,, 2020.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arxii preprint arXiv:1708. 00080, 2017.
Burr Settles. Active learning literature survey. 2009.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Shruti Singh, Rajesh Kumar, Shuvasree Payra, and Sunil K Singh. Artificial intelligence and machine learning in pharmacological research: Bridging the gap between data and drug discovery. Cureux, 15(8), 2023.
Silviu-Marian Udrescu and Max Tegmark. Ai feynman: A physics-inspired inspired method for symbolic regression. Science Advances, 6(16))eay2633, 2020.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Mohamed Akram Zaytar and Chaker El Amrani. Sequence to sequence weather forecasting with long short-term memory recurrent neural networks. International Journal of Computer Applications, 143(1 1117-11 2016.
13

---

Appendix
Table of Contents
A Additional Related Work
15
B Benchmark Dataset Environment Details
15
B.I Cancer PKPD
15
COVID-19
B.3 Plankton Microcosm
................................
17
B.4 PK Wafarin Dataset
17
C Evaluation Metries
17
C.I. Model Optimization Losses
18
Pharmacologists Feedback Statements
18
E
Benchmark Method Implementation Details
19
F
D3
Implementation Details
20
F.1
D3 Pseudocode
20
F2
Training Models
20
F3
System Description
21
D3 Prompt Templates
................................
21
F5
D3 System Description Prompts
................................
21
G
Additional Experiments
21
G.1 Warfarin Case Study Additional Results
21
G.2 Insight Experiment: Feature Acquisition Setup
23
G.3 Other Datasets Discovered White-Box Models
32
Future Work
Additional Sections
34
1.1 Cost Function Considerations
34
14

---

A Additional Related Work
We expand on the related work contained within the main paper.
ODE learning methods. Symbolic regression methods [Koza, 1994, Kacprzyk et al., 2024, Holt et al., 2023b] such as SINDy and D-CODE [Brunton et al., 2016, Qian et al., 2022] can automatically discover closed form ODEs f E Mc through genetic algorithms. In principle, they can incorporate features a,, achieving refinement R2, but they struggle with many variables (c.g., more than 20) due to computational complexity. On the other hand, Neural ODE leverages neural networks E My to capture a large number of variables [Chen et al., 2018, Dupont et al., 2019, Zaytar and EI Amrani, 2016, Devlin et al., 2018, Sehovac and Grolinger, 2020, Holt et al., 2022, 2023a, 2024a]. However, Neural ODEs involve many free parameters and cannot automatically reduce the level of refinement when there is insufficient training data, leading to overfitting. In comparison, D3 can capture f E Mc U My and R1-RA R3 levels of refinement; it can further adapt the level of refinement based on data availability. Furthermore, in both Symbolic regression and Neural ODE, human experts need to manually incorporate the problem context through the specification of many hyperparameters. Finally, neither of these methods assumes fixed training data and does not address the Data Acquisition Challenges.
Al for automated modeling. Al Feynman [Udrescu and Tegmark, 2020] improves upon the symbolic regression methods by leveraging a specific type of information c, i.e. the physical unit of each variable (e.g., meter). In applications in physies, knowing the units provides additional constraints on the variable's 's relationship, thus helping narrow down the search space. However, its utility outside physical sciences has been limited. More recently. Eureqa [Ma et al., 2023] applies LLMs to modeling the reward function of reinforcement learning algorithms. Eureqa is similar to D3 in its ability to integrate unstructured context and represent the model code. However, Eureqa cannot automatically adjust the level of refinement or acquire new data and it does not easily apply to learning temporal dynamics, c.g. ODEs. Furthermore, other LLM code generation [Holt et al., 2024b,c] approaches cannot acquire new data.
Data Acquisition. Methods have been developed to address specific data acquisition challenges. Active Feature Acquisition (AFA) attempts to measure additional feature variables h on existing training samples to improve predictive performance [Ma al., 2018, Gong al., 2019]. On the other hand, Active Learning [Sebastiani and Wynn, 2000, Settles, 2009, Sener and Savarese, 2017, Imberg et al., 2020] and Active Testing [Lowell et al., 2018, Kossen et al., 2021] attempt to acquire new samples to increase the size of training or evaluation data N while keeping the features constant. Neither field has proposed method to address the zero-shot setting, where the data acquisition decision must be made before observing any target data (Equation 7). Moreover, most existing methods in Active Learning focus on acquiring labels for supervised learning given unlabelled samples, which is not immediately applicable to temporal modeling tasks.
B Benchmark Dataset Environment Details
In the following, we present six clinically relevant datasets, each of which either sourced from real-world data or generated from high-fidelity simulator developed by domain experts.
B.1 Cancer PKPD
Three of our environments are derived from a ofate-of-the-art biomedical Pharmacokinetic- Pharmacodynamic (PKPD) model of lung cancer tumor growth, used to simulate the combined effects of chemotherapy and radiotherapy in lung cancer [Geng et al., 2017]. This model has been extensively used in other works [Seedat et al., 2022, Bica et al., 2020, Melnychuk et al., 2022]. We use this bio-mathematical lung cancer model to create three variations: no treatments (Lung Cancer), chemotherapy only (Lung Cancer (with Chemo.)), and both chemotherapy and radiotherapy (Lung Cancer (with Chemo. & Radio.)). For each model, we sample respective dataset. Below, detail the general case of Lung Cancer (with Chemo. & Radio.), which originates from the general Cancer PKPD Model, and then describe the variations.
Cancer PKPD Model. This model simulates the combined effects of chemotherapy and radiotherapy on lung cancer tumor growth [Geng et al., 2017], shown in Equation (11). The model predicts
15

---

tumor volume rrt) over time in days post-diagnosis. The model includes two binary treatments: (1) radiotherapy un and (2) chemotherapy u..
dx(t) = (plog dt
K
(tt
- [a,d(tt+ )z((t
(11)
xxt)
Chemotherapy
Radiotherapy
Tumor growth
The parameters K, P, Bo, Cry, 3, for each simulated patient are specified in Geng et al. [2017] and sum- marized in Table 4. Additionally, the chemotherapy drug concentration C(t) follows an exponential
Table 4: Cancer PKPD parameter values.
Model
Variable
Parameter
Parameter Value
Growth parameter
7.00 x 10
Tumor growth
K
Carrying capacity
30
Radio cell kill (ox
0.0398
Radiotherapy
Radio cell kill (3)
Set such that o/ 10
Chemotherapy
Chemo cell
Be
0.028
decay with a half-life of one day:
dC = -.5C
(12)
dt
The chemotherapy binary action represents increasing the C(t) concentration by 5. 5.0mg m & of Vinblastine given at time L. The radiotherapy concentration d(l) represents 2.0 Gy fractions of radiotherapy given timestep I, where Gy is the Gray ionizing radiation dose.
Time-dependent confounding, We introduce time-varying confounding by modeling chemotherapy and radiotherapy assignment Bernoulli random variables. The probabilities pe and pr depend on tumor diameter as follows:
(l(1) ( (
(+(()= =
(1()= (
D 8,)
(13)
D
where Dmax 13cm is the maximum tumor diameter, 6 8, D 12, and D(t) is the average tumor diameter. The parameters To and control the extent of time-varying confounding, with
Ta = 7
Sampling datasets. Using the Cancer PKPD model, we sample N = 1, 000 trajectories, corre- sponding N 10, 000 patients. Initial tumor volumes are sampled from uniform distribution x00) ~ 2000, 1149). and patient trajectories are forward simulated for 60 days using the Cancer PKPD Equation (11) and the action policy of Equation (13), employing a Euler stepwise solver. This process generates one dataset sample. We repeat this with independent random seeds to create D Dusin Dval, Duss- For each benchmark method run with random seed, we re-sample the datasets. Each variation includes either chemotherapy, both chemotherapy and radiotherapy, or neither. We provide further details of this dataset's system description and variable descriptions Appendix F.5.
B.2 COVID-19
We utilize the sophisticated epidemic agent-based simulator COVASIM [Kerr et al., 2021] to model COVID-19 epidemics. This advanced simulator is capable of simulating both non-pharmaceutical interventions (such lockdowns, social distancing, and school closures) and pharmaceutical interven- tions (such as vaccinations). In this agent-bacce based simulator, each agent represents an individual within the population, and can be in of several states: susceptible to COVID-19, exposed, infectious, recovered (including deaths).
We use COVASIM with its default parameter settings provided by the open-source implementation COVASIM simulates population of individuals, and ensure accuracy, we simulate 24 countries,
*COVASIM is Inatiiutbbfforissssseeeeeeeeeeed
open-source
simulator,
available
https: thip.....// com/
16

---

collecting trajectories for each. For each simulation, we use population size of 1, 000, 000 individu- als, simulating each individual separately (disabling simulation rescaling). The simulation begins with random number of individuals initially infected with COVID-19, T(0) = (110, 000, 100, 000), and proceeds for 60 days.
We repeat this process with independent random seeds to generate Prain: Dval. Prest- For each benchmark method run with different random seed, we re-sample the datasets. Detailed descriptions of this dataset's 's system and variables are provided in the prompt template outlined in Appendix F.5.
B.3 Plankton Microcosm
This subsection describes ecological model of microcosm consisting of algae, flagellate, and rotifer populations, replicating an experimental three-species prey-predator system [Hiltunen et al., 2013]. We use the dataset provided by [Bonnaffé and Coulson, 2023J9. The dataset contains single trajectory of 102 time steps. We split the data into training, validation, and test sets with proportions of 70%, 15%, and 15%, respectively, ensuring that the splits maintain the chronological order to preserve temporal causality.
Detailed descriptions of this dataset's system and variables are provided in the prompt template outlined in Appendix F.5.
B.4 PK Wafarin Dataset
Here describe the real PK Wafarin dataset from clinical trial that is publicly available [Janssen et al., 2022]. We use the NOMEN dataset available at https: / com/Janssena AIEAAIIPPpppee This publicly available dataset consists of 32 patients who received warfarin in clinical trial, which was originally designed to determine how to predict drug concentrations in pa- tients. The dataset includes total of 251 warfarin concentration measurements, with median of six measurements per patient. Each patient received a single dose of warfarin at t = O, and measurements were taken & f0.25, 0.5, 1.0, 2.0, 4.0, 6.0, 12.0, 24.0, 48.0, 72.0, 96.0, 120.0}. The available co- variates in the dataset are patient weight, age, and sex. We follow the dataset's 's original pre-processing code, which is provided https: : /t//tttttbbb//// thub....... com/ Janssena/SI SI--IIP- papper. We split the data into training, validation, and test sets with proportions of 70%, 15%, and 15%, respectively, ensuring that the splits maintain the chronological order to preserve temporal causality. This dataset is released under a GPL-3.0 license.
Detailed descriptions of this dataset's system and variables are provided in the prompt template outlined in Appendix F.5.
C Evaluation Metrics
We utilize mean squared error (MSE) to assess the benchmark methods on a separate test dataset comprising individual trajectories, denoted as Drest This assessment is based the loss defined in Equation (14) and reported as MSE. The metrics are averaged over ten runs with different random seeds, and we provide these averages along with their 95% confidence intervals. Specifically provide the mean confidence interval?. For each random seed, we independently generate new training, validation, and test datasets when simulator is available. Specifically, for each simulation, we ensure that the validation and test datasets contain the same number of trajectories as the training set. Each baseline model is trained on the training dataset, with early stopping applied using the validation dataset when supported by the method. Subsequently, we evaluate the performance of each baseline on the test dataset. This entire procedure repeated for each random seed. All experiments and training were conducted using single Intel Core i9- 1290K CPU @ 3.20GHz, 64GB RAM, and an Nvidia RTX3090 GPU with 24GB of memory.
'The Plankton Microcosm and Hare Lynx datasets are both open source and available at https://github. com/Wi httm//iileeBoomaffe//// NODDEEGGK
'We use the following code htttp://stackovrrfloocccccc/ wwtp:.cco/mmuueeeeee wwtpc//eettinns//0033/11111 compute- www.coinneieeeeeoooeeeee www.erinneeeiiieeee www.o-rammlleee sample-data
17

---

C.1 Model Optimization Losses
We evaluate the optimization loss using mean squared error (MSE) on a dataset D, and also consider a component-wise MSE loss for more detailed analysis.
MSE Loss. Specifically, optimize the following mean squared error objective:
T
[(,,D) N N
Σ a=1+=0
Σ
(t))
(t,(()))
(14)
where N x T represents the total number of state-action pairs in the dataset. The goal is find the parameters 8* that minimize this loss, ie., A* = arg ming [(, Duain) ). We optimize 0 using stochastic gradient descent with the Adam optimizer [Kingma and Ba, 2014], although other optimization algorithms, such as black box optimizers, could also be employed.
MSE Loss per component. To gather detailed quantitative statistics on the performance of the trained system model, we collect the validation loss per component. Using (i) to denote the predictions for the jth component, we define:
Nis!
wy(0 Dea)) Nval
(ft( (t)
(15)
X
7
x=11=0
We aggregate these scalar validation losses per component into vector w = [W1 w2, ................................ Wm), and compute the mean validation loss as m th
1
(B"",, Deal).
D Pharmacologists Feedback Statements
We sought feedback from three expert pharmacologists on the discovered model. They have validated and commented on the discovered model and the impact of such a tool as D3.
Prof. Eoin Mckinney, Clinician. "This model is significant, as consortiums are dedicated to improving Warfarin modeling [Consortium, 2009]. The updated PK model (Equation (10)) adds complexity with interaction terms and introduces two new variables: the Michaelis constant and the natural decay rate of warfarin concentration. The Michaelis constant, which varies between individuals or over time due to factors like temperature and pH, is presumably estimated from observed data. The natural decay rate is likely derived from prior reports but also varies between individuals. The model has proposed minimal-coss new features (a constant and interaction terms), but it's s worth noting the genotype of certain warfarin- metaloiiiimaaiiiiiingggg enzymes (CYP2C9 and VKORCI) dictates clearance rates and levels, though the model did not recommend measuring these, possibly due to the associated costs. The principal clinical relevance of the improved model better prediction of individual warfarin doses. Numerous algorithms attempt this, supported by consortium focused on dose- puse-predintion models. The data used, including public datasets, supports demonstrating the clinical utility of the improved PK model, highlighting the relevance of this problem despite newer alternatives to warfarin."
lean-Baptiste Woillard, Pharmacologist. "The model promising and pharmacokinetically plausible. I have data for mycophenolic acid, which has a challenging, complex absorption profile that would be interesting to test. The challenge extends to PK/PD, linking exposure and effect (e.g., tumor growth). Monolix, based on the SAEM algorithm, includes relevant datasets such as the PK/PD Warfarin dataset and PK dataset for tacrolimus in heart transplants, which could serve as valuable benchmarks for comparison. Additionally, TMDD models, where clearance changes with tumor size, are of particular interest. It would be beneficial to see performance comparisons with these datasets using your approach."
Richard Peck, Clinical Pharmacologist. "This model reasonable and potentially superior. represents significant advance in clinical pharmacology by automatically identifying robust PK models. While I am not modeler or an expert in warfarin modeling, the discovered model appears suitable for your purpose. The most interesting aspect of this work is that D3 identified an alternative model that could be superior. While improving PK models warfarin might not have much clinical use since warfarin dosing is monitored by INR rather than concentration, the ability to automatically identify robust PK models is a tremendous advance. Publishing about DI3 in clinical pharmacology literature would be significant. haven't seen anything like this before. However, to show superiority,
18

---

consider studying a drug with a more challenging PK. For such a publication, involving an expert pharmacometrrcian would be beneficial."
E
Benchmark Method Implementation Details
To evaluate the performance of D3 and establish its competitive performance, we conduct com- parisons with leading modeling methods for ODEs. Specifically, we benchmark against advanced black-box models that have many parameters, such as neural ODEs with action inputs, known as DyNODE [Chen et al., 2018, Alvarez et al., 2020]. Also in this class are recurrent neural networks (RNN) and a sfate-of-the-eee art transformer model (Transformer). In addition, we include white-box transparent dynamical systems models identified through equation discovery techniques, such as Sparse Identification of Nonlinear Dynamics (SINDy) [Brunton et al., 2016]. Moreover, D3 sup- ports two discovery modes, of which we compare against both, discovering only white-box models (D3-white-bol and discovering hybrid models (D3-hybrid). We also perform ablations of D3, of zero-shot generated model from D3 as (ZeroShot) and the same model with optimized parameters (ZeroOptim).
DyNODE DyNODE is neural network-based model that integrates control inputs into the neural ordinary differential equation (ODE) framework [Chen et al., 2018], as outlined by Alvarez et al. [2020]. Our implementation of DyNODE features 3-layer Multilayer Perceptron (MLP) with hidden layers consisting of 128 units each, utilizing tanh activation functions. We initialize weights using the Xavier method [Kumar, 2017]. For consistency and competitiveness, we apply the same objective, optimizer, and hyperparameters as used in D3. Specifically, we utilize the Adam optimizer [Kingma and Ba, 2014) 4] with learning rate of 0.01, batch size of 1,000, and early stopping with a patience of 20. The model is trained for 2,000 epochs to ensure convergence.
RNN Recurrent Neural Networks (RNNs) [Graves et al., 2007] serve as a standard benchmark for autoregressive time series next-step prediction. our implementation, the input dataset normalized according to the training dataset. The model consists of a Gated Recurrent Unit (GRU) RNN that maps the state-action dimension to a hidden dimension of 250 across two layers. This hidden representation is then passed through linear layer to convert it back to the state dimension, enabling next-step prediction. For a fair comparison, we employ the same objective, optimizer, and hyperparameters used in D3. Specifically, of 0.01, a batch size of 1,000, and early stopping with patience of 20 epochs. The model is trained for 2,000 epochs to ensure convergence.
utilize the Adam optimizer [Kingma and Ba, 2014] with learning rate
Causal Transformer The Causal Transformer represents a cutting-edge model designed for estimat- ing counterfactual outcomes, detailed by [Melnychuk et al., 2022]. Given its intricate structure, which includes three distinct transformer networks for processing covariates, past treatments, and past outcomes, we opted for streamlined approach more suitable for our datasets and task domains. Specifically, we employed a single transformer to model past outcomes. This implementation utilizes standard transformer encoder, where the input data is normalized to the training dataset. The state-action input dimensions are encoded into a 250-dimensional embedding vector via linear layer, followed by the application of a standard positional encoder [Melnychuk et al., 2022]. This data is then processed through a transformer encoder layer with a head size of 10 and a dropout rate of 0.1. The output from this layer is passed through linear layer reconstruct the next state, matching the state dimension size. Training this model involves the A optimizer [Kingma and Ba, 2014] with learning rate of 0..0005,, complemented by a step learning rate scheduler with step size of 1.0 and gamma of 0.95. Gradient clipping is applied at 0.7, and the model is trained with a batch size of 1.000. Early stopping is used with patience of 20 epochs, and training is conducted for up to 2,000 epochs to ensure convergence.
SINDY Sparse Identification of Nonlinear Dynamics (SINDy) [Brunton et al., 2016] is data-driven methodology designed to uncover the governing equations of a dynamical system directly from time-series data, resulting in transparent, closed-form mathematical model. The SINDy algorithm operates by performing sparse regression iteratively on a library of candidate functions to find the most parsimonious and accurate representation of the system's 's dynamics.
In our approach, we employ second-order polynomial library, L = (1,
zo 31, 11 }. as the feature library. To calculate the time derivatives from the input time-series data, use finite difference
19

---

approximations of first order. We maintain the alpha parameter at 0.5 for all experiments, with sparsity threshold 0.02, except for the COVID-19 dataset where is set to 10
Data-Driven Discovery (D3) We defer to Appendix F for implerentaaionnspecffii details. Specifi- cally, ZeroShot and ZeroOptim are ablations of D3, where these use the exact same setup, hyperpa- rameters, and prompts. First, ZeroShot generates one model without optimizing it's parameters 0 to the training dataset split, thereby assessing the model's output loss directly from the LLM. Second, ZeroOptim re-uses the same setup as ZeroShot, however now optimizes the parameters to the data, using the training setup described in Appendix F.2.
D3 Implementation Details F
The data-driven discovery method follows the framework as outlined in Section 3. We present pseudocode in Appendix F.1, how the code-generated models fo are trained in Appendix F.2, prompt templates Appendix F.4, system description prompts in Appendix F.5 for each dataset. Specifically,
we find top-K, where K = 16 is sufficient. Additionally, we use the LLM of with temperature of 0.7.
F11 D3 Pseudocode
Algorithm 1 Pseudocode for D3 Framework
1: Input: Context = {system description) and {feature description} }; training
dataset Dusin. validation dataset Dval, maximum generations G, top K models to consider
2: Output: Best fitting model fo
3: 8; < 0. 5-
Initlize top-k model memory s, and feedback
4: for = 1 to G do
5: f; G(c, fi-1, Gic.,i--,,,,.i-1.8111 11,
b Modeling Agent G generates model fa > Fit the model fa
6:
arg ming ((f.(0), (((0),Diuuus Di,uain)
7:
Compute validation loss ((f:(0), Di.val)
8:
5; + s; U 0, ((f((),, Pvai))}
L Add model to top- models D Generate self-reflection and feedback from Evaluation Agent E
9:
TO Elc, fú, D.) if features possible acquire then
11:
h; + A(c, fi. Di,
Optionally Acquire feature h,, decided by the Feature
Acquisition Agent
12:
Update dataset: Di.uain: Ds. + include h;
Return: Best model from s with the lowest validation loss
F22
Training Models
Upon generation by the Modelling LLM Agent, the model f is output as code, specifically a PyTorch [Paszke et al., 2017] neural network module. This code is then executed and the module is trained on the provided training dataset. The Modelling LLM Agent observes a code skeleton within the system description context examples of which are given in Appendix F.5. The code skeleton must be 'torch.nn. Corch.nn.Module named 'StateDiffierential with initialized parameters and a forward function that computes the state differential using the state and action as input variables. The LLM completes, but does not alter, the skeleton, simplifying text processing, module execution, and model training.
The model is trained using the standard MSE loss function Equation (14), optimized with the Adam optimizer [Kingma and Ba, 2014], using a learning rate of 0.01, batch size of 1,000, and early stopping with a patience of 20 epochs. Training proceeds for up to 2,000 epochs to ensure convergence and fair comparison.
After training, the validation MSE and per-component MSE (validation loss per state output di- mension) are computed Equation (15). The trained model appended back into 8; with a string representation including initialized parameter values. Providing these optimized parameters aids the L.L in suggesting effective initial values in subsequent generations, which are then refined through further optimization.
20

---

Computational Efficiency. The computational efficiency of D3 overall arises from two sources: Using a Large Language Model (LLM) to generate the completions of the respective agents, and fitting the model, we now discuss each of these separately. First, using a capable enough L.M such as GPT4, for performing the completions for the agents can be computationally demanding, however, this computer costs with the computational cost per token of the underlying LLM that is used, and scales in the aspect of data with the number of tokens required to process, which we could envisage arises from having the LLM utilize dataset with many features, each with their own textual description scaling the number tokens in the input that the LLM has to process. One mitigation for this is to LLMs that can handle larger context window sizes, and or be more selective about the input tokens fed into the LLM, such as restricting the number of in-context examples in the top-k examples, by reducing k, hyperparameter. Second, represent the models with parameters PyTorch module and use Pytorch to optimize the parameters, using a standard MI. pipeline to train neural networks using stochastic gradient descent. Such approaches should scale with the number of input features, and parameters, however as with training any large eventual parameter model will scale well any other neural eetwork-based approach in the same data pipeline. We note that more complicated distributed neural network training paradigms exist, however emphasize the focus of this work was on an initial framework, and leave the specific implementation for scale for future work.
F.3 System Description
Our method is initiated by expert who provides structured prompt that provides the system {system descriptions and (feature description).
The system description provides a holistic description of the system, specifying the state variables (t) and actions a(t) that are used model the system. The variables are described semantically. feature description define features, names, units, and their ranges.
In addition these components, the structured prompt includes skeleton code program {skeleton program) to instruct the LLM to synthesize executable code in pre-determined format. We refer this prompt as the context
E4
D3 Prompt Templates
For all prompt templates used, please see the code https://githhb: http:://ggttuuucooo/eeeomee com/samhott. samholt/ DataDrivenDiscoveryy
E5 D3 System Description Prompts
By following our proposed system requirements format Appendix F.3 we constructed prompts for each of the datasets that we evaluated against, which are provided in the code, https: / github. com/ sambolmmlllloooo httn//amoottDaaaDrriiin h/DaaaaDaaieeiiisoreerrrrr
G Additional Experiments
G.1 Warfarin Case Study Additional Results
Hybrid PKPD Warfarin Model with Neural Network Integration. This model incorporates simplified pharmacokinetic hatpr/oookeetii-hhhmacyyyyayaaa achieving validation loss of 0.276. The model defined as:
(PKPD) structure with additional neural network complexity,
dC = F. V k. - C (1 Ba (A-40)) (1+ B. - -.5) + MLP(CC, D, R,
dt
R = A (Kke +
(16) where F represents the bioavailability of the drug, V denotes the volume of distribution, k. is the clearance rate of warfarin, B. the slope of the age effect on drug clearance, 3, is the offset for the sex effect on drug clearance, MIP(C. D, A, is multi-layer perceptron output, representing complex relationships in the data captured by the neural network, and R is regularization term to prevent overfitting, with À being the regularization weight. The addition of neural network
21

---

allows the model to learn nonlinear relationships directly from data, providing improved predictive performance.
Existing Wafarin PK Model. Test MSE Loss: 0.6461.
Difference elimination per year age class Sifferiiforantiaal Modulo):
def -.iii (self);
super (StateDiffrreniaa
Parameters for abserption and elimination rates
xelf ta Paramctzr; loreh. (c) Absorption rate climination rate
self buse nn Parameter (torch. (0.1o)
age
(enorrt-0.081))
mos_ffiiiie Paranetii teaso:
def forward selr. warferir, concentration. warfarin,
Calculate the climisation rate adjasted for are and
self. bel self. mocooiii paiient_age self melhifir paticat_ses
Differential eguatios for warfaris concentration
dawariarin, conceniration dt self. k.a warfarin k.z werfarin.ccmcentrriios rotore 1d, da_warrorin, td_sattorio_coscente coscestratios dt.).
D3-white-box, Test MSE Loss: 0.39
class StateDiffermniialln Medulc):
def __jait__ (self);
(mpr(3tttteliiierennta calf). Init....
Defins parameters for 1010 model
self clearasce_ Parameter 16666111111177771111411111111
self oge.ogoeeeeeee effect. Parometer (torch. tessr(0411119607768888
Parameter (torch, tensor (00.or(0771197700220
dowss__poerr
celf michaolis
Para Parameter(tora (e3xor (619954-77799))
xelf docage_ cex_iniirettinn Paramaier (tarrterrrrrrrr...ee (tarch.tensod (e.osrroo
xelf iesteeteitttttii Parameter(urch (aw.rr/-..09002252--00060053
self dosoge_age ieteeerrccccciooonn nn Pacameter(toro) wenshheeerrrrrr (tc:or(0.0110000077777755
def sell,
Tesse. warfaria_dossat:: torch Tenser, patieni_aze: Tesso, paei
torch Tensor) > tophe Tense:]:
Caleslate the change is warferin constatration over time
sge_cifeci ape_effzet__bs [paiizrt_age 422)
effcc ars_elfzet_.bic [patient,
dosage effect torch, (gnth dosage) Using square root
dowere_power
dreay
desage
dasage
dnsage (patirat (paiient_eex
olf_.gge_ee jeg_iex__itttrcttion ja__eatccttooo wariaarin concentration_... conteeeottiiiiiiinn J tallf michaelit, age_effect sex_effect
age ittersctioo micharlis, mentes_term varfaris_
dosage warlaris ccnsestrationn)
ifferetffect clerseeees rate intttretttti)))) michselis, sactellie_mesten_tcoo
dosage_ess interacttion roture (4 warforin concentration di
interrttrtiioon
opineized_sremntees pacometers l'clearance_rat, 1.1119995775444, "age, offeef base 6:0110950446600000, "sex_effeet__bue bass L561199543366. "dosoge poser '. 0.3, naterall dreay "; 014441411112380011 michuelis, cossiont "; 10. 10.199944444336666 dosage sox interaction 0.00000001 'ago. sex_se__nee sex_intteacttion -0.00390222 2001000000444444550
ddosee_aag age Interaction
White box model with reintroduced interaction terms, square fremstormttion Mesten salaration kinctics
dosage effect, nstural deesy
D3-hybrid. Test MSE Loss: 0.271
class StoreDifforemtiall Module):
def i. (teff):
super (Statebifferenniil self)....iii init.....
Define the parameieess for the RRD model with sensenable initial values velf clearanes, rate Parameter (torch tenser
velf biuavailability = Paramzter (lorch, (3.00)
self volavoolee.... disiribation Paaameter(torhh (37.0) age age stope
Simplifted
tersorco.on
(0.1))
Simpliffed NLP self Sequenitat
Linearte. 10).
LestyRcLU... n. Lineearr(10 1)
LI Resuiarization
self wwwpiiaisation..eeo... weigh: an. Parameter(tord) tensor (0.0)).b(()
forward self. varfarin sxrarin_.encettatiies
Tenser, dora::i turch Temor, paticnt_age: tarch. pes: tnrch Femsr] = Torleftorch. Temor]:
abcorption change
dasage
Lineer effect
age. selr.
effect self offee (palicsi_ 0.5)
diuxize_clcaranee
climisation
agc_effect) tex_effect)
MP for canturing complex relationsiips
nlp tapor torch ( warfarin conconration, warfaris dosage, pationt pationt sex), dimal)
rip murper self aigimip.ing ispat) qqeee()
Regulacitotion teom
regolarization self. repalaiiaaziionnnocooooooo weight (self. (lelfeclearanee (tttrane_ratte.dbr self. bisoveilability..h absi) self. volume of distribution, abs(13
d_wsrfarin_ d_werfarin_coonccttta conccctraationn__..
absorption sdjostei,cltaracc mlp_output tscularizaaionn
returs (d_warferio, w...aaffiinccneeetaaaiiiii coneeettatiinn___...
22

---

oplimized,
0,8011659955601199 "bicavailabiliit 3.1901441342315674 vollemc-of.distrbiiiii regalarication weight -000077400000 0000160 white and btack hos model with cimplified linsar and cas affecte, simplified MI archirectare for capturing complet relationesipe.. and LE regelarization prevent evertitiing.
3434933861224125 "sse_effictt -007019594977774, "sex_cffett_offee"" -0,16804412229599
G.2 Insight Experiment: Feature Acquisition Setup
For this experiment, we adapted the Lung Cancer (with Chemo. & Radio.) dataset simulator, to form a new variation of this dataset, where D3 starts with two features, which are the main state x(t) features that it intends to model, and can optionally acquire new features a(t) from of possible 22 features, which are listed below. Of importance, two features directly affect the underlying behaviour of this dynamical system, these being radi otherapy.. _dsage and themotherapy- _opaeee and the rest of the features we model as random white noise features, hence have no effect on the underling system, however D3 is unaware of this, and must use the feature description, from the features name to determine relative value of improved performance that it can attain by acquiring that feature, and propose new features to acquire along with their expected improvement.
Specifically, in this setup, we allow the feature acquistion agent to select from the following set of features at each iteration:
[blood_type. smoking_statte statess pat weight patient_heizhhr sex. radiotheraps _doaaee dlcehol_conuuppiin chemotherapy desage, family History psychategical tevale.
previoons treatments dittee.. gexette axpasorn, medicarion diatary hahite.. phystcatt. actisily
performance
rexidenital_ locaatinn suppor_ncteerr,
We also make the assumption that each feature that can be acquired has equal cost I of acquisition, and that the agent has an unlimited feature acquisition budget, however these settings can be readily changed depending on the specific application.
To provide comparable basclines, we compare D3 against ablations of itself, which is a random feature acquirer Feature Acquisition Agent, that selects feature randomly for acquisition, and null, or no Feature Acquisition Agent, that only observes at all iterations the target states x(t) to model. We observe experimentally that D3 that can leverage the LLM to quantify the value of information for features, where no training data exists for them in the dataset. We observe, as in Figure 2, that D3 converges the fastest, and achieves the overall highest performance, whereas the other feature acquisition methods fall short.
We provide part of the logs from one of the random seed runs, including part of it due to the logs being extensive.
2024-05-10 09:26
mulciprocsssing
Canctr-randon Di-featare
MataPresess 2024-05-22 00:26 1844,.4 mutcsprocsssing (Rezsing gensration Di-featare Sampling not
feetures
samples
MaisPrezncel 1034-01-30 07:36:23, 185,185 185 mutriprocassing INM [Syden]
Osjective: Write cnde to create effective differencial equatios cimulstor for given tack Please note that the code should be fally fanctional, No placeholders.
You mast act astonomossly and you will receive ne hunan inpat at any stage. You have to reture as exipst the complete code for completing tais task, and carrectly improve the code to create the meat accurate sad realistie simulator possible.
You always write cal the code contents. You always indent code with tabs. You caneot vissalize ay grophical ouput. You rsist withis machine. The code ar include black hot mstti-layer percaptions whene required.
Use fuactions provided. calling fenctions only pooside deviation.
compliast IS0N request following this format withost
MuiaPrecess! 2024 05 01.26.22.185.185 multiproccssingg [User] You will syete descrigtion cnds diffarential equation cimulator for.
Sydee Description **.
Prediction of Treatmennt Response for Combined Cheno and Radiation Therapy for Nor-Small Cell Lung Cancer Patients Using Bie- Mathomatical Model
Here you most model the state differential of tonorvolame, and chemotterspy_dug digg with the inpat actions of
Dexeription of variahlex:
Volemz of tonor with units en'3
Concentration of the chenetherapy drig vinblastine osits mgaint
days.
Additionally these variables
raages
volume: 10.01435. 1170.861]
caemetherapy pre.omoetterttomttrrttiinin
concentration: [0. 9.9975)
The training dataser consicts of 1000 patiznte, where each patiext is abserved for 60 days.
Modelling foels
The parameters of the model optimized to an observed training dataxet with the ziven simulator.
The observed traising datsset has very fer samples, and the model muat be able to pencralize to unsces data.
23

---

Specification;
The code generared chorld achieve lowedt poccible validation lost. of 1e-10 lest The code generated chould interpretable, and fir the daraset accuraizly poscible..
Skeletor code to IIII in:
class StateDifferenisa lnn. Modale): def -.ii (self);
super Statteittiferrannlaall self)...ioit__.. 1000: Fill is the zoda here
sell, tomer, volume: torch. Tensor, chenstteragyyddg drug_ooxeenttaiie:: torch. Tetsor) Taple[torch Tensor, torch, Tessor retuce (d_uaoo
Usefal to know:
You code evolving machise, and will be called 20 times to generate code, and improve the code to sehieve the lowest
possible validation loss.
The model defines the cists differential and will he und with an are salver to fir the obsrrved training dataset.
You con. use any parameters you vos: and my black bos neural newerrk composcuts (mulli layer pereeptrozs). loverer, you have to
definc these.
is prefereble to decompose the systen into differeatial equations (compatimett) if possible.
You can use my unary fanctiots. for exampe leg. esp, power etc.
Under ap circomatance cma you change
vectors shope skelztos code functios defisitions. , only fill the cods.
inpat
initially
byhrid
residusis, only
prugram
ingrovemest skeletou specification.
Make your code follows
Use PyTorch.
...
Think step by- step. and then give the complete full working code. You generating code for iteration out of 20.
MainPreceass 3624-05-20 03:26-22,115 185, malciproezesinn INFD (Progresx: Step 1/b Retrize: 9:00 Token Capacity Used: 0.00%
Takens remotsing 126980]
MalaProcess 2034-05-20 07:27 26 602 6022 mutriprocessing INFO (UM RESPONSE MESSACE
MalsPrezess 3034-05-10 01:27 26 602 602 mutriprocsssing INF0 (Ferction Call] [complete Statefifferenttal cade] i
StatcDifferratill..
"import terchtsiapot terch. satafren tspins impost Tuplelatnciaxs StateDiffeicatialle Module) Defise the parameters the self-alphs Farameteri fensor (0.1)) Tonor soowth rate
def waiii... ise
sepert SiateDiffeessttid self) ititt__. (11n
tomer growth modelin
bete Ea. self alpha torch 1og(1170.36] tumor volsme)
Parameier drrud beng_cenccntaatiin, d volimeta
efficacy
based volume's alf bets chemotherapy drug censentration drag dorsy
l:ke
Gompertzian growth
Drng affect Asuming coserant concantration decayln = dres__ecsy_robl
volone
ummrr volumeiria
dechenotherapy. deuperrpy__uup_eee pou__ccoetetttooo.....
chemotzerapy doug_ooscenttatie cocuttatiisuuu tuke
ceturn
dummmor, volune d:. L_hhemobberapy conceettatttti,oon _d1""
_ode_descrippie ; This is a wsie bas model that use a Gomperisian grouth model for the tomor volume and a timple dreay model for the eiemoiherapy drug conceniration. The model includes parancierrs for the unnr goowh rate and drug offiracy. which can be optimised based en the observed traising dataset.". model inpor variables" ["remor volume". "chemotherayy drag concestrarinn")
MoiaProcess 2024
01:27.33,445, multiprocessits Eatly stopping tiagered epech 1050
MotaProcess 2024-00 07:27 33,444455555 muttiprocessing Fitness: 632215663116671
randon features 1211 Gencration (D3 featere select Fituesses: (43.215860588719] Current Gen
Pitness: $3.21585008866119
Programs:
MuisPresess 2024
33:27.33,.454.4 multipecccssitg "anv name": " "Caserr-raddon- fratstes ', "sred" B2 "generation "top. finnas 13.21560088617
[Di-fcatare-- sscleii
randon-feeatuees 12]|GEN RESULT) ('method
to "eferarre
"num
mnean_tirece" :: 11.118893116619 'fitazee [43 2588188666622)) "current_gen, wal.eel___veeeee..eee_a
43 4321155008888
MolsPrecess 2024 05 20 07:27 33,455 455 multiprocessing INFO [Syston]
Objective: Write oode to create as effective differential equatics simulator for given task. Please note that the code should be fally fanctional, No placeholdzrs.
You mast act estonomorsls end you will receive ae homn inpat at any stage. Yoe have to reture as exipet the zemplete code for completing this tadk, and carrectly improve the code to create ite most arcurate and realistie cimulator passible. You always write est the ende contents. always indent code with tabs.
code cte include black malti-layer perceptions where You cusnot viseclize ay grophtcal output. You exist withis machine. required.
Use the fuactions proxided. What calling fanctions proside RFC8259 compliatt request following this format without deviation.
MainPrecen 2024-05-20 00:29:33 455,44555,555 multiprocznoing INED [Uher] You will system description ende differential equation. simulator for.
Sysie pescriprion -os
Prediction of Treatmentt Responce for Combinzd Chens and Radiation Therapy for Nm--Saal Call ling Cancer Patients Using liie- Motsenitical Model
Here you mod model the sists differential of volor, and chemotterrpy coneecnnccttriiien: with thz inpat actions
Description of vatiahles:
chenstherapy drig vinhlaccinz mglnt))
The dine enits is in days.
Additionally these variables have the razzes of: volune: 10.01433, 1170,861] chemetherapy dreg. concentration: [0. 9.9975]
The training datasot cossists of 1000 patients. where each patiest is observed for 60 days.
Modellisa ".
The paremetess of model aptimized on observed training dataset with the given simulater. generalixe
traising datssei very samples, and
Requiremnnt Specification
The code generaved chorld acaieve the lowed possible validation lose of 1e-10 laxs The code geaccated should intespretable, and fi: the dacaset accurately possible.
24

---

Skeletor code 7111 le:ris
clase State Siffeniffernntsll Module]:
def Inin calf):
( (State
init.______.
1000: FIll it the code here
def sell, yoluer torch. Temor. terch.Temor.
dbenolaenexx..uugcceeeeom lorck, Tecaor) > Teple Trnser, Tenaor
1000 the here
reture calumm dt. L.chemoherapy.drug dragg www..eooennttrriicc_.....
You velidation loss
will called
and
code schieve lovest
The model defines the differentisi and will use with ODL salver fir the obxerved teaining dataxet.
You
parameters define these.
black bus neural netwerk components (mylii-layer perceptrons); husever, you have
It is preferable to decompore the syten into differential equations compariments) ) if possible.
You can use ay maryy fenatiors for asamle log. exp power atc.
Under no circumstunce em you change the skelsioz code fusctios defisitions only fill in the codz.
The inpat tensors are vectors of shape (batch_size
Use ieitially white box models first and thea switch to hybrid white and black bot models for the cosiduals, only after so further best program iteration improvemest with while bas models.
sure your code follows the exact code skeleten apecification.
PyTarch
Thisk by
complete full working
senerating iteration 20.
MalnPresess 2024-05-20 07:27-33, 33,455 muttiprecassing INFD [therr] You generated the followinz code completions whick thes had their parameters optimized to the traininr dataset. Please reflect
low you cmn improse the code minimize the validation loss le-6 less. The code examples are delineated by ree.
Here are your previsse iterations the best program generated. the it to see if you have eshaneted whine bos models 1:6, whet while bas model repeats with the ume val lea and isea only add black bas modsls to the white bos models:
Trocation a. Best Val Less: 43 2331666656675
Selected Feeteres: nomor, volume , chemotherapy dayg concatration Model for the chemetheragg dres concetration.
desoription This is white box model that nes Gompertziis grouth model the renor volume ard simple doeay nodel
inclodes parameters for tamer grovis and dres efficucy. whick can optimized based on the observed traising datasci.
features
sich
validation less is misimized treaimenrs comorbizition genatic markere.
less. The height.
possible additionsl festures celect beed type smoking
radiotherapy dosage slonhal conompilar,
daenotherasy Josage. family histeryy_to_oeeteee cancee, performance, states, neterry_abbic plssical_activig letvvl
environmestol..ooo exposure, uedication adbeerszeee psychological, stress_levels vccoocenomic, stlus, ceidextiel_Jecattiee serwor..
MaiaPrecess 2024-05-20 00:27:33 (322:333455,,44 .455 multiproeensing INFO [Progress: Step 100 Retries: 9/0
Token Capacity Used; 1.665
Takens remaleing 126638]
MainPrecuss 1031-01-10 01:28
matriprozzoingg (IMM BISPONS: MISIARI
MoiaProcess 2024 07:28 021222044993.....0.... 04.903.903 multiprocessing (Fasetion (asiuueecqqissttoon
"feateres".
iimporanne_scers
"nmm"" "chemetherapy.. dosage". importance Stere 0.20
name" "xesetic_markie importanee_scene 0.15
"nane": "patient Tingortance score"
"amoking_statuss iipprrtane__soo
"nmm"" "comorbidities" Imporiance scose" 0.01
Tome" "performance status". Saperiaace_sseer
imporiance
"psychological levols". imporimee_ 0.05
MainPreceuss 2024-05-10 09:28 04, 903 malssproesoiing INFO (Selected new festure] SELECTED radiotkerapy dotage No using: ["
tenor, volume' "chemotherray drug. ccicentration 'ndiotherapy dosage Malssrecess 3034-01-00 07:25 na 965 mulriprocassing IN00 [Sysom]]
Objective: code create effective differential. cquatics simulator for ziven task. Please note that code should fally fanctional, placeholdzrs..
You completnng
you improv inpai any siage. Indent
asipst compleie
possible.
You
You viceztize grophical ourput withis machine. code include black mshillsyer perceptions required.
25

---

Use provided.
MainPrecnce 3024-05-10 07:25-04,90 205, 905 mutriprezzoiing INT [ther] You will
dexcription cnde differzniizl equation emmlator for.
System Desoription:" ca
Prediction of Treazient Responac for Combinad Cheno and Radiation Therapy for Saa-Small Cell Leag Cancer Paticats Using Bic- Muliematical Model
you med model the cistz differential of velor, and chemotterapy inggeonnenttatien: with the ingat actions radiotberapy
Description
Concentraton of chemetherapy vinblastiae usits matar? rediotherapy
Gy
The time enits daya.
Additionally those variables have the ranges of: romor valume: 10.01435. 1170 861) cleneiherapy, sies__oonenttaann [0, 9.9975]
addietherap_ddosp 10, 2.0]
The training dataect cansists 1000 patizats, where each patiest observed for days.
Midelling
The of model will optimized observed training The observed traising dateset very samples, and the model be able
the sinulator.
data.
Requrrement Specification="
The code generated chould achieve the lowes pascible validation loss of 1e-10 or texs The code generated chould be interpretable, and fir the dataset a accurately as possible..
Skeletor code to fi3l Module):
class StateDiffermniial inini ssll);
supert super Diatefieeii...l self)....init_...
der forward selr.
Tensor. chenooherrgy drug coscentratio:: torch radiotherapy dosage: terch. Tensor)
Toplef torch Tonsor, tnrch. Temor]]:
1000 Fill the here.
reture temor yolume__ dt, berroooteerryydddooooeeeeeteeeee
Uselul to know:"
You are code evolving machise. and you will be called 20 tims to generate code, and imppove the code to achieve the lovest
paccible validarins lose
The model defisze the ciss differential will he med with an volver fir the obsrrved training datacer.
You con
define passmeters you vos: my bluck bos neural netwerk components (nulli layer perceptross). loverer, you have to
preferable
decompose sysee differential equatioss (compatimett) ) possible.
esp,
Under
definitions.
(hatch the initially white bas models first then switch byhrid white and black has models for rexidusls, after further best prugram tteration inprovemest with white bax modele.
The
Make sure your code follows the exoct code skeletou specification.
Use PyTorch.
Thiak step by step and then give the complete full working code. You are senerating code for iteration out of 20.
Maintreceus 3024-0i-10 07:28:04, 905 malcipreezsing INFD [Uher]
You generated following complations.
their parameters optimized traising doteset, code exanples are delineated by
reffect io
you omn improve code minimize validation loss
bos nodel iterations
exhausted
models white bos t1:
Iteration Best Val Loss: 8.2118660886611 Selected Features: blck models dropp__dre_oooeeeoee cosvenraation. Model description:
then
chenotleiapy. drepeeppp_dep___.. deesy madel for the
Gomperizian growth model for the volame chemotherapy drez concentration The model incledes parameters for the tumer grewis rate and drug effiency, which be optimized based in the observed training dataser.
This white bos model that
Here are the top code completions so for that you have generated, serted for the lowest volidation loss lest.
Val Loves 43.2 (Where the val lose per dimension voner_ val loss: 79, chrmothsropy_drug drug concentrrrtt val loss: 2.42)
Iscration:
import
typing import Teple
class StuteDiffereniial Medulc):
def ini. (self);
soper (Staeeiiiiirenntal
salf)....nn 1nit....
befine the parameiers for the ramor growch model
self alpha z in Parameize (torch 1enso118.133 12) # Timer growt rate
self bota a. Parenoter( torch, tosceh. (0 01;) Drig efficaey
def forward selff volume: torch. Tensor, chemetherapy, Aaraetteraps_drgg_oossceee cogg_cosennttaiiie: torch. Tetaor) Teple/turch Temer, torch. Tensor
nowtt model
Goupertzian
__umer_volume__.
siphs (11177033
vmmor.volame)
Drag effect
volune.
chemotherapy drug soncentration volome
Asaming constont concrntration datay
drug. 4ecay__o__ 0.1
26

---

-dar_atnn
roture (d temor volumr dr. d. d.chonotherapy. drig coscentration
upimizell.ppammeeeee parametere ('slpta" 0.809877199455775 'beta" 063100682567779555
Cude Skeleton:"
clase SiafeDifferential Medule]:
(self); super StateDiiferential self). Init
def forward self.
torch. Tensor. cbessebbrupp.duugo.sseeen tosch. Tesser, rndiotbrray_dossee terch. Tensor)
Tuplel ierch. Teser,, tnrch. Temor]:
1900 Fill the code here
reture td_temor golume d.chemoteerssydlnyyycee org.eettti
Please reflect on how you can improve the cede to fi: the dataset is accurately as possible. and be intespretable. Us the code skeleton ahove to proside the code. Please also specify only the isput parameters that you west to include is the model is from the skeleton code. Think step by step. Where epplicable use the values of the optinized paranetors reason how the code cat improved to the dalaset as accerately as possible. This for generating new code for the nest ileratios cst of 20.
953406-20 03:28-04
mutriprornssing
(Prograss: Step
Capacity 1.25%
rematning
MaiaProuess 2024 20 01:29. $37.577 muttiprocessing INFO RESPONSE MESSAEE
MaldProcess 2024-06 -0---0--00 20-20 07:29.41 41,97.57777.77 mutriprocessing
tarchseinport terch (Fecetion Call] Icomplete. StateDifferential. codel def inii (self)cin
SStaeDiifeernnia
matatrn rsplng import Tugle'ntnolass StateDifferential Modele) superi StateDiffereatiil self) -iiii_____ (1in
Defiss the parameters for the 8 Opiinized tumor growth rateto # Optimizdd drag effiracy
lumor srowth modelle self hera na Parameirriarr tarch izseorj0 921666888777990 parameser for the radincherapy effeetin
self.aiphe is Patancter torch lecaor
a befine the tell gamm un Parramcerr (tnrch remorr (0. 11) a Radimbberagy effirseylnin del (orward ummor volame: torch Temor, chemmbhrrap..ruee drag__oncesiratin
torch. Tenerr, radiotherapy. douge: 8 Tuncc growth model based on Gonjerizian grewthin ummor colome dt self, alpha tore) (11 861 renor volume) a tumor volemeto "_tumer_velate veluss... self. bets chemmtherapy,due diup_ccnccattation tsmor_volumele d_tumer Assuning constast diug consentration decas's -drug_decay. ratz chenoherapy.duu conpeeratyy cra "odee description". This white model that Gompersiion growth model for the volune. simple decay model the themotheriyy dreg concrstratior, and includes the affect radiotherapy deege The model inclades
torch Tensor] >> Tople[torch, Tensor, torch Teaser 130
* Drg efrect mn uiner Radiotherapy
velame'a
effect tomer volumela
Yusss_yolumzaale self ganm rediotherspy chensthecapp conggcooccttrtiinn___. deoy_ 0.11a rd_tumor_
for the toeer
drig officacy. and radinherrpy afficey. which he optimized based the eborredd
texinins dataset." "appel_iaput_ variables
voluue"" "chemotherapy. draperrppyydrrr.._o cnn__concettaaionn.. "ndietherapy_doee dosage"]
MainPresess I 2024-05-20 09:29:50 478,478 malciprocsssing INFO Earls stopping triggered at epoch 1110
MaiaPrezess! 2024 65 20-2 03:29.50, 45.99.50.49........ 489. mullipocessiigg INFO se--featsr--eeles I seseer--anddu----e features 1211 Gensration I] Too
Birnass: 7-4633 7-6194062244411
Nan Progams:
Men Pinnss: 25 2933995136677000 70854
Pinnases: 17 77-8320663444122
43 21118008356711 MoiaProcess! 2024
Eurrent Ge Val Lose: 74612402274111
07:29. 48,9944999.49 multiprocessits
(D3 festure select Crsces randon features 12]|GEN RESULT] l'methed
festare selectt.
name". 'Cascer candon festures' "sced". 12, "gencration" ": 1. "top. fitness 7.40340623441221
"oon programss
21,3999355670055
"fisnesses" (7.4614663374111. 43 2156605336719)
7,461246634444221 MuisPresess 2024 65 20 03:29 59..509990.000 490
carrent_gen
effective differencial [System]
Osjective: Writs ande
cimulator for given tack
Please coiz that zode shorld fully functiussl, placeholdere.
You mast act estonomorsly end you will receive ne hamn inpat at any stage, Yor have reture as oxtpst the complete code for compeeinn this task, and correctly improve the code to create the most accurate and realistic simulator possible.
You always write eat the code contents. You always indent code with tabs. You casnot visuslize any arophical output. You zxist withis a machine. The code cas include biack box milti layer perceptions where requised.
Use the functions provided. Whm calling fanctions only provide RICRIS compliast I8ON requext following chis format without deviation.
MalnPrecnce 3034-05-1 You will system description
differential
for.
System Deseription: .os
Prediction of Treatmentt Responte for Combined Cheno and Ridiation Therapy for Sm--Smal Cell Lung Cancer Patiente Using Bio- Matsematical Model
Hem you mast model the eistz. differentiat of temor volums, and chamotrerayy drigg concentration: with the ingat actions of
Deseription of the variebles:
a Voluae of the tonor with units en'S
Concentration of the chenstherapy dreg vinblastine with usits miint
The tine eaiss is in days
Additionally
volene. 10.01433. 1170.861) ranges
chemetherapy
9.9975)
The training dataiet coasists of 1000 patiznts. where each patiest is observed for 60 days.
Modelling **
a The parameters of the model will be optimized to an observed training dataset with the given simulator.
The obverved traising dateset has very fow samples and the medel most be ablo to genoralize to unsecs data.
Requitement Specification
generated
possible valicatios of interpretable,
generated
accuratzly possible.
Skeletor cnde to fi1l tass class StuteDifferentiale (nn. Molale).
27

---

1200 Fill the code
def foreard self, sumer, volume: torch. Tentor,
torch. Terxor) Teple[tuch Temor, turch, Texcor
I 1000: Fill is the code here
reture voluus dt, htgp:e/rers...uugoooeeeeiaeee...
Osafal Insw....
You are code cvalving machize, and you will be called times to generate code, and improve the code to achieve the lnvest
possible velidation
state differential my
solver fir observed training dataser.
define patamaters you
componenss (multi-layer patoeptrons); towseer, you
preferable fenetiors, for
the syste into differeatial equatioss (compartncets) possible.
You
Under circomstance en you change the skelzios ande fasction definitions. only fill the endz.
The input tensors
vectors shape (batch_size),
Use initially white hos models first and then switch to lybrid white and black hos models for the residusls, only after to further best program iteration ingrovement with whitr hax models.
Male sure your code follows lhe exect code skeleton specification.
Use PyTorch.
Thisk step step-by-step. and then give the complete full working code. You are generating code for iteration out of 20.
MaisPrecnes 3134-06-11 generated following completions which validation loss le-6 low you
muttipreassingg [ther]
parameters aptimized training
code
code examples delineated
Here presiors iterations
best white box model repeats with the val lao taen only old black box models to the white bos models:" tes
generated. Use it
oshansted white box models. 1.t, wtea
lteration 0. Best Val Losa: 43.21566885511
Selected Features: lumor_yolune, chenotherapy drtrrrpyy_drr_ceccctte dour__onesttraion deseription: This is white box madel that mes a Gompertiies growth model for the temor volume and cimple sansy model for the chemotheragy drag concentration. . The model includex parametert for the nmmrr grown tae and drag efficsey. whick can be upismized based on the observed training datasel.
Model
Iteration 1. Best Val Loss: 7.46320023444121
Selected Features: tenor volane. chenttorrpy dreg concestratios. , radiotherapy dosage Model description: This is white hot model that uses a Gonperrian goowth model for the uoarr volume. simple decay model for the chemetherapy drez cosseatrationn and iscludes effect rediotherapy deage, The medel includes parsuaterrs timor nowth rale. dreg efficacy. , ad radiotheropy efficsey. which can be optimized based the obseeved traising doteset.
Please
list in order rasted inportance (with the first festure to be aequired first) and percentage espected imprexement
ouc that the validation less is misimired le-5 less The possible additionsl festures to celezt mes bled type smoting amm. aicohol_ cossoepciisiic..... periess,cancer, caaccceceeeeeeeeeeeeeeeeee treatments conorbidities merker, chemstherapy dosgge.
features
patient weight patient height
family history ceseer, performance steles, stelus, dietury_habils
arsiccllrcctvay level. cnvironitstoll exposure. socinccomemic Hatus, residentill lecattion xppoor._eewwrr.
medication addherenceee
MainPrecess 2024-05-20 09:29:50 490.440 maltipreeensing INFO (Progress: Step 1/0
Retries: 9/30
Token Capacity Used; 1.44%
Takens remaliing 126118]
MainPrecuss 1031-01-10 01:30-35.119022
matriprezzoingg INT (IEM BESPONS: MISAARI
MoiaProcess 2024 27.30.23.219.219 multiprocessing (Fasetion Call] [Feature_ [cquisitiio]
Testeres":
("mane": "chemotherapy dosage", "importence, score": 0.25).
("nanr": "patiest_age". sreatments" ",
9.15).
0.10).
("tanr": 'comoriidities' 'imparisare "core": 0.10].
("tanr": sstoking__tttto" "iapartates_ 0,05).
("mane: "performance, staius". " "importance, score " b.05).
("nane": medicatiin odherence".. " "importzace "; 0 0.05].
[ ["mmee : "pachellogieal, sirennn "inporisnce_scorr" 0.05]
M 3024-05-10 07:30-25,310 219 malsipreezaonng IST (Selected new feature] SMEETD chumstherrpy Nox uxing: [*
MolaProcess semor_volume "chemthherary coneeennncnn rndiotheray
"chmmttrrayy
multiprocessing [Systen]
Osjective: that create effective differencial equatics cimulator for given task.
rally finetiossl. placehaldars.
You extonomossly and will improe the
stage.
complete
completing this
realistie simulator possible.
You always krite the code contents. You indent with You eassot viseslize any praphical output. You esist withis machine. The code can include black bos malli-layer perceptions where required.
Use the functions prosided Wha calling fanctions only provide a RiCEss compliani ISON request following this format withort deviation.
MainPrecens 2024-05-20 00:30:23,115, .25 malsiprecsssing INFO [User]
You will xel system description to cods differential equation simulator for.
Systen Description Motienetical Model
Responce Combined
Therapy Non-Snall ling Cancer Using
Here you most nodel the state differential of tanor volume, ard chemotterrayy concentration: with actions radiotherapy_ dosage. chenotherssy_douge
Deseription of the variahles:
1 Volem: of the tumor with unite ants
* Concentration of the chenotherapy crag vinhlastine with usits melor))
Dosage of the rediotherapy with units Gy
Dosage of the chenetherapy drag vinblascine with uxits mignt3
The tine eaiss in days.
Additionally
ranges
chemetherapy
[0. 9.9975)
radiaibarapy desage: 10.
chenetherapy dosage: 10,
28

---

The to
patizats.
obszeved for
Modelling goaler
The parameters of the model will be optimized to on observed training dataset with the given simulator. The observed traising dataset has very fee samples, and the model most to ablo generalize to unsees data.
Requirement Specification.
The code generated thould achieve the lowed poscible valication loss, of te-1l or lex. The code generated thould interpretable, and fir the dataset a accuratzly as poscible..
biteeiiferratiall Module):
iiit sself);
oupert StateDiiieeeeniial ielf)....ittt...))
10000 Fill the code here
def forward solt tumer volume: torch Tonsor, chemotheragy drug concentattio:: torch. Tesser, radiotherapy dosage: terch, Tersor, chemorherapy dotage: torch Tessor) -s Tuplattorch Tontor, torch Trasor]. 1000 Fill is the sode hetee
reture tesortt dt, j_chemotherrpps slggsooeeontatios_..
kaow....
code avalving
you
ganarate improve
schleve
poccints
training componeuts layer perceptrons). lovever, you
You
black bot
deffce
preferable decompore the syttn into differential equatioss ccomartimets)) possible.
You
fanctiots for eumple lex, power etc.
Under ao circemstunce cmn you change the skelzios cods functios defisitions. only fill in the codz.
The input rensers are vectore of shape (hatch, sixe).
Use initially white his models first and then switch to byhrd white and black hos models for the residusil, only after so further best pougram iteration improvemest with while bax models.
Make sure your code follows the exect cede skeleton specification.
Use PyTorch.
Thisk step by step. then give the complete full working
senerating for iteratinn 20.
MainPrecess 2024-05-20 02:30:23.125 You generated following code complations, which then had their
[User]
optimized the traising dateset, Please reflect you improve cade minimise the calidation loss le-6 less The code exanples delinested by 6ee.
Here are your previous iterations the best progzams genercted. Ue it to see if you hare exkausted white bat models i.c. when white bos model repeats with the une val loss and then only old black box models to the white bos models: ... Model cescription:
lirration o, Bee Val Los: 43 51215800556611
Szieeied Pratures: tomor voleme. chemotserapy. drugapp_drg_..o coneeeorcenttaiicc This is a white bes model that uses a Gomperixian sowwt model for the tumer votume and simple dresy model for the chemotherapy drez concentration. The model inslades parameters for the lomer grewi rate add érz effieasy. whith can be optimized based on the obsrredd training dataser-
Iieration 1. Bed Las:: 7467950039111111 Selzeied Fraturre tomor voleme. chemotherayy drsg concentraties, radiotheragy dosage tomer volume. simple docy rediotherapy desgge. The nodel includes parausters tanor poowth rate drag efficacy. and rdditheropy efficacy which can optimized based the observed traising
description. This is model the chemetherapy dreg concestiaioon
bot model that uses Gonpertzian nowth nedel
iscludes
Hers code completions for that you have generated. serred for the lowest validation las last:"..
Val. Loss: 43.2 (Where the vol. Toss per dimeasioa is tumor volume val loss: 79, chenotheryyy dropgnyppyyuoooooooeeeooeeeer soscentttnt.... val loss: 7.42)
Iteration: 9
import torch
impori torch mn ra
from syping import Tople
Module):
(self):
superi StateDifferential. _iif___))iit__
Define the Parametzr self self bets m. Parametert turch leneer (1))
goouth model
growd
def forward selr tmmer volume: torch Tonsor, chemetheragy drug concentratios: torch. Tersors > Taple Temor, torch, Teasor J:
Tumer growch motel baaed on Compentian growth
4 Lemerr volune di = self clpha torch. torch.log (117006) / tumor_ volsne) tumer_volane
Drig effect on tamor volune
4_ __tmmrr_ rolums...d
self. beis chemsiberapy brapeapppppruur_cssstttttttt temor_volume
Assumins coastan: dres concentration desay srgg deeay stng_eossemrrttios__..
c_chemotheraayyco wroggyy_e_g__oeeeeee
dicay_deccy_rait chemotherapy
(d tenor volume c.chmmoteerapy wow.eottttiiis.....
parantee ('ulpka") 030991874994555755 'beta". 003106682855775503 dd
Val Loss: 7.40 (Whore The vol loss per dimension is tamor, tunor, volume val loss: 7,51. chemetheropy, wwtp:dddoooooooooeeee concentration vol loss: 7.43) Iteration:
import torch
typing import Tople
class Sifferifferentiill Medale):
def init (salf):
Btateiittiill self)...iiit__.
29

---

Define parameeerss 1em.0: (.999445799945575555
model
Optimized Optimiedd officacy
celf bots Paramater 10 0110642283779903)
Define the parameter for the rndiotherapy effret
self
mn Parameter (0..0r(0.1)) Radiotherapy efficacy
def forward selt. toner, volume: torch Tensor, chenotherayy drug concentratio:: torch Teaser, radiotherryy dosege: terch. Tensor)
-> Topleftorch. Terrch.TTenn Temor, torch.
Tunor gowlt model based sa Gonpenteian keewth
[_tenor_ voluueeleee.... di = self-slpha torch bochh.log110 (1170.466 / [umor__ volane) volame
lrng effect on ismor volune
d_umerr valems.. di telf, beis chemsiberapy. dtabbrapp.tugg..ssettaaiii temor_volume
Radiotherapy effect tanor volune
tumer volume
ganna rediotherapy dosage uierr volume
Assumins
decay
drog_
=0.1
s_chenotberragy.oo wrrpe._ddgg_oooooeo srmg_ooncenrattior_... di -drar_daccy_rait chemetheragy htt.:errrgy..tg...cceaaaaaa
reture C_itemorrouums___. d_chemotherrapy, d_chemoteerry,dyyyeeeeeeea wtw..ooegoseccntaiiis...uu .t)
oprimized paramoters a ("lgga" ": 0.9988782221311377 "hata': 607556651169177906 "pmmu" 0 0017101411007173
Module)
intt...
StateDifferential. 1000: Fill the ()
def forward self, lumer, volume: torch Tensor, chemstberaggy, bannolberey_dduu_ooccce dtwpccccccettatiiee jorch. Texser, radiolberay_ddosgee terch. Tersor, chemothcrapy douage: Tessor) Tuple Temor. torch. Teaser]: TERE Fill is the sode here
reture (d_remor volume dt, d_chemotherapy. ammg.essenttatiir_....
Please reffect on how you can impoove the code to fi: the dataset as aecurately as possible and he intergretable Tw the code skeleton above proside code. Please also specify onls the isput paramiters that you west inclede is model is from the akelcton code. Think step step. Where applicable use the values of the optinized parsucters reason how dataset is securately is possible. This for generating sew code for the next iteratios
code cat improsed to
MalsProcoss 2024-05-00 07:30 23.335.233 mutriprocessing INFO (Progress: Step 1/0 Retries: 0/30 Token Capacity Used: 1.77.
Tokens remateing 125608]
MainPrecnce 1054-05-10 07:31-55 571 621 mutripronzosing INTI (IFM BISONa MESAARy
MaiuProcess 2024 05 20 07:33 55.55,557 6766776611 muttiprecessits INFO (Fasction Call] (complete SateDifferential.odde I
StateDifferentiall code" " "import terchsiinpot terch, an intafron tsping impost Tuplelatncless StateDiffeientialin Module) Defize the parameiere for the a Optimized lumer growis rate e Define the Oprimized
del inin (selfr:in lumor growis modeltn
supert sepert StateDiifereanal sepf.aiphs a m. Farameterr/eers tem.er00023580022222
self) -iit)-..111---- 1st1__ (]In
self bets m. Parameter(iond (e ( 9238622 (8999970066) a Optimized drux efficasy la
paramater for the radiatherapy effeatin
all gamm an Paramater (tnrch remsor (0 03110141119911111)
Define the parametar for the chenathrrapy efferita tocch tensor (0. 1)) Chemotherapy dosage afficacylala chemotberasp soug_soncettiaiion torch. redietherayy dossge: torch. Temsrr chenotlereyy
rddiiiheray efficasy'n
call dalta nn. Parametert def fel. yoloryyoome: torch, Tenser.
torch. Tensor) duton volume
Taple torch Teasor,
Teasor
Tunor growth nodd based Gompertzien growthle
lumor_rolzme)
valome aecnotberayy_dras.coom radiacherayy dosage timrr volumeia d_ voleme__.fi di cilf delia themolherpy Asuuing consisei drig concestratien dscsyls
__iumor_vllun___
all wrrperapwydrra_nnsttttttttiiieee
tunor_volumelo
RRdiotheraps
volumees
4 Chemoharrg dooaee
effect romor volumeta
uumrr volumeroum dreg. cwn.ettannnnnnttiiiinnn....
drug.dreay_ 6.1
decay_decyy rate chemotherapy. drug ienecettaaiionlnn d chemotherapy drug. concestration dr)".
retarn (d. dd_tumor_voo (d_tunor_yolune___ voluue_____.. B.
descripicss : This is a waite bos model that uses a Gomperizian gooth model for the tomor volume. a simple decay model for the ahmotherass dres cenventzaion, cad incledes the effeets of zeth radieiaerapy and chenotherapy dossgts. The model includes parameieis for the tenor gosth rale. dug effiensy. , radiotherapy efficass. and semotberesy dosage efficscy. . which cas be optimixed bated on the observdd itaining daisser "mdel_ variables' I" ("rmmor volume". " "chemotherapy.drgg cor
"ndietterrppy Josaee'. "chemothrray,ddaae
M
muttiprocessing mattiprorssiigg
Early stopping triggered opech 1120 (BS-
Snnor-radon-ftarne Programs: 3 Mea Fituess: 19.04921025777011
Top
Fitaess: 6.3366216499911
Fituesses. 1633566265559177.
146324662344413 3321586003366191
[BS 65386236699717
MaiaPrecess 2024 of 01:34 560.560 multiprocessies
Craces randoit-fettur features 12]]GBN RESULT] ('methed is fextare exelect "enyname"" :: "Cascer-rnnoon-- feaisses ', "seed" 12, "zencration :: 2. "top fiiness 6.3588226499711
poggaat 3. fitnen" : 19 0100449292577711 "fitsesses ": (6.135521549977), 7.45110622474121, (3.219660899771))
"carrett gen val lass " 6.336621449977
MainProcncel 253-001-1 07:34 06 561, 561 mulriproesosiigg INM [Syaam]
Objectiee: Write onde to creste at effective differentill rquatics cimulator for a given tack. Please note that the code should be fally fanctiond, No placeholdzrs.
You mast act axtonomoddl snd you will receive be homn inpal any stagt. Yos have reture is exiput the complzte code for completing this taik, and carrectly improve the code to create the meat accurate snd realistic simulator possible. You always write est the code contents. You always indent code with tubs,
viseolize any graphical muppat
include
milti-isyer percaptions
reguized.
Use the fuactions proxided. Whet calling fenctions proside RFC8255 compliatt ISON request following this format withort deviation.
MaisPrecess 2024 05 20 07:34.06,561 561 multiprecessing INFO [User] You will get system description to codz differential equation simulator for.
Sydiem Description:
Prediction of Treattient Response for Combined Cheno and Radiation Therapy for Na--mall Cell Lung Cancer Patieats Using Bie- Mathenatical Model
Here you moit model the state differential of volune, and chemotterrpy congeenncen;;niinn; with ths inpat actions
Deseription of of
an's
Concentration of chenetherapy drag vioblastine with usits night33
The time anits is in daya.
30

---

Additionally variables 10.01433, 1170,861]
chemetherapy dreg concentration: [0. 9.9975)
The training dataset coasists of 1000 patients. where each patiest is abserved for 60 days.
Modellisz gouls:"..
a The parameteis of the model will be optimized to an observed trainiag dataset with the ziven simulator.
The observed training dataset has very lew sampler, and the model madt be able to generalixe to unsess data.
Reqitement Specification
grnarared should interpreiable,
paccible dataset accurately possible.
Skcletor cndz to fill in:
class StuteDifferntial (nc. Medulc): def init (self):
super (Stanniifffrrnrllll calf). 1) 1000 Fill is the code here
def self, tomer_volume: torch. Tensor, chemotherapy. prr.ooppdddoggoocctttttteeee Terch.Tecsor) > Teyle Tensor, torch. Tensor
1000 Fill the here
(d
chemotheragy drug concntration
Useful
You code evolving machite. and will be called times generate code, and improve code schieve the lowest
possible validation loss
The model defises the siale differential and will be used with ODE solver to fit the observed training dataset. black bos ncural netwerk composents Eaulii-lage perceptrons). loweier, you have to
You con
arameters you vos: defles thise.
It ix prefzrable to decompose the syste into differential equations ccomparimesn) if postible.
You can use my unary fanctiors, for esample leg, esp, power etc.
Under no oircuisttnce om you change the skeletos code fuectior defieitions only fill in the code.
The inpat censors are vectars shape (hatch sixe): box models first
Use initially
then switch kybrid white black bos models for residuals, oaly after so further
best poogram iteration improvemeat
white box modeis.
sure your code follows the exoct code skeletoa specification. PyTarch.
Thisk crep-hy-step.. and then the complete full working onde You genorating code for iteration of 20.
MoisProzess 2024 05 20 07:34 06, 561 multiproesssing INFO [User]
You senerated the following code completions whick thet bad their parometers optimized to the trainiag dataset. Please reflect on how you can improse the code to mimmise the validation lose to le-h or less The code examples are delineated by oee.
Here are your presioss ittratioss the best programs gencrated. the it to see if you have eahaested white bos models i.c. whea white bax modal ropears with the ome cal last and thn only ald black has modnls to the white has models
Ireration Best Val Loxe 43 2191600811111 Seleated Beatee nimer valume, chomatharry. dray conerstration Model is
deseription:
box model use Gompeitzice goot model inclodes parameters for lomer growth rate and dres efficacy. the chemetterrag dreg concentration. training dataset. the bos model concestration. and incledos the effea radiatharapy desage. The node inclndes paramztere rale. efficacy. and radietherapy efficacy. which be optimized based ihe abserved traizing
tenor volume and simple doess uodd
lieration Beat can optimized
Model description: 4412440624471221
caemetherapy..ee draperaayydrrr__nneettttttt
medal for the chenatbergy
for the temor
dataazi.
Iteration 2. Best Val Loss: 6. 6.366666999777 Selected Festures: tanor tunor volune. denetherapy brrg_concesttttttee chenotherapy dosage, Medel description: This is white bos nodel that ases Gompentiion growth nodel for the timor volume simple deess model for the chemotherupy doug concentration. and includes the effecis of both radiatheraps and chemotherapy dossges. The model includes paranciens for its uaer nowth mee dru effiessy. radiotheragy efficeey. and chemetherapy doagge effictey. which eu le optimized based o the observed training dalaset.
oddetherapy. dosege
Please list order
ingortance imprevement potentiat features zequire sext, sich that possible
percentage expected patizat age. patient weight pationt height type snoking marktic_marken fanily bistors cariiiss_caeeettettteee comortidities. pichol_cessuapic. haabity pbysical activity_ auvioamental_espe espoosuee. medicatics slleiincee residentiol_ location.. support
first festure be aequited first)
festures
psyclological, leves.
MaisPrecess 2024-05-20 09:34:06,561 561,561 muttiprocnniing INFO (Progress: Step 110
Retries: 0/20
Token Capacity Used: 1.3%
Takens remalring 126435)
MainPrecnce 3054-05-11 07:34 35, 705.703 701 mutriprorseainng IMT (IMM RESPOGE MINART
MainPrecnaee 3034-05-10 09:34:5 55544-5566,,77000 704 malriproezosinng INT (Faection Call] "eatere_acqssiiin
"fexteres". "
"name" pesstti_markers"
iinporinne_sceee" 0.25
"pravinse, trzalmenic" iimportanee_scoe 0.2
"ame": "comorbidities" importaees_scere" 0.15
nume": "paticat_age". importance score 0.1
"oane" "amoking__tatoo iimportance 0.1
"oane": "performance status". Imporiance score"
31

---

importance_scers
nime": paycholagical. atraaaa
importance siere 0.05
nane "carineancaial_.s Eaposar:"
importance 0.05
festure] genetio morkers using Prosss cchmmtheragy drug concentration "radiotherapy desage "chemothrrapy dosage" "gonetic
muttiprecessing
G.3 Other Datasets Discovered White-Box Models
In the main paper presented the best Warfarin discovered white-box model, however after running D3-white-bon across all the other datasets, we include a few of the white-box models that it discovered and generated for each other respective dataset used that compared against. Of particular interest, that these white-box models fit the dataset well, and can even outperform some of the other neural network based baselines-ae we highlight that with the right functional form of the white-box model, we can achieve better performance than a largely over-parameterized neural network model, that has signficiantly more parameters (such RNN with 18,000 parameters), than white-box model with 5-15 parameters.
Lung Cancer
Losc: (Wherz the val loss per dimension remor volome val loxs: 0,353) Ireration:
StoreDifferentiall Module):
Define
model
Tettiatize parameters with valees closer the optimized values. from ineracion
velf alpha
Parametzr (enxor (m.oor(0.00281) 0028))
self bete an Pareucter(
(100
Taitialize gamn
small positive value
self gamn in ttraneterctorch (ensor (0.0013)
Lnstielize delta with a valez closes to the optimized valee from itecation $
delia an,
Introdace epsilen reprasent new binlogical trom P.D.. immane response
self epsiton mn. arameter(iorc) (em00)().011)
forward volume:
Teple[terch. Temso]]:
mn-neeative
Calculate self. venor_volume
secrosis -self
rsgiogenests_ter
self, delta
sqri volume)
immase resporse term -self epsilox tanor volune
Rate of change of tomar voleme
d romer valome di growth tern nncrosis, rorm aegloganncts, tern inmonz inmonz wwwponee..nn...... Jern
reture (d_tsmor_ C.itaoor_olums__.d
earmeteed..araaeeerr parametere ('alpta 0.006657665544119990 0.74225185129258,
"spiilen" 9.0012:9256019449853 85509493338 "gmmma -0.0022193500004495
Lung Cancer (with Chemo.)
Val Lose: 2:45 [Where the val. loss per dimension is iemor_volume val los: 4,91, chemetherapy htw.:erappy.ttro..mmmnnrrttiti val loss: 1.65:-06) Iteration:: 0
4d0
class StateDifferrntial Module):
oopert Siateeiiiierennial
parameiers ieiff....mtt_....
celf
Parameter ramorr
xelf c.chmmo effect Frramiee(torch tensar
def forward self. tomer, volume: volums:
torch Tensor, chemotherrgy dogg_oooscnntttiie:: torch. Teaser, Meeuetheragy_dossgg terch. Tensor)
-> Topleltorch. Tomor torch. Tensoc
Calculaie the rate of chane of tumor velene d_tmoor_ volune = self self kateekih caner_vluus self k_chsno_effeet Siemooherupy_irrs_ceeeaaaatt jomos_volume
Caleulaic the rate change of etematherryy drng concentration Lehhmotheragy.com www.emmccncetaatiinnn......
chrmotherapy, dorage seif.k, keeayy thematergy..rog..cnecaee
roture temor
chemotterapy drug
044999066360000551
0.2759057572445544
Lung Cancer (with Chemo. & Radio.)
(Where the val per dimeasion vuner_vouume 2.85. chemetherapy_ val los: 1.96-111)
Lteratinn:
32

---

torch torch
from typing import Tople
class StateDifforential (nn. Module]:
def (xelf)...eel));
super puper (tttteiiiiireeiial _elf)....nttt_..
Define the parciscieis for the model
telf growih
Parameier (trrch. (0nsor(0.1))
telf L_eaeay m. Pammeier temor[0.11)
self K_chesto, radio sersitivity
_ceeshheffeec effect Prameter(torch (ensor(-B.055)
(errmeterctorch (ensor(-0.033)
def forward self.
torch Temor chemotherayy dro.__oseeetttiiit Teaser, cheaellerryy terch. Tersor.
rodiotheropy dossze
Teasor) Tuple[torch Tensor torch. Teaser].
Caleslate the difierential of volume
4_mmrr velume..dd a self k.rrowih cumor_yolums self L.chemoeeffee
bemncherapy.drgg.ommcce tumor_yulume radiotherspy, dousg yuner_yoluns
self. L_radiosseastttiii
Calcalato the diffarantial of chematharapy drig concestration L_clenotberrrr.co Btcpiass____oooeeeeeee diog_coscentreiioo
chemothciopy, dosage seca.k_decyy clenodderapssduuu.cceseaae
reture (d_teoor__ee td_tumor_solum____ dt, d, L.chemotbeerpps.oom digg_ooseotrraioo_.o .t)
optinized_sarantter ('k.rrowh'' 0.9644599199518955 "k_deess" 0.9999109932285, 'k_chene_effect"" 00272095535112923 consinivity
Plankton Microcosm
Val. Loss; 5. The 06 (Where the vel loss per diaeesion is pves_ population val. loss; 1.18 as, perraediate_ppsalaaiii val loss: L.3e
-6. top_ predators__opuheee val. loss: 4:07e-06) Iteration: 13
Module):
supert (StateBiiiirrattaa
Define parciseeers celf)....mtt....
model with initial values
optimized parameters
self
Parameter te.soo
self bets an. Parameter( torch, iemsor (0. 231)
Pres death rate
predation
self
an. Parameter torch 10n.or (0.23)
Intermediate predstor grewth rate t Intermediate predetor death
self delia an. Parameter (torch tensoz (0.13))
eelf ar na. Farametar morch (0 06)3
Top prodator growit rate
eilf thei
1n. Parameler (tnrch tensor(-B.133
* Top predator deat rate Piey carrying capacity
self kapps in. Paraneter (torch. 1enso/ 2:)
self signs 10. Parameter
tensor11 4;)
Intermedizte predator carcying capacity Tep predator carryteg espacity
self doo ta. Parameter/toral torch. tessor 13)
xelf emera an. Parametea (torch tensor 02))
lep preduters on intermediate predater nortality prsy shundasse intermediste predute: growth
xelf psi ta. Paraucter: torch. tessor 12);
phi
payy prodators top predaters
prodstort
def forward self. prey proy_ popolation:
Tesser, intermediate pepulation: tarch, Tewor, top, predators popstation: terch. Tensor)
Topleftorch
torch Tonsor tarch Temsr]]
Prey nowlh limited carssing capacity
pepry_populattoo di self ripha prey_pepelstios (1 pres_popelltion / self kespo) self.beta prpyppootlation istermedicte_ peppeiicttteppplttio population
Intermediate predator grouth with noslincs: intersctios tern peteeediaeepppuiatiin dt rell-pmma
pama intermediate, intermedintt.pepplaate pepulation (1 intermedizie pogelation self sigma) a (self.
pel a pray pepolatios) delf. Intermediste popolation top predators popolation selt. phi prey pogulation 44 Intermediare popolation
Top prodaior groath with effect of intermadiare predstors norliazar interaction torn d_top_ predatreattrss_pee peppllain__..li..______
self.ete- top_predotors_pee populatss thete top, rep_prelktors_ppe popullatoo intermediate popuulatiopooiiii self.si intermediste peppuediiteppullatooeeee pp.prrraters.pppplattt
pwwppaators_oppllltetee self. self.
popaat_porslatiee__.. L_inernacdiie_pppullaioo
oprimized parametere ('alpta 0.000000004117 'batas 023000000 'rmmss 0.3390002012977 "delts
0. 6.189999955444555 "et"": 3.0590031994441222.
"Ileto" ". 0.12999279979011 'keps"". ". 2.2999979972830 "sigae".
33990000099966626 "cho". " 0.09003353353331
"amege" b.199999552965111 "psi b.129999052214464, "phi"
0.00 0399000037622544 "xl": 039800099622977777
NdV
H
Future Work
The Data-Driven Discovery (D3) framework demonstrates initial success in generating interpretable pharmacokinetic (PK) models using an LLM-based iterative process. However, future work can expand and deepen the D3 approach in multiple directions:
1. Tree-Based and Graph-Based Generation Strategies: Extending D3 with tree-based generation strategies, such as Tree of Thought [Yao et al., 2024] or Graph of Thought [Besta et al., 2024], could provide a more structured exploration of model space. Such approaches could enhance D3's capacity to generate increasingly complex models over time, aligning with a growing need for adaptive modeling frameworks that account for intricate biological dynamics and drug interactions.
2. Robustness Through Retrieval-Augmentt Generation: Incorporating Retrieval- Augmented Generation (RAG) techniques could bolster D3's robustness by enabling on- demand access to external domain knowledge. RAG techniques [Lewis et al., 2020] can
33

---

serve as mechanism to manage LLM ballucinations and ensure that model suggestions re- flect state-of-the pharmacological principles. This capability would minimize reliance on the initial system descriptions and provide more flexible, context-driven model adjustments.
3. Adaptive Feature Acquisition:
The current D3 implementation assumes that newly ac- quired features apply uniformly to all individuals in dataset, which may not be feasible in practice due to data variability and differing acquisition costs across individuals. Future iterations could focus on more nuanced feature acquisition strategies that adapt to individual data constraints, potentially exploring patient-specific feature prioritization algorithms to reduce acquisition costs in clinical settings.
4. Enhanced Tool Use by LLM: While D3 utilizes GPT-4 for model discovery, integrating more advanced tool usage capabilities could further enhance its performance. Developing mechanisms for LLMs to independently call and evaluate external computational tools could extend D3's evaluation capacity. leading to richer and more rigorous model feedback, particularly for complex. multi-stage PK processes.
5. Extending Applications to Multi-Phase PK/PD Studies: Expanding D3's applicability to multi-phase PK/PD studies, such as those involving dynamic drug interaction effects or multi-compartmentall absorption models, could broaden the impact of the framework. These applications would push D3 toward modeling scenarios with temporal and interventional complexity, where capturing delayed and interactive effects becomes paramount.
These advancements would not only strengthen D3's theoretical underpinnings but also enhance its practical applicability in clinical pharmacology, potentially contributing to improved drug efficacy and safety.
I Additional Sections
L.1 Cost Function Considerations
In our feature acquisition process, the cost function I(h:) plays central role in determining the feasibility of acquiring new features. This function captures the computational, logistical, and practical costs associated with acquiring additional data for model refinement, thereby guiding the D3 framework to make efficient, data-driven decisions.
While our primary focus has been on computational costs and feature utility, it is important to recognize that feature acquisition often involves tradeoffs, particularly when biomarkers other data types are costly, time-consuming, or invasive to obtain. We outline several key considerations and potential tradeoff's below:
1. Cost-Benefit Analysis of Biomarker Acquisition: In pharmacokinetic (PK) modeling, biomarkers can provide high-impact data that significantly improve model accuracy. How- ever, some biomarkers (e.g., genetic markers, certain plasma proteins) are costly or require invasive procedures. Thus, the acquisition cost I(h:) should reflect not only monetary or computational expenses but also potential patient discomfort and clinical feasibility. This factor can be critical when evaluating the value of acquiring such features in real-world applications.
2. Marginal Utility and Redundancy Avoidance: Including a cost function allows D3 to weigh the marginal utility of new features against their acquisition cost, thereby avoiding redundancy. For instance, if a biomarker's 's expected contribution to model improvement is low, D3 can prioritize less costly or more readily available features, optimizing overall efficiency. This approach minimizes unnecessary data collection, thus reducing the risk of overfitting and keeping model complexity manageable.
3. Scalability of Feature Acquisition: In large- scale clinical studies, acquiring high-cost features across population sample may be prohibitive. D3's cost function can be adjusted to prioritize scalable data collection strategies by favoring features with lower per-patient costs or those that require fewer resources to standardize across study participants. This balance enables broader, real-world applicability while still allowing D3 to leverage high- value data where feasible.
34

---

4. Dynamic Cost Function Adaptation: As D3 iterates and learns from acquired features, the cost function ((h;) can dynamically adapt based on the observed impact of previous acquisitions. For example, if certain features consistently yield low model improvement relative to cost, the function can adjust to deprioritize similar features. Such dynamic adaptation helps D3 achieve a more nuanced balance between cost and utility as it iteratively refines the model.
These cost function considerations are integral to D3's feature acquisition approach, supporting more balanced model refinement process that aligns predictive accuracy with practical feasibility. Future iterations of D3 could further incorporate adaptive mechanisms to fine-tune these tradeoffs, potentially optimizing feature acquisition across varying clinical and resource settings.
35

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Answer: [Yes]
Justification: Yes, in this paper introduce the data-driven discovery (D3) framework, verify experimentally against existing modeling ML methods, and verify the new novel discovered PK model Wafarin with pharmacometriccan experts. We detail the framework in Section 3, provide full implementation details in Appendix F, provide the experimental results to verify our core claims of discovering well-fitting models in Section 6, and our discovered Wafarin PK model as case study in Section 6.1. We further discuss key assumptions of the method in Section and outline important limitations in Section with the "limitations & future work section".
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have dedicated limitations section in Section titled "limitations future work section". We also discuss the key assumptions of the method when we introduce it, such as in Section 3, for example Section 3.3 we state the assumptions made, and how relevant they are in clinical practice, where D3 is designed to work. We also discuss the computational efficiency of D3 in Appendix F.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ted system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
36

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include any theoretical results, therefore this question is not applicable.
Guidelines:
The answer NA means that the paper does not include thcoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
To ensure thorough reproducibility, in addition to D3's description within the main paper (Section 3), we include full implementation details of D3 in Appendix To fully reproduce the experiments we detail the exact datasets, and how they were pre- processed generated in Appendix B, how the respective evaluation metries were computed in Appendix C, along with the corresponding full implementation details of the implemented benchmark baseline methods Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
37

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Code is available at https://gi thub. com/samholte Regarding each dataset, we provide full implementation details to either re-recreate the dataset or links to exactly where the dataset comes from, this is all detailed the extensive Appendix B. To reproduce the experimental results we provide detailed implementations of D3 in Appendix F, the evaluation metries in Appendix and the baseline implementation details in Appendix E.
Justification:
SataDrivenDiscovery
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.. ca/ public/ Sublic/guides.
htt.://eed/eoeedmmiiiiiaac hwt////ooeeSsssiieeeeeee wwwsssssssoooooooiooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wwtppcccppppppeecccccccccccccctt gwwgggudd///eeee /wt///euoieSsiiiiicc..eaee
humbbiiicguddee/oddeuuuiiii wtt.suummssiooooolol.eoo (eiionooicy)) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide this full information within the dataset implementation section in Appendix B, evaluation section Appendix C, and the training details for each method in the respective implementation section of Appendices E and
Guidelines:
The answer NA means that the paper does not include experiments.
38

---

The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We accompany results with error bars in the main results, shown in Section 6. We explain how these are computed, and that they performed over random seed runs, which provided in the main experimental results section, Section 6, and expanded in the evaluation metrics section, Appendix C.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етгог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide full details of the compute used in Appendix C, and further detail the LLM used in Appendix F and for the respective baselines in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: :/tee////eeeeee /t///nnurppps httppppc/cppppppieitttccceeeeeee
39

---

Answer: [Yes]
Justification: Yes, and we provide an explicit ethical consideration discussion section in Section 7.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: Yes, and we provide an explicit broader impact discussion section in Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: It is not applicable, however, we attempt cover such discussions in the "Broader Impact and Ethical Considerations" section in Section 7.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
40

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Yes we provide full creditiation the original creators of the synthetic datasets, and or provide the links directly alongside licenses for the public datasets used within the paper, and detail all of this in Appendix B.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithhee thcode. danaaatsetss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets, therefore this is not applicable.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We do not perform any crowdsourcing, research directly with human subjects, therefore this is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
41

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
42


Reviews:
[{"review_id": "94fK3W7Ffs", "reviewer": "Reviewer_Re2Z", "summary": "This paper develops an LLM-assisted equation discovery framework especially for pharmacokinetic process. Three agents of Modeling Agent, Evaluator Agent and Feature Acquisition Agent are built to explore, refine and iterate vast model space, including three levels of initial conditions, observed features and possible acquired features. Several experiments mainly on simulated benchmarks are conducted to evaluate the performance of the proposed framework. Overall, this paper investigated whether LLM agents can help effectively search and determine the model space for dynamics modeling and process discovery in pharmacology.", "questions": "1. How is the memory module implemented in this paper, and what exactly is the memory $s_i$? Are there several choices for memory and how would they affect the performance?\n2. How much is the cost for training a D3 framework on certain datasets, especially how much would it cost for calling GPT API? Can other open-sourced LLM be used for same tasks and how is their performance?\n3. RNN and Transformer models usually perform similarly in prediction tasks. However, in Table 2, RNN and Transformer perform differently on several datasets. For example, on Lung Cancer, RNN's MSE is 1.16e6, which I am wondering if the training converges successfully.", "limitations": "The pharmacologists feedback statements highly depend on the personal knowledge and scope, as well as their understanding to the dataset or modeling. In my opinion, this part especially with only three experts are not worth for evaluation and should be excluded to avoid bias.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper investigates an interesting problem of LLM agents for pharmacological dynamical system discovery, which show some promising results of LLM for pharmacokinetic modeling. \n2. The feature acquisition could be a novel part, which leverages the knowledge capability of LLM and well matches with the application demand in pharmacokinetic modeling.\n3. The paper is well-organized, with clear sections and detailed explainations. The use of diagrams and examples, such as the iterative process involving the three agents, aids in understanding the complex interactions within the D3 framework.", "weaknesses": "1. The benchmark methods are not recent works, which might not well identify the effectiveness of the proposed framework. For example, the symbolic regression methods like D-CODE [ICLR'22] are not compared. \n2. To my understanding, the D3 model as well as other symbolic regression models discover equations from data-driven training, and then evaluate the performance based on the discovered equation. If I understand correctly, to me the experiments are not enough to prove the model effectiveness because of only five datasets with five equations. I would like to set several simulated datasets with different simulation equations for further investigation.\n3. There is no ablation study on model design or case study on different module choice, which makes it hard to evaluate the robustness of the proposed framework. Also, can LLM agents provide explanations on their output results to make a more transparent solution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and are glad the reviewer finds the results promising for using LLMs for pharmacokinetic modeling and the paper well-organized.\n\n> symbolic regression methods like D-CODE [ICLR'22] are not compared.\n\nThank you for raising additional related works. Allow us to kindly re-iterate that the focus of the paper is to propose a method that can both discover an interpretable model, incorporate textual prior (context), and have the possibility to acquire new features and samples. Existing symbolic regression works such as D-CODE and others cannot acquire new features on demand, as outlined in the related works table in Table 1. We did compare against SINDy as a baseline, and believe D-CODE to be out of scope.\n\n>  I would like to see several simulated datasets with different simulation equations for further investigation.\n\nWe agree with your invaluable suggestion and created **five additional semi-synthetic datasets** running D3-Hybrid across them, combined with some of the baselines, the results of which can be seen in the additional rebuttal PDF, as outlined in the global response. We observe that D3-Hybrid still performs well, especially when the LLM has never seen such a model. This provides empirical evidence to mitigate the concern of potential knowledge leakage issues due to pre-trained LLMs, as D3-Hybrid is still able to discover well-fitting models of the underlying synthetic equation it has never observed before.\n\n> no ablation study on model design or case study on different module choice\n\nWe can see how the existing ablations were overlooked. We did include ablations in most experimental results as ablations of D3, specifically with the additional baselines of a model zero-shot generated from D3 called **ZeroShot** and the same model with its parameters optimized called **ZeroOptim**. We have revised the baseline method description and tables to make these ablations more prominent to the reader now. Thank you for the suggestion. In terms of ablation results, these ablation results verify the components and complexity of D3 to achieve better performance.\n\n> can LLM agents provide explanations on their output results to make a more transparent solution\n\nWe agree that this is indeed possible and already achieved by the evaluator agent, see Figure 1.\n\n## Questions\n\n> how is the memory module implemented in this paper\n\nThe memory module is simply a buffer of the top-k performing programs represented as code.\n\n> Cost for training D3, API cost\n\nThe cost for training models is equivalent to training a standard 3-4 layer MLP with 128 hidden unit parameters. The API cost is around $0.075 per D3 run in total. Open-source LLMs could be used, however, D3's performance would correlate with the underlying LLM's ability.\n\n> RNN and Transformer perform differently on several datasets.\n\nWe agree some datasets are difficult to model due to the large variation of the features and complex underlying feature interactions, especially for the Lung Cancer model when analyzing the underlying model. This leads to pure parameter optimization techniques getting stuck in local minima, and hence, starting with random seeds of random weight initialization can produce different final models on these complex datasets.\n\n> Ethics review flag\n\nThank you for being cautious. However, we would like to clarify that we did not perform any research involving human subjects, only analyzing existing collected open-source medical data that is available in the public domain and providing links and descriptions to all datasets used within the paper.\n\n---\n*We hope that most of the reviewer’s concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZPpjV5TbwM", "reviewer": "Reviewer_b77T", "summary": "The paper proposes the Data-Driven Discovery (D3) framework, which leverages Large Language Models (LLMs) to iteratively discover and refine interpretable models of pharmacological dynamics. This approach enables the LLM to propose, acquire, and integrate new features, validate, and compare pharmacological dynamical systems models. The framework is demonstrated on a real pharmacokinetic datasets, highlighting its potential for clinical applications.", "questions": "1. Given that models like GPT-4 can generate incorrect information, what measures have you considered or could implement to mitigate this risk, especially for clinically relevant tasks? I think implementing a RAG like model to address hallucinations will be highly relevant in this case and make the paper better. \n2. How do you ensure consistency and reliability in the prompts across different datasets? Have you considered strategies to standardize the prompting process to minimize variability and improve predictability? Did you experiment with prompts multiple times to arrive at a reasonable framework?\n3. Can you provide more information on the scalability and computational efficiency of your framework? How does the iterative model refinement process impact computational resources compared to other baselines, and what are the practical implications for larger datasets ?\n4. The paper shows that models like RNN and transformers perform better on tasks such as COVID-19 and Warfarin. Can you explain why this might be the case and what limitations the D3 framework has in these scenarios? How might the performance be improved?\n\nIf the authors address some of the concerns and reevaluate the frameworks with the clinicians, I will be happy to change the score. I get a sense from the expert comments that the datasets evaluated upon might not be challenging enough.", "limitations": "The authors have acknowledged the limitations of their model. However, I think some of the points mentioned, such as retrieval-augmented generation, must be implemented in the framework to make the model more robust.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The D3 framework’s ability to iteratively improve models and acquire new features enhances its performance and robustness. The writing is clear and well-structured, and the methodology is well-explained. Figures and tables effectively convey key information. The problem setup, where LLM agents act together for clinically relevant tasks, is interesting and potentially useful for developing better models the current baselines. The authors demonstrate the framework’s performance on a clinically relevant warfarin dataset and show effective results. Additionally, they validate the model by obtaining feedback from clinicians, which improves the robustness of the evaluation.", "weaknesses": "1. Limited Novelty: The paper has very limited novelty in terms of machine learning and does not propose any new tools. It effectively uses existing toolboxes for a clinically relevant task. While applying an existing toolbox in the context of a new task is perfectly fine, similar methods have been published before, particularly those related to automated science labs(self driving labs).\n2. Lack of Guardrails for Hallucinations: Models like GPT-4 are prone to hallucinations and can generate incorrect information. The framework does not appear to have guardrails to mitigate this risk, which is a serious concern given the clinical relevance of the task. While the authors have acknowledged this as a limitation, it remains a critical issue.\n3. Prompt Dependency: The model’s performance is highly dependent on the prompts provided. It is crucial to formulate the prompts correctly for the LLM to generate reasonable responses. This dependency suggests that the prompting framework might need adjustments for each dataset, leaving users vulnerable to the LLM’s unpredictability.\n4. Scalability and Computational Efficiency: The framework’s scalability and computational efficiency are not thoroughly discussed. Given the iterative nature of the model refinement process, it could be computationally intensive compared to other baselines and may not be practical for larger datasets without significant computational resources.\n5. Better performance of other baselines : The performance of the D3 framework might be dependent on the data. The paper shows better performance of models like RNN and transformers on tasks such as COVID-19 and Warfarin.  The paper does not any comment on why that might be the case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and are glad the reviewer finds the D3 framework’s ability to iteratively improve models and acquire new features enhances its performance and robustness. We also appreciate that the reviewer acknowledges the clear writing, well-structured methodology, effective figures and tables, and the interesting problem setup involving LLM agents for clinically relevant tasks.\n\n> Limited Novelty: automated science labs (self-driving labs)\n\nThank you for raising additional related works. Allow us to kindly re-iterate that the focus of the paper is to propose a method that can both discover an interpretable model, incorporate textual prior (context), and have the possibility to acquire new features and samples. Existing automated science labs (e.g., The Automatic Statistician) do not leverage LLMs for iterative optimization and generally do not integrate feature acquisition in a dynamic manner. Our approach uniquely combines the strengths of LLMs with an iterative discovery process that can adjust to new data and contexts, providing a more flexible and scalable solution for pharmacological modeling.\n\n> Lack of Guardrails for Hallucinations\n\nWe agree with your suggestion and highlight that any LLM-based method or paper should have its outputs always checked by a human expert before producing the final result. We highlight that this is a common problem across any LLM-based method paper. To mitigate this risk, we have implemented a human-in-the-loop framework where clinician feedback is integrated into the iterative process. Additionally, future work could incorporate retrieval-augmented generation (RAG) techniques to further reduce hallucinations by grounding the LLM’s outputs in factual data.\n\n> Prompt Dependency\n\nAllow us to highlight that any LLM-based paper depends on the prompts that are input into it. However, we emphasize that our method’s strength lies in the combination of model feedback and LLM iteration. By iteratively refining prompts based on model performance and expert feedback, we mitigate the risks associated with prompt dependency and enhance the reliability and consistency of the generated models.\n\n> Scalability and Computational Efficiency\n\nWe find the method to still be scalable and the inner loop to train a small parameter hybrid model, where the parameter count on average is 245 parameters, to be feasible and a scalable approach. Compared to large black-box models like transformers with millions of parameters, our framework is computationally efficient and practical for larger datasets with reasonable computational resources.\n\n> Better performance of other baselines\n\nThank you for raising this. The focus of the paper is to propose a method that can both discover an interpretable model, incorporate textual prior (context), and have the possibility to acquire new features and samples. Existing black-box methods such as RNNs or transformers can fit datasets well but are unable to perform the feature acquisition and interpretability functions of D3, making them not directly comparable. Our method offers a unique blend of interpretability, adaptability, and performance, making it suitable for clinical applications where understanding the model is as crucial as its predictive accuracy.\n\n## Questions\n\n> Incorporate RAG?\n\nWe agree that incorporating RAG-based techniques could improve the method; however, we mark it as out of scope for our initial paper. We acknowledge the potential benefits and will include a discussion of RAG techniques in the final paper to highlight future directions for enhancing robustness against hallucinations.\n\n> Did you experiment with prompts multiple times to arrive at a reasonable framework?\n\nYes, we experimented with multiple prompts and iteratively refined them based on model performance and expert feedback. This process allowed us to develop the final proposed framework in the paper, ensuring that the prompts are well-suited to the task and yield reliable results.\n\n> Can you provide more information on the scalability and computational efficiency of your framework?\n\nThis is a great question. We find our hybrid models on average contain 245 parameters, making them more scalable than existing large black-box methods such as transformers with over a million parameters. Our approach balances computational efficiency with the ability to iteratively refine models, making it practical for real-world applications with limited computational resources.\n\n> How might the performance be improved to black-box methods?\n\nIt is a tradeoff to get an interpretable, hybrid/white-box model compared to a purely data-driven black-box method. Sacrificing some overall MSE can be beneficial when the process is interpretable and understandable by humans. Interpretable models provide insights that are crucial for clinical applications, where understanding the model’s behavior and its underlying assumptions is as important as its predictive accuracy.\n\n---\n\n*We hope that most of the reviewer’s concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G9Q5E9UQYe", "reviewer": "Reviewer_QDKe", "summary": "The paper presents the Data-Driven Discovery (D3) framework, a novel approach that iteratively discovers and refines interpretable pharmacological dynamical models using LLMs. This framework is novel and innovative in its domain, it is designed to address limitations in traditional pharmacokinetic (PK) modelling, which often relies on human expertise, and prior data in various formats. It is also constrained by existing knowledge. D3 leverages LLMs to propose and integrate new features, validate suggested models, and uncover new insights in PK. The framework demonstrates its effectiveness through experiments on various PK datasets, including a real-world Warfarin dataset, where it identified a new, well-fitting PK model outperforming existing literature.  The D#  framework utilises the following agents:\nModelling agent: Writes Python code for an AI model with the information the LLM has acquired    \nEvaluator agent: Evaluates the performance of the model by the modelling agent. The metric used is MSE (Mean Squared Error) \nFeature acquisition agent: Based on the evaluation of the model, more features are added to the model to improve accuracy (reduce MSE). A feature is acquired based on an estimation of the feature’s value using existing frameworks and available information such as text-based description of the feature, summary statistics and the feature’s existing data.", "questions": "* Why was MSE used as the metric for accuracy?  And on the same topic, in Table 2, what would be an interpretation of units in MSE? \n* Do you have ideas as to how to combat the limitations mentioned such as hallucinations? \n* Do you have a view on how to address the cost/ability of the LLM to distinguish between “easier” and “harder” features?  Such as: “don’t include biomarker x as it’s difficult to acquire from patient”? \n* Have you considered the inverse bias problem when models such as GPT-4 are restricted not to use biased features but it could be informative in the pharmaceutical environment to a PK model, such as ethnicity? \n* I'm still not fully clear how the LLM acquires the unstructured context/metadata about each feature.", "limitations": "Outlined in the previous sections. They are outlined and very helpful but some of them spark interesting questions and would like to see some of them elaborated on a bit more like missing data.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The Data-Driven Discovery (D3) framework introduces a novel approach to pharmacokinetic modelling by leveraging Large Language Models (LLMs). This method appears to be highly original within its domain, representing a unique application of well-known techniques to pharmacology-specific challenges. The framework's ability to explore multiple models and incorporate unstructured data distinguishes it from traditional methods that rely heavily on human expertise and predefined knowledge bases.\n\nI would judge the quality of the work as high, with claims well supported by theory and experimentation. The paper provides detailed information on D3 implementation, prompts, metrics, and compares its performance against other methods. The results demonstrate that D3 is capable of identifying well-fitting models and providing valuable insights, such as in the Warfarin case study where it uncovered less-intuitive features, interactions and combinations of them.\n\nThe paper is clearly written and well-structured, with a logical flow of sections. It effectively uses tables, graphs, and equations to communicate information, although some text could be improved for readability. The methodology is explained with sufficient detail, covering how each component works.\n\nThe significance of the framework lies in its potential to accelerate pharmacological modelling by automating the discovery and refinement of interpretable models. While it still requires some human expertise, D3 reduces dependency on experts to a degree, assisting them in their work. Its ability to uncover new insights into pharmacokinetic processes could have important implications for optimising drug dosing and minimising adverse effects.\n\nIn terms of the novelty in AI, while agentic workflows are not new, the usability of D3 by non-AI specialists may be significant. The \"evolving\" nature of the system for non-AI domain specialists can be impactful, potentially bridging the gap between AI capabilities and pharmacological expertise.", "weaknesses": "In general, the paper is inspiring and is solid in its experimentations and validations. There are a few remarks, some of them for future consideration, while others might improve the clarity of the paper:\n\n1. Cost function considerations: the current description of the cost function l(h_i)  might be too simplistic and not fully capture the complexity of acquiring new features. Cost functions are often difficult to quantify accurately. Adding a few sentences or a small subsection to discuss the aspects of the cost function and its potential impact and tradeoffs will provide a more comprehensive understanding. \n2. The D3 framework assumes that once a feature is selected, it is available for all individuals in the dataset. For example, certain biomarkers can be measured only in a small sample of patients. It would be interesting to discuss potential performance and limitations in light of missing data. Note: the authors briefly outlined it but elaboration on how to address it would be good given the commonality of this scenario. \n3. While authors called out the scalability and computational complexity of the framework, it would be interesting to see specific numbers around time to completion wrt the numbers of features. \n4. In line 212 it is mentioned that ‘The Evaluation Agent dynamically assesses both model performance and plausibility <…>’. Not sure if understood how plausibility is measured (couldn’t find it in the prompt in the Appendix either). Would be nice to see a bit more elaboration.\n5. Table 2 has mixed formatting: mixing scientific and regular notation: like 0.000245 and 2.47e+03 are both present. Unifying the notation will be more consistent. In addition, the variability of the results in terms of MSE is concerning. Might be useful to think of a more coherent metric or to provide an explanation for such extreme variability (e.g. 5780 vs. 7.07 for Lung Cancer or 719 vs. 0.3 for Lung Cancer with Chemo ).   \n6. The choice of LLM is quite significant and it is stated for the first time in Limitations on page 9. This could be brought earlier.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and are glad the reviewer finds our work introducing the Data-Driven Discovery (D3) framework novel, innovative, and significant in its application to pharmacokinetic modeling.\n\n> Cost function considerations: Adding a few sentences or a small subsection to discuss the aspects of the cost function and its potential impact and tradeoffs will provide a more comprehensive understanding.\n\nWe agree with the reviewer's suggestion and appreciate the opportunity to clarify this point. The current description of the cost function \\( l(h_i) \\) captures the complexity and computational resources required to acquire new features. However, it may not fully account for the practical challenges of acquiring specific biomarkers or data types that are costly or invasive to obtain. To address this, we have added a subsection discussing the potential impact and tradeoffs of different cost function considerations, emphasizing the importance of balancing accuracy improvements with practical feasibility. This addition provides a more comprehensive understanding of the cost-benefit analysis inherent in our feature acquisition process.\n\n> Certain biomarkers can be measured only in a small sample of patients. It would be interesting to discuss potential performance and limitations in light of missing data.\n\nWe agree, and this is indeed an assumption of the current presented method. Allow us to kindly re-iterate that the focus of the paper is to propose the Data-Driven Discovery (D3) framework, a novel approach leveraging Large Language Models (LLMs) to iteratively discover and refine interpretable dynamics models, advancing pharmacokinetic modeling. While we acknowledge that handling partially missing data is a critical aspect of real-world applications, we believe such a detailed analysis falls beyond the scope of this paper. However, we plan to explore this interesting topic in future work, investigating robust methods to address missing data scenarios within the D3 framework.\n\n> See specific numbers around time to completion wrt the numbers of features.\n\nOn average, a complete run of D3 with a feature size of 3 features takes approximately 1 hour. This duration includes the iterative processes of model generation, evaluation, and feature acquisition, ensuring a thorough exploration and optimization of the pharmacokinetic models.\n\n> How plausibility of the model is evaluated.\n\nThe plausibility of the model is evaluated by the LLM through reflective analysis. Specifically, the LLM assesses the generated model based on domain knowledge, prior literature, and logical consistency. This reflective process involves checking the alignment of the model's predictions with known pharmacological principles and empirical observations. Additionally, the evaluation includes examining the model's ability to generalize across different datasets and its interpretability by human experts, ensuring both accuracy and clinical relevance.\n\n> mixing scientific and regular notation\n\nThank you for highlighting this inconsistency. We will revise the tables to ensure uniform notation, using scientific notation consistently throughout the manuscript. The mixed notation resulted from the automatic generation of tables from raw results using Pandas dataframes, and we appreciate your attention to detail in this matter.\n\n> The choice of LLM is quite significant and it is stated for the first time in Limitations on page 9. This could be brought earlier.\n\nWe agree and have now included the choice of the LLM in the introduction. Clearly stating this earlier provides context for the framework's capabilities and limitations, helping readers understand the significance of the LLM in driving the iterative discovery and refinement processes.\n\n## Questions\n\n> Why was MSE used as the metric for accuracy?\n\nMean Squared Error (MSE) is a standard metric for assessing model accuracy in pharmacokinetic modeling, as demonstrated in the cited PKPD model papers. It quantifies the average squared difference between observed and predicted values, providing a clear measure of model performance. We have defined and explained the use of MSE in the Appendix, ensuring transparency in our evaluation criteria.\n\n> Do you have ideas as to how to combat the limitations mentioned such as hallucinations?\n\nTo combat limitations such as hallucinations, we could employ Retrieval-Augmented Generation (RAG) techniques. RAG combines the retrieval of relevant documents or data with generative models, grounding the LLM's output in factual information. This approach reduces the likelihood of hallucinations by ensuring the generated content is based on verified sources, enhancing the reliability and accuracy of the models produced by D3.\n\n> Have you considered the inverse bias problem.\n\nWe acknowledge the inverse bias problem, where models like GPT-4 are restricted from using biased features that could be informative in the pharmaceutical context, such as ethnicity. While we have not addressed this issue in the current paper, we recognize its importance and plan to explore it in future work. Investigating strategies to balance the ethical considerations of bias with the need for accurate and individualized pharmacokinetic models is a promising area for further research.\n\n> I'm still not fully clear how the LLM acquires the unstructured context/metadata about each feature.\n\nThe LLM is provided with the feature's name, of which it is empirically observed that is enough information to understand the relevance and value of that feature given the current context and progress seen during operation. \n\n---\n*We hope that most of the reviewer’s concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cBMuDejK66", "reviewer": "Reviewer_2i8U", "summary": "This paper presents the D3 Data Driven Discovery framework which uses GPT4-1106-Preview in a framework to iteratively consider modifying the features used in the ODE.  The dynamical systems are evaluated on MSEs of held-out state-action trajectories.", "questions": "What happens when the Modeling Agent produces an invalid model? What models are possible?", "limitations": "There is a limitation section, though the limitations appear more to be feature requests, rather than considerations that users would need to be aware of in using or building upon the work.", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper presents an innovative general strategy for searching the space of pharmacokinetic models in terms of which features to use.  The high level approach is accessible.  In their experiments, the D3-Hybrid (proposed) approach has the lowest MSE on 4 of 6 datasets.", "weaknesses": "The model space for the Modeling Agent could be better scoped---it appears to be whatever code the GPT-4 response is to the query.  What happens in event of failure?  Is this an automated framework?\n\nTypically you want interpretable and performance pharmacokinetic models, so there is a tradeoff between validation performance (e.g. MSE) and simplicity (measure in some way, e.g. MDL, number of features, etc), which creates a frontier for each model class.  Could this be another way to compare the performance of these models?  An analogous question would be related to acquisition costs $l(h_i)$ as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and are glad the reviewer finds our paper's presentation of the innovative general strategy for searching the space of pharmacokinetic models through our D3 framework to be accessible and acknowledges the superior performance of our D3-Hybrid approach on the datasets.\n\n> What happens when the Modeling Agent produces an invalid model?\n\nThank you for raising this important point. When an invalid model is generated, typically due to a coding error, we have implemented a robust mechanism to address this. Specifically, if the generated code contains syntax errors or logical inconsistencies that prevent it from running correctly, the system automatically re-queries the LLM up to 10 times to regenerate the model. In practice, this iterative querying resolves most coding errors and ensures that the model is valid and executable. We have now included a detailed description of this procedure in the experimental setup section in Appendix F to enhance the transparency and reproducibility of our approach. This method effectively mitigates the issue of invalid models, ensuring continuous and reliable model generation.\n\n> What models are possible?\n\nThe models generated by our framework are represented as PyTorch models. Therefore, any model that can be expressed using PyTorch's capabilities is possible within our framework. This includes a wide range of mathematical white-box models that can incorporate various operations such as logarithms, maxima, minima, and trigonometric functions. Additionally, our framework supports complex neural architectures, including multi-layer perceptrons with various regularization techniques (e.g., dropout), parameter initialization schemes, and activation functions. In practice, we observe that the generated models often consist of a combination of interpretable mathematical components and neural network elements, optimized for both performance and interpretability. This flexibility allows our framework to adaptively explore a diverse model space and discover well-fitting models tailored to the specific dataset and task at hand.\n\n> Typically you want interpretable and performance pharmacokinetic models, so there is a tradeoff between validation performance (e.g., MSE) and simplicity.\n\nWe completely agree with this observation, and it aligns with one of the central goals of our research. Our D3 framework is specifically designed to navigate the tradeoff between model performance and interpretability. To achieve this, we employ a hybrid approach that combines the strengths of both white-box and black-box models. Our white-box models are inherently interpretable, providing clear insights into the underlying pharmacokinetic processes. At the same time, our hybrid models incorporate neural network components to capture complex dynamics that might be missed by purely white-box models.\n\nIn our experiments, we systematically evaluate both types of models to ensure that we achieve a balance between accuracy and simplicity. For example, in our case study on Warfarin pharmacokinetics, we discovered a new, interpretable model that outperformed existing literature models in terms of validation MSE while maintaining clinical plausibility. This demonstrates the framework's ability to produce models that are not only accurate but also easy to understand and interpret by domain experts. Detailed insights into how we balance these tradeoffs can be found in Sections 4 and 5 of our paper, where we discuss the model evaluation metrics and provide expert clinical commentary on the interpretability of the discovered models.\n\n---\n\n*We hope that most of the reviewer’s concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M92PY2lWs5", "reviewer": "Reviewer_QBVa", "summary": "The paper  presents the data discovery framework. It uses a LLM to iteratively discover and refine interpretable models of pharmacological dynamics. The D3 framework consists of three agents, which work collaboratively to propose, acquire, and integrate new features, validate models, and uncover new insights into pharmacokinetic processes. The framework is designed to address the limitations of traditional pharmacokinetic models that are often constrained by human expertise and scalability issues. The D3 framework was validated using a real pharmacokinetic dataset of Warfarin, demonstrating its ability to uncover a new, plausible pharmacokinetic model that outperforms existing models.", "questions": "NA", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This paper is interesting. It innovatively applies a LLM to generate an interpretable the skeleton of a dynamic system and optimize the system via tranining dataset. By progressively adding relevant variables, the authors ultimately obtaining a precise and interpretable closed-form ODE. The paper leverages the extensive knowledge and self-reflection capabilities of LLMs, and the writing is clear and the experiments are comprehensive. I have learned a great deal from this paper, particularly admiring the authors' adept use of LLM agent capabilities.", "weaknesses": "All simulated datasets (and the ODE parameters) used in this study were publicly available before GPT-4's knowledge cutoff date. It is likely that OpenAI trained GPT-4 on relevant literature, leading to a significant knowledge leakage issue. I appreciate the author's perspective that LLMs encompass a vast amount of potentially usable knowledge. However, I do not believe that allowing LLMs to see the standard answers during training is an appropriate experimental setting. It will be better If the authors could demonstrate the feasibility of this method using real complex high-dimensional datasets whose dynamic is really unknown. After all, this is the kind of task faced in real pharmacokinetic studies. Meanwhile, the difference between this paper and many previous pharmacokinetic studies is that this paper does not conduct experiments on causal discovery. Instead, the model performance is evaluated through the MSE. It seems that this paper does not actually need to rely on simulated data and could use real high-dimensional time-series observational data for experiments.\n\nThe Table 3 did not report the performance of baseline models. I believe that comparing only the so-called \"standard model\" in Table 3 is inappropriate. Researchers in many other disciplines do not prioritize performance to the same extent as those in the machine learning community. They often trade off performance for model simplicity, valuing clear and concise formulas over marginal gains in performance. Thus, it is not very convincing that the authors achieved a performance advantage by using a method that resulted in a much more complex system compared to the standard model.\n\nThe baselines used in this paper are somewhat outdated. More recent studies published in the last two years, such as D-CODE and PGM (arxiv 2105.02522), could perform similar tasks as this paper. It is unclear why these were not included in the comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and are glad the reviewer finds our paper interesting and appreciates our innovative application of LLMs to generate interpretable models of pharmacokinetic dynamics. We are also pleased that the reviewer acknowledges the clear writing and comprehensive experiments, and that they have learned from our work.\n\n> It is likely that OpenAI trained GPT-4 on relevant literature, leading to a significant knowledge leakage issue.\n\nWe appreciate this insightful concern regarding potential knowledge leakage. To address this, we conducted additional experiments using five semi-synthetic datasets specifically designed to ensure that the LLM had no prior exposure to their underlying equations. We ran D3-Hybrid on these datasets and compared the results with some of the baselines. The results, which are included in the supplementary material, demonstrate that D3-Hybrid consistently performs well even when the LLM has never encountered such models before. This empirical evidence strongly mitigates the concern of knowledge leakage and reinforces the robustness of D3-Hybrid in discovering well-fitting models from novel data.\n\n> It seems that this paper does not actually need to rely on simulated data and could use real high-dimensional time-series observational data for experiments.\n\nThank you for this suggestion. We agree with the importance of using real datasets and highlight that the Warfarin dataset used in our experiments is indeed a real-world dataset. Additionally, we conducted experiments involving up to 22 features, as shown in Figure 2 of Appendix G.2. The core focus of the D3 framework is to identify the most relevant features for modeling while disregarding irrelevant ones. This approach is particularly beneficial in real-world scenarios where distinguishing significant features from a large pool is crucial for accurate modeling.\n\n> The Table 3 did not report the performance of baseline models.\n\nWe acknowledge the oversight and appreciate the reviewer pointing this out. We have updated Table 3 in the manuscript to include the performance of baseline models for a comprehensive comparison. This additional information highlights the competitive edge of our proposed D3 framework over traditional baseline models, reinforcing the novelty and effectiveness of our approach.\n\n> Baseline methods like D-CODE [ICLR'22] and PGM are not compared.\n\nWe appreciate the suggestion to compare our work with D-CODE and other symbolic regression methods. However, our paper aims to introduce a framework that not only discovers interpretable models but also incorporates textual priors (context) and enables the acquisition of new features and samples on demand. While symbolic regression methods like D-CODE are powerful, they do not address the integrated feature acquisition and context utilization capabilities of D3. For comparison purposes, we included SINDy, a well-regarded symbolic regression method, and provided a detailed evaluation against it. We believe that a direct comparison with D-CODE, which focuses solely on symbolic regression, would not fully capture the broader capabilities and contributions of the D3 framework. Furthermore, the referred to PGM paper is out of scope as well, as it focuses on probabilistic graphical modeling of dynamical systems, whereas we focus on a deterministic best-fitting model for the dataset at hand.\n\n---\n\n*We hope that most of the reviewer’s concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to the reviewers for their insightful feedback. The reviewers broadly agree that our approach is novel and effective in leveraging Large Language Models (LLMs) for pharmacological dynamical system discovery.

$\color{red} Re2Z$: “This paper investigates an interesting problem of LLM agents for pharmacological dynamical system discovery, which shows some promising results for pharmacokinetic modeling.”

$\color{green} b77T$: “The D3 framework’s ability to iteratively improve models and acquire new features enhances its performance and robustness.”

$\color{blue} QDKe$: “The Data-Driven Discovery (D3) framework introduces a novel approach to pharmacokinetic modeling by leveraging LLMs. This method appears to be highly original within its domain.”

$\color{magenta} 2i8U$: “The paper presents an innovative general strategy for searching the space of pharmacokinetic models in terms of which features to use.”

Reviewers also had concerns about the potential knowledge leakage issue due to pre-trained LLMs ($\color{magenta} 2i8U$).

We address this concern below, and address reviewers individual concerns in each separate rebuttal.

## **[A1]** Performance on procedurally generated synthetic models.

We would like to deeply thank the reviewers for bringing this up. We’ve performed a further analysis that considerably improves the paper by re-running D3-Hybrid and some baselines across five additional synthetically generated datasets. The results are provided in the attached one-page PDF. We observe that D3-Hybrid still performs well, especially when the LLM has never seen such a model. This provides empirical evidence to mitigate the concern of potential knowledge leakage issues due to pre-trained LLMs, as D3-Hybrid is still able to discover well-fitting models of the underlying synthetic equation, it has never observed before.

The above point, in combination with the reviewer's addressed concerns, we believe significantly strengthens our paper. Thank you for your valuable feedback.

Sincerely,
The Authors

Meta-review: As the Area Chair, I have carefully reviewed the submission, the reviewers' comments, and the authors' rebuttal. The paper presents a novel framework called Data-Driven Discovery (D3) that leverages Large Language Models (LLMs) to discover and refine interpretable models of pharmacological dynamics. The reviewers generally agree that the approach is innovative and shows promise in the field of pharmacokinetic modeling.

The main strengths of the paper include its well-structured presentation, clear methodology, and the framework's ability to iteratively improve models and acquire new features. The authors demonstrated the effectiveness of their approach on both simulated benchmarks and a real-world Warfarin dataset, which adds to the practical relevance of the work.

However, the reviewers raised several concerns. One reviewer pointed out the lack of comparison with recent symbolic regression methods like D-CODE. In response, the authors clarified that their focus was on proposing a method that can discover interpretable models, incorporate textual prior, and acquire new features on demand, which existing symbolic regression methods cannot do.

Another concern was the potential for knowledge leakage due to pre-trained LLMs. The authors addressed this by conducting additional experiments on five semi-synthetic datasets, demonstrating that D3-Hybrid performs well even on models the LLM has never seen before. This provides evidence to mitigate the concern of knowledge leakage.

The reviewers also noted the absence of ablation studies and case studies on different module choices. The authors clarified that they did include ablations in most experimental results, specifically with the additional baselines of ZeroShot and ZeroOptim models. They have revised the baseline method description to make these ablations more prominent.

Some concerns were raised about the scalability, computational efficiency, and potential for hallucinations in LLMs. While the authors acknowledged these limitations, they did not fully address how they plan to mitigate these risks in future work.

In conclusion, while there are some limitations and areas for improvement, the paper presents a novel and promising approach to pharmacokinetic modeling using LLMs. The authors have addressed many of the reviewers' concerns in their rebuttal and provided additional experimental results to support their claims. 

Based on the overall assessment, I recommend accepting this paper for publication at NeurIPS. The innovative use of LLMs in pharmacokinetic modeling, combined with the authors' thorough response to reviewer concerns, makes this work a valuable contribution to the field. However, I suggest that the authors incorporate their additional experiments and clarifications into the final version of the paper to address the reviewers' concerns more comprehensively.
Decision: Accept (poster)