Timely Clinical Diagnosis through Active Test Selection
Authors: Silas Ruhrberg Estévez
Nicolás Astorga
Mihaela van der Schaar
Venue: NeurIPS 2025 poster ()
Keywords: LLMs
Clinical Diagnosis
Bayesian Experimental Design
BED
Active learning
Primary area: deep_learning
TL;DR: This study introduces a framework combining Bayesian Experimental Design with Large Language Models (LLMs) to actively inform medical test selection.

Abstract: There is growing interest in using machine learning (ML) to support clinical diagnosis, but most approaches rely on static, fully observed datasets and fail to reflect the sequential, resource-aware reasoning clinicians use in practice. Diagnosis remains complex and error prone, especially in high-pressure or resource-limited settings, underscoring the need for frameworks that help clinicians make timely and cost-effective decisions. We propose ACTMED (Adaptive Clinical Test selection via Model-based Experimental Design), a diagnostic framework that integrates Bayesian Experimental Design (BED) with large language models (LLMs) to better emulate real-world diagnostic reasoning. At each step, ACTMED selects the test expected to yield the greatest reduction in diagnostic uncertainty for a given patient. LLMs act as flexible simulators, generating plausible patient state distributions and supporting belief updates without requiring structured, task-specific training data. Clinicians can remain in the loop; reviewing test suggestions, interpreting intermediate outputs, and applying clinical judgment throughout. We evaluate ACTMED on real-world datasets and show it can optimize test selection to improve diagnostic accuracy, interpretability, and resource use. This represents a step toward transparent, adaptive, and clinician-aligned diagnostic systems that generalize across settings with reduced reliance on domain-specific data.

Timely Clinical Diagnosis through Active Test Selection
Silas Ruhrberg Estévez University of Cambridge
Nicolás Astorga University of Cambridge Cambridge, UK nj nja460cam.ar ac.
Mihaela van der Schaar
University of Cambridge
Cambridge, UK
Cambridge, UK mv4720cam. ac. uk
sr933@cam.ac. ac. uu
Abstract
There is growing interest in using machine learning (ML) to support clinical diag- nosis, but most approaches rely on static, fully observed datasets and fail to reflect the sequential, aisource-wwar reasoning clinicians use in practice. Diagnosis remains complex and error prone, especially settings, underscoring the need for frameworks that help clinicians make timely and cost-effective decisions. We propose ACTMED (Adaptive Clinical Test selection via Model-based Experimental Design), diagnostic framework that integrates Bayesian Experimental Design (BED) with large language models (LLMs) to better emulate real-world diagnostic reasoning. At each step. ACTNED selects the test expected to yield the greatest reduction in diagnostic uncertainty for a given patient. LLMs act flexible simulators, generating plausible patient state distributions and supporting belief updates without requiring structured, task-specific training data. Clinicians can remain in the loop; reviewing test suggestions, interpreting intermediate outputs, and applying clinical judgment throughout. We evaluate ACTMED real-world datasets and show it can optimize test selection to improve diagnostic accuracy, interpretabilityy and resource use. This represents a step to- ward transparent, adaptive, and clinician-aligner diagnostic systems that generalize across settings with reduced reliance on domain-specific data.
high-pressure or resource- resturce-limited
Introduction
Clinical diagnosis is fundamental step in modern medical practice [1], providing the framework for future investigations and guiding treatment decisions, often determining patient outcomes. Yet remains complex and error-prone, especially in fast-paced or resource-limited settings [2], where delays, misdiagnoses, and over-testing pose persistent global challenges [3]. [4]. Additionally, the WHO projects global shortage of more than 12 million qualified health professionals by 2035 [5]. Machine learning (ML) has emerged as a promising tool to support clinicians by improving diagnostic accuracy, optimizing test selection, and enabling earlier disease detection [6]-[9] However, many current MI models operate under unrealistic assumptions, such as complete data availability [10], and fail to reflect the iterative, context- decision-making used by human clinicians [11].
Clinical diagnosis. Clinical diagnosis has traditionally followed local or national guidelines based clinical trials and expert consensus [12], [13]. Although such guidelines improve outcomes by standardizing care, they are population-based and often inefficient at the individual level, with an estimated 40 to 60% of diagnostic tests being unnecessary [14]. Resource constraints further hinder access; for example, around 15% of clinician-orderet genetic tests go unperformed due to financial barriers [15]. In response, machine learning (ML) models have been proposed support more personalized and efficient test ordering [16]. However, for these models to gain trust and adoption, they must be transparent and aligned with clinicians' reasoning processes [17], [18].
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Current models for diagnosis. The diagnostic process is inherently sequential, involving stepwise information gathering from patient examinations and tests [19]. Clinicians aim to achieve accurate early diagnoses while minimizing diagnostic costs, as timely intervention can significantly improve outcomes [20]-[22]. Prior models for early diagnosis often target single disease and rely on specific modalities such as blood tests or imaging [23]-[25], which typically require large training data sets and assume full availability of modality [26]. reality, this assumption rarely holds, and while imputation methods can handle missing data, they often introduce bias [10]. Additionally, many models treat diagnosis as a static classification task, overlooking the inherently dynamic and progressive nature of disease development and clinical reasoning. State-space models have been developed to capture disease trajectories [27]-[29]. but they also require extensive retraining and struggle with balancing information acquisition costs [30]. Consequently, there remains gap for generalizable frameworks that can reason dynamically under uncertainty and resource constraints [22], [31].
LLMs for elinical diagnosis. Recent advances in large language models (LLMs) have sparked interest in their use as general-pune purpose tools for medical decision meriiignnmmkinggi [32], [33]. LLMs perform well on medical licensing exams [18]. [34], [35] and are particularly effective in zero-shot settings [36], [37]. However, their direct deployment in clinical contexts faces challenges, including limited transparency and interpretability [38]. While chain-of- Shain-of-thought prompting improves interpretability [39], L.MS often deviate from the probabilistic optimum, although fine-tuning can improve their probabilistic reasoning [40]. Furthermore, recent work shows that LLM explanations often do not reflect their true internal reasoning processes, raising concerns about the faithfulness of chain-of- thought outputs [41]. It has also been shown that LLMs can approximate structured decision- tasks, such as Bayesian optimization or decision tree induction, by leveraging latent inductive biases learned from large-scale scale text corpora [42]- [24]. Additionally, shifting LLM reasoning to the natural language solution space has been shown enhance decision quality [45].
Contributions. ! We motivate and formalize transparent, stepwise diagnostic framework that aligns with clinical reasoning. ② We propose ACTMED, probabilistic approach to timely diagnosis that uses Bayesian Experimental Design with LLMs to adaptively select tests based on their expected diagnostic utility. 3 We show that shifting reasoning from the LLM the natural language output space can improve clinical decision-making real-world datasets, demonstrating its ability to optimize test selection and improve diagnostic accuracy, interpretability, and resource use. This framework ultimately contributes more transparent, adaptive, and clinician-aligned diagnostic processes.
④ We validate ACTMED on
2
Problem formalism
Agent-based diagnosis model. T =1,,,,,,....... Tmax} denote the discrete time horizon repre- senting stages in the decision process. The space of natural language is denoted by E, and S E E represents the natural language instructions provided to the agent at each stage. The agent must return diagnosis d, E Dj Σ, where D. is the set of possible diagnoses at time t. We assume that patient may present with subset of all possible diagnoses Dine.: C Dr. The agent independently estimates the posterior probability P(ys, = 1 Kę) for each diagnosis de E Dr, where K, is the information available at time t, and the belief is updated dynamically as new information is acquired.
At each time step E T, the agent observes subset Ke C X₁ of ground truth information X₁ C E, and may request additional information U// U4 from an external source. with 24 X, \ Ke. Here, U/ denotes the random variable corresponding to the requested test, while its realized outcome is written u/-- The agent's objective is to minimize diagnostic error while penalizing costly information acquisition through a joint Lagrangian objective:
Cly, U4 A) Σ Uly + va. + Σ c((u.)
(I)
where c(u.) denotes the cost of the requested information and À is a weighting parameter controlling the trade-off between diagnostic accuracy and test cost.
Optimal diagnostic test selection. The challenge of optimal diagnostic test selection within Bayesian Experimental Design (BED) is to identify the test, ((()), that provides the greatest informa-

---

A. Prior True
B: Sampled Posteriors
C. Posterior Selection
20
25
Prior Bellef
Density
2.0
20
2.0
True Distributon
0.5
15
1.5
Sam pled Posterior
3.-
10
1.0
Selected Posterior (max KL)
05
Finsll Posterior after Testing
0000
0.5
0.50 Probability
9.79
100
5.00
0.25
0.55
0.75
1.00
000000
0.2g
0ua0 0.15
L.0
Probability
Probabilty
Figure 1: Illustration of KI. divergence-based diagnostic test selection. (A) The agent begins with an initial prior belief about the likelihood of each diagnosis (blue), which represents current uncertainty, and seeks approximate the unobserved true posterior (black dashed).
(B) For each candidate diagnostic test, the agent uses surrogate model to simulate possible posterior distributions (gray) conditioned on hypothetical outcomes, and computes the Kullback-Leibeer (KL) divergence between each simulated posterior and the prior. (C) The test whose expected KL divergence is greatest (green) selected, is expected yield the largest information gain. After the test result observed, the agent updates its belief to a new posterior distribution (orange).
tional utility regarding diagnostic label, yd.- We consider binary classification (e.g., sick/not sick), though this framework extends to multiple diagnoses by independent simultaneous application. The core objective is to select tests that maximally reduce epistemic uncertainty, the uncertainty stemming from our limited knowledge or model imperfections, which reducible with new data, as opposed to aleatoric uncertainty, which is inherent system randomness.
To model the impact of potential information U,"). employ surrogate model draw M hypothetical realizations u, ~ P(U("). For binary classification, we assume the posterior distribution P(4d, the success probability. clinical practice, a test's value lies not just reducing uncertainty, but in meaningfully shifting the probability of disease presence, especially across decision thresholds relevant to treatment decisions. While entropy-based formulations can be used, they may sometimes prioritize tests that reinforce confident but incorrect predictions. For instance, when patient is initially assigned a very low disease probability, a truly informative test may increase this belief substantially. However, test with real diagnostic value might keep the prediction near zero, deceptively minimizing entropy (see Appendix B).
Ke, ui ((())) follows Bernoulli distribution B(p;), where ps [0,1] 1] is
The information gain can also be expressed as the difference in KL divergence between the posterior and prior distributions yd. Maximizing this expectation ensures the selection of tests whose outcomes, average, induce the most significant and diagnostically meaningful shifts in belief. This aligns with the core BED principle of maximizing information gain and directly addresses the clinical need understand how test will alter diagnostic probabilities, especially concerning critical decision thresholds. Given the unknown nature these prior and posterior distributions, we utilize surrogate model to generate samples j representing hypothetical test results. Let Petior represent
the prior, and Ppost the posterior probability distribution: Ppnor ~ P(ys, 1 Ki), Ppost
(i)
(5)
P(ya, K2, uz (b.1)). The expected KL divergence is then computed as the average KI divergence over the M samples from both distributions:
M
G)
( Prust
E (B(Ppo))
B (Pprior))]
= M
Posut
log
Ppost
+ Ppust (1-ppt)) log
(3) Priurr
1=1
Pprior
1
The optimal test (((") is the one that maximizes this expected KL divergence, i.e. 1" arg max; E (B(((pppoa) | [B The process of KL-guided diagnostic test selection and belief updating illustrated in Figure 1, A full derivation is included in Appendix B.
agent is tasked with diagnosing chronic kidney disease (CKD), aiming to establish the correct diagnosis d, as early as possible while minimizing additional diagnostic evaluations U:(0) due budget constraints and test delays. At each time point the agent has access to clinical information Kú, including demographics and previous test results, and can request further information ((()) such lab tests imaging, refine the diagnosis d,-
Example 1:
3

---

Clinical environment
LLM-problem solving agent
Natural language output space
Problem stalement 5 knowledge
Sample from prior
Prior distribution
Pytter PlAy xi)
Syree
Sample results
Test results
4 - PM4
Updated problem statement Kis1 + K, U %
*i
Sample from posterior
Posterior distribution
2 ~ Plye KG 4
Select with highest utility
Obtain last result
!
Figure 2: Overview of ACTHED. A clinician queries the system (Step 1), prompting the agent to estimate prior disease risk. The agent simulates test outcomes (Step 2), updates beliefs (Step 3), and computes expected KL divergence to select the most informative test (Step 4). The clinician reviews and conducts the test (Steps 5-6), updates the context (Step 7), and the process repeats iteratively.
3 ACTMED: Probabilistic reasoning for clinical diagnosis
Surrogate model sampling. ACTMED estimates the expected utility of potential diagnostic tests by evaluating their hypothetical impact on posterior diagnostic beliefs. At each time step, for every candidate test (²( UR the agent draws a set of plausible outcomes u, (t.)) P(U Kt), where
K, denotes the current patient knowledge base. Each hypothetical outcome u, (6-5) induces a posterior probability P(ya, 1 Kr, u, (,"))) allowing estimation of the expected information gain associated with performing that test.
This process requires reliable mechanism for approximating the conditional outcome distribution that is, for predicting what values a test might realistically produce given the patient's current profile. In well-characterized physical systems, such distributions can be modeled mechanistically, and Bayesian Experimental Design (BED) has been successfully applied in domains such as engineering [46]. physics [47], and neuroscience [48]. In clinical and biomedical contexts, however, the underlying dynamics are rarely known closed form: interactions among physiological, biochemical, and behavioral variables nonlinear, nigh-dimensional, and only partially observable [49]. This renders direct physical modeling infeasible and motivates the use of data-driven surrogates capable of capturing complex empirical relationships between tests and patient states.
P(U Kh)-that
LLM-driven sampling for ACTMED. To overcome the absence of explicit mechanistic models, introduce Large Language Models (LLMs) as flexible generative surrogates that encode prior medical knowledge learned from large-scale clinical and biomedical corpora. When prompted with structured patient information K,, an LLM acts as a conditional sampler from an implicit joint distribution over
diagnostic variables. In this setup, the model generates plausible realizations u. ((-) for unobserved tests U effectively approximating ((U K() without requiring explicit mechanistic equations or task speciie finettunigg These generated samples allow ACTMED simulate how new evidence would alter diagnostic beliefs and, consequently, to compute the expected information gain before any real test is performed. Prior studies have demonstrated that LLMs can anticipate clinical test outcomes and model disease trajectories with high fidelity [50], suggesting that they encode inductive biases suitable for probabilistic reasoning in diagnostic decision-making Figure 2 illustrates how ACTMED supports clinician-driven diagnostic reasoning.
We quantify the diagnostic value of each candidate test using an information-theoreii cost-benefit criterion. Let ((U(()) := (B IB (P)p)rr))] denote the expected information gain obtained from performing test (,(() To account for practical constraints such as test invasiveness, financial cost, and time delay, we include a cost term c(U{)) in the decision objective. Relaxing the Lagrangian formulation in Eq. (1) and assuming logarithmic scaling of cost, c(U:()) log c* (U(()) we define

---

the normalized utility:
I(U,0)
F(U!))) = c(U)
This formulation prioritizes tests that maximally reduce epistemic uncertainty relative to their expected cost, enabling efficient, interpretable, and resource-aware diagnostic reasoning.
ACTMED selects the next diagnostic test at each time step as follows:
1. Initialize the prior belief Pprior P(ys, K,) based on current knowledge Kg.
2. For each candidate test U,(), sample M possible outcomes ( {ut ~ )(U
3. Compute posterior beliefs Ppost
(i,f) P(yd: Ke, ())
4. Estimate expected information gain ( B(praie)) and define utility F(U())
5. Select the most informative test U((T) arg max; F(U, and observe outcome u, (*")
Update the knowledge base: Ket1 K. )
Deciding when to stop information acquisition. Our KL divergence- -bssed criterion naturally supports adaptive test acquisition by recommending new test only when is expected to significandly update the current belicf. At each step, the agent maintains a discase belief Prncr E [0. 1]. Diagnosis proceeds until this belief sufficiently confident, measured relative to a decision threshold 0 = 0.5, regardless of the expected results of any further tests. Wc define the confidence gap 5 1ppiice 0, and set target posterior belief Garget + -yō, where 0 S 7 S 1 hyperparameter that controls the desired confidence margin before stopping. The sign chosen to move the target posterior toward decision boundary the hyperparameter y controls how much that distance is required to justify acquiring the test. A test is acquired only if the expected KI. divergence from the current belief to the candidate posterior satisfies: [[KL((B(ppo)) Further tests acquired only if at least remaining test expected meaningfully shift the current belief; otherwise, the model is sufficiently confident that no additional information, regardless outcome, would alter the prediction.
[B( ])] ])] 2 (B (uuggs) || B (p ))]
Mitigating LLM hallucinations. We utilize LLMs as surrogate models for BED, employing structured
prompts with three components: ! Context specification; A brief description of the clinical scenario and discase. 2 Known information ( Clinical observations and test results available at time formatted in
a clinical vignette (see Appendix C). 3 Task-specific instruction: Directives the model's output, such
as predicting test outcomes u, (t-)) P(u)), diagnosis probabilities P(ya, Kt). or selecting the most informative next test. Full prompt examples are given Appendix D.
Encouraging diverse test result sampling. We enhance model robustness and capture diagnostic uncer- tainty with three strategies: ! Avoiding population averages: The model is prompted sample from broader distribution of possible outcomes. 2 Increased sampling temperature: This introduces greater randomness, reflecting higher uncertainty and improving prediction diversity. 3 Sampling both disease presence and absence: The model is instructed to sample outcomes under both conditions to ensure balanced and varied predictions.
Experiments
We evaluate two central hypotheses derived from the discussions in the previous sections:
HI) Large Language Models (LLMs) can accurately approximate the distributions of diagnostic test outcomes based on available patient information.
H2) Incorporating LLM--bsedd Bayesian Experimental Design (BED) enables more accurate and efficient diagnosis through adaptive, information-driven test selection.
examine these hypotheses, we design experiments spanning three progressively challenging diagnostic tasks, each corresponding to distinct and clinically relevant disease context.
Chronic Kidney Disease (CKD) affects over 700 million people worldwide and is responsible for more than million deaths annually. Early detection is essential for slowing disease progression and preventing renal failure [51]. Hepatitis C infects approximately 57 million people and causes around 300,000 deaths each year. Its
5

---

diagnosis is often delayed because it relies on specialized confirmatory tests, underscoring the need for surrogate blood-based indicators [52]. [53]. Diabetes, which affects roughly 500 million individuals globally, remains leading cause of cardiovascular disease and premature mortality. These three single-condition datasets provide controlled environments to assess ACTMED's s diagnostic reasoning. and test-selection efficiency across patient cohorts. They allow fine- -en-grr comparison of how well -LM-driven surrogate sampling aligns with empirical data and whether adaptive BED reasoning leads to more efficient diagnostic strategics.
To evaluate generalization beyond these controlled settings, we further introduce custom OSCE-style (Objective Structured Clinical Examination) dataset derived from AgentClinic [54]. This dataset comprises diverse, multi- condition clinical cases designed to mimic real-world diagnostic encounters, where the agent must reason across varying clinical contexts. It serves as a robust stress test for zero-shot diagnostic generalization, assessing whether the proposed framework can transfer its reasoning beyond narrowly defined single- -inlle-dseaase tasks.
4.1 LLMs accurately predict distributions of diagnostic tests
Surrogate sampling evaluation. The fidelity of surro- gate samples generated by LLMs is critical for the reliability of our Bayesian diagnostic framework. For each patient, the model was tasked with sampling ten plausible outcomes for every missing laboratory feature, conditioned on the available clinical context. The resulting synthetic feature distributions were then compared to the empirical distributions observed in the real datasets. Table 1 reports normalized Wasserstein and Energy distances between LLM-generated and empirical feature distributions, providing quantitative measures of dis- tributional alignment. Across all datascts, GPT-4o achieved the lowest average distances, indicating strong agreement with real-world feature distributions. GPT- showed slightly higher divergence, consistent with its smaller model capacity. These results confirm that both models generate physiologically coherent and statistically realistic samples suitable for surrogate-based Bayesian Experimental Design. Feature-level deviations are visualized in Fig. 3, which illustrates per-test mean absolute error (MAF) across five random seeds, highlighting that most biomarkers are reproduced with high fidelity, with slightly higher errors observed for high- high-variance biochemical markers. The complete distributions produced during the sampling are shown in Appendix E.
Table I: Normalized distribution matching metrics (lower is better). Average Wasser- stein and Energy Distance (mean + std).
Dataset
Model
Wasserstein
Energy
Diabetes
GPT-4o
0.110 0.038
0.256 1: 0.079
4P mini
0.117 1 0.046
0.269 1: 0.094
Hepatitis
GPT-4o
0.130 + 0.157
0.266 + 1.1
GP mini
0.173 1 0.237
0.330 1. 0.254
Kidney
GPT.4o
0.082 + 0.055
0.203 + 102
GPTT-4o-mini
0.082 1 0.057
0.203 1 0.104
4.2 LLM-based BED improves diagnostic accuracy and efficiency
Timely diagnosis under resource constraints. We evaluate diagnostic accuracy under the constraint of acquir- ing only three clinical tests per patient, simulating real- world limitations such as acquisition delays and resource costs. Although our framework can accommodate arbitrary test cost functions, we use uniform costs across tasks to maintain consistency, as defining lask-specific costs would require expert clinical input. Importantly, ACTMED is model- agnostic: its performance depends on the quality of the underlying surrogate model rather than any specific LLM architecture. We validate this by applying it across models of varying capacity: GPT -inii and GPT--4o. Details for all datasets given in Appendix C. We also provide a de- tailed failure analysis of ACTMED when using less powerful models that do not produce accurate samples in Appendix E.
AIo Pesetto
1.57
3.00
our Preaaao
10:3
6.01
Ast Peantio
2.57
65A
au heswtto
30.21
5:34
3 soc plicos undam resmop)
3:42
2.03
2.35
3.39
Bocchruuuee or idabeteo
2.05
2.09
Mon Aboe Mo
ciss Pesuttto
3.44
3.0
cen Peacaro
3.50
4.00
ont Fraitii
9.27
5.60
Gucas iduseta
1.43
5.00
Heroopon voo1t Prantio
3.6
1:34
Peautiu lives i0stap)
251
3:2
Rad sod conntia
224
1.00
Senthiarass idabelen
450
5.00
Nakar (iliin))
0.60
0.00
SP 80
Mode
02 981
Figure 3: Feature sampling performance. Heatmap showing the best MAE percentage for each test averaged across seeds.
We benchmark ACTMED against the following baselines that the same models and risk prediction prompts show how our approach can improve the performance of specific models:
LLM classifier: No three-feature constraint; uses all features for direct classification.
Random selection: Picks three features randomly as stochastic baseline.
Global best fixed subset: Selects a predefined set of three features prior to observing individual patient data for the task, mimicking diagnostic guidelines.
Impliit selection:
Selects three features actively using L.M reasoning each step without Bayesian modelling.

---

Methods
ACTMED
Random
Global Best
Implicit
All
GPT-4--M
1.0
0.9
Accuracy
0.6
0.5
0.4
Diabetes
Kidney Dataset
Hepatitis
Diabetes
Kidney
Hepatitis
Dataset
Figure 4: Model accuracy across methods and datasets. Bars show mean accuracy over five seeds; error bars indicate standard deviation.
Chronic Kidney Disease (CKD). Chronic kidney disease is typically diagnosed using biomarkers such as serum creatinine or glomerular filtration rate (GFR), with well-established clinical thresholds [51]. As a result, this represents relatively straightforward classification task. Both models achieved near-perfect accuracy, even under feature selection constraints, indicating that LLMs possess strong prior knowledge of CKD's clinical presentation. This highlights their potential support diagnostic decision maisiionmmaiing settings where key features are well understond. Consequently. we focus subsequent analysis on more challenging tasks, such as hepatitis and diabetes, where diagnostic ambiguity is higher and performance depends more heavily on effective test. selection. Full performance metries for all datasets and feature selection evaluation are provided in Appendix E.
Hepatitis. This task focuses diagnosing hepatitis C using liver function tests [52]. Unlike CKD, hepatitis often produces subtle and non-specific alterations in blood biomarkers, which can also be influenced by a range of other conditions. In experiments, ACTMED consistently outperformed other feature selection methods across all model types (see Figure 4), Small improvements in GPT-44o compared to GPT-4o-mini are also noted. Interestingly, it even surpassed the full-information baseline, indicating that the targeted selection of informative features improve diagnostic precision. This effect is visualized Figure 5, which shows a clear increase in the average diagnostic accuracy across all datasets using ACTMED additional features are acquired sequentially.
0.700
0.675
0.650
0.625
Model
Number of diagnastic tests
Figure 5: Sequential diagnosis refinement. Accuracy improves additional tests quired; error bars show standard deviation across datasets.
Diabetes. Diagnosing diabetes the challenging the three tasks, as no single definitive test is available
the datasets. Instead, diagnosis must be inferred by synthesizing multiple indirect indicators; such as blood glucose, body mass index (BMI), and blood pressure; to estimate overall disease risk [55]. Across both models, ACTMED consistently outperformed all other baselines, including the full-information setting. Performance further improved more capable GPT-4o model, highlighting the benefit of stronger underlying surrogate models. These results suggest that deliberate, targeted test acquisition not only enhances diagnostic accuracy but also improves gencralization by reducing the influence of irrelevant or misleading features [56].
4.3 Implicit Selection Lacks Personalization
For the datasets where the baseline feature selection by the model performed notably worse than ACTMED, we further analysed the tests selected by cach method by evaluating how frequently models chose features identified as globally optimal prior to observing any patient-specific data (see Figure 6). Random selection serves as a stochastic baseline. Across all model-dataset pairs, implicit selection methods showed a strong bias toward globally optimal features, significantly diabetes dataset, where both models almost exclusively selected globally optimal features, despite ACTMED achieving higher predictive accuracy. These findings suggest that L.LM-based implicit selection may struggle to capture patient- specific uncertainty and adaptively personalize test acquisition, relying instead on prior knowledge about general test utility. A more detailed feature selection analysis performed in Appendix E.
than ACTMED. This effect especially pronounced in the

---

4.4 KL-Based Stopping Minimizes Redundant Testing
Mathod
ACTMED not only provides greater transparency than implicit feature selection methods by generating in- termediatc outputs and utilizing Bayesian test selee- tion, but also introduces a principled stopping crite- rion for diagnostic testing based on the expected shift
ACTMED
MarhhBata
0.8
0.6
in the posterior distribution, measured by KL diver- gence. A representative example in Table 2 shows that KL divergence consistently decreases across se- lection rounds, naturally allowing the stopping crite- rion to trigger when the expected informational gain hecomes negligible. We evaluated this criterion us- ing thresholds 7 E {0.3, 0.5, 0.7), which represent varying levels of evidence required to continue test- ing. Baselines based on implicitly selected features or global optima did not allow early stopping and had access to all three selected tests. Our KL-based stopping method achieved superior accuracy with sig- nificantly fewer diagnostic steps (sec Figure 7), As expected, higher values of y reduce the number of diagnostic tests selected. At the conservative threshold (7 0.6), it reduced the average number of tests to under two across all datasets and models, except for the hepatitis dataset with GPT-4o. The method reduces overall diagnostic burden by nearly 50% while providing comparable or superior accuracy compared to baseline feature selection (see Table 3).
0.2
0.0
Hepostss
Diabetes
Osce
Hepatitis Diaiettes Kidnyy Osce Dataset
Dataset
Figure 6: Feature selection analysis. Average se- lection frequency and variability across five ran- dom seeds are compared against the three globally optimal features. The implicit model consistently favours these global features with low variability, indicating limited personalization across patients.
4.5
Wiinicin--in-the-loy evaluation
We conducted diiiccan-in-th---oe evaluation involving three experi- enced clinicians and two senior medical students, assessing a total of 450 diagnostic test decisions (10 simulated diagnostic traces across the three datasets, each reviewed by five evaluators with three decisions per trace). Experts judged ACTMED's 's test selections and resulting risk adjustments clinically reasonable in 94.5 = 1.4% of cases, underscoring its reliability as a decision-support tool. Clinicians emphasized that they would be reluc- tant to trust diagnostic suggestions from a purcly black-box LLM without transparent reasoning. In contrast, ACTMED's step-by-step decision process was repeatedly praised for fostering confidence in the system's 's reasoning.
Table 2: Representative fea- ture selection rounds from the Hepatitis C dataset. Darker blue indicates higher KL diver- gence.
Selection
Selection
Selection
ALB: 0,065
ALB: 0.014
ALa: 0.016
ALP. 0,041
ALP: 0,005
Several clinicians remarked that ACTMED's 's Bayesian decision framework closely reflects their own reasoning process when selecting diagnostic tests under uncertainty, particularly in situations lacking explicit clinical guide- lines. They also noted instances where real-world practice departs from a purely Bayesian rationale- for example, when standardized protocols pre- scribe specific test (such as PCR for COVID-19 diagnosis) or when certain screening procedures are routinely performed despite limited diagnostic yield (e.g., broad cancer panels). Nonetheless, participants emphasized that decision-suppon systems are most valuable in non-trivial clinical scenarios, where genuine uncertainty exists and clear diagnostic pathway defined.
ASTO 0.115
BAL: 0.191
B1LL 0.025
B1LL: 0,012
CHE 0.137
CH2 0.016
CEEB 0,015
DI2L: 0.124
C10L: 0.522
CINLL: 0.016
6am (11 18
B0m 0.005
BGm 0.015
4.6 OSCE-style evaluation
The three datasets discussed above each represent single-diagnosis task. While this setting is well-suited to analyze ACTMED's behaviour in controlled diagnostic contexts such as test selection, it does not fully capture the diversity of clinical reasoning required practice. To evaluate ACTMED's performance in more realistic scenario, we constructed a custom OSCE-style dataset based on cases from the AgentClinic tramework [54]. We selected 114 representative cases with sufficiently rich test results and generated corresponding synthetic negatives by adjusting laboratory values toward physiological ranges that make the diagnosis unlikely. This allowed us assess both diagnostic discrimination and generalization under realistic clinical variability. Across these multi- malii-diagnosis settings, ACTMED continued to outperform baseline methods, maintaining superior calibration and diagnostic accuracy (see Figure 4). These results suggest that ACTMED's 's sequential Bayesian reasoning scales effectively to diverse and complex diagnostic tasks, further supporting its potential for integration into clinical assessment frameworks such as OSCEs.
8

---

5 Discussion
BED improves LLM-based clinical diagno- sis. ACTMED, integrating Bayesian Experimental Design (BED) decision-making with large language models (LLMs), outperforms naive LLM- fea- ture selection methods in accuracy, cost-awareness, personalization, and interpretability across several datasets. In the hepatitis and diabetes datasets even surpasses the baseline model that uses all avail- able features. While this may seem counter-intuitive, it aligns with traditional machine learning findings where feature selection reduces noise and over- fitting [57]. Implicit sequential and global selection strate- gies provide strong baselines but occasionally fail to identify the most informative features. We hy- pothesize this arises from the model's difficulty in accurately representing how test results influence di-
ACTMSD
0P5404851 C7540
65
64
Figure 7: KL-based early stopping criterion eval- uation. Bars represent the mean accuracy across seeds, with error bars indicating standard devia- tion.
Dacasit
Secaett
agnostic risk due to limited Comain-specific data. By shifting reasoning from the input space to the solution space, our framework enables better probabilistic inference for test selection, improving decision-making under resource constraints and leveraging LLMs' strength in contextual prediction while compensating for their limitations in Bayesian reasoning.
Comparisons to other diagnostic frameworks. Conventional clinical diagnosis relics on rule-based, population-derived guidelines that generalize well but often lack personalization and may delay detection, especially in asymptomatic patients [58]. While machine learning (MI. and deep learning (DL. approaches promise earlier detection, they are typically spec-spe,, lack individualized reasoning. and offer limited transparency. Even interpretability techniques such as attention maps fall short in addressing the broader need for explainable and collaborative decision-makine making. Recent efforts have begun applying LLMs to clinical diagnosis, but naive implementations have proven inadequate [38], particularly in our study, where LLMs struggled with personalization and had limited ability to assess the effectiveness of candidate tests. Our framework addresses these challenges by leveraging the generative strengths of LLMs within a BED framework. This enables explicit reasoning under uncertainty, personalized diagnostic pathways, and principled test selection based on expected information gain. Importantly, it also supports transparency by incorporating clinicians in the process [39]. A summary comparison of capabilities is presented in Table 4.
Clinical relevance. Our frame- work's key advantage over standard end-to-end classifiers is its ability to involve clinicians directly in the diagnostic process. At each deci- sion step, clinicians can review sim- ulated test outcomes and assess their impact on diagnostic probabilities (see Appendix E). The framework issues recommendations rather than fixed decisions, allowing clini- cians to override suggestions based on context and re-query with alter- native test results. While capable of providing final discase prediction, the model serves primarily a decision-support tool, leaving diagnostic judgment to the clinician. By highlighting the most informative tests, it streamlines clinical reasoning and reduces cognitive burden. We emphasize that current LLMs are not yct ready for direct autonomous diagnosis but may be better suited as decision-support tools, assisting clinicians in structuring and refining the diagnostic process. Importantly, safety can be enhanced by constraining the model to recommend only tests that clinically approved for the suspected conditions, ensuring alignment with established diagnostic pathways and regulatory guidelines.
Table 3: Average number of tests selected under KL-based termination for varying values. Higher y requires stronger evidence to continue testing. Results are reported as mean + stan- dard deviation across five random seeds.
Model
Hepatitis
Diabetes
Kidney
0.3
2.36 + 0.68
2.45 = 0.71
1.60 + 0.75
GPT-40-MINI
0.5
1.87 = 0.77
2.09 = 0.75
1.48 + 0.68
0.7
1.48 + 0.73
1.90 = 0.74
1.28 + 0.52
0.3
2.67
+ 0.60
2.27 = 0.84
1.35 + 0.64
GPT-40
0.5
2.41 + 0.67
1.90 = 0.89
1.09 + 0.35
0.7
2.27 = 0.65
1.69 = 0.84
1.04 + 0.24
Limitations. Our evaluation is currently constrained by dataset scale and scope. The combined sample size across all datasets approximately 1,000 patients, with limited number of covariates per condition. Each task is framed as a binary diagnostic decision, though the framework is applied across multiple distinct diseases rather than within a single multi-label setting. Extending ACTMED to larger, more heterogeneous datasets with richer feature spaces and overlapping comorbidities would better reflect the complexity of real-world clinical diagnosis.

---

Table 4: Capability matrix for clinical reasoning approaches. Only our proposed framework satisfies all key criteria for transparent, timely diagnosis under resource constraints.
Methed
Timeh Diagaoss
Personalized Reasoning
Reworre- Constraine
Zero-Shot Geocraliation Unstractured Data Transparent Explanatioos
Rule- Systems
Clastcal ML
Deep Leaning
Nave
Ours (ACTKED)
/
/
Our framework is also currently limited to binary classification. Future work should extend evaluation multi-label datasets and co-morbidities for broader clinical applicability. The diagnostic process considers only categorical and numerical features, with free-text outputs (e.g.. imaging or pathology reports) not yet incorporated, though structured representations could be included. While LLMs can interpret free-text, this adds complexity beyond our current structured setting. Additionally, our method requires more LLM queries than simpler heuristics, which may pose challenges in couurceeoonttriieed environments. However, in high stakes domains such as healthcare, the added computational cost is justified by improved decision-making, reduced uncertainty, and fewer unnecessary tests.
ACTMED also depends on strong, high-capacity L.Ms to generate physiologically coherent feature distributions and uncertainty-aware predictions. Simpler models fail to reproduce accurate surrogate samples or stable posterior estimates, underscoring the current reliance foundation-scale models. LLMs often criticized for their black-box nature and susceptibility to hallucinations. While these issues persist, our framework mitigates them by generating interpretable intermediate outputs, such as sampled feature distributions and uncertainty- calibrated diagnostic probabilities, thereby improving transparency in the reasoning process. Biases in model behavior also remain concern, particularly for under-represented patient subgroups. By explicitly exposing how test results influence posterior diagnostic beliefs, clinicians can better identify and assess such biases in real time, enabling oversight and corrective action when needed.
Clinical diagnosis generally follows a two-step process [19]: (1) an initial assessment, where the clinician formulates a primary diagnosis and a list of differential diagnoses, and (2) targeted testing phase to confirm or refute these hypotheses. Our implementation of ACTMED addresses the second step by optimizing test selection across individual diseases, independent of how the differential list is generated. This one-vs vs-al formulation mirrors established clinical reasoning practices such as the Wells score for deep vein thrombosis [59] and the CH22DS2- VASc score for atrial abrillation-relatte separately. While ACTMED can in principle he extended to sequentially handle multiple conditions, our present experiments focus exclusively binary discase-speciff tasks and do not include multi-label datasets.
stroke [60] where each disease likelihood is evaluated
Conclusions and Impact. We present ACTMED, prohabilistic framework for clinical diagnosis that uses sequential, teemmation-theorttii decision- diagnostic models, our approach actively selects and interprets tests to inform serving as flexible surrogate simulators that predict test outcomes and update beliefs without task-specific training. The framework is model-agnostic. its performance depends on the fidelity of the surrogate model rather than any particular LLM architecture. Beyond improving diagnostic accuracy, ACTMED demonstrates how probabilistic reasoning and large-scale language models can be integrated to support data-cfficient, uncertainty- aware clinical decision-making As LLMs generative models continue advance, frameworks like ACTMED could enable clinician-in- the-loop systems that optimize diagnostic efficiency, improve interpretability. and mitigate resource constraints in healthcare. Future research should benchmark range of model architectures and training paradigms quantify their impact on diagnostic reasoning, extend the framework to multimodal and longitudinal data.
refine beliefs about disease states. Unlike traditional
maaiiinnnmaking.... with LLMs
6 Acknowledgments and Disclosure of Funding
We thank the anonymous NeurlPS reviewers, members of the der Schaar Lab, and Andrew Rashbass for their insightful comments and suggestions. N.A. gratefully acknowledges support from the W.D. Armstrong Trust. This work was supported by the Microsoft Al for Good Research Lab, including Azure sponsorship credits.
7 Reproducibility
The code and datasets to reproduce the main findings this paper are available under https: : http//ggttttuhccc com/ Sr933/actt /acteed.
10

---

References
[1] A. Jutel, "Sociology of diagnosis: A preliminary review." Sociology of Health &amp; Illness, vol. 31, no. 2. pp. 278-299, Feb. 2009, ISSN: 1467-9566.
[2] Z. I. Vally, R. A. Khammissa, G. Feller, R. Ballyram, M. Beetge, and L. Feller, "Errors in clinical diagnosis: A narrative review," Journal of International Medical Research, vol. 51,
no. 8. Aug. 2023, ISSN: 1473-2300.
[3] D. E. Newean-Toker, N. Nassery, A. C. Schaffer, et al., "Burden of serious harms from diagnostic error in the usa," BMJ Quality &amp; Safety, vol. 33, no. 2, pp. 109-120, Jul. 2023, ISSN: 2044-5423.
[4] J. J. M. Müskens, R. B. Kool, S. A. van Dulmen, and G. P. Westert, "Overuse of diagnostic testing healthcare: A systematic review," BMJ Quality &amp; Safety, vol. 31, no. pp. 54-
63, May 2021, ISSN: 2044-5423,
[5] WHO, "No health without a workforce," World Health Organisation (WHO) report, vol. 2013,
pp. 1-104, 2013.
[6] J. Hällqvist, M. Bartl, M. Dakna, et al., "Plasma proteomics identify biomarkers predicting parkinson's disease up to 7 years before symptom onset," Nature Communications, vol. 15, no. 1, Jun. 2024, ISSN: 2041-1723.
[7] S. Fouladvand, F. R. Gomez, H. Nilforoshan, et al., "Graph- based clinical recommender. Pre- dicting specialists procedure orders using graph representation learning," Journal of Biomedical Informatics, vol. 143, p. 104 407, Jul. 2023, ISSN: 1532-0464.
[8] M. Khalifa and M. Albadawy, "Artificial intelligence for clinical prediction: Exploring key domains and essential functions," Computer Methods and Programs in Biomedicine Update, vol. 5, p. 100 148, 2024, ISSN: 2666-9900.
[9] S. A. Alowais, S. S. Alghamdi, N. Alsuhebany, et al., "Revolutionizing healthcare: The role of artificial intelligence clinical practice," BMC Medical Education, vol. 23, no. 1, Sep. 2023, ISSN: 1472-6920.
[10] M. Liu, S. Li. H. Yuan, et al., "Handling missing values in healthcare data: A systematic review of deep scarning-basee imputation techniques," Artificial Intelligence in Medicine, vol. 142,
p. 102 587, Aug. 2023, ISSN: 0933-3657.
[11] L. Tikhomirov, C. Semmler, M. McCradden, R. Searston, M. Ghassemi, and L. Oakden- Rayner, "Medical artificial intelligence for clinicians: The lost cognitive perspective." The Lancet Digital Health, vol. 6, no. 8, c589-e594, Aug. 2024, ISSN: 2589-7500.
[12] P. G. Shekelle, S. H. Woolf, M. Eccles, and J. Grimshaw, "Clinical guidelines: Developing guidelines," BMJ, vol. 318, no. 7183, pp. 593-596, Feb. 1999, ISSN: 1468-5833.
[13] S. Misra and H. Barth, "How good is the evidence base for test selection in clinical guide-
lines?" Clinica Chimica Acta, vol. 432, pp. 27-32, May 2014, ISSN: 0009-8981.
[14] C. Koch, K. Roberts, C. Petruccelli. and J. Morgan, "The frequency of unnecessary testing in hospitalized patients." " The American Journal of Medicine, vol. 131, no. 5, pp. 500-503, May 2018, ISSN: 0002-9343.
[15] D. J. Erwin, C. LaMaire, A. Espana,
Eble, and S. U. Dhar, "Financial barriers in county genetics clinic: Problems and solutions," Journal of Genetic Counseling, vol. 29, no. 4, pp. 678 678-688, Apr. 2020, ISSN: 1573-3599.
[16] M. M. Islam, T. N. Poly, H.-C Yang, and Y.-C, (Li, "Deep into laboratory: An artificial
intelligence approach to recommend laboratory tests," Diagnostics, vol. 11, no. 6, p. 990, May 2021, ISSN: 2075-4418.
[17] H. Lakkaraju, D. Slack, Y. Chen, C. Tan, and Singh, "Rethinking explainability as a dialogue:
A practitioner's perspective," NeurIPS Workshop Human Centered AI, [18] Z. Sadeghi, R. Alizadehsani, M. A. CIFCI, et al., "A review of explainable artificial intelligence
in healthcare," Computers and Electrical Engineering, vol. 1 118, p. 109 370, Aug. 2024, ISSN: 0045-7906.
[19] E. P. Balogh, B. T. Miller, and J. R. Ball, Eds., Improving Diagnosis in Health Care. National Academies Press, Dec. 2015, ISBN: 9780309377690.
[20] D. Crosby, S. Bhatia, K. M. Brindle, et al., "Early detection of cancer," Science, vol. 375. no. 6586, Mar. 2022, ISSN: 1095-9203.

---

[21] A. Kumar, D. Roberts, K. E. Wood, et al., "Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock*." Critical Care Medicine, vol. 34, no. 6. pp. 1589-1596, Jun. 2006, ISSN: 0090-3493.
[22] T. Schubert, R. W. Peck, A. Gimson, C. Davtyan, and M. van der Schaar, A foundational framework and methodology for personalized early and timely diagnosis, 2023.
[23] Y. Tang. Y. Zhang, and J. Li, "A time series driven model for early sepsis prediction based on transformer module," BMC Medical Research Methodology, vol. 24, no. 1, Jan. 2024, ISSN: 1471-2288.
[24] D. J. Park, M. W. Park, H. Lee, Y.J. Kim, Y. Kim, and Y. H. Park, "Development of machine learning model for diagnostic disease prediction based on laboratory tests," Scientific Reports, vol. 11, no. 1, Apr. 2021, ISSN: 2045-2322.
[25] P. G. Mikhael, J. Wohlwend, A. Yala, et al., "Sybil: A validated deep learning model to predict future lung cancer risk from single low-dose chest computed tomography," Journal of Clinical Oncology, vol. 41, no. 12, pp. 2191-2200, Apr. 2023, ISSN: 1527-7755.
[26] X. Liu, L. Faes, A. U. Kale, et al., "A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: A systematic review and meta-analysis," The Lancet Digital Health, vol. 1, no. 6, e271 Oct. 2019, ISSN: 2589-7500.
[27] A. M. Alaa and M. van der Schaar, "Attentive state-space modeling of disease progression." in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelz- imer, F. d' Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32, Curran Associates, Inc., 2019.
[28] A. M. Alaa, J. Yoon, S. Hu, and M. van der Schaar, "Personalized risk scoring for critical care prognosis using mixtures of gaussian processes," " IEEE Transactions on Biomedical Engineering, vol. 65, no. 1, pp. 207-218, Jan. 2018, ISSN: 1558-2531.
[29] J. Jiang. W. Yang. E. M. Schnellinger, E. Kimmel, and W. Guo, "Dynamic logistic state space prediction model for clinical decision making," Biometrics, vol. 79, no. 1, pp. 73-85,
Nov. 2021, ISSN: 1541-0420,
[30] A. M. Alaa and M. van der Schaar, "Balancing suspense and surprise: Timely decision making with endogenous information acquisition," in Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, Eds., vol. 29, Curran Associates, Inc., 2016.
[31] H. von Kleist, A. Zamanian, I. Shpitser, and N. Ahmidi, "Evaluation of active feature acqui- sition methods for time-varying feature settings," " Journal of Machine Learning Research, vol. 26, no. 60, pp. 1-84, 2025.
[32] J. Clusmann, F. R. Kolbinger, H. S. Muti, al., "The future landscape of large language models in medicine," " Communications Medicine, vol. 3, no. 1, Oct. 2023, ISSN: 2730-664X.
[33] K. Singhal, S. Azizi, T. Tu, et al., 'Large language models encode clinical knowledge," Nature, vol. 620, no. 7972, pp. 172-180, Jul. 2023, ISSN: 1476-4687.
[34] D. Brin, V. Sorin, E. Konen, G. Nadkarni, B. S. Glicksberg, and E. Klang, "How gpt models perform on the united states medical licensing examination: A systematic review," Discover Applied Sciences, vol. 6, no. 10, Sep. 2024, ISSN: 3004-9261.
[35] H. Zong, R. Wu, J. Cha, al., "Large language models in worldwide medical exams: Platform development and comprehensive analysis," Journal of Medical Internet Research, vol. 26, e66114, Dec. 2024, ISSN: 1438-8871.
[36] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, "Large language models zero- shot reasoners," in Proceedings of the 36th International Conference on Neural Information Processing Systems, ser. NIPS $22, New Orleans, LA, USA: Curran Associates Inc., 2022, ISBN: 97817778887 17211711871000 1088.
[37] H. Meshkin, J. Zirkle, G. Arabidarrehdor, et al., "Hamessing large language models' zero-shot and few- shot learning capabilities for regulatory research." " Briefings in Bioinformatics, vol. 25, no. 5. Jul. 2024, ISSN: 1477-4054.
[38] P. Hager, F. Jungmann, R. Holland, et al., "Evaluation and mitigation of the limitations of large language models in clinical decision-making Nature Medicine, vol. 30, no. 9. pp. 2613-2622, Jul. 2024, ISSN: 170X.
[39] T. Savage, A. Nayak, R. Gallo, F. Rangan, and J. H. Chen, "Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine," npj Digital Medicine vol. 7, no. 1, Jan. 2024, ISSN: 2398-6352.
12

---

[40] L. Qiu, F. Sha, K. Allen, Y. Kim, T. Linzen, and S. van Steenkiste, Bayesian teaching enables probabilistic reasoning in large language models, 2025.
[41] X. Chen, H. Yi, M. You, et al., "Enhancing diagnostic capability with multi-agents con- versational large language models," npj Digital Medicine, vol. 8, no. 1, Mar. 2025, ISSN: 2398-6352.
[42] T. Liu, N. Astorga, N. Seedat, and M. van der Schaar, "Large language models to enhance bayesian optimization." " in The Twelfth International Conference on Learning Representations. 2024.
[43] T. Liu, N. Huynh, and M. van der Schaar, "Decision tree induction through LLMs via remantically-awan evolution," " in The Thirteenth International Conference on Learning Repre- sentations, 2025.
[44] N. Huynh, K. Kacprzyk, R. M. Sheridan, D. L. Bentley, and M. van der Schaar, "Decision tree induction with dynamic feature generation: A framework for interpretable DNA sequence analysis," in Learning Meaningfua Representations of Life (LMRL) Workshop ICLR 2025, 2025.
[45] K. Kobalczyk, N. Astorga, T. Liu, and M. van der Schaar, Active task disambiguation with Ilms, 2025,
[46] C. Papadimitriou, "Optimal sensor placement methodology for parametric identification of structural systems," Journal of Sound and Vibration, vol. 278, 4-5, pp. 923-947, Dec. 2004, ISSN: 0022-460X.
[47] S. Dushenko, K. Ambal, and R. D. McMichael, optically detected magnetic resonance of nitrogen- vacancy centers," Physical Review Applied, vol. 14, no. 5, Nov. 2020, ISSN: 2331-7013
"Seqiuentl "Sequential bayesian experiment design for
[48] B. Shababo, B. Paige, A. Pakman, and L. Paninski, "Bayesian inference and online experimen- tal design for mapping neural microcircuits," in Advances in Neural Information Processing Systems, C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, Eds., vol. 26, Curran Associates, Inc., 2013.
[49] A. White, M. Tolman, H. D. Thames, H. R. Withers, K. A. Mason, and M. K. Transtrum, "The limitations of model-based experimental design and parameter estimation sloppy systems," " PLOS Computational Biology, vol. 12, no. 12, A. Csikász-Nagy, Ed., e1005227, Dec. 2016, ISSN: 1553-7358.
[50] N. Makarov, M. Bordukova, R. Rodriguez Esteban, F. Schmich, and M. P. Menden, "Large language models forecast patient health trajectories enabling digital twins." Jul. 2024.
[51] A. Francis, M. N. Harhay, A. C. M. Ong, et al., "Chronic kidney disease and the global public health agenda: An international consensus," Nature Reviews Nephrology, vol. 20, no. 7, pp. 473-485, Apr. 2024, ISSN: 1759-507X.
[52] N. Abu-Freha, B. Mathew Jacob, A. Elhoashla, et al., "Chronic hepatitis c: Diagnosis and treatment made easy." European Journal of General Practice, vol. 28, no. 1, pp. 102-108. May 2022, ISSN: 1751-1402.
[53] M. Martinello, S. S. Solomon, N. A. Terrault, and G. J. Dore, "Hepatitis e," The Lancet, vol. 402, no. 10407, pp. 1085-1096, Sep. 2023, ISSN: 0140-6736.
[54] S. Schmidgall, Ziaei, C. Harris, E. Reis, Jopling, and M. Moor, Agentclinic: A maltimodal agent benchmark to evaluate ai in simulated clinical envirouments, 2024. arXiv: 2405.07960 [cS.HC]
[55] K. L. Ong, L. K. Stafford, S. A. McLaughlin, et al., "Global, regional, and national burden of diabetes from 1990 to 2021, with projections of prevalence to 2050: A systematic analysis for the global burden of disease study 2021," " The Lancet, vol. 402, no. 10397, pp. 203-234, Jul. 2023, ISSN: 0140-6736.
[56] G. Chatziveroglou, R. Yun, and M. Kelleher, Exploring Um reasoning through controlled prompt variations, 2025.
[57] N. Pudjihartono, T. Fadason, A. W. Kempa-Liehr, and J. M. O'Sullivan, "A review of feature selection methods for machine learning-basee based disease risk prediction," Frontiers in Bioinfor- matics. vol. 2, Jun. 2022. ISSN: 2673-7647.
[58] WHO, "Guide cancer early diagnosis," World health Organisation (WHO) report, 2017.
13

---

[59] D. Anderson, M. Rodger, J. Ginsberg, et el., "Derivation of simple clinical model to categorize patients probability of pulmonary embolism: Increasing the models utility with the simplified d-dimer," Thrombosis and Haemostasis, vol. 83, no. 03, pp. 416-420, 2000, ISSN: 2567-689X.
[60] J. B. Olesen, G. Y. H. Lip, M. L. Hansen, et al., "Validation of risk stratification schemes for predicting stroke and thromboembolism in patients with atrial fibrillation: Nationwide cohort study," BMJ, vol. 342, no. jan31 1, pp. 1114-d1244 Jan. 2011, ISSN: 1468-5833.
[61] H. Nori, Y. T. Lee, S. Zhang, al., Can generalist foundation models outcompete special- purpose tuning? case study in medicine, 2023.
[62] A. Toma, P. R. Lawler, J. Ba, R. G. Krishman, B. B. Rubin, and B. Wang, Clinical camel: An open expert-level medical language model with dialogue-based knowledge encoding, 2023, [63] J. Wu, W. Deng. X. Li, et al., Medreason: Eliciting factual medical reasoning steps llms via
knowledge graphs, 2025.
[64] E. Goh, R. Gallo. J. Hom, et al., "Large language model influence on diagnostic reasoning: A randomized clinical trial," JAMA Network Open, vol. 7. no. 10, e2440969, Oct. 2024, ISSN: 2574-3805.
[65] Y. Ling, S. A. Hasan, V. V. Datla, et al., "Learning to diagnose: Assimilating clinical narratives using deep reinforcement learning," in IJCNLP(1), 2017, pp. 895-905,
[66] 7. Yu, Y. Li, J. C. Kim, K. Huang, Y. Luo, and M. Wang, "Deep reinforcement learning for cost-effective medical diagnosis," in The Eleventh International Conference on Learning Representations, 2023.
[67] C. Zhong, K. Liao, W. Chen, et al., "Hierarchical reinforcement learning for automatic disease diagmosis," " Bioinformatics, vol. 38, no. 16, J. Wren, Ed., pp. 3995-4001, Jul. 2022, ISSN: 1367-4811.
[68] G. Mulier, S. Chevret, R. Lin, and L. Biard, "Bayesian optimal designs for multi-arm multi- stage phase ii randomized clinical trials with multiple endpoints," Statistics in Biopharmaceu- tical Research, vol. 16, no. pp. 315-325, May 2024, ISSN: 1946-6315.
[69] A. Giovagnoli, "The bayesian design of adaptive clinical trials." International Journal of Environmental Research and Public Health, vol. 18, no. 2, p. 530, Jan. 2021, ISSN: 1660-460)
[70] P. Nguyen, A. Rathod, D. Chapman, el al., "Active semi-supervised learning via bayesian experimental design for lung cancer classification using low dose computed tomography scans," Applied Sciences, vol. 13, no. 6, p. 3752, Mar. 2023, ISSN: 2076-3417.
[71] A. Ananda Rao, M. Awale, and S. Davis, "Medical diagnosis reimagined a process of bayesian reasoning and elimination," Cureus, Sep. 2023, ISSN: 2168-8188 8184.
[72] J. Kornak and Y. Lu, "Bayesian decision analysis for choosing between diagnostic/prognosiic prediction procedures," " Statistics and Its Interface, vol. 4, no. 1. pp. 27-36. 2011, ISSN: 1938-7997.
[73] M. Kukar and C. Grošelj, "Machine learning in stepwise diagnostic process," in Artificial Intel- ligence in Medicine. Springer Berlin Heidelberg, 1999, pp. 315-325, ISBN: 9783540487203.
[74] O. Alagoz, T. Ayer, and F. S. Erenay, Operations research models for cancer screening. Jan. 2011.
[75] K. Ahuja, W. Zame, and M. van der Schaar, "Dpscreen:
Dynamic personalized screening," in Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, et al., Eds., vol. 30, Curran Associates, Inc., 2017.
[76] O. Ginsburg, C.-H. Yip, A. Brooks, et al., "Breast cancer early detection: A phased approach to implementation," Cancer, vol. 126, no. S10, pp. 2379-23933 Apr. 2020, ISSN: 1097-0142.
[77] N. Fisemann, S. Bunk, T. Mukama, el al., "Nationwide real-world implementation of ai for cancer detection in population-basee mammography screening," " Nature Medicine, vol. 31,
no. 3. pp. 917-924, Jan. 2025, ISSN: 1546-170X.
[78] B. Settles, "Active learning literature survey." University of Wisconsin Computer Sciences Technical Report 1648, 2009.
[79] J. Li, L. Wu. H. Dani, and H. Liu, "Unsupervised personalized feature selection." " Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, Apr. 2018, ISSN: 2159-5399.
[80] Y. Gal, R. Islam, and Z. Ghahramani, "Deep Bayesian active learning with image data," " in Proceedings of the 34th International Conference on Machine Learning, D. Precup and Y. W. Teh, Eds., ser. Proceedings of Machine Learning Research, vol. 70, PMLR, Aug. 2017,
pp. 1183-1192.
14

---

[81] N. Astorga, T. Liu, N. Seedat, and M. van der Schaar, "Partially observable cost-aware active- learning with large language models," in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[82] D. Reker, "Practical considerations for active machine learning in drug discovery," Drug Discovery Today: Technologies, vol. 32-33, pp. 73-79, Dec. 2019, ISSN: 1740-6749.
[83] J. Piskorz, N. Astorga, J. Berrevoets, and M. van der Schaar, "Active feature acquisition for personalised treatment assignment," The 28th International Conference on Artificial Intelligence and Statistics, 2025.
[84] L. Rubini, P. Soundarapandian, and P. Eswaran, Chronic Kidney Diseuse, UCI Machine Learning Repository, DOI: http://dbi.rrg/1022433///
2015.
[85] R. Lichtinghagen, F. Klawonn, and G. Hoffmann, HCV data, UCI Machine Learning Reposi- tory, DOI: https://doi.org/// 242244/3///C 2020.
[86] J. Smith, Everhart, W. Dickson, W. Knowler, and R. Johannes, "Using the adap learning algorithm to forecast the onset of diabetes mellitus," Proceedings Annual Symposium on Computer Applications in Medical Care, vol. 10, Nov. 1988.
[87] D. B. Rubin, "The bayesian bootstrap." " The Annals of Statistics, vol. 9, no. 1, Jan. 1981, ISSN: 0090-5364.
[88] S. Im, J. Oh, and E. Choi, Labtop: A unified model for lab test outcome prediction on electronic health records, 2025.
15

---

Supplementary Material
A Extended related works
LLMs and Reinforcement Learning for Clinical Diagnosis. There has been growing interest in validating large language models (LLMs) on clinical diagnosis tasks, alongside efforts to fine-tune these models for biomedical datasets. Interestingly, recent work has shown that general-purgy purpose LLMS can sometimes outperform specialized models even on domain-specific henchmarks [61]. To improve clinical reasoning, some approaches have involved training LLMs on medical dialogues [62], while others have incorporated structured knowledge sources such medical knowledge graphs, particularly for tasks resembling medical licensing exams [63]. Beyond static evaluation, researchers have explored interactive diagnostic frameworks that allow LLMs to collaborate with clinicians. For instance, agentic networks of LLMs have been shown to enhance diagnostic reasoning compared to individual models, though their effectiveness strongly depends on the quality of the underlying language model [41]. Conversely, simply providing clinicians with access to unrefined LLMs has not yielded significant improvements in diagnostic performance [64].
Reinforcement learning (RL) has also heen proposed as a way to teach models elinically grounded decision- making hehaviours. Farly work by Ling et al. demonstrated how deep reinforcement learning can he used to learn diagnostic reasoning directly from clinical narratives [65]. Subsequent studies have extended this paradigm to cost-sensitive and hierarchical frameworks, where the model learns to balance diagnostic accuracy against the cost of additional tests or procedures [66]. [67]. Such approaches typically require substantial sask-specific training and supervision, which can limit generalization to new diagnostic settings. contrast, ACTNED combines Bayesian Experimental Design with LLM-based generative priors to simulate plausible test outcomes at inference time. This enables it to operate without any task-specific retraining, while still providing adaptive and interpretable test selection. We therefore view DRL-based diagnostic agents complementary approach.
Bayesian methods in medicine
Bayesian methods have long been influential in clinical research, particularly for optimizing trial design, such as adaptive patient allocation and dose-finding strategies in early-phase drug de- velopment [68]. [69]. In medical imaging. Bayesian active learning approaches have been used to strategically acquire informative data points, improving model efficiency and performance [70]. More broadly, the diagnostic process itself is naturally aligned with Bayesian reasoning, where clinical evidence incrementally updates probabilistic beliefs about possible conditions [71].
diagnostic settings, Bayesian approaches have been specifi- cally applied to optimize test selection by leveraging the known sensitivity and specificity of various diagnostic tools [72]. Early foundational work in this domain illustrated the potential of step- wise Bayesian integration test results, where each successive diagnostic query is strategically chosen based on its expected di- agnostic value [73]. These methods effectively utilize probabilis- tie reasoning over potential test outcomes, exploiting the inherent accuracy profiles of available tests make efficient decisions. Furthermore, prior work on the development of timely diagnosis tools has shown significant benefits in applying structured ap- proaches to population-base basedd screening efforts [74], [75]. Such diagnosis of asymptomatic patients is critical as it enables ear- lier intervention, which has been consistently shown improve treatment outcomes for various discases [22], [76]. Machine learning approaches have further enhanced the effectiveness of this screening in specific diseases, including breast and lung cancer [25], [77].
A-6 aedioearode
The mast intrrmtri
degrests
What stulli 3o mard?
Figure 8: Example of a ML assistant guiding a clinician through sequential diagnostic refinement for suspected chronic kidney disease (CKD) case. The agent recommends the most informative next test and iteratively updates its sug- gestions as new results arrive.
Despite these valuable applications and advances, a unified Bayesian framework that fully supports the entire sequential decision making process inherent in clinical diagnosis, from the initial formation of hypotheses to the targeted acquisition and sophisticated integration of heterogeneous clinical data, remains largely underdeveloped. Our current work directly addresses this critical gap by demonstrating how large language models (LLMs) effectively serve powerful surrogates for Bayesian Experimental Design, thereby providing novel
16

---

computational approach to support and improve real-world diagnostic tasks by facilitating this comprehensive sequential reasoning and data integration process.
Active learning in medicine
Active learning subfield of machine learning in which models strategically select the most informative data points to observe, thereby minimizing need for costly human labelling [78]. has been successfully applied in both unsupervised settings to identify informative features [79] and in conjunction with deep learning architectures to improve data efficiency [80]. Central to active learning is the definition of acquisition functions data. Large language models (LLMs)
utility metries that quantify the expected benefit of observing have recently been explored as surrogate models for guiding such selection, particularly in settings where data acquisition expensive or sparse [81]. hiomedicine, active learning has heen applied to optimize test ordering and experimental design, where the cost of procedures is often limiting factor. For example, in pharmacology. it has been used to prioritize compound testing [82], and in clinical treatment planning, active feature selection methods have enabled for active test selection the context of sequential clinical diagnosis remains unexplored. Figure illustrates how such an ML-based framework can support clinicians by optimizing the selection of diagnostic tests to sequentially refine the diagnosis.
efficient personalization of care [83]. Despite these advances. the use of LLMs
B Extended works
Summary of formalism
summary of the mathematical formalism introduced is given in Table 5,
Table 5: Summary notation used in the agent-based based diagnosis and Bayesian experimental design
framework.
Symbol
Definition
Σ
Space of natural language strings
SEE
Natural language input or instruction to the agent
D
Set of all possible diagnoses at time f
Dine.t C Dt
Subset true diagnoses for a patient at time Single diagnosis considered time t Discrete time horizon for the decision process Ground-trute information available at time t Known (observed) information at time Unknown (unobserved) information at time t
de E Dr
T= (1,2, Tmax)
X. E
K. C X
U 4= Xi Ke
Random variable representing candidate diagnostic test at time Sampled realization j from candidate test
(W P(U(
a
Observed (true) outcome of test i time
P(ya,
K2)
Posterior belief over diagnosis de given current knowledge K₁ Shannon entropy of probability distribution
(P())
(I
Expected information gain of test i
Pptior
Prior belief distribution before test observation Posterior belief distribution after observing test result Kullback -Baeekk-Liiiee Leibler divergence between two distributions and p Number of Monte Carlo samples used to estimate expectations
Prost
KL(a P)
M
F(
Utility or information -thooretic value of diagnostic test
Cost associated with performing test (:(0)
Trade-off parameter weighting diagnostic accuracy vs. test cost Lagrangian objective combining diagnostic error and cumulative test cost
L(y, A)
17

---

Formulation of the Bayesian Experimental Design (BED) Objective using KL Divergence
Epistemic vs. Aleatoric Uncertainty in Active Learning. In standard Active Learning (AL), which represents common instance of BED, the goal reduce epistemic uncertainty by querying the label of the most informative observation To. The expected information gain can be written
1(Y; 1 2 2o. d)
HY d, x.) H(Y L, d, 2.),
(3)
epistemic
alcutoric
where Z is random variable representing the latent hypothesis model parameters, d denotes the currently observed data, (-) Shannon entropy, and (-) denotes mutual information.
The first term quantifies the total predictive uncertainty, while the second term captures irreducible (aleatoric) uncertainty given the true latent model. Their difference isolates epistemic component:
Epistemic Predictive Aleatoric.
Active Feature Acquisition (AFA). Our clinical test selection setting closely related to Active Feature Acquisition (AFA), where the agent selects next feature or diagnostic test F3 maximally reduce uncertainty about the disease label Y. The corresponding mutual information criterion is
1(); H(Y) H(Y 1 FG).
(4)
Since H() is constant across candidate features, maximizing I(Y; Fÿ) is equivalent to minimizing H(Y F2); thus, the most informative test is the one that most reduces predictive entropy.
KL-Based Expected Information Gain. Information gain can be equivalently expressed in entropy or KL-divergence space. Equation (2) in the main text evaluates the expected divergence between posterior and prior predictive distributions
E|KL] H(Y F3) + CE Yprir).
(5)
where CE denotes cross-entropy between the posterior predictive distribution and uninformative prior.
Interpretation.
Term (-H(Y Pj)): corresponds the entropy-reductinn term AFA, rewarding features that reduce predictive uncertainty.
Term B (CE(Y, )): encourages divergence from the prior, preventing redundant queries by favoring features that meaningfully update prior beliefs.
Note notation. The prior probability Pyeior is constant and longer indexed a correction applied in the current manuscript revision.
Concrete Derivation For single Monte Carlo sample, the divergence between posterior and prior discase probabilitics is computed as
(Ppoo PPproo)) = Pposs log Prost + (1 Pposs) log
Prest
(6)
Pprior
Ppeie
Rearranging terms yields
KL KL (poos Pprice)
Ppost log Ppost + (1 Ppost) log(1 Ppon)
Term H(()FF)
Ppos log Perior + (1 Ppess) log(1 Ppriar)]
(
Term B: CEpY))))
Averaging M Monte Carlo samples gives the final expected value used in Eq. (2):
EM ( lppcier)
(8)
18

---

This formulation provides principled Bayesian information- -ainn objective that decomposes into uncertainty reduction and prior divergence terms, bridging the entropy-basee and KL-based views of active test selection.
Directly calculating the EIG from entropy
The information gain (IG) from an additional piece of information ut
(6.)) is defined as:
(G ) H[P(va 1 K.)] P[ =1 K., u, ((f
(9) where HI denotes Shannon entropy. The first term represents the uncertainty about the diagnosis Dr given the current knowledge Kt, while the second term represents the uncertainty after observing the additional piece of information u,"). Since this quantity is difficult to compute directly, we approximate it using expectation over samples obtain the Expected Information Gain (EIG):
EIG( Py = K.)] E (1.) P(( [P 1 Kr, {
(10)
While it is possible to directly estimate the expected information gain (EIG) using changes in entropy, this approach sometimes favour diagnostic tests that are not truly informative. For binary classification tasks, model the posterior distribution P(ya 1 Kr, " as Bemoulli distribution TB(x,) with success prohability p. E [0, 1]. The entropy of Bernoulli distribution is given by:
([.)) -p, log(r,) (1 pt) log(1 - pp:).
(11)
We approximate the expected entropy using samples ut (5.5) P(24) and corresponding posterior probabilities p.g = P(ya- = 1 Kr, u, ).:
M Σ j=1
EIG(u(") & H[P(ya 1 K.)
Pis log (1 -pis) log(1 - -pis)).
(12)
M
Finally, the optimal piece of information query is:
arg max (IG (+)
(13)
However, entropy-based selection can inadvertently prefer tests that preserve an already confident be- lief-even when that belief is wrong-ove challenge it. Consider patient with an initial dis- ease probability 0.1. J. truly informative test might raise this probability to 0.6. while an uninformative test would leave it at 0.1 regardless of outcome. Be- entropy-
Table 6: KL vs. entropy selection. Mean accuracy (# std) across datasets.
tests that
Model
Dataset
KL-baaed
Entropy-hased
GPT-4o-mini
Diabctes
0.593
0.039
0.572 0.005
GPT-4o-mini
Kidney
0.992
0.006
0.972 0.010
cause entropy is low near the extremes, based EIG can assign little value the informative test (which moves the belief away from a low-entropy region) and may instead tolerate the non-informative one that keeps the posterior near 0.1. In contrast, expected Kullback-Leibler (KL) divergence criterion explicitly measures the magnitude of change in the predic- tive distribution and thus correctly prioritizes the test that meaningfully shifts beliefs. Empirically, our KL-based selection outperforms the entropy-based variant across datasets (Table 6).
GPT-4o-mint
Hepuiiis
0.825 : 0.013
0,770 1: 0.010
GPT-4o
Diahetes
0.682 0.012
0.673 0.011
GPT-4o
Kidney
0.999 + 0.003
0.994 + 0.000
GPT-40
Hepatiis
0.839 1: 0.013
0.770 1: 0.017
C Experimental details
Datasets
We evaluated the performance of our model on three clinical datasets of varying complexity:
Chronic Kidney Discase. The first task involves the prediction of chronic kidncy disease (CKD) from symptoms and laboratory results [84]. The model is provided with demographic variables such as age and gender, as well as signs observable on physical examination (e.g., oedema). It is then tasked with diagnosing CKD by selectively ordering lab tests such serum albumin serum creatinine. We filtered the dataset retain only instances without missing values, resulting in total of 157 patients. Categorical clear clinical interpretation, such as Pus Cells: Abnormal, were excluded. The dataset is available from the UCI Machine Learning Repository under CC BY 4.0 license: https: / wawchiic.eieeeeeeeeeeeeeaeeeeeee ics uci, adu/dataset//S5 chronic prronicckidin <iddey--dseasse.
tests lacking a
19

---

The second task requires the model to diagnose Hepatitis C virus (HCV) infection using liver function test results [85]. Age and were provided as known demographic features, while laboratory tests (e.g.. ALT, AST. GGT) could queried by the model. We included all patients with confirmed HCV and selected a random subset of 56 healthy individuals to form a balanced datasct. The dataset is publicly available from the UCI Repository under CC BY 4.0 license: https: // //t//aachhaa aarie.ieee.... ics. .uc. edu/ http/dataset/// 17tc///111//////
Hepatitis.
Diabetes. The third task uses a random subset of 100 patients from the Pima Indians Diabetes dataset, originally collected by the National Institute of Diabetes and Digestive and Kidney Discases [86]. The model asked to predict the presence diabetes based on clinical measurements. All individuals are female, at least 21 years old, and of Pima Indian heritage. Age is treated as a known feature; the remaining features are unknown and can be selectively queried. The dataset is available on Kaggle under a CCO Public Domain license: attps://wwa kaggle. homm/daaaets//utt htcttt////////////e/////////tllt /pmma indi
We construct custom OSCE-style dataset derived from AgentClinic [54] evaluate zero-shot generalization in multi-condition setting. The cohort comprises 114 clinically diverse cases spanning a range of presentations and diseases. To align with our focus on sequential test acquisition, we retain only laboratory-style diagnostic features and the ground-truth diagnosis. The model is tasked with predicting the primary diagnosis given limited initial context (demographics and brief presentation), while remaining lab tests are unknown and can be selectively queried. For each case, we additionally create matched synthetic counterpart by replacing inneass-inicettvee labs with clinically normal values that point away from the true diagnosis, yielding case-control pairs for evaluating test selection policies. We release the modified OSCE dataset alongside our code to facilitate replication and comparison.
OSCE.
Input data formatting
LLMs have been shown to struggle interpreting tabular clinical data accurately [38]. To mitigate this limitation, we converted all available clinical information from structured tabular form into concise natural language vignettes. Each vignette integrates demographic information, diagnostic test results, and measurement units where appropriate, providing a more interpretable and context-rich format for the model. Table 7 presents an example of the raw tabular input used for diagnosing chronic kidney disease. This input is automatically converted into clinical vignette using a rulc-based preprocessing script (see Clinical Vignette 1).
Clinical Vignette 1
The patient is 63 years old. The patient's diastolic blood pressure is 70 mm/Hg. The patient has a poor appetite. The patient has pedal oedema. The patient has hypertension. The patient has diabetes mellitus. The patient does not have coronary artery discase. The patient does not have anacmia. Specific gravity was measured at 1.01. Albumin levels in urinc was measured at 3.0. Sugar levels in urine Blood glucose random was measured 380.0 mg/dL. Blood urea was measured at 60.0 mg/dL. Serum creatinine was measured at 2.7 mg/dL. Sodium levels was measured at 131.0 mEq/L. Potassium levels was measured at 4.2 mEq/L.. Haemoglobin levels was measured at 10.8 g/dl. Packed cell volume was measured at 32.0.
measured 0.0.
Experiment
For each patient in a preprocessed dataset subset, evaluated risk predictions using all features, as well under four feature selection strategies: Bayesian, Random, Global Best (predefined), and Implicit. At each iteration, one additional feature from the unknown set was revealed, and corresponding risks were computed. Pseudocode for the Bayesian selection using the KL-divergence is given in Algorithm 1. Importantly, the risk prediction prompt remained unchanged between feature selection methods other than the clinical information added the end. All experiments were implemented using GPT-4o (Version 2024-11-22 (Version 2024-07-18) as provided on the Azure OpenAl Service. ensure rohustness, each experiment was run across 5 different random seeds. Opens-source experiments were run suing Biomistral-7B and Llama70B version 3.3 as provided on HuggingFace using vLLM. For all experiments, we set the number of sampled test outcomes risk probability distributions to 10. To ensure the sampling produced more diverse of responses, we used a temperature of and specifically instructed the model the prompts to simulate randomness. Performance metrics were averaged across runs with their corresponding standard deviation.
and GPT-4o-mini
D LLM Prompts
For each dataset, we use four distinct prompt types. The first prompt, shared across all methods, is used for risk prediction based on known patient data, ensuring consistency in evaluation. The second prompt is issued at the start of the experiment, before observing any patient-specific information, to identify globally optimal features.
20

---

Table Structured clinical data for an example patient prior to natural language transformation and inclusion of appropriate units.
Category
Test Name
Result
Demographics
Age
63
Vital Signs
Blood Pressure
70
Specific Gravity
1.010
Albumin
3
Sugar
0
Urine Tests
Red Blood Cells
Abnormal
Pus Cells
Abnormal
Clumps
Present
Bacteria
Not Present
Blood Glucose
380
Blood Urea
60
Creatinine
2.7
Blood Tests
Haemoglobin
10.8
PCV
32
WBC Count
4500
RBC Count
3.8
Sodium
131
Electrolytes
Potassium
4.2
Appetite
Poor
Symptoms
Pedal Edema
Yes
Anemia
No
Hypertension
Yes
Comorbidities
Diabetes Mellitus
Yes
Coronary Artery Disease
No
Algorithm KL-guided Diagnostic Test Selection
Require: Initial knowledge K,, unknowns (u, N /i= number of samples M
1: Sample M draws from P(ya, = K.): (p 1ri
1M
2: for 1 to N do
3: for j = to M do
4: Sample v. (i-5)
((u (1)
5: Compute posterior: Pposterior P(ya, = 1 Kes u (t.1))
1-p (
6: Compute KL (1) = Pprior log 1
(1)
+ (1 - D P log
pr
1-p p G1
posterion
7:
end for
8: EJKL( Uz (i)1 = M Σ j=1 K)
1
9: end for
10: 1" arg max; B[KL((u))) )
11: Request ((()) and update Ke+1 = K, ((a
21

---

The third prompt is used in the implicit selection baseline, where the model is queried at each decision step choose the most suitable test based the information observed far. The final prompt is specific to the Bayesian Experimental Design setting and is used sample plausible outcome candidate diagnostic test, conditioned on the currently available patient data.
Risk Prediction Prompt: CKD
You are an expert nephrologist. Based on the following clinical data and the patient's history, provide an estimate of the patient having chronic kidney disease single number between 0 and 1. Consider key laboratory markers and other pertinent values. When these values indicate kidney disease damage damage-assign number closer I, indicating higher probability of chronic kidney disease. If the laboratory values are within normal ranges, assign value closer O. Return only the number that can be converted to Python float, without any additional commentary.
$known _n
Global Feature Selection Prompt: CKD
You are an expert nephrologist. Based on the following list of features: which $n features you believe are the most informative and critical for diagnosing chronic kidney disease. Return your answer Python list of exactly $n feature For example: if return ['age if n return l'age', 'bp']. Do not include any additional commentary.
please indicate
Feature Selection Prompt: CKD (Implicit)
You are an expert nephrologist. Based solely on the following known clinical data, determine which additional feature from the list below would be the most informative sample next for diagnosing chronic kidney disease.
Known Data: $known_ca data
Unknown Features: $unknown, Sanknoim_featuree features Return only the name of the feature strictly in the form shown in the list as string, without any additional commentary.
Sampling Prompt: CKD Lab Test Simulation
You expert nephrologist. Based on the following clinical data and the patient's 's history, please simulate random draw from the full range of clinically plausible values for $feature. co_sample. The value should not simply he the average or a central tendency, from a realistic distribution. Consider the following description: $ref innf.
should vary as sampled random
Avoid returning the same value repeatedly across multiple draws, and ensure the value varies as if sampled from plausible distribution. Introduce randomness by considering edge cases, typical values, and outliers within the plausible range.
Return your answer as a single numeric value that can he parsed into a float in Python, with no additional commentary or units. IMPORTANT: Assume that the patient may may not have chronic kidney disease, and your sampling should reflect that uncertainty.
$known_
IMPORTANT: Under NO circumstances provide explanations, commentary, or text beyond the single numeric float or string requested. The response MUST be parseable strictly as float, c.g., 0.512, with no extra words. If a string is requested no float is required.
Risk Prediction Prompt:
Hepatitis C
You expert hepatologist. Based the following clinical data and the patient's history, please provide single number between and
an estimate of the patient's risk of being infected with hepatitis Consider key laboratory markers and other pertinent values. When these values indicate liver inflammation or damage assign number closer to 1, indicating a higher probability of hepatitis C infection. If the laboratory values are within normal ranges, assign value closer to 0. Return only the number that can be converted to Python float, without any additional commentary. $known_info
22

---

Global Feature Importance Prompt: Hepatitis C
You are an expert hepatologist. Based on the following list of features: $all_ fealufes please indicate which šn features you believe are the most informative and critical for diagnosing hepatitis. Return your answer as Python list of exactly $n feature names (for example, if is 1, return ['ALT']: i is return ['ALT', >AST']), without any additional commentary. ixnown_info
Feature Selection Prompt: Hepatitis C (Implicit)
You are expert hepatologist. Based solely the following known clinical data, determine which additional feature from the list below would be the most informative sample for diagnosing hepatitis. Known Data: $smowm_data,
Unknown Features: $unknown featuurrsee
Return only the name of the feature as string, without additional commentary.
Sampling Prompt: Hepatitis C Lab Test Simulation
You are an expert hepatologist. Based on the following clinical data and the patient's history, please simulate random draw from the full range of clinically plausible values for $feature sacrrre___sseeeeeeeeeee to_sample.
Consider the possible range as deseribed: $ref_infe info. Ensure that the value you return is realistic and reflects clinical variability. Avoid returning same value repcatedly across multiple draws, and ensure the value varies as sampled from a plausible distribution. Introduce randomness by considering edge cases, typical values, and outliers within the plausible range.
Return your answer single numeric value that can be converted Python float, without any additional commentary.
IMPORTANT: Assume that patient may or may not have hepatitis C, and your sampling should reflect that uncertainty.
$known_ info
IMPORTANT: Under NO cireumstances provide explanations, commentary, text beyond the single numeric float string requested. The response MUST be parseable strictly float, c.g., 0.512 with extra words. If string is requested no float is required.
Risk Prediction Prompt: Diabetes
You are expert endocrinologist. Based on the following clinical data and the patient's 's history, provide estimate the patient's risk diabetes as single number between 0
It is known that all patients are females at least 21 years old of Pima Indian heritage. Focus on key markers. Assign value closer to if the data indicate high risk, and closer to 0 if within normal limits, Return only the number, without any additional commentary. $known_infe
Global Feature Importance Prompt:
Diabetes
You are expert endocrinologist. Based on the following list of fcatures: which $n features you believe the most informative and critical for diagnosing diabetes. Return your answer as Python list exactly feature names (for example, i return [ Gluuoose if 2, return ['Glucose', BMM''], without any additional commentary.
please indicate
Feature Selection Prompt: Diabetes (Implicit)
You are an expert endocrinologist. Based solely the following known clinical data, determine which additional feature from the list below would be the most informative to sample next for diagnosing diabetes.
Known Data: $ data Unknown Features: $unknown features Lanknonn_features Return only the name of the feature string, without any additional commentary. $known_info info
23

---

Sampling Prompt:
Diabetes Lab Test Simulation
You are an expert endocrinologist. Based on the following clinical data and the patient's history, please simulate a random draw from the full range of clinically plausible values for $feature tt__t__s_ppleeeeee Consider the following unit for the sampled value: $ref_ info. Ensure that the value you return is realistic and reflects clinical variahility. Avoid returning the value repeatedly across multiple draws, and ensure value varies as if sampled from a plausible distribution. Introduce randomness by considering edge cases, typical values, and outliers within the plausible range.
Return your answer single numeric value that can be converted to Python float with units or additional commentary.
IMPORTANT: Assume that the patient may or may have diabetes, your sampling should reflect that uncertainty.
$known_ info
IMPORTANT: Under NO circumstances provide explanations, commentary, text beyond the single numeric float string requested. The response MUST he parseable strictly float, e.g., 0.512, with extra words. If string is requested no float is required.
Risk Prediction Prompt: OSCE
You are an expert clinician. Given the following case details, estimate a realistic and conservative probability of the suspected diagnosis of $potential, s_lddaannsiii single number between and 1. Return only the number that be converted to Python float, without any additional commentary. $known_info
Global Feature Importance Prompt: OSCE
You are clinical assistant. Given these case details: Sknown_date Which additional feature from the list Dunknown_feature would be most informative next for the suspected diagnosis of Return only the feature name without commentary.
Feature Selection Prompt: OSCE (Implicit)
You clinical assistant. Given these details: $known data, which feature from the list Sunknoun_featuree would provide most information next? Return only the feature name without commentary."
Sampling Prompt: OSCE Lab Test Simulation
You expert clinician. Based the following case details, simulate plausible value for $feature caatuuee_t__saee so__ample Return only the value without explanation."
IMPORTANT: Assume that patient may or may not have diabetes, and your sampling should reflect that uncertainty.
$known info
IMPORTANT: Under NO cireumstances provide explanations, commentary, text beyond the single numeric float or string requested. The response MUST be parseable strictly as float, e.g., 0.512, with extra words. If a string is requested no float is required.
E Extended results
Predicting plausible test outcome distributions
To evaluate model performance, we analyzed how the mean absolute error (MAE) changed with increasing numbers of generated samples. For computational efficiency, we limited the number of samples to per query. As shown in Figure 9, we observe a consistent decrease in MAE as more features are acquired, indicating that the model is producing a diverse set of samples, some of which closely approximate the ground truth. Furthermore. the larger GPT-4o model outperformed the smaller GPT-4o-mini variant benefits of scale in both predictive accuracy and sample quality.
all three datasets, highlighting
assess the ability large language models (LLMs) to generate diverse samples from hypothetical test outcome distributions, we analysed the generated values for all numerical features across the three datasets. Categorical features were excluded from this analysis. For numerical features, the LLM produced non an-deterministic outputs that varied meaningfully between samples, rather than issuing static averaged responses. This behaviour

---

Kidney
Diabetes
Hepatitis
Mcan Absoluze Errer (1)
8
GP7-40
6
GPT-40 MINI
4
10
10
Number Samples
Number of Samples
Number Samples
Figure 9: Increasing sample size improves the MAE. Model performances are averaged across 5 different seeds and are shown with their corresponding standard deviations.
indicates that the model is capahle of representing distributional uncertainty in a clinically meaningful way. This behaviour was consistent across all three datasets, illustrated in Figures 10, 11 and 12.
Sample Value Distributions for Diabetes
Pregna
BloodPressure
12000
Frequency
15403
30007
10003
0000
7566
0000
3003
2506
2002
Diabeteshccigeefoontiii
20003
22000
10000
Frequency
35003
20200
0100
0003
0100
0003
4100
4003
5003
3106
Sample
Sample Value
Sample
Figure 10: Sample variability of LLM predictions in the diabetes dataset. Each bar chart displays the distribution of values sampled for numerical features. The heterogeneity across samples indicates the model's 's capacity to avoid mode collapse and to reflect uncertainty consistent with clinical variation.
Diagnostic Performance Evaluation
We evaluate ACTMED against multiple baselines using accuracy, precision, recall, score, and ROC-AUC (see Tables 8 and 9). Performance improves across all metrics when moving from GPT-4o-mini to the more capable GPT-4o. On the kidney dataset, differences between models minimal, suggesting both models perform reliably. In contrast, the hepatitis and diabetes datascts, ACTMED consistently outperforms baseline feature selection methods and the full- feature classifier, demonstrating ability to adaptively select informative subsets.
We further assess ACTMED under varying feature budget constraints (Table 10), controlled via the stopping threshold y. Across all datasets and both models, diagnostic accuracy remains stable even as 7 varies. However, stricter thresholds (higher ?) lead to more conservative test acquisition and substantially fewer tests. For example, on the kidney dataset, both models typically require only 1-2 tests per patient when using the stopping criterion. At conservative threshold of 7 0.7, GPT-4o achieves near-perfect accuracy while querying just in nearly all cases.
Feature selection impact on performance
This section investigates the impact of feature selection frameworks on diagnostic performance by analysing the features most frequently selected and their empirical informativeness. Using random baseline to mitigate confounding, we first identified empirically informative features based their accuracy when selected among three features. For GPT-40-mini. the most informative features were GGT, AST, and ALT for hepatitis; BloodPressure, Pregnancies, and Insulin for diabetes; and Random Blood Glucose, Potassium Levels, and Serum Creatinine for CKD. For GPT-4o, BIL, CHE, and GGT were most informative for hepatitis; BloodPressure, Skin Thickness, and Insulin for diabetes; and Blood Glucose, Red Blood Cell Count, and Potassium Levels for CKD.
25

---

Sample Value Distributions for Kidney
Blood
Blood glu
Blood
6400
6030
Inqquency
3400
5030
4300
6033
3300
3033
2030
1030
3300
503
60
100
102
1155
200
2.0
310
25
50
300
125
Serum ineatin
Sodium levels
Potassium levels
4030
Frequency
6000
4300
2000
2300
2000
1939
3300
132
135
140
Sample Value
Hemoglabii in levels
Redd blood cell count
0020
frequency
5033
0030
2033
Sample Value
Sample
Figure 11: Sampled feature distributions in the chronic kidney disease (CKD) dataset. The LLM produces diverse and physiologically grounded value ranges across all features. This supports its utility as a surrogate model in capturing uncertainty for Bayesian evidence diagnostics.
Table 8: Performance metrics (mean + sid) for GPT-4o-mini across datasets. Best-performing methods are in bold; if All Features best, the second-best is also indicated.
Dataset
Method
AUC
Accuracy
F1
Precision
Recall
All Features
0.9997 + 0.0002
0.975 + 0.006
0.956
0.009
0.915 + 0.017
1.000
0.000
ACTMED (ours)
0,9999 0.0001
0.992 - 0.006
0.986
0.011
0.978 + 0.020
0.995
0.009
Kidney
Random
0,9990
0.0002
0.981 0.004
0.966
0.007
0,947 0.021
0.986
0.011
Global Best
1.000 + 0.000
1.000 - 0.000
1.000 + 0.000
1.000 1 0.000
1.000
1: 0.000
Implicit
1.000 + 0.000
0.997 = 0.005
0.995 + 0.009
0,991 + 0.018
1.000
0.000
All Features
0.874 = 0.002
0.793 = 0.009
0.758
0.011
0,910 = 0.012
0.650 0.014
ACTMED (ours)
0,891 = 0.006
0.825 0.013
0.806 0.015
0,903 = 0.017
0.729 0.017
Hepatitis
Random
0.743 + 0.018
0.670 = 0.020
0.558
0.031
0,843 = 0.045
0.418 + 0.029
Global Best
0.824 + 0.007
0.718 + 0.018
0.668
0.022
0.812 + 0.031
0.568 0.021
Implicit
0.856 0.016
0,770 + 0.007
0.743 0.005
0.023
0,664 0.013
All Features
0.727 + 0.011
0.417 = 0.006
0.545 0.003
0,374 + 0.003
1.000 1: 0.000
ACTMED (ours)
0.743 + 0.009
0,569 = 0.004
0.601 + 0.002
0.444 = 0.002
0.930 1 0.005
Diabetes
Random
0.663 0.008
0.493 = 0.008
0.560 0.005
0.402 = 0.004
0.923 0.004
Global Best
0.725 0.006
=
0.563
0,393 = 0.001
0.992 0.001
Implicit
0.733 + 0.004
0.460 = 0.002
0.562 0.001
0.392 + 0.001
0.993 0.000
All Features
0.755 + 0.009
0.684 + 0.015
0.744
0.012
0.625 + 0.010
0.919 0.018
ACTMED (ours)
0.755 0.028
0.705 = 0.016
0.760 0.014
0.641 = 0.011
0.933 0.023
OSCE
Random
0.653 + 0.019
0.604 = 0.020
0.670 0.013
0.575 + 0.018
0.804 0.026
Global Best
0.756 + 0.018
0.667 - 0.021
0.735 0.017
0.610 + 0.014
0.923 +: 0.024
Implicit
0.727 + 0.026
0.665 - 0.007
0.732 0.006
0.610 1 0.004
0.916 1 0.013
We also identified globally optimal features selected by both models Glucose, BMI, and Insulin for diabetes, ALT, AST, and BIL for hepatitis, and BloodPressure, Serum Creatinine, and Haemoglobin CKD.
We then examined the feature selection patterns of ACTMED and the implicit baseline. The random baseline served as a control, showing no significant selection bias. For hepatitis, ACTMED with GPT-4o-mini preferentially selected ALT, AST, and GGT. while with GPT-4o it favoured AST, GGT, and BIL. The implicit method selected ALT, AST, and GGT for GPT-4o-mini and ALT, AST, and BIL for GPT-4o. On the diabetes dataset, GPT-40-
26

---

Sample Value Distributions for Hepatitis
2000
2300
Frequencc
2500
2000
1500
2000
300
CHE
4000
4000
125-
2000
2000
903
1000
1000
253
50
160
1.5
60
66
10
12
CHOL
CREA
GCT
2500
Trequency
2003
1000
5.5
6.5
Sample
Sample
PAOT
2000
1500
560
Sample Value
Figure 12: Distributions LLM- LLL-eeenrratd samples for numerical features in the hepatitis dataset. Bar plots show the spread of sampled values for each feature, illustrating the model's ability to represent plausible clinical variability. The distributions vary across features, reflecting both physiological ranges and disease-specifee uncertainty.
mini with ACTMED selected Glucose, Blood Pressure, and Skin Thickness, whereas GPT-4o preferred Glucose, Diabetes Pedigree Function, and BMI. The implicit method consistently selected BMI, Glucose, and Insulin across both models. For the CKD dataset, GPT-4o-mini with ACTMED selected Red Blood Cell Count, Blood Pressure, and Creatinine, while GPT-4o selected Blood Urea, Creatinine, and Blood Pressure. The implicit method selected Serum Crcatinine, Blood Urca, and Blood Pressure for GPT-44-mini, and Serum Creatinine, Blood Urea, and Hacmoglobin for GPT-4o.
Collectively, these results indicate that neither ACTMED nor the implicit method consistently selects fixed set features that are empirically superior across all scenarios. The observed performance advantage ACTMED likely stems not from identifying a universally optimal feature subset, but rather from its ability to perform more varied and potentially petter-personllized test selections compared to the more constrained implicit or global methods, thereby optimizing the diagnostic process for individual cases.
Importance of L.LM quality for ACTMED
ACTMED is inherently model-agnostic, as the framework only requires access to a simulator capable of generating test outcomes and corresponding risk distributions. Our contribution is test-time computational improvement selecting tests maximize information gain rather than modification of the underlying language model. Consequently, the primary consideration not whether the model is open- source or closed-soures source, but whether it is capable of generating sufficiently accurate and diverse outcome distributions. To assess the impact of model quality, we evaluated two smaller open-source models, BioMistral-7B and LLaMA-70B, as representative lower-bound baselines.
27

---

Table 9: Performance metrics (mean + std) for GPT-40 across datasets. Best-performing methods are in bold; if All Features is best, the second-best is also indicated.
Dutuset
Method
AUC
Accuracy
Precision
Recall
All Features
1.000 + 0.000
1,000 + 0.000
1.000 + 0.000
1,000
0.000
1,000 + 0.000
ACTHED (ours)
1.000 + 0.000
0.999 + 0.003
0.998 1 0.005
1.000 + 0.000
0.995 1 0.009
Kidney
Random
0.999 1 0.001
0.995 1 0.003
0.991 + 0.005
1,000
0.000
0.981 1 0.009
Globul Best
1.000 + 0.000
0,999 + 0.003
0.998 0.005
1.000
0.000
0.995
0.009
Implicil
1.000 + 0.000
0.003
0.998 0.005
1.000
0.000
All Teatures
0.876 + 0.014
0.807 + 0.004
0.771 +:: 0.006
0.948
0.001
0,650 +: 0.009
ACTHED (ours)
0.917 + 0.013
0.839 + 0.013
0.814 + 0.016
0.966 + 0.012
0.704 + 0.021
Hepatitis
Random
0.728 + 0.039
0.657 + 0.018
0.507 0.036
0.904 0.052
0.354 + 0.035
Global Best
0.799 + 0.013
0.757 0.007
0.699 0.009
0.199
0.011
0.564 0.009
Implicit
0.805 + 0.013
0.729 0.016
0.646 0.030
0,927 0.008
0.036
All Features
0.815 + 0.004
0.592 + 0.004
0.623 + 0.003
0,460
0.003
0.967 + 0.005
ACTHED (ours)
0.803 + 0.009
0.668 = 0.009
0.648 + 0.008
0.514 0.008
0.878 + 0.012
Diabetes
Random
0.703 + 0.011
0,581 0.010
0.593 0.007
0.448 0.007
0.875 + 0.014
Globul Best
0.781 + 0.004
0,548 : 0.005
0.599 0.003
0.434 0.003
0.969 0.005
Implicit
0.783 0.003
0.350 0.006
0.601 0.003
0.435 0.003
0.971 0.003
All Features
0.805 +: 0.006
0.719 + 0.008
0.772 + 0.007
0.650 + 0.006
0.951 +: 0.013
ACTHED (ours)
0,791 +: 0.015
0.746 1 0.006
0.785 1 0.008
0.680 0.006
0.930 1 0.029
OSCE
Random
0.696 + 0.024
0.640 + 0.021
0,701 + 0.021
0.600 0.015
0.842 + 0.035
Global Best
1 0.011
0.723 0.012
0.771 0.010
0.657 0.0TO
0,933
Implicit
0.730 1 0.026
0.681 0.016
0.736 0.013
0.627 0.012
0.891 1 0.017
Table 10: Average number of tests selected and accuracy (mean standard deviation) under KL- based termination for varying y values. Higher 7 requires stronger evidence to continue testing. Results are reported across five random seeds.
Model
7
Hepatitis
Diabetes
Kidney
Tests
Accuracy
Tests
Accuracy
Tests
Accuracy
0.3
2.36
1: 0.68
0.832 1 0.374
2.45 1: 0.71
0.560 di 0.496
1.60 1. 0.75
0.997 1: 0.050
GPT-40-MINI
0.5
1.87
0.77
0,841 + 0.366
2.09
0.75
0.534 + 0.497
1.48 0.68
0.999 + 0.U36
0.7
0.73 1.48
0.845 1 0.363
1.90
0.74
0.542 1 0.498
0.52 1.28
0.997 1 0.050
0.3
0.60
+
2.27
0.84
0.674
1.35 0.64
GP
0.5
2.41
1:
0.67
0.827
1 0.379
1.90
1 0.89
0.683 1. 0.465
1.09 1. 0.35
0.996 1. 0.062
0.7
2.27
0.65
0.816 + 0.388
0.684 + 0.465
1.04
0.996 +
Table 11: Accuracy versus number of sequentially added tests. Values denote mean + standard deviation of predictive accuracy across evaluation folds.
Dataset
Model
1 Test
3 Tests
5 Tests
All Tests
GPT-4o
0.994 + 0.004
0.995 0.003
0.995 + 0.003
1.000 + 0.000
Kidney
GPT-4o-mini
0.975 + 0.011
0.983 0.005
0.990 + 0.010
0.972 0.006
GPT-4o
0.589 + 0.023
0.679 0.022
0.718 + 0.013
0.796 0.010
Hepatitis
GPT-4o-mini
0.588 + 0.030
0.698 0.022
0.732 + 0.010
0.798 0.011
GPT-4o
0.730 + 0.138
0.662 + 0.170
0.664 + 0.170
0.678 + 0.161
Diabetes
GPT-4o-mini
0.546 + 0.015
0.524 1: 0.025
0.528 +: 0.016
0.458 1 0.007
Failure to generate accurate samples. The BioMistral-7B model frequently fails to generate realistic outcome samples, often producing implausible outcomes such as non-integer values for the number of previous pregnancies. As result, the observed Wasserstein distances between the podel-generated and empirical distributions substantially higher compared GPT-40 and GPT-4o-mini (see Table 13).
Failure to perform BED accurately. The LLaMA-70B model also fails to achieve accuracy comparable the closed source GPT models, though degradation arises from different failure mode. While its Wasserstein distance is not substantially worse than GPT-4, inspection of the predicted outcome distributions reveals that the model often collapses to deterministic outputs, producing the same value across all Monte Carlo samples even when increasing the sampling temperature beyond I. This lack of distributional diversity
28

---

Table 12: Impact of additional tests risk estimation. Mean + standard deviation of normalized risk estimates as the number of tests increases. Lower values indicate better calibration.
Dataset
Model
1 Test
3 Tests
Tests
Features
GPT-4o
0.259 + 0.007
0.267 0.003
0.273 + 0.004
0.274 0.002
Kidney
GPT--4-minii
0.292 + 0.007
0.330 0.010
0.349 + 0.009
0.377 0.003
GPT-4o
0.209 + 0.012
0.255 + 0.008
0.285 + 0.018
0.326 + 0.005
Hepatitis
GPT-44-mini
0.275 + 0.019
0.307 + 0.009
0.323 + 0.015
0.374 + 0.003
GPT-4o
0.575 + 0.062
0.638 + 0.013
0.698 + 0.096
0.687 + 0.082
Diabetes
GPT--4omini
0.606 + 0.014
0.696 + 0.018
0.725 + 0.013
0.768 + 0.004
Hepatiiis GPT-40-MMINI
Kidney -P1-40-NINI
Boed
Disc glace sandam
Fessuur
Peaturs
81.
PXOT
Accursdy
Accuracy
Accuracy
Hopatitis GPT-40
Diabetes GPT-40
Kidney GPT-40
A4T
Ruature
Serum
Miced
PRT
Sodium meb
A05
Hemogean aeal Bied
0.3
1.3
Accuracy
Figure 13: Feature performance. Mean accuracy per feature, measured across multiple seeds, for two models (gpt-4o-mini and got-4o) three datasets. Bars represent mean accuracy (0-1), and black error bars denote +1 standard deviation. Features are ranked in descending order of mean accuracy.
Table 13:
Wasserstein distance between true empirical and model- test outcome distribu- tions. Reported mean + standard deviation (lower is better).
Dataset Model
Avg. Wasserstein (Mean + Std)
GPT-4o
0.110 + 0.038
Diabetes
GPT-4o-mini
0.117 + 0.046
BioMistral-7B
0.125 + 0.048
GPT-4o
0.130 + 0.157
Hepatitis
GPT-4o-mini
0.173 + 0.237
BioMistral-7B
0.425 + 0.453
GPT-4o
0.082 + 0.055
Kidney
GPT-4o-mini
0.082 + 0.057
BioMistral-7B
0.215 + 0.225
prevents the model from capturing uncertainty in test outcomes. Consequently, Bayesian Experimental Design (BED) provides no measurable performance improvement, and the overall diagnostic accuracy remains low (see Table 14).
29

---

10
15 Selection Percentage (%)
20
25
Hepatitis ACTMED
Hepatitis Implicit
Hepatitis
Random
A08
ALB
6.C
2.4
A1P
10.6
122
81
11.1
86
CHCC
AST
153
253
CHOL
101
GST
GGT
254
G3T
GPT-4G-MINI GPT-40 Model
OPT-4O-NINI GPT40 Model
GPT-4C-MRI CPT40
Model
Diabetes ACTMED
Diabetes Implicit
Diabetes
Random
BloedPressure
BMI
16.1
22.3
33.3
Presuree
BMI
147
1:4
145
phabetesPedigreef unction
2233
DibbetesPedgreeuucciin
giabeteseeggreefueeeee uncctontoo
148
Glucose
210
133
Glucase
137
13.
Insulin
Glucose
11.1
Insulin
137
11.
Pregnancles
Pregnancies
SkinThikkees
11.3
SkinThickowss
GPT-40
GPT-4D NINI
GPT400
GPT 40-MNI
GPT40
Kidney ACTMED
Kidney
Implicit
Kidney Random
Blood glucose random
Blooe pressure
random
10.5
Blood glucase random
Blood pressure
Hemoglooin Potassium levels
Bined urea
30-3
20.7
Hemoglobin
Potassium levels
12.
Red blood cell count
283
Hemoglobin levels
3.4
25.4
Red blcod cell count
11.
12.1
Serum creatinine
Sodium
Scrum creatinine
23-5
35.5
Sodium livels
Model
Model
Model
Figure 14: Feature selection by method. Heatmaps of feature-selection frequencies (0-100%) across three datasets (hepatitis, diabetes, and kidney) and three selection methods (ACTMED Implicit, Random). Columns represent GPT-4O-Min and GPT-4O. Each cell shows how often a feature was included across all seeds
Table 14: Accuracy of different models on the Hepatitis, Diabetes, and Chronic Kidney Disease datasets using all features and ACTMED-selected features. Results are reported as mean + standard deviation.
Dataset
Model
ACTMED (Mean + Std)
All Features (Mean + Std)
GPT-4o
0.682 0.012
0.584 + 0.011
GPT-4o-mini
0.593 + 0.039
0.451 + 0.062
Diabetes
Biomistral-7B
0.529 + 0.025
0.450 + 0.008
LLaMA-70B
0.540 + 0.013
0.474 + 0.012
GPT-4o
0.999 + 0.003
1.000 + 0.000
GPT-4o-mini
0.992 + 0.006
0.975 + 0.006
Kidney
Biomistral-7B
0.657 + 0.021
0.564 + 0.033
LLaMA-70B
0.984 + 0.003
0.986 + 0.000
GPT-4o
0.839 + 0.013
0.807 + 0.004
GPT-4o-mini
0.825 + 0.013
0.793 + 0.009
Hepatitis
Biomistral-7B
0.489 + 0.029
0.511 + 0.027
LLaMA-70B
0.699 + 0.023
0.793 + 0.010
Bayesian bootstrap
To quantify uncertainty in diagnostic predictions, we performed 10 independent diagnostic trials patient for each model and dataset. The resulting prediction accuracies were aggregated using Bayesian bootstrapping [87].
30

---

which provides non-parametric estimates of posterior distributions over mean accuracy values and corresponding 95% credible intervals. This approach captures both inter-patient and intra-model variability and enables estimation worst-case performance bounds. Table reports the mean standard deviation predicted risk probabilities, together with lower and upper bounds of the 95% Bayesian bootstrap credible intervals, for each datasct, model, and true disease label. GPT--4o and GPT-4o- monm both show strong stratification between positive and negative classes, with narrower credible intervals in GPT-4o-mil mini for highly separable discases (e.g.. chronic kidney disease).
Table 15: Average model prediction accuracy across the Hepatitis, Diabetes, and CKD (Kidney) datasets, Values show the mean and standard deviation of predicted risk with 95% Bayesian bootstrap credible intervals, stratified by true disease label.
Dataset
Model
Label
Mean Avg. Risk
Std. Avg. Risk
95% CI [Lower, Upper]
GPT-4o
0.548
0.275
[0.512, 0.583]
GPT-4o
0.799
0.143
[0.782, 0.815]
Diabetes
GPT-4o-mini
0.716
0.223
[0.686, 0.744]
GPT-4o-mini
0.856
0.039
[0.836, 0.875]
GPT-4o
0
0.093
0.129
[0.081, 0.106]
GPT-4o
0.561
0.320
[0.539, 0.583]
Hepatitis
GPT-4o-mini
0.167
0.147
[0.157, 0. 178]
GPT-4o-mini
0.581
0.289
[0.555, 0.605]
GPT-4o
0.039
0.033
[0.029, 0.051]
GPT-4o
0.907
0.060
[0.898, 0.916]
Kidney (CKD)
GPT-4o-mini
0
0.179
0.119
[0.161, 0.11
GPT-4o-mini
0.908
0.061
[0.897, 0.917]
Example of ACTMED's Test Evaluation Process with Clinician-in-the-LLorr
To illustrate ACTMED's 's information-theomm
selection process, present simplificd synthetic binary diagnostic task involving chronic kidney disease (CKD) with two tests. At each stop, ACTMED supports clinician decision-making by maintaining transparency over test evaluations and allowing human review.
Step 1: Prior Belief. The system starts with a diagnostic prior based on available information:
P(ya, 1 K.) ((prrii)) where
Preior 0.20
Clinician role: The clinician inspect the current risk estimate and adjust the prior based on additional context currently captured in the structured data.
Step 2: Candidate Test Outcomes. ACTMED considers candidate and simulates plausible results using a surrogate model:
Creatinine (high) 2.3, mg/dL
Creatinine (normal) : 1.0, mg/dI.
Sodium (normal) 140 mmol/I.
Sodium (low): 130 mmol/L.
Clinician role: The clinician may review the simulated outcomes for plausibility, reject irrelevant or infeasible tests, and flag preferred tests based on domain knowledge or patient-specific factors.
Step 3: Posterior Beliefs, ACTMED computes updated diagnostic beliefs for each hypothetical outcome:
Creatinine
high: P(ya, 1 K,, high) 0.65 normal: P(ya. K,, normal) = 0.22
Sodium
low: P(ys, Ko, 0.45
normal: P(ya. Kr, normal) = 0.18
31

---

Clinician role: The clinician can examine the impact of each test result on the diagnostic belief, and assess whether these posterior shifts clinically meaningful or likely influence treatment decisions.
Step 4: Utility of Each Test. We compute the expected information gain from each test using the KL divergence between posterior and prior diagnostic belicfs. This utility expressed as:
Flu = E[KL(P = Kr, ") P(Ud, = K.)) = 0.134,
F(ut [PKL = K2, u7") || P(yd: K.))] = 0.056.
Since F(u F(ut)) ACTMED selects the serum creatinine test it provides higher expected diagnostic value.
Clinician role: Before confirming the selected test, the clinician may override the choice if the utility estimate contradicts clinical judgment, safety concerns, or logistical constraints.
Once test acquired, the diagnostic process iteratively continues until diagnosis achieved. Figure 15 details how the model supports this by generating intermediate outputs that clinicians can review throughout the process.
1. Generating prior distributions
Selecting test with the highest utility
P(yar 1| K2) 0.20
F(
Reviow and polentially updale suggested
F(! 0.056
Revizw
Reviow samples
samples
Review sampled test outcomes
2. Generating hypothetical test outcomes
Generating posteriors distributions
Crcotinin: (bigh)
Pleu =1 Ri.ccm 2..) 0.53
Oreatinlne (normol)) 1.4, mg/
R..mms n.22
Sodium (normal) 1/0mm///
Raad
(low) zumel/L
Pleus Ri,
Figure 15: Illustrative example of ACTMED 's diagnostic reasoning. The current belief (prior) about CKD updated based on hypothetical outcomes for two candidate tests (serum creatinine and sodium). Posterior probabilities differ for each outcome, and the expected KL divergence determines which test offers the greatest diagnostic value. Serum creatinine yields higher expected information gain and is selected.
F
Computational Cost Analysis
Our framework relies on repeatedly querying a LLM to simulate plausible estimates for candidate diagnostic test results and estimate disease posteriors. Consequently, the computational cost is dominated by the number of LLM queries required per patient episode.
32

---

At each decision step 4, the agent evaluates set of candidate diagnostic tests U, C 24. where [Ua] denotes the number of available tests, For each candidate test ut Ut. the agent samples M possible outcomes and the resulting hypothetical posterior probability from the LLM approximate the expected KL divergence. Thus, the computational cost per decision step O U4 MM LLM queries.
I' denote the maximum number decision steps per patient before either termination or diagnosis. Then, the total computational cost per patient is:
T
C= O
Σ
(14)
In the worst case, where no early termination occurs and all tests are considered at every step, the complexity simplifies to:
C C-O(TMM),
(15)
where N is the total number of possible diagnostic tests. In practice, N may already be quite small there will available tests that can ordered for diagnosing conditions due existing guidelines and the model only needs to determine which of those offers the highest utility. Table 16 summarizes the asymptotic computational complexity of our method compared to bascline approaches. While our method incurs higher per-patient computational cost compared static classifiers, this is justified in domains like clinical medicine where information acquisition expensive and decision quality is paramount. Inference time depends primarily on LLM latency hardware availability. In our setup, estimating the expected value of a single diagnostic test takes approximately 20 seconds using either GPT-4o or GPT-4o-mini. Running the full suite of experiments across five random seed, parallelized under shared API key for each model, takes roughly 60 hours. In deployment, inference time could be significantly reduced through parallelization of the independent API requests. This cost and time delay is negligible relative the clinical and financial burden unnecessary or delayed testing. Furthermore, computational demands can be significantly reduced by training lightweight surrogate models specialized for predicting test outcomes, as demonstrated in prior work [88].
only be subset
Table 16: Comparison of per-patient computational complexity across methods.
Method
Complexity
Description
Static classifier
O(1)
Single forward pass using all features Random subset of all features selected
Stochastic feature acquisition
O(1)
Greedy feature acquisition
(TT)
Selects top-k features in static order Actively selects using KL divergence
BED with KL divergence (ours)
O (TNN)
33

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduce and formalize a framework for timely diagnosis using BED and LLMs in Sections 2 & 3. We validate the performance of our model real-world medical datasets in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can he expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of the model are discussed in Section 5, Computational cost is discussed in Appendix F.
Guidelines:
The answer NA means that paper has no limitation while answer means that paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
paper should point out any strong assumptions and how robust results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification,
asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to- text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicabll,, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't I acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The assumptions underlying the Bayesian Experimental Design are discussed in Section 2. The paper does not include novel theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
34

---

All the theorems, formulas, and proofs in the paper should be numbered and cross tross-reference..
All assumptions should he clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: provided detailed information on the publicly available datasets as well the prompts, pseudocode for the KL divergence based test selection and specific models used to run the experiment. Source code to reproduce the main findings can be found under https: // //g thub. com/ Sr933/ac 3e33//
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments. No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can accomplished various ways. example, contribution novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results. access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source source models, it may be that access to the model is limited in some way (e.g.. registered users). but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data code, with sufficient instructions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
The datasets are all open access, with details provided in Scction C. The experiments and prompts used are also described. Source code and the datasets to reproduce the main findings can be found under https:/ / thub. wtt.cooo////eeeeeeee Sr3333/ctmmmtt..eeeeeeaaeaaaa
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines cut.tes/Cooeeuuumiiesssoo/o// for more details.
35

---

While we encourage the release of code and data, we understand that this might not be possible. "No" acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g.. for a new open-soure benchmark).
The instructions should contain the exact command and environment needed the results. See the NeurlPS code and data submission guidelines (httpe: / /ttps/ipps...//. cc/ ec/publcc/
reproduce
Saaees//oodSubmieee ssionooiccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental sefting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Details on the datasets used the LLM setup are provided Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We performed all experiment across 5 different random seeds and report all results as the mean of the runs with the corresponding standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 96% CI. if the hypothesis of Normality of errors
a 2-sigma error than state that they not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric emror bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer: [Yes]
36

---

Justification:
The bulk of the experiments other than the API calls was performed on the Azure Open Al services, detailed in Appendix C & F.
Guidelines:
The answer NA means that paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminay or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code Ethics https:/ bttps://neuripp: httpppullie/eccsssuuueeeeeeeeeee
Answer: [Yes]
Justification: The research conducted conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts the work performed?
Answer: [Yes]
We discuss how our approach may lead to more rational test selection through personal- ization in Section 5. We also acknowledge limitations of directly deploying LLMs in clinical practice and highlight that our model maintains mamaann---tte-looe approach and clinicians override test suggestion and review the models outputs at each step.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considcrations (c.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considcrations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications. the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback time, improving the efficiency accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
37

---

Answer: [NA]
Justification:
We release any pretrained language models or datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model. for example by requiring that adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators the datasets as well as the licenses under which these are available are provided in Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright terms of service of that source should be provided.
license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
assets are released,
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should he provided.
If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We do not create novel datasets or pretrained models. Source code and the exact datasets used to reproduce the main findings can be found under https: //gtthub. com/ St//////333/a/a actmed
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate details of dataset/code/moodl as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
38

---

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least minimum wage country data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurlPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The use of L.MMs as generative models for Bayesian Experimental Design is described Section 3.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttees///euuuriee wwwwoncceeencee///cccccccccccce/ should should not be described.
for what
39


Reviews:
[{"review_id": "2WN12MNpcC", "reviewer": "Reviewer_tmUM", "summary": "This paper presents an LLM-based clinical diagnosis framework that actively selects and updates its belief on tests to make decisions on disease states. Experiments were conducted on three different datasets using two GPT models and the results were also compared with various feature selection-based baseline approaches using the same models and risk prediction prompts. Results showed the effectiveness of the proposed methods compared to considered approaches and baselines across various capabilities including timely diagnosis and transparent explanations.", "questions": "see above.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The paper addresses an important problem of clinical diagnosis using LLMs and a probabilistic framework, where LLM is used as a flexible simulator to predict test outcomes to update the disease belief states while it does not need any task-specific training.\n\n- The proposed approach is agnostic to LLM model architecture and shows promising results within the considered experimental setup and comparisons with various baselines.\n\n- Rigorous experiments have been carried out on 3 datasets demonstrating the generalizability of the proposed methods.\n\n- Extended results with clinician-in-the-loop review process and computational cost analysis have been reported that would be beneficial for the community. \n\n\nWeaknesses:\n\n- Evaluation is weak given the paper considers only two closed source GPT-based models for experimentation lacking other open-source models, hindering detailed understanding of why the proposed methods work or do not work in certain cases and reproducibility of the experiments.\n\n- The experiments have been conducted in a strictly controlled setting, where the task is formulated as a binary classification task making it far from the real-world clinical diagnosis inference task requiring multi-label categorization and dependencies.\n\n- The paper only considers categorical and numerical features ignoring free text EHR reports that raises questions about the usefulness of using LLMs for this task. \n\n- The related work section seems to be weak as well, as it omits various state-of-the-art approaches using DRL that also mimics similar iterative clinical decision-making process (e.g., https://aclanthology.org/I17-1090/)\n\n- Although clinician-in-the-loop evaluation process was proposed but the paper lacked a thorough qualitative evaluation of the proposed approaches along with detailed examples of how the methods could have been improved further.\n\n- The overall writing, structure, organization and content of the paper can be improved, as at times the proposed objectives, frameworks, experimental setup etc. seemed confusing and not coherent.", "was_revised": true, "final_justification": "I've reviewed the rebuttal responses and other reviewers' comments and author responses that helped with additional clarity and context about the key contributions of the paper, limitations, and some efforts/ongoing work to address those limitations. However, I am still not convinced on the value-add of ACTMED as it requires a strong LLM to perform better, so not sure if the results would be model agnostic as mentioned in the paper. Regardless, based on the new information and results shared by the authors, I've adjusted my rating accordingly.", "rebuttal": "*We thank the reviewer for their constructive feedback and insightful comments. We appreciate that the reviewer recognizes both the importance and clinical relevance of our work, as well as the detailed evaluation of its performance across three datasets. Below, we address each concern raised.*\n\n---\n\n## **Experimental evaluation and model choice**\nAs we emphasize in the paper, ACTMED is inherently model-agnostic: it does not rely on any model-specific fine-tuning or internal access, but only on generated distributions of test outcomes and disease risk estimates. Our contribution is a test-time computational improvement — selecting tests to maximize information gain — rather than a modification of the underlying language model. Consequently, the primary consideration is not whether the model is open-source or closed-source, but whether it is capable of generating sufficiently accurate and diverse outcome distributions.\n\nTo clarify this point, we reran all experiments using Biomistral-7B, an open-source biomedical LLM. The results (Tables 1 and 2) show that while ACTMED is compatible with Biomistral, its diagnostic performance depends heavily on the quality of the generated distributions. Biomistral’s outputs frequently diverged from real data — for example, predicting non-integer values for the number of prior pregnancies in the diabetes dataset — leading to weaker downstream performance.\n\n**Table 1:** Wasserstein distance between the true empirical distributions and the model-generated test outcome distributions.\n| Dataset   | Model        | Avg. Wasserstein (± SD) |\n|-----------|--------------|--------------------------|\n| Diabetes  | GPT‑4o       | 0.110 ± 0.038           |\n| Diabetes  | GPT‑4o‑mini  | 0.117 ± 0.046           |\n| Diabetes  | Biomistral      | 0.125 ± 0.048           |\n| Hepatitis | GPT‑4o       | 0.130 ± 0.157           |\n| Hepatitis | GPT‑4o‑mini  | 0.173 ± 0.237           |\n| Hepatitis | Biomistral      | 0.425 ± 0.453           |\n| Kidney    | GPT‑4o       | 0.082 ± 0.055           |\n| Kidney    | GPT‑4o‑mini  | 0.082 ± 0.057           |\n| Kidney    | Biomistral      | 0.215 ± 0.225           |\n\nLower values indicate that the generated samples more closely reflect the real-world data. GPT models produce distributions that closely match the data, while Biomistral-7B shows substantially higher distances, suggesting its generated outcomes are less realistic.\n\n**Table 2:** Classification accuracy using all features versus ACTMED-selected and sequential implicit-selected features for each model.\n| Dataset   | Model       | ACTMED (Mean ± Std)   | All Features (Mean ± Std) | Implicit (Mean ± Std)     |\n|-----------|-------------|------------------------|----------------------------|----------------------------|\n| Diabetes  | GPT-4o      | 0.682 ± 0.012         | 0.584 ± 0.011              | 0.544 ± 0.004              |\n|           | GPT-4o-mini | 0.593 ± 0.039         | 0.451 ± 0.062              | 0.488 ± 0.040              |\n|           | Biomistral     | 0.529 ± 0.025         | 0.450 ± 0.008              | 0.554 ± 0.034              |\n| Kidney    | GPT-4o      | 0.999 ± 0.003         | 1.000 ± 0.000              | 0.999 ± 0.003              |\n|           | GPT-4o-mini | 0.992 ± 0.006         | 0.975 ± 0.006              | 0.997 ± 0.005              |\n|           | Biomistral     | 0.657 ± 0.021         | 0.564 ± 0.033              | 0.620 ± 0.045              |\n| Hepatitis | GPT-4o      | 0.839 ± 0.013         | 0.807 ± 0.004              | 0.729 ± 0.016              |\n|           | GPT-4o-mini | 0.825 ± 0.013         | 0.793 ± 0.009              | 0.770 ± 0.007              |\n|           | Biomistral     | 0.489 ± 0.029         | 0.511 ± 0.027              | 0.501 ± 0.042              |\n\nACTMED improves diagnostic accuracy when paired with models that generate realistic outcomes (low distances in Table 1). For models like Biomistral-7B, where outputs diverge significantly from the data, ACTMED cannot provide meaningful gains.\n\n---\n\n## **Binary vs. Multi-Label Diagnosis**\nClinical diagnosis generally follows a two-step process (1). First, the clinician conducts an initial assessment, forming a primary diagnosis alongside a list of differential diagnoses. Second, tests are ordered to both confirm the leading diagnosis and rule out competing conditions.\n\nThis process naturally reduces to a series of one-vs-all evaluations, where each condition is assessed independently. Such an approach is common in clinical practice and machine learning when handling complex multi-class problems. Widely used clinical scores, such as the Wells score for deep vein thrombosis (2) and the CHA₂DS₂ VASc score for atrial fibrillation-related stroke (3), each estimate the likelihood of a single disease. Clinicians routinely apply these sequentially, allowing patients to be assessed for multiple conditions in parallel.\n\nACTMED addresses the second step by optimizing test selection across diseases, independent of how the differentials are generated. This distinction has been clarified in the revised manuscript, and we include results on a multi-disease dataset derived from AgentClinic (see response to reviewer fFNv), where ACTMED continues to perform strongly.\n\n---\n\n## **Handling of Categorical and Free-Text Features**\nNumerical and categorical variables are converted into natural language descriptions resembling EHR notes (see Appendix D). This ensures that values are presented in a clinically interpretable way, accounting for differences in units and context. At the same time, ACTMED relies on a tabular structure so each feature can be sampled and recombined consistently, allowing us to dynamically generate clinical vignettes and counterfactual diagnostic pathways for the same patient — something not feasible with free-text records alone.\n\nTo address the reviewer’s concern, we added an experiment using a dataset derived from AgentClinic that combines structured features with free-text EHR-style notes. ACTMED operates effectively in this mixed-modality setting without modification, demonstrating its flexibility.\n\n---\n\n## **Comparison with DRL-Based Approaches**\nWe thank the reviewer for pointing out additional related work. In the revised manuscript, we now cite Ling et al. (4) along with related DRL-based clinical decision-making systems by Yu et al. (5) and Zhong et al. (6). These works demonstrate how reinforcement learning can iteratively guide test selection and decision-making in healthcare, and we view them as complementary to our approach.\n\nACTMED differs in that it combines Bayesian Experimental Design with LLM-based generative priors to simulate plausible test outcomes at inference time. This allows ACTMED to operate without task-specific retraining, whereas DRL-based methods typically require substantial training to adapt to new tasks.\n\n---\n\n## **Clinician-in-the-Loop Evaluation**\nWe conducted a clinician-in-the-loop evaluation with three experienced clinicians and two senior medical students, evaluating 450 test decisions (10 traces across 3 datasets, each assessed by 5 evaluators with 3 decisions per trace). Experts judged ACTMED’s test selections and the resulting risk adjustments as clinically reasonable in 94.5 ± 1.4 % of cases, underscoring its reliability as a decision-support tool.\n\nClinicians emphasized that they would be reluctant to trust diagnostic suggestions from a black-box LLM without transparent reasoning. ACTMED’s step-by-step decision process was repeatedly praised for building confidence in the model reasoning. Several clinicians noted that ACTMED’s Bayesian decision framework aligns with the reasoning they themselves use when selecting tests under uncertainty, particularly in the absence of clear guidelines.\n\nThey also pointed out cases where clinical decision-making diverges from a purely Bayesian approach — for example, when clear guidelines mandate a specific test (such as PCR for COVID 19 diagnosis) or when certain tests are routinely performed despite limited informativeness, such as screening to rule out cancer. However, clinicians agreed that decision support is only needed for the less straightforward cases, where test selection involves genuine uncertainty and no clear protocol exists.\n\n---\n\n## **Writing and Presentation**\nIn response to Reviewer pTAQ and others, we have significantly improved the manuscript’s clarity and presentation. Sections 2 and 3 were rewritten with clearer, standardized notation, while Figures 1 and 2 were clarified for readability. Section 4.1 now reports Wasserstein and Energy distances to better characterize the generated sample distributions, addressing requests for more rigorous evaluation. Additional reviewer suggestions have also been incorporated throughout, resulting in a clearer, more consistent, and more informative presentation.\n\n---\n\n## **References**\n1. Improving Diagnosis in Health Care (Committee on Diagnostic Error in Health Care National Academies Press (US) 2015)  \n2. Derivation of a simple clinical model to categorize patients probability of pulmonary embolism: increasing the model’s utility with the SimpliRED D-dimer (Wells et al. Thrombosis and haemostasis 2000)  \n3. Validation of risk stratification schemes for predicting stroke and thromboembolism in patients with atrial fibrillation: nationwide cohort study (Olesen et al. BMJ 2011)  \n4. Learning to Diagnose: Assimilating Clinical Narratives using Deep Reinforcement Learning (Ling et al., IJCNLP 2017)  \n5. Deep Reinforcement Learning for Cost-Effective Medical Diagnosis (Yu et al. ICLR 2023)  \n6. Hierarchical reinforcement learning for automatic disease diagnosis (Zhong et al. Bioinformatics 2022)  \n\n---\n\n*Thank you for your thoughtful feedback. We appreciate your insights, which have helped improve the quality and clarity of our work. We believe ACTMED provides a practical step toward generalizable, interpretable AI in clinical diagnostics, and we welcome continued dialogue.*"}, {"review_id": "0MRu6JnmEJ", "reviewer": "Reviewer_2RXP", "summary": "The work tried to balance the medical cost and diagnostic accuracy in clinical diagnosis tasks. Based on previous work showing LLM gives very accurate prediction on diagnostics, the authors propose a new pipeline based on greedy selection of test with most information gain with the help of LLM estimating the prior and posterior distribution. Solid experimental studies were conducted to verify the effectiveness of the proposed approach.", "questions": "See weakness", "limitations": "NA", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- Strength\n\nThe methodology itself is quite intuitive and the paper is overally well presented. \n\n- Weakness\n\nGiven that the quality of proposed approach heavily relies on the fact that LLM gives accuracy prediction on diagnostic, is there any quantitative characterization of possible error rate if LLM has some possibility to give wrong predictions? The reviewer believe if such error rate can be well controlled than the proposed method can be more convincing.", "was_revised": false, "final_justification": "", "rebuttal": "*We thank the reviewer for their positive assessment of our work and for recognizing both the intuitive methodology and overall presentation. Below, we address the specific concern regarding potential LLM prediction errors.*\n\n---\n\n## **Quantifying LLM Uncertainty and Error Rates**\nWe agree that the reliability of the framework depends on the accuracy of the LLM-generated diagnostic predictions. To address this, we now provide two complementary evaluations:\n\n### 1.Repeated sampling and bootstrapping: \nFor each patient, we perform 10 independent diagnostic for risk classification and report the mean prediction accuracy with 95% confidence intervals using Bayesian bootstrapping (1). This approach captures the variability of model outputs and allows us to estimate worst-case bounds on prediction accuracy. Table 1 shows the mean, standard deviation, and Bayesian bootstrap confidence intervals for both models across the three datasets with patients stratified by true disease label. Table 2 summarizes the metrics across the entire dataset. A more detailed discussion is included in the Appendix E.\n\n**Table 1:** *Average model prediction accuracy across the hepatitis, diabetes, and CKD datasets (positive and negative cases, both models), with corresponding 95% confidence intervals.*\n\n| Dataset   | Model        | Label | Mean Avg. Risk | Std. Avg. Risk | CI Lower | CI Upper |\n|-----------|--------------|-------|----------------|----------------|----------|----------|\n| Diabetes  | GPT‑4o       | 0     | 0.548          | 0.275          | 0.512    | 0.583    |\n|           | GPT‑4o       | 1     | 0.799          | 0.143          | 0.782    | 0.815    |\n|           | GPT‑4o‑mini  | 0     | 0.716          | 0.223          | 0.686    | 0.744    |\n|           | GPT‑4o‑mini  | 1     | 0.856          | 0.039          | 0.836    | 0.875    |\n| Hepatitis | GPT‑4o       | 0     | 0.093          | 0.129          | 0.081    | 0.106    |\n|           | GPT‑4o       | 1     | 0.561          | 0.320          | 0.539    | 0.583    |\n|           | GPT‑4o‑mini  | 0     | 0.167          | 0.147          | 0.157    | 0.178    |\n|           | GPT‑4o‑mini  | 1     | 0.581          | 0.289          | 0.555    | 0.605    |\n| Kidney    | GPT‑4o       | 0     | 0.039          | 0.033          | 0.029    | 0.051    |\n|           | GPT‑4o       | 1     | 0.907          | 0.060          | 0.898    | 0.916    |\n|           | GPT‑4o‑mini  | 0     | 0.179          | 0.119          | 0.161    | 0.197    |\n|           | GPT‑4o‑mini  | 1     | 0.908          | 0.061          | 0.897    | 0.917    |\n\n### 2.Distribution-level validation: \nWe compute Wasserstein and Energy distances between the generated and true test distributions (see also response to reviewer fFNv), providing a measure of how well the LLM replicates the observed data beyond point predictions.\nAcross all three datasets, ACTMED maintains low distributional distances (Wasserstein ≤ 0.15) and stable performance under repeated sampling. For smaller models that fail to produce realistic distributions, overall accuracy declines and ACTMED does not yield performance gains, as effective Bayesian Experimental Design depends on accurate sampling (see also response to reviewer tmUM).\n\n---\n\n## **Strengthening Experimental Validation**\nIn addition, we have incorporated several experiments requested by other reviewers, including:\n1. Evaluations using open-source models.\n2. Multi-disease diagnostic tasks derived from the AgentClinic dataset.\n3. Expanded clinician-in-the-loop evaluations.\n\n---\n\n## **References**\n1.\tThe Bayesian Bootstrap (Donal B. Rubin, The Annals of Statistics, 1981)\n\n---\n\n*We appreciate the time and effort you have dedicated to reviewing our paper and hope these additions and clarifications address your concerns satisfactorily. We’d be happy to engage in further discussions.*"}, {"review_id": "zAiIadq6zF", "reviewer": "Reviewer_fFNv", "summary": "The paper proposes ACTMED, a step-wise diagnostic reasoning framework combining LLMs generations with Bayesian Experimental Design / Informativeness of actions.\n\nACTMED proposes to take a probabilistic view on diagnostic reasoning and to consider actions (i.e. further diagnostic tests) only if they promise to shift the posterior over diagnoses. This way a sequential sampling over most informative tests is implemented.\n\nOverall, this paper has a strong clinical motivation - in clinical decision making we need to carefully select diagnostic actions that provide most insights and have the potential to reduce uncertainty. \n\nExperiments are quite convincing, even if conducted on maybe a bit simplified tasks. \n\nOverall the paper adds value to the community in my view.", "questions": "- Eq 1 has a in my view weird notation under the \"min\", i would ask to use more standard notation\n- Fig 1 is poorly explained what is shown, it feels quite out of context, and I don't know what it is trying to tell me (and what exactly the distribution is trying to show)\n- Fig 2 again has a bit cluttery and unclear notation in the green boxes on the right (both step 2 and 3).\n- l.112 it's a bit unfortunate to use u_t with different upper indices to indicate both the test and its value/outcome --> this makes the notation a bit overloaded and hard to read - i would disentangle that or use RV vs realization lingo\n- How does this approach compare to general medical experts / clinicians in solving diagnostic tasks based on same / limited input information? \n- I see that lagrange formulation was used for weighing informativeness vs cost, but wouldn't it make sense to actually tune the lambda between those constraints? (Maybe i misunderstood this)\n- Please show how and how accurately the LLM predicts the lab test results.\n- Fig 3: \"Mean predictive performance\" -> i have no clue what is shown and meant here.\n- For surrogate sampling evaluation: pls explain why this minimal MAE / distance approach is justified (as i believe it will go down automatically with enough samples drawn) --> instead distributional metrics would be more interesting\n- l.206 Performance improves with more samples --> this is exactly trivially so if you consider minimal distance to closest generated sample\n- l 227 what is \"all features\" exactly? explain in detail\n- what does ACTMED perf > all features mean exactly? What's the mechanism? Which additional info in all features is hurtful and why?\n- Reader wants dataset details -> ref to App C when introducing tasks in Sec 4.2\n- it's not clear how the \"stop testing and go for final diagnosis\" procedure is implemented -> e.g. prior works like AgentClinic let an agent state \"request test\" vs \"arrive at final diagnosis\". How this is solved here is not so clear\n- Fig 7 what does it mean that implicit method is better than proposed? How to interpret this?\n- Where are CKD results? Fig 5 and 7 only report Hepatitis and Diabetes\n- l 352 says framework allows for clinicians to override suggestions: has this been tested with few clincians? This would strengthen the paper a lot\n- l 354 this paragraph sounds a bit ChatGPT written, please reprhase.\n- Tab 3: ML and DL column are not very clear what is meant: Machine learning and Deep learning? pls explain more clearly the reasoning in caption\n- l 259: Diabetes diagnosis is made (gold standard) with Oral Glucose Tolerance Test (OGTT), not BMI..", "limitations": "-Interaction with human clinician is not properly described.\n- i don't expect negative societal impact", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Probabilistic reasoning is currently underemphasized by the communtiy in the medical LLM space\n- BED makes a lot of sense to use here\n- The KL-based loss is an elegant way to determine the value of diagnostic tests\n- The framework also allows for modeling of costs of individual tests, which makes it more amenable for deployment.\n\nWeaknesses:\n- The experimental part is not super strong: three diagnostic benchmarks are used that are not very typical benchmarks in the medical reasoning / medical LLM community, but rather look like UCI and kaggle benchmarks - which raises the question: why these 3 exactly? (e.g. how do we know that not 6 were tried and 3 reported etc.) Just for sake of comparing to prior literature, it would have been easy to leverage more typical medical reasoning benchmarks such as MedQA, MedExpertQA, AgentClinic etc.\n- One aspect that is not clear to me: to which degree is this KL-based formulation for selecting informative tests novel? I haven't seen this done before with LLMs in medical context, but KL and information theoretic aspects seem quite straight forward, so what exactly is the novelty / most similar related work here?\n- It is not clear to me how costs are modeled? (i.e. simply $ cost of lab test, which would be oversimplifying, as there are many types of costs to diagnostic tests: invasiveness / health risk introduced by test/intervention (e.g. sampling amniotic fluid), opportunistic cost in triage: who will get into CT first in the rural emergency room? \n- Humans are trained to do this sequential diagnostic reasoning, but no comparison to a human baseline is provided to my knowledge: this would strengthen the paper and support the claims made.\n- I do not trust the surrogate sampling evaluation metric: predicting lab tests from other tests should be somewhat doable, but I believe it's misleading to report it to be 99% within reasonable range: to evaluate whether a predicted test is reasonable one should report distributional metrics (distribution matching, wasserstein dist etc.) - not the minimal distance to samples (as this flawed metric will trivially degenerate to 0 if enough samples are drawn for 2 distributions with joint support, to my understanding).", "was_revised": true, "final_justification": "Overall, I think the paper can be accepted. It adds a intuitive and interesting perspective of applying Bayesian experimental design / decision making to LLM-based diagnosis. During rebuttal my concerns were adressed, e.g. around \"percentage in reasonable range\".", "rebuttal": "*We thank the reviewer for their time reviewing our work and their constructive comments.*\n\n---\n\n## **Choice of datasets**\nThe chosen datasets were selected using strict criteria:  \n1. Open source (to avoid restrictions on LLM use and ensure reproducibility).  \n2. Primarily numerical features (as sampling narrow categorical features offers little benefit from LLM reasoning).  \n3. Well-defined clinical meaning and units, enabling realistic simulation.  \n4. The datasets intentionally do not include a single definitive diagnostic test (e.g., a PCR test for COVID-19), as such features would trivialize the decision-making task.  \n\nWhile these differ from typical benchmarks such as MedQA or AgentClinic, few open-source datasets meet these criteria and support dynamic, sequential test acquisition. Traditional benchmarks generally lack the necessary structure for sequential testing, as ACTMED is designed for the second step of diagnosis (optimal test selection once differentials are known) rather than the initial generation of differentials (see response to reviewer tmUM).  \n\nTo address concerns about simplicity, we created a dataset derived from MedQA, selecting 120 patients with sufficient test results. For each, we provide a brief EHR-style summary, including synthetic controls with normal lab values, and compare ACTMED’s performance against baselines. Results (Table 1) show ACTMED achieves performance gains across varied diseases, though differing test panels limit detailed population-level sampling analysis.\n\n**Table 1:** *Performance of ACTMED on the AgentClinic derived dataset against baselines.*\n\n| Model      | Method         | Mean Accuracy | Std. Dev. |\n|------------|----------------|---------------|-----------|\n| GPT‑4o‑mini | ACTMED         | 0.697         | 0.015     |\n|            | Random         | 0.596         | 0.015     |\n|            | Global Best    | 0.673         | 0.016     |\n|            | Implicit       | 0.673         | 0.009     |\n|            | All Features   | 0.674         | 0.012     |\n| GPT‑4o     | ACTMED         | 0.730         | 0.010     |\n|            | Random         | 0.643         | 0.024     |\n|            | Global Best    | 0.714         | 0.005     |\n|            | Implicit       | 0.694         | 0.016     |\n|            | All Features   | 0.712         | 0.007     |\n\n---\n\n## **Novelty of BED with LLMs**\nWhile Bayesian Experimental Design (BED) and KL-divergence–based selection are well-established tools, ACTMED is, to our knowledge, the first framework to integrate BED with large language models as generative priors for clinical reasoning in a zero-shot setting. Unlike prior reinforcement learning or decision tree–based approaches, which demand extensive task-specific training and retraining to accommodate new diseases or tests, ACTMED uses LLMs to generate plausible, patient-specific test outcomes on demand. This enables rapid, adaptive reasoning across diverse tasks and clinical domains without any fine-tuning or reliance on large, labelled datasets—a capability not demonstrated in prior work. By decoupling diagnostic reasoning from model retraining, ACTMED fills a critical gap: enabling sequential, uncertainty-aware diagnostic decisions in contexts where labelled data is scarce or clinical conditions shift rapidly. To our knowledge, the only related work applies BED with entropy gains to coding task disambiguation (1), but not to medicine, and does not employ KL divergence.\n\n---\n\n## **Diagnostic test cost modelling**\nAs the reviewer notes, clinical decision-making often involves balancing the diagnostic value of a test against factors such as invasiveness, availability, and cost. In practice, these trade-offs are typically determined by panels of experienced clinicians rather than by automated systems.  \n\nIn the current study, ACTMED serves solely as a decision-support tool, recommending tests based on their estimated diagnostic benefit alone. For simplicity and comparability, we assumed unit cost for all tests, as the baseline methods used for comparison do not incorporate cost modelling.  \n\nNevertheless, our framework is fully compatible with cost-aware decision-making. Institutions adopting ACTMED could assign test-specific costs, and the model can directly integrate these weights into its test selection procedure.\n\n---\n\n## **Clinician involvement**\nThe performance of LLMs against humans in diagnostic settings has been evaluated in prior studies suggesting that experienced clinicians still perform better in complex scenarios (2) and this is discussed in the paper. The authors would like to reemphasize that ACTMED is designed as a test selection tool to support clinicians not as a standalone diagnostic tool. To strengthen this, we conducted a qualitative clinician review of ACTMED’s reasoning pathways (see response to reviewer tmUM), which confirmed its alignment with expert reasoning under uncertainty.\n\n---\n\n## **Evaluation of LLM-generated samples**  \nWe appreciate the reviewer’s concerns regarding the quality of LLM-generated samples. We emphasize that the purpose of the sampling procedure is not to exactly predict the most likely test result for a given patient but to draw plausible values under both disease and non-disease assumptions. A well-calibrated test will naturally produce roughly half of its simulated values deviating significantly from the true outcome.  \n\nThe “percentage in reasonable range” metric is intended to detect hallucinations rather than assess per-sample accuracy. The reasonable range is empirically defined based on observed data, and the LLM generates samples that fall within this range for 99% of cases, confirming that outputs remain realistic. The full distributions produced are also shown in Appendix E.  \n\nThe “minimum distance” metric will, as noted, tend toward zero if the model can generate sufficiently diverse samples and enough draws are taken (Appendix E). Since ACTMED is constrained to 10 draws per query, this metric serves primarily to ensure that the number of samples drawn for the experiment is large enough for enabling reliable BED computations.  \n\nAddressing the reviewer’s suggestion, we conducted a distribution-level evaluation of generated versus true samples using Wasserstein and Energy distances (see Table 2). We have also updated Figures 3 and 4 to represent these distribution metrics. The manuscript text has been revised for clarity to reflect these results.\n\n**Table 2:** *Normalized Distribution Matching Metrics (Lower is Better)*\n\n| Dataset   | Model        | Avg. Wasserstein (± SD) | Avg. Energy Distance (± SD) |\n|-----------|--------------|--------------------------|------------------------------|\n| Diabetes  | GPT‑4o       | 0.110 ± 0.038           | 0.256 ± 0.079               |\n| Diabetes  | GPT‑4o‑mini  | 0.117 ± 0.046           | 0.269 ± 0.094               |\n| Hepatitis | GPT‑4o       | 0.130 ± 0.157           | 0.265 ± 0.191               |\n| Hepatitis | GPT‑4o‑mini  | 0.173 ± 0.237           | 0.330 ± 0.254               |\n| Kidney    | GPT‑4o       | 0.082 ± 0.055           | 0.203 ± 0.102               |\n| Kidney    | GPT‑4o‑mini  | 0.082 ± 0.057           | 0.203 ± 0.104               |\n \n\n---\n\n## **Interpretation of model results and stoppage criterion** \nCKD results, originally included only in the appendix due to near-perfect performance across all methods, are now presented in Figure 5 for completeness. Figure 7 examines why the implicit baseline underperforms relative to ACTMED. For datasets where ACTMED shows clear performance gains (diabetes and hepatitis), we further analyzed feature selections relative to the globally optimal sets identified prior to observing any data.  \n\nACTMED’s stopping criterion is not based on the model’s state but is determined by the KL divergence between successive belief states. For example, with γ = 0.5, the framework will acquire another test only if the expected shift in belief is at least half the distance to the decision boundary. If this threshold is not met, the system deems itself sufficiently confident that no additional test would meaningfully alter the prediction and halts testing.  \n\nClinical experts involved in the review noted that this rule closely mirrors their own decision-making in practice, where additional testing is pursued only when it is likely to significantly change diagnostic confidence.\n\n---\n\n## **Comparison of ACTMED and full information baseline**\nWe agree with the reviewer that ACTMED outperforming the full features baseline, which is not limited to 3 diagnostic tests but has access to all information in the dataset may seem surprising. We have conducted an experiment involving classification using sequential increases in the number of features to support this (see response to reviewer pTAQ).\n\n---\n\n## **Formatting and minor comments**\nWe standardized mathematical notation (per Reviewer pTAQ’s suggestions) and clarified figure content. Figure 1 now clearly indicates that the distributions shown represent prior and posterior disease probabilities. The green box notation was simplified for clarity. We clarified that “ML” and “DL” in Table 3 refer to “machine learning” and “deep learning,” with examples, and corrected the note on the diabetes dataset: we do not claim BMI is diagnostic, only that OGTT is absent, necessitating indirect metrics. The paragraph on line 354 has been rewritten to better explain how ACTMED supports clinicians.\n\n---\n\n## **References**  \n1. Active Task Disambiguation with LLMs (Kobalczyk et al., ICLR 2025)  \n2. Evaluation and mitigation of the limitations of large language models in clinical decision-making (Hager et al, Nature Medicine 2024)  \n\n---\n\n*We greatly appreciate the time and effort you’ve dedicated to reviewing our paper. We hope that the resulting additional experiments will strengthen the contributions of this paper. We’d be happy to engage in further discussions.*"}, {"review_id": "Mc8oimKBAe", "reviewer": "Reviewer_pTAQ", "summary": "This paper casts clinical diagnosis as a sequential decision making problem: where every step is an action of selecting the next diagnostic test with the aim of diaganosing the patient successfully. To achieve this, they propose ACTMED (Adaptive Clinical Test selection via Model-based Experimental Design) - this method combines Bayesian Experimental Design (BED) with large language models. LLMs are used to (1) simulate plausible test outcomes based on the available information so-far and (2) the disease-probability distribution given the information so far, effectively turning LLMs into on-demand generative priors.Tests are chose based on expected KL div between prior distribution and posterior distribution if a test is conducted (normaized by their costs). This procedural loop continues until no remaining test can meaningfully refine the belief, after which a clinician can inspect, accept, or override the transparent intermediate reasoning—keeping humans firmly “in the loop.” Experiments on three real-world tabular datasets show that ACTMED delivers higher diagnostic accuracy than baseline strategies, cuts the average number of tests, demonstrating gains in accuracy, interpretability, and resource use simultaneously.", "questions": "**Major comments**\n\n(1) Please correct or explain (2) and (3)\n\n(2) It would be helpful to justify the use of KL divergence (as opposed to entropy criteria) as a selection criteria by some concrete theoretical proof and experiment.\n\n(3) How is the LLM calibration (some evaluation on that), and how it impacts performance would be helpful in more concretely bear out the impact underlying LLM accuracy on algorithm's performance.\n\n(4) Improve the writing of Section 2,3 and 4.1\n\n\n\n\n**Minor comments**\n\n(1) Line 142 - Is it possible that prior is not confident enough and none of the test satisfied condition presented in Line 142 - what to do in that case?\n\n(2) Why using all features is worst (Figure 5) is not entirely clear to me - why this should be the case? What if we ask ACTMED do sequentially sample all the features (without early stopping) - will its performance (accuracy) start decreasing after some steps?", "limitations": "Yes there is a dedicated section on the Limitations of the paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**STRENGTHS**\n\n(1) **High-impact, well-motivated problem** – frames diseases diagnosis as personalized, cost-aware decision-making problem that promises earlier detection, lower expenditure, and wider access to care.\n\n(2) **Flexible framework** –  Model agnostic and runs with any without task-specific fine-tuning, suggesting easy portability (although performance depends on the underlying LLM accuracy) ; allows for clinician to be in loop - simulated lab values and belief updates are readable, so experts can audit or override recommendations..\n\n(3) **Thorough empirical study** – on three public datasets, two different models, and comparisons against guideline-style, random, implicit-LLM, and full-feature baseline (although I have concerns with some of them - please see questions section). Additionally Section 4.3 is an interesting read. \n\n\n**WEAKENESSES**\n\n(1) **Notational inconsistency** - following terms are not clearly defined and are hard to read - prior, posterior over ''disease probability`` are not clearly defined and are used interchangeably with prior and posterior over ''disease label''  (which conflates epistemic and aleatoric uncertainty). Similarly for the distribution of test outcomes and other quantities.  \n\n(2) **Equation (2) seems incorrect** - Expectation should be over different posteriors (due to different test outcomes). KL divergence should be between prior and posterior of the ``disease probability''. And not on samples basis as done in Equation (2). Explained below: Let $\\theta$ be the probability that person has disease or not ($Y=\\{0,1\\}$), that is $Y \\sim p_\\theta$. And we have a prior $\\pi(\\theta|K_t)$ on $\\theta$ at time step $t$ given information $K_t$. Overall $P(Y) =\\int p_\\theta (Y) \\pi(\\theta|K_t) d\\theta$. Now, the test outcome $u_t^{i} \\sim P(u_t^{i}|K_t)$, then we are intersted in\n$$ E_{u_t^{i} \\sim P(u_t^{i}|K_t)} KL(\\pi(\\theta|K_t) || \\pi(\\theta|K_t,u_t^{i}))$$ which can be approximate by\n$ \\frac{1}{K} \\sum_{j=1}^K KL(\\pi(\\theta|K_t) || \\pi(\\theta|K_t,u_t^{i,j}))$.\n\nNow, $KL(\\pi(\\theta|K_t) || \\pi(\\theta|K_t,u_t^{i,j}))$ will depend on distributions $\\pi(\\theta|K_t)$ and $\\pi(\\theta|K_t,u_t^{i,j})$ which are not bernoulli but something else. \n\n(3) Rationale behind Equation (3) is not given.\n\n(4) Applications are limited to tasks on which GPT-4o performs good - it seems for other tasks finetuning would be crucial.\n\n(5) Overall writing and presentation needs improvement - especially Sections 2 and 3, Subsection 4.1", "was_revised": true, "final_justification": "The authors have addressed most of my concerns and added additional experiments to support it. The paper’s writing and exposition still needs some refinement, I am raising my score by 1 point to weak accept.", "rebuttal": "*We thank the reviewer for their helpful feedback, particularly on enhancing the clarity of the mathematical formulation.*\n\n---\n\n## **Formulation of the BED using KL divergence**\n\nWe now clearly distinguish between priors and posteriors over *disease probabilities* versus *disease labels*, and we clarify that the KL divergence in Eq.​(2) is computed between *probability* distributions (not between the labels themselves).\n\n### 1. Epistemic vs. Aleatoric in Active Learning  \n\nIn Active Learning (common BED framework), we aim to reduce the epistemic uncertainty by acquiring the label of the most informative sample $x_o$, which.  \n\n$\\underbrace{I(Y;Z\\mid, x_o, d)}_{\\text{epistemic}} = H(Y\\mid d, x_o)-   \\underbrace{H(Y\\mid Z,d, x_o)}\\_{\\text{aleatoric}} ,$  \n\nwhere $Z$ is a R.V. representing the latent hypothesis (e.g. parameters distribution), $d$ denotes current data, $H(\\cdot)$ is entropy and $I(\\cdot)$ mutual information. \n* **Epistemic** $=$predictive $-$aleatoric\n\n---\n\n### 2. Active Feature Acquisition (AFA)  \n\nOur setting is closer to Active Feature Acquisition, where we ask how much a candidate feature $F_j$ reduces **predictive** uncertainty in $Y$:  \n\n$\\displaystyle I(Y;F_j)=H(Y)-H(Y\\mid F_j).$  \n\nBecause $H(Y)$ is constant across features, maximizing $I(Y;F_j)$ is equivalent to minimizing $H(Y\\mid F_j)$; the most informative test is the one that lowers entropy most.\n\n---\n\n### 3. Our KL‑Based Criterion and Its Derivation  \n\nWhen computing expected information gain we may work interchangeably in entropy or KL‑divergence space (1).  Equation (2) evaluates  \n\n$E[\\mathrm{KL}] = -H(Y \\mid F_j) + \\mathrm{CE}\\bigl(Y, Y_{\\text{prior}}\\bigr).$\n\n\nwhere  \n* **Term A** ($-H(Y\\mid F_j)$) is exactly the uncertainty‑reduction term in AFA.  \n* **Term B** is a cross‑entropy that rewards tests which move beliefs away from an uninformative prior, thus avoiding redundant queries.  \n\n> **Note on notation:** the prior probability $p_{\\text{prior}}$ is constant and no longer carries an index $j$, this has been corrected in the manuscript.\n\n#### *Concrete derivation*  \nFor a single Monte‑Carlo sample we compute  \n\n$p_{\\text{post}}\\log \\frac{p_{\\text{post}}}{p_{\\text{prior}}}+ (1-p_{\\text{post}})\\log \\frac{1-p_{\\text{post}}}{1-p_{\\text{prior}}}$  \n\nRe‑arranging,\n\n$= \\bigl[p_{\\text{post}}\\log p_{\\text{post}} +(1-p_{\\text{post}})\\log(1-p_{\\text{post}})\\bigr]$\n$- \\bigl[p_{\\text{post}}\\log p_{\\text{prior}} +(1-p_{\\text{post}})\\log(1-p_{\\text{prior}})\\bigr]$\n\nWhere the term in the first brackets corresponds to $A$ and the second to term $B$. Averaging over $M$ such samples yields the expectation in Eq.​(2). The full derivation has now been included in Appendix B.\n\n---\n\n## **Clarity on the utility function notation**\nFirst, **Equation (1) is now expressed in standard minimisation form** as suggested by reviewer fFNv:\n\n$\\mathcal{L}(y,u_t,\\lambda)=\\sum_t \\mathbb{I}[y\\neq y_{d_t}]+\\lambda\\sum_t c(u_t)$\n\nSecond, **to prevent symbol overload we separate the random variable from its realisation**:  \nthroughout Section 2 (lines 110–114), $u_t$ denotes the *type* of diagnostic test (a random variable), while $u_t'$ denotes the *observed outcome* of that test.\n\nWe relax the dual objective with  \n$\\mathcal{F}(u_t^i)=\\frac{I(u_t^i)}{c^*(u_t^i)},$\n\ntaking \n$c(u)=\\log c^{u}$ \nto model diminishing marginal penalty.  All experiments assume unit costs \n$(c^*=1)$ \nso baselines remain directly comparable; real monetary or invasiveness costs can be inserted transparently.  Normalizing information gain by cost gives an intuitive “information‑per‑dollar” score and removes the need to hand‑tune a dataset‑specific $\\lambda$. The gain per unit cost formulation is also common in medicine (2).\n\n## **Use of KL Divergence versus Entropy for BED**\n\nIn Appendix B, we present a theoretical example illustrating how an entropy-based criterion can fail to identify the most informative test, specifically in cases where entropy is high but concentrated on outcomes that provide little diagnostic value. Both entropy and KL divergence can, in principle, be used for test selection (1), and there is no universal proof favoring one over the other; their relative effectiveness depends on the specific setting and data distribution. However, in our experiments, we provide both a theoretical justification and empirical evidence supporting KL divergence for our application. Across all datasets, the results (Table 1) demonstrate that KL-based selection consistently improves diagnostic accuracy and reduces redundant testing relative to entropy-based selection.\n\n**Table 1:** *Performance of ACTMED using KL divergence versus entropy for test selection.* \n\n| Model       | Dataset   | KL-based (Mean ± Std) | Entropy-based (Mean ± Std) |\n|-------------|-----------|------------------------|-----------------------------|\n| GPT-4o-mini | Diabetes  | 0.593 ± 0.039         | 0.572 ± 0.005              |\n| GPT-4o-mini | Kidney    | 0.992 ± 0.006         | 0.972 ± 0.010              |\n| GPT-4o-mini | Hepatitis | 0.825 ± 0.013         | 0.770 ± 0.010              |\n| GPT-4o      | Diabetes  | 0.682 ± 0.012         | 0.673 ± 0.011              |\n| GPT-4o      | Kidney    | 0.999 ± 0.003         | 0.994 ± 0.000              |\n| GPT-4o      | Hepatitis | 0.839 ± 0.013         | 0.770 ± 0.017              |\n\n---\n\n## **Performance Degradation with All Features**\n\nWe confirm that ACTMED’s accuracy can decline when too many features are presented at once. Following the reviewer’s suggestion, we added an experiment examining performance as features are added sequentially (Table 2).  \n\nThe results show that ACTMED avoids this “information overload” by stopping early when further tests offer little diagnostic value and only selecting informative tests. Similar degradation with excessive input has been observed in prior work in different contexts (3, 4).\n\nOne contributing factor is that the models tend to assume higher disease risk as more lab results are presented, even when results are normal (Table 3). This bias is especially evident with GPT‑4o mini.\n\n---\n\n**Table 2:** *Accuracy versus Number of Sequentially Added Tests*\n\n| Dataset   | Model        | 1 Test            | 3 Tests           | 5 Tests           | All Tests         |\n|-----------|--------------|--------------------|--------------------|--------------------|--------------------|\n| Kidney    | GPT‑4o       | 0.994 ± 0.004     | 0.995 ± 0.003     | 0.995 ± 0.003     | 1.000 ± 0.000     |\n|           | GPT‑4o‑mini  | 0.975 ± 0.011     | 0.983 ± 0.005     | 0.990 ± 0.010     | 0.972 ± 0.006     |\n| Hepatitis | GPT‑4o       | 0.589 ± 0.023     | 0.679 ± 0.022     | 0.718 ± 0.013     | 0.796 ± 0.010     |\n|           | GPT‑4o‑mini  | 0.588 ± 0.030     | 0.698 ± 0.022     | 0.732 ± 0.010     | 0.798 ± 0.011     |\n| Diabetes  | GPT‑4o       | 0.730 ± 0.138     | 0.662 ± 0.170     | 0.664 ± 0.170     | 0.678 ± 0.161     |\n|           | GPT‑4o‑mini  | 0.546 ± 0.015     | 0.524 ± 0.025     | 0.528 ± 0.016     | 0.458 ± 0.007     |\n\n---\n\n**Table 3:** *Impact of Additional Tests on Risk Estimation*\n\n| Dataset   | Model        | 1 Test            | 3 Tests           | 5 Tests           | All Features      |\n|-----------|--------------|--------------------|--------------------|--------------------|--------------------|\n| Kidney    | GPT‑4o       | 0.259 ± 0.007     | 0.267 ± 0.003     | 0.273 ± 0.004     | 0.274 ± 0.002     |\n|           | GPT‑4o‑mini  | 0.292 ± 0.007     | 0.330 ± 0.010     | 0.349 ± 0.009     | 0.377 ± 0.003     |\n| Hepatitis | GPT‑4o       | 0.209 ± 0.012     | 0.255 ± 0.008     | 0.285 ± 0.018     | 0.326 ± 0.005     |\n|           | GPT‑4o‑mini  | 0.275 ± 0.019     | 0.307 ± 0.009     | 0.323 ± 0.015     | 0.374 ± 0.003     |\n| Diabetes  | GPT‑4o       | 0.575 ± 0.062     | 0.638 ± 0.013     | 0.698 ± 0.096     | 0.687 ± 0.082     |\n|           | GPT‑4o‑mini  | 0.606 ± 0.014     | 0.696 ± 0.018     | 0.725 ± 0.013     | 0.768 ± 0.004     |\n\n---\n\n## **LLM Performance within ACTMED**\n\nWe agree that the performance of ACTMED depends on the underlying LLM’s ability to generate plausible test outcomes. As noted in our response to Reviewer tmUM, a 7B parameter Biomistral model produced unrealistic samples, limiting ACTMED’s gains. While fine‑tuning could improve small models for specific tasks, our results suggest that ACTMED benefits most from larger, general models capable of diverse, accurate sampling.\n\n---\n\n## **Test Stopping Criterion**\n\nIf no test meets the KL divergence threshold, ACTMED stops acquiring tests. This reflects many clinical situations, where a confident diagnosis can be made based on history and presentation alone (often referred to as diagnosed clinically by doctors).  \n\nIn our experiments, this situation was rare, as each patient was provided with only basic demographic and symptom information initially, so additional testing was typically required. However, in principle it would be possible for the model to diagnose a patient “clinically” without requesting further tests.\n\n---\n\n## **Writing and Presentation Improvements**\n\nSections 2 and 3 have been revised to include the more consistent notation. Section 4.1 has also been changed to incorporate the distribution analysis metrics as suggested by Reviewer fFNv. Figures 1 and 2 have been simplified and re-captioned as requested by Reviewer fFNv. Figures 3 and 4 now reflect distributional evaluation metrics.\n\n---\n\n## **References**\n\n1. Modern Bayesian Experimental Design (Rainforth et al. Statist. Sci. 2024)\n2. Cost-effectiveness Thresholds Used by Study Authors, 1990-2021 (Neumann  et al. JAMA 2023)\n3. Large Language Models Can Be Easily Distracted by Irrelevant Context (Shi et al., ICML 2023)  \n4. Inverse Scaling in Test-Time Compute (Gema et al., arXiv 2025)\n\n---\n\n*We appreciate the time and effort you have dedicated to reviewing our paper and hope these additions and clarifications address your concerns satisfactorily. We’d be happy to engage in further discussions.*"}, {"review_id": "7d8iFML9ec", "reviewer": "Reviewer_JEyC", "summary": "This paper introduces ACTMED, a model-agnostic, adaptive diagnostic framework that supports timely and resource-aware clinical decision-making. The core idea is to integrate Bayesian Experimental Design (BED) with large language models (LLMs) to guide sequential test selection for individual patients. Rather than relying on static, fully observed datasets, ACTMED mimics how clinicians reason over time—prioritizing tests that are expected to reduce diagnostic uncertainty the most (see Figure 2 of the paper).\nAt each step, ACTMED uses LLMs (e.g., GPT-4o) as surrogate models to simulate plausible patient test outcomes and to estimate the diagnostic utility of candidate tests via expected KL divergence. A stopping rule is introduced to avoid redundant testing, based on whether further tests are expected to meaningfully shift diagnostic beliefs.", "questions": "1. Why does the proposed method outperform models that use the full set of features in the simulation?\n2. How does the method address potential overfitting, especially given the small sample sizes used in evaluation?\n3. How does this approach compare to a pipeline where LLMs are used for imputing missing features, followed by conventional classification models?\n4. In the binary diagnostic setting, how are the “true” diagnostic probability distributions defined and validated?\n5. How does the proposed framework scale to and perform on larger, more complex clinical datasets such as MIMIC-III/IV, UK Biobank, or All of Us?\n6. How would the clinician-in-the-loop component operate in practice—can clinicians override the model’s test selection, and is this interaction supported in the current framework?", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper addresses a well-motivated and clinically important problem: how to make timely diagnoses in a cost-effective manner.\n2. The use of large language models (LLMs) as a flexible sampling engine for test selection is an interesting/potentially novel idea.\n\nWeaknesses\n1. The diagnosis task is limited to binary classification. It is unclear where the “true” diagnostic probability distributions come from, especially given the reliance on LLM-generated samples.\n2. The method is framed as an agent-based decision process, but it lacks comparison to simpler alternatives such as using LLMs for missing data imputation followed by standard prediction.\n3. The paper claims clinicians can be kept in the loop, but this is not demonstrated or evaluated in the experiments.\n4. The real-world examples used for validation (e.g., hepatitis, diabetes, CKD datasets) are small and outdated, with only ~100 patients and ~20 covariates. This limits the assessment of scalability and clinical relevance.", "was_revised": true, "final_justification": "I appreciate the authors' efforts and have increased my score to 3. However, the concern remains: the statement that “scaling ACTMED to very large clinical datasets is challenging” raises questions about whether the reported performance gains may be due to overfitting.", "rebuttal": "*We thank the reviewer for their constructive comments. We have addressed the reviewer’s concerns below:*\n\n---\n\n## **Binary classification and diagnostic probability distributions**\nBinary classification in a one vs all manner is a common approach in medicine once a differential diagnosis list has been established for test selection (see response to Reviewer tmUM). ACTMED does not assume access to a ground-truth diagnostic probability distribution. Instead, it leverages LLM-generated samples to approximate plausible posteriors under different hypothetical test outcomes. This does mean performance depends on the accuracy of the LLM generated samples and probability estimates (see response to Reviewer tmUM), but we demonstrate that this approach improves downstream classification accuracy. The test that most meaningfully shifts the model’s disease probability estimate is recommended. Final predictions are then evaluated against ground truth labels, allowing comparison of test selection quality across methods.\n\n---\n\n## **Comparison to LLM-based imputation pipelines**\nWhile pipelines that combine LLM based imputation with classifiers (1) can work well, they typically require task specific labelled training data, risk introducing distributional biases, and do not support sequential, adaptive feature acquisition. ACTMED is designed to operate zero shot and focuses on optimizing which tests to perform rather than replacing clinician driven diagnosis. We have added a more detailed discussion of imputation methods, including those using LLMs, in the related work section.\n\n---\n\n## **Clinician-in-the-loop**\nWe have now included a qualitative evaluation with experienced clinicians and senior medical students, who reviewed ACTMED’s reasoning pathways (see response to reviewer tmUM). Clinicians valued ACTMED’s ability to suggest tests when they were uncertain, its intuitive stopping rule, and its supportive nature. As detailed in the study, clinicians can examine the intermediate outputs and assess how the model is changing its risk in response to the lab value changes. Clinicians can intervene at three points: by adjusting the predicted lab values, modifying the assigned risk estimate, or overriding the final test recommendation. In practice, reviewing distributions and risk changes can be time consuming, so most clinicians under time pressure would focus primarily on evaluating the suggested test. After incorporating the clinician’s input, whether it is confirming the suggested test or selecting a different one, ACTMED updates its predictions and proceeds accordingly.\n\n---\n\n## **On Dataset Scale and Realism**\nScaling ACTMED to very large clinical datasets is challenging due to (a) privacy restrictions on LLM usage and (b) the fact that many datasets only record a single test trajectory per patient, limiting their suitability for sequential testing studies. We address this by dynamically constructing multiple synthetic trajectories from tabular data to generate EHR style notes as tests are selected.\nWe emphasize that the cited datasets contained roughly 100 patients only but this was the size of the test set used in the model evaluation. ACTMED is not subject to overfitting in the conventional sense, as it performs no fine-tuning and does not update its parameters based on the evaluation data. Its zero-shot design ensures it avoids learning dataset-specific shortcuts. In contrast, baselines that rely on full-feature classification are more prone to overfitting, especially when operating on small, fully observed datasets. While the datasets are older, the clinical challenges and measurements remain relevant, as discussed in Section 4.\n\nTo strengthen the evaluation, we extended the diabetes analysis to include all 768 patients (Table 1) and added a more complex evaluation on an AgentClinic-derived dataset (see response to Reviewer fFNv). Together, these bring the total number of evaluated patients to over 1,000 and cover a wider range of covariates.\n\n**Table 1:** *Performance of GPT-4o and GPT-4o-mini on the full diabetes dataset (n=768) against the baselines.*\n| Model       | Method         | Mean   | Std   |\n|-------------|----------------|--------|-------|\n| GPT-4o-mini | ACTMED         | 0.593  | 0.039 |\n| GPT-4o-mini | Random         | 0.505  | 0.027 |\n| GPT-4o-mini | Global Best    | 0.485  | 0.042 |\n| GPT-4o-mini | Implicit       | 0.488  | 0.040 |\n| GPT-4o-mini | All Features   | 0.451  | 0.062 |\n| GPT-4o      | ACTMED         | 0.682  | 0.012 |\n| GPT-4o      | Random         | 0.565  | 0.014 |\n| GPT-4o      | Global Best    | 0.540  | 0.008 |\n| GPT-4o      | Implicit       | 0.544  | 0.004 |\n| GPT-4o      | All Features   | 0.584  | 0.011 |\n\n---\n\n## **References** \n1.\tActive Learning with LLMs for Partially Observed and Cost-Aware Scenarios (Astorga et al. NeurIPS 2024)\n\n---\n\n*Thank you for your thoughtful feedback. We appreciate your insights, which have helped improve the quality and clarity of our work. We believe ACTMED provides a practical step toward generalizable, interpretable AI in clinical diagnostics, and we welcome continued dialogue.*"}]

Author rebuttal:

Meta-review: The paper introduces a diagnostic framework that integrates (Bayesian) experimental design with LLMs to guide adaptive clinical test selection; its goal is to improve accuracy, interpretability, and cost-effectiveness across datasets. Reviewers appreciated the novelty and the application relevance of the approach, as well as the extensive experiments and clinician-in-the-loop evaluations. However, they expressed reservations about the evaluation design, including reliance on closed-source models, simplified binary tasks, omission of free-text EHR features, and questions about whether the approach is truly model-agnostic. Overall sentiment is cautiously positive: reviewers found the idea important and timely, but felt the claims and experiments needed stronger grounding. To strengthen the work, the authors should candidly discuss its limitations—such as dependence on strong LLMs, restricted evaluation settings, and gaps in multimodal or multi-label clinical contexts—so that readers can better understand the scope, generalizability, and future directions of the proposed framework.
Decision: Accept (poster)