Title: Timely Clinical Diagnosis through Active Test Selection
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLMs

Clinical Diagnosis

Bayesian Experimental Design

BED

Active learning
Abstract: There is growing interest in using machine learning (ML) to support clinical diagnosis, but most approaches rely on static, fully observed datasets and fail to reflect the sequential, resource-aware reasoning clinicians use in practice. Diagnosis remains complex and error prone, especially in high-pressure or resource-limited settings, underscoring the need for frameworks that help clinicians make timely and cost-effective decisions. We propose ACTMED (Adaptive Clinical Test selection via Model-based Experimental Design), a diagnostic framework that integrates Bayesian Experimental Design (BED) with large language models (LLMs) to better emulate real-world diagnostic reasoning. At each step, ACTMED selects the test expected to yield the greatest reduction in diagnostic uncertainty for a given patient. LLMs act as flexible simulators, generating plausible patient state distributions and supporting belief updates without requiring structured, task-specific training data. Clinicians can remain in the loop; reviewing test suggestions, interpreting intermediate outputs, and applying clinical judgment throughout. We evaluate ACTMED on real-world datasets and show it can optimize test selection to improve diagnostic accuracy, interpretability, and resource use. This represents a step toward transparent, adaptive, and clinician-aligned diagnostic systems that generalize across settings with reduced reliance on domain-specific data.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ACTMED, a model-agnostic, adaptive diagnostic framework that supports timely and resource-aware clinical decision-making. The core idea is to integrate Bayesian Experimental Design (BED) with large language models (LLMs) to guide sequential test selection for individual patients. Rather than relying on static, fully observed datasets, ACTMED mimics how clinicians reason over time—prioritizing tests that are expected to reduce diagnostic uncertainty the most (see Figure 2 of the paper).\nAt each step, ACTMED uses LLMs (e.g., GPT-4o) as surrogate models to simulate plausible patient test outcomes and to estimate the diagnostic utility of candidate tests via expected KL divergence. A stopping rule is introduced to avoid redundant testing, based on whether further tests are expected to meaningfully shift diagnostic beliefs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper addresses a well-motivated and clinically important problem: how to make timely diagnoses in a cost-effective manner.\n2. The use of large language models (LLMs) as a flexible sampling engine for test selection is an interesting/potentially novel idea.\n\nWeaknesses\n1. The diagnosis task is limited to binary classification. It is unclear where the “true” diagnostic probability distributions come from, especially given the reliance on LLM-generated samples.\n2. The method is framed as an agent-based decision process, but it lacks comparison to simpler alternatives such as using LLMs for missing data imputation followed by standard prediction.\n3. The paper claims clinicians can be kept in the loop, but this is not demonstrated or evaluated in the experiments.\n4. The real-world examples used for validation (e.g., hepatitis, diabetes, CKD datasets) are small and outdated, with only ~100 patients and ~20 covariates. This limits the assessment of scalability and clinical relevance.\n\nQUESTIONS:\n1. Why does the proposed method outperform models that use the full set of features in the simulation?\n2. How does the method address potential overfitting, especially given the small sample sizes used in evaluation?\n3. How does this approach compare to a pipeline where LLMs are used for imputing missing features, followed by conventional classification models?\n4. In the binary diagnostic setting, how are the “true” diagnostic probability distributions defined and validated?\n5. How does the proposed framework scale to and perform on larger, more complex clinical datasets such as MIMIC-III/IV, UK Biobank, or All of Us?\n6. How would the clinician-in-the-loop component operate in practice—can clinicians override the model’s test selection, and is this interaction supported in the current framework?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper casts clinical diagnosis as a sequential decision making problem: where every step is an action of selecting the next diagnostic test with the aim of diaganosing the patient successfully. To achieve this, they propose ACTMED (Adaptive Clinical Test selection via Model-based Experimental Design) - this method combines Bayesian Experimental Design (BED) with large language models. LLMs are used to (1) simulate plausible test outcomes based on the available information so-far and (2) the disease-probability distribution given the information so far, effectively turning LLMs into on-demand generative priors.Tests are chose based on expected KL div between prior distribution and posterior distribution if a test is conducted (normaized by their costs). This procedural loop continues until no remaining test can meaningfully refine the belief, after which a clinician can inspect, accept, or override the transparent intermediate reasoning—keeping humans firmly “in the loop.” Experiments on three real-world tabular datasets show that ACTMED delivers higher diagnostic accuracy than baseline strategies, cuts the average number of tests, demonstrating gains in accuracy, interpretability, and resource use simultaneously.\n\nSTRENGTHS AND WEAKNESSES:\n**STRENGTHS**\n\n(1) **High-impact, well-motivated problem** – frames diseases diagnosis as personalized, cost-aware decision-making problem that promises earlier detection, lower expenditure, and wider access to care.\n\n(2) **Flexible framework** –  Model agnostic and runs with any without task-specific fine-tuning, suggesting easy portability (although performance depends on the underlying LLM accuracy) ; allows for clinician to be in loop - simulated lab values and belief updates are readable, so experts can audit or override recommendations..\n\n(3) **Thorough empirical study** – on three public datasets, two different models, and comparisons against guideline-style, random, implicit-LLM, and full-feature baseline (although I have concerns with some of them - please see questions section). Additionally Section 4.3 is an interesting read. \n\n\n**WEAKENESSES**\n\n(1) **Notational inconsistency** - following terms are not clearly defined and are hard to read - prior, posterior over ''disease probability`` are not clearly defined and are used interchangeably with prior and posterior over ''disease label''  (which conflates epistemic and aleatoric uncertainty). Similarly for the distribution of test outcomes and other quantities.  \n\n(2) **Equation (2) seems incorrect** - Expectation should be over different posteriors (due to different test outcomes). KL divergence should be between prior and posterior of the ``disease probability''. And not on samples basis as done in Equation (2). Explained below: Let $\\theta$ be the probability that person has disease or not ($Y=\\{0,1\\}$), that is $Y \\sim p_\\theta$. And we have a prior $\\pi(\\theta|K_t)$ on $\\theta$ at time step $t$ given information $K_t$. Overall $P(Y) =\\int p_\\theta (Y) \\pi(\\theta|K_t) d\\theta$. Now, the test outcome $u_t^{i} \\sim P(u_t^{i}|K_t)$, then we are intersted in\n$$ E_{u_t^{i} \\sim P(u_t^{i}|K_t)} KL(\\pi(\\theta|K_t) || \\pi(\\theta|K_t,u_t^{i}))$$ which can be approximate by\n$ \\frac{1}{K} \\sum_{j=1}^K KL(\\pi(\\theta|K_t) || \\pi(\\theta|K_t,u_t^{i,j}))$.\n\nNow, $KL(\\pi(\\theta|K_t) || \\pi(\\theta|K_t,u_t^{i,j}))$ will depend on distributions $\\pi(\\theta|K_t)$ and $\\pi(\\theta|K_t,u_t^{i,j})$ which are not bernoulli but something else. \n\n(3) Rationale behind Equation (3) is not given.\n\n(4) Applications are limited to tasks on which GPT-4o performs good - it seems for other tasks finetuning would be crucial.\n\n(5) Overall writing and presentation needs improvement - especially Sections 2 and 3, Subsection 4.1\n\nQUESTIONS:\n**Major comments**\n\n(1) Please correct or explain (2) and (3)\n\n(2) It would be helpful to justify the use of KL divergence (as opposed to entropy criteria) as a selection criteria by some concrete theoretical proof and experiment.\n\n(3) How is the LLM calibration (some evaluation on that), and how it impacts performance would be helpful in more concretely bear out the impact underlying LLM accuracy on algorithm's performance.\n\n(4) Improve the writing of Section 2,3 and 4.1\n\n\n\n\n**Minor comments**\n\n(1) Line 142 - Is it possible that prior is not confident enough and none of the test satisfied condition presented in Line 142 - what to do in that case?\n\n(2) Why using all features is worst (Figure 5) is not entirely clear to me - why this should be the case? What if we ask ACTMED do sequentially sample all the features (without early stopping) - will its performance (accuracy) start decreasing after some steps?\n\nLIMITATIONS:\nYes there is a dedicated section on the Limitations of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes ACTMED, a step-wise diagnostic reasoning framework combining LLMs generations with Bayesian Experimental Design / Informativeness of actions.\n\nACTMED proposes to take a probabilistic view on diagnostic reasoning and to consider actions (i.e. further diagnostic tests) only if they promise to shift the posterior over diagnoses. This way a sequential sampling over most informative tests is implemented.\n\nOverall, this paper has a strong clinical motivation - in clinical decision making we need to carefully select diagnostic actions that provide most insights and have the potential to reduce uncertainty. \n\nExperiments are quite convincing, even if conducted on maybe a bit simplified tasks. \n\nOverall the paper adds value to the community in my view.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Probabilistic reasoning is currently underemphasized by the communtiy in the medical LLM space\n- BED makes a lot of sense to use here\n- The KL-based loss is an elegant way to determine the value of diagnostic tests\n- The framework also allows for modeling of costs of individual tests, which makes it more amenable for deployment.\n\nWeaknesses:\n- The experimental part is not super strong: three diagnostic benchmarks are used that are not very typical benchmarks in the medical reasoning / medical LLM community, but rather look like UCI and kaggle benchmarks - which raises the question: why these 3 exactly? (e.g. how do we know that not 6 were tried and 3 reported etc.) Just for sake of comparing to prior literature, it would have been easy to leverage more typical medical reasoning benchmarks such as MedQA, MedExpertQA, AgentClinic etc.\n- One aspect that is not clear to me: to which degree is this KL-based formulation for selecting informative tests novel? I haven't seen this done before with LLMs in medical context, but KL and information theoretic aspects seem quite straight forward, so what exactly is the novelty / most similar related work here?\n- It is not clear to me how costs are modeled? (i.e. simply $ cost of lab test, which would be oversimplifying, as there are many types of costs to diagnostic tests: invasiveness / health risk introduced by test/intervention (e.g. sampling amniotic fluid), opportunistic cost in triage: who will get into CT first in the rural emergency room? \n- Humans are trained to do this sequential diagnostic reasoning, but no comparison to a human baseline is provided to my knowledge: this would strengthen the paper and support the claims made.\n- I do not trust the surrogate sampling evaluation metric: predicting lab tests from other tests should be somewhat doable, but I believe it's misleading to report it to be 99% within reasonable range: to evaluate whether a predicted test is reasonable one should report distributional metrics (distribution matching, wasserstein dist etc.) - not the minimal distance to samples (as this flawed metric will trivially degenerate to 0 if enough samples are drawn for 2 distributions with joint support, to my understanding).\n\nQUESTIONS:\n- Eq 1 has a in my view weird notation under the \"min\", i would ask to use more standard notation\n- Fig 1 is poorly explained what is shown, it feels quite out of context, and I don't know what it is trying to tell me (and what exactly the distribution is trying to show)\n- Fig 2 again has a bit cluttery and unclear notation in the green boxes on the right (both step 2 and 3).\n- l.112 it's a bit unfortunate to use u_t with different upper indices to indicate both the test and its value/outcome --> this makes the notation a bit overloaded and hard to read - i would disentangle that or use RV vs realization lingo\n- How does this approach compare to general medical experts / clinicians in solving diagnostic tasks based on same / limited input information? \n- I see that lagrange formulation was used for weighing informativeness vs cost, but wouldn't it make sense to actually tune the lambda between those constraints? (Maybe i misunderstood this)\n- Please show how and how accurately the LLM predicts the lab test results.\n- Fig 3: \"Mean predictive performance\" -> i have no clue what is shown and meant here.\n- For surrogate sampling evaluation: pls explain why this minimal MAE / distance approach is justified (as i believe it will go down automatically with enough samples drawn) --> instead distributional metrics would be more interesting\n- l.206 Performance improves with more samples --> this is exactly trivially so if you consider minimal distance to closest generated sample\n- l 227 what is \"all features\" exactly? explain in detail\n- what does ACTMED perf > all features mean exactly? What's the mechanism? Which additional info in all features is hurtful and why?\n- Reader wants dataset details -> ref to App C when introducing tasks in Sec 4.2\n- it's not clear how the \"stop testing and go for final diagnosis\" procedure is implemented -> e.g. prior works like AgentClinic let an agent state \"request test\" vs \"arrive at final diagnosis\". How this is solved here is not so clear\n- Fig 7 what does it mean that implicit method is better than proposed? How to interpret this?\n- Where are CKD results? Fig 5 and 7 only report Hepatitis and Diabetes\n- l 352 says framework allows for clinicians to override suggestions: has this been tested with few clincians? This would strengthen the paper a lot\n- l 354 this paragraph sounds a bit ChatGPT written, please reprhase.\n- Tab 3: ML and DL column are not very clear what is meant: Machine learning and Deep learning? pls explain more clearly the reasoning in caption\n- l 259: Diabetes diagnosis is made (gold standard) with Oral Glucose Tolerance Test (OGTT), not BMI..\n\nLIMITATIONS:\n-Interaction with human clinician is not properly described.\n- i don't expect negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The work tried to balance the medical cost and diagnostic accuracy in clinical diagnosis tasks. Based on previous work showing LLM gives very accurate prediction on diagnostics, the authors propose a new pipeline based on greedy selection of test with most information gain with the help of LLM estimating the prior and posterior distribution. Solid experimental studies were conducted to verify the effectiveness of the proposed approach.

STRENGTHS AND WEAKNESSES:
- Strength

The methodology itself is quite intuitive and the paper is overally well presented. 

- Weakness

Given that the quality of proposed approach heavily relies on the fact that LLM gives accuracy prediction on diagnostic, is there any quantitative characterization of possible error rate if LLM has some possibility to give wrong predictions? The reviewer believe if such error rate can be well controlled than the proposed method can be more convincing.

QUESTIONS:
See weakness

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an LLM-based clinical diagnosis framework that actively selects and updates its belief on tests to make decisions on disease states. Experiments were conducted on three different datasets using two GPT models and the results were also compared with various feature selection-based baseline approaches using the same models and risk prediction prompts. Results showed the effectiveness of the proposed methods compared to considered approaches and baselines across various capabilities including timely diagnosis and transparent explanations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper addresses an important problem of clinical diagnosis using LLMs and a probabilistic framework, where LLM is used as a flexible simulator to predict test outcomes to update the disease belief states while it does not need any task-specific training.\n\n- The proposed approach is agnostic to LLM model architecture and shows promising results within the considered experimental setup and comparisons with various baselines.\n\n- Rigorous experiments have been carried out on 3 datasets demonstrating the generalizability of the proposed methods.\n\n- Extended results with clinician-in-the-loop review process and computational cost analysis have been reported that would be beneficial for the community. \n\n\nWeaknesses:\n\n- Evaluation is weak given the paper considers only two closed source GPT-based models for experimentation lacking other open-source models, hindering detailed understanding of why the proposed methods work or do not work in certain cases and reproducibility of the experiments.\n\n- The experiments have been conducted in a strictly controlled setting, where the task is formulated as a binary classification task making it far from the real-world clinical diagnosis inference task requiring multi-label categorization and dependencies.\n\n- The paper only considers categorical and numerical features ignoring free text EHR reports that raises questions about the usefulness of using LLMs for this task. \n\n- The related work section seems to be weak as well, as it omits various state-of-the-art approaches using DRL that also mimics similar iterative clinical decision-making process (e.g., https://aclanthology.org/I17-1090/)\n\n- Although clinician-in-the-loop evaluation process was proposed but the paper lacked a thorough qualitative evaluation of the proposed approaches along with detailed examples of how the methods could have been improved further.\n\n- The overall writing, structure, organization and content of the paper can be improved, as at times the proposed objectives, frameworks, experimental setup etc. seemed confusing and not coherent.\n\nQUESTIONS:\nsee above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-21"}