ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoning
Authors: Shuyue Stella Li
Jimin Mun
Faeze Brahman
Pedram Hosseini
Bryceton G. Thomas
Jessica M. Sin
Bing Ren
Jonathan S. Ilgen
Yulia Tsvetkov
Maarten Sap
Venue: COLM 2025 (2025)
Keywords: Information Seeking
Question Asking
Reliable LLM
Clinical Reasoning
Structured Rewards
TL;DR: Novel alignment recipe to teach LLMs perform complex goals by (1) decomposing it into more tangible attributes, (2) creating synthetic data, and (3) integrating the attributes.

Abstract: Large language models (LLMs) often fail to ask effective questions under uncertainty, making them unreliable in domains where proactive information-gathering is essential for decision-making. We present ALignment via Fine-grained Attributes, (ALFA) a framework that improves LLM question-asking by (i) decomposing the notion of a “good” question into a set of theory-grounded attributes (e.g., clarity, relevance), (ii) controllably synthesizing attribute-specific question variations, and (iii) aligning models via preference-based optimization to explicitly learn to ask better questions along these fine-grained attributes. Focusing on clinical reasoning as a case study, we introduce the MediQ-AskDocs dataset, composed of 17k real-world clinical interactions augmented with 80k attribute-specific preference pairs of follow-up questions, as well as a novel expert-annotated interactive healthcare QA task to evaluate question-asking abilities. Models aligned with ALFA reduce diagnostic errors by 56.6% on MediQ-AskDocs compared to SoTA instruction-tuned LLMs, with a question-level win-rate of 64.4% and strong generalizability. Our findings suggest that explicitly guiding question-asking with structured, fine-grained attributes offers a scalable path to improve LLMs, especially in expert application domains.

Published as a conference paper at COLM 2025
ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoning
Shuyue Stella Li
Jimin Mun
Faeze Brahman
Pedram Hosseini
Bryceton G. Thomas
Jessica M. Sin
Bing Ren
Jonathan S. Ilgen'
Yulia Tsvetkov! Maarten Sap²
UUniversity of Washington
Carmegie Mellon University
Allen Institute for Al
LLavita AI Dartmouth Medicine
stelliêcs, stelli@cs.washinggeee washingtt..nn....ccc.o edu, jmun@andrew.com cmwpea cmu. edu
O
https:///github wtt. sem/sslllalssy/AAFAA
https://hugginggfacee http://www.....aeeee////aaaaaaaa
Abstract
Large language models (LLMs) often fail to ask effective questions under un- certainty, making them unreliable in domains where proactive information- gathering is essential for decision-making We present ALignment via Fine-grained Attributes, (ALFA) a framework that improves LLM question- asking by (i) decomposing the notion of "good" question into set of theory-grounded attributes (e.g., clarity, relevance), (ii) controllably synthe- sizing attribute-speciffs question variations, and (iii) aligning models via preference- optimization to explicitly learn to ask better questions along these fine-grained attributes. Focusing on clinical reasoning case study, we introduce the MediQ-AskDocs dataset, composed of 17k real- world clinical interactions augmented with 80k attribute-speccfic preference pairs of follow-up questions, as well as a novel expert-annotated interactive healthcare QA task to evaluate question-asking abilities. Models aligned with ALFA reduce diagnostic errors by 56.6% on MediQ-AskDors compared SoTA instruction-tunet LLMs, with a question-level win-rate of 64.4% and strong generalizability. Our findings suggest that explicitly guiding question-asking with structured, fine- tine-grained attributes offers scalable path to improve LLMs, especially in expert application domains.'
Hi doctor, am 35 yo, femalc, and have fatigue and night sweats, What happoning, what should
Vanilla LLM Agent
ALFA-aligned Expert Agent
Has anyone your family becn sick?
Ambiguous, broad vs. Clear, focused
What is temperature!
Um... My g grandpa has back pain, my dad has lung cancor:
Hard Straightforward
temperature is 98"F now but was 102"r last night.
for patient to respond
How are feeling? Anxious?
W/ DDX bios vs. Relevant
Have traveled recently?
am worried my symptoms lead to something serious.
Unuseful vs. Useful response
am travel blogger, just came back from Africa last month.
have anxicty sorder
Wrong Correct
You have a viral infection.
diagnosis
Figure 1: Effective information-seekle questions are crucial for clinical reasoning. ALFA-aligned models can ask better questions and lead to more accurate diagnosis.
Introduction
Interactive language models have demonstrated remarkable capabilities across numerous domains (OpenAl et al., 2024), yet proactive interaction abilities in high-stakes scenarios- clinical reasoning, legal analysis, investigative journalism remmann a challenge (Fung et al., 2024). A key obstacle is the ability of these models to recognize and anticipate missing or ambiguous information and proactively seek clarification (Li et al., 2024; Deng et al.,
We release all data, code, and models for further research.

---

Published as a conference paper at COLM 2025
"Good" Question
1. Decompose
2. Synthesize
3. Align
ONCST
Ganeral Question-Aaionn Aariiogn-kkinggga.. Att busos
Cinical Question Adking Atbributss
Clarity
Focus
Answerability
Accuracy
Relevance
Blas
Enhanced
Original
Compptte
Arty
weerage modelss
Crasterreee Syythes
Wew of you Oheve Aomide
Anarage
W
aaggugge
Prefernce Date
Comppted
Oriinnll
Oomypted
Data Mixing
Reward Fuston
Pollcy Fluston
Figure 2: ALFA: decompose, synthesize, align.
2024). In clinical practice, for instance, physicians systematically ask patients questions to rule out or confirm relevant diagnoses (Richardson et al., 1995; Proffit, 2013). This iterative, information-seekinng behavior is essential for accurate and safe decision-making. Similarly, for large language models (LLMs) to serve as reliable decision-support tools for clinicians, they must learn not only to provide answers, but also identify when additional information is needed, and to ask follow-up questions that effectively explore and adjust possible hypothesis or reduce uncertainty (Figure 1).
However, there are two main challenges in building LLMs that ask good questions, especially in expert domains, First, defining a "good" question is inherently complex and context-dependent. In general, attributes such as clarity, and answerraiility (Heritage & Maynard, 2006; Roter & Hall, 1987; Freed, 1994; Searle, 1969); however, in domain-specific scenarios such clinical reasoning, additional properties- meddial accuracy, diagnostic relevance, and mitigating differential dingnosis (DDX) biases-are necessary for reducing diagnostic uncertainty (Richardson et al., 1995; Silverman et al., 2016; Heritage, 2010; Hall et al., 1995; West, 1984; Stivers& & Majid, 2007; Ong et al., 1995). Second, instilling the ability
essential
ask good questions in LLMs is technically non-trivial (Li al., 2024; Johri et al., 2025; Zhang et al., 2024a). Naive prompting strategies such as "Ask a follow-up question if needed."" " may enhance interactivity but lack a principled foundation for defining good question. We propose leveraging well-established general and task-specific principles from communication theory and psychology to improve LLMs' information-seedt seeking abilities.
Methodologically, we introduce general recipe to incorporate question-asking abilities into LLMs, focusing on clinical reasoning as a case study. Our Beehod-ALignmen Fine-grained Attributes (ALFA)-relies on the idea that question quality can be improved by explicitly training models with data grounded in structured, theoretically motivated attributes. Our recipe proceeds in three steps:
via
Decompose the goal of asking "good" questions into structured, grounded attributes.
2. Synthesize counterfactual data by controllably altering any attribute (e.g. make clearer).
3. Align models using preference optimization algorithms to integrate the attributes and produce final policy.
Since labeled conversational datasets containing follow-up questions along a variety of important attributes are scarce, ALFA exposes models to much broader range of question- asking behaviors than what can be typically found in the wild, especially in specialized domains like clinical interactions.
We instantiate the above recipe with a focus on clinical reasoning, where question-asking is central to reducing diagnostic uncertainty and preventing errors. To this end, we construct a novel dataset, Mesio--AsDDoss, containing 17k clinical interactions with follow-up ques- tions from the r/AskDocs subreddit?, paired with 80k synthesized counterfactual variants of these questions highlighting each attribute. These counterfactual pairs provide fine-grained training signals for preference learning (Li et al., 2025). Finally, we integrate the attribute- specific signals into a unified policy by combining all synthetic data into a single model, training separate reward models and merging them, or fusing attribute-specin policies. ALFA contrasts with boarse-grained preference learning, offering a more targeted way to refine question-asking behaviors.
With medical experts verified following subreddit policy. See 57 for further discussion on setting.
2

---

Published as a conference paper at COLM 2025
As part of MediQ-AskDocs, we introduce novel healthcare QA task of 302 expert-annotated clinical interaction scenarios to evaluate the proposed method. These scenarios are passed into MediQ (Li et al., 2024), an interactive clinical simulator, in which the models asks questions to the patient agent. ALFA-aaligne models achieve a 64.4% win-rate in question- level evaluation and 56.6% reduction in diagnostic errors, relative to baselines of SoTA instruction tuned LLMs. Beyond these empirical gains, our work presents a new paradigm for aligning language models to specialized domains by decomposing the complex goal of question-asking into attributes, synthesizing pairwise data in each attribute dimension, and aligning the model to jointly optimize the overall complex goal. This general approach to aitribute-baser question-asking alignment can be extended to many other domains where systematically eliciting information is key to reliable, effective decision-making.
Problem Statement
Aligning LLMs to ask good question requires reasoning along multiple attributes (e.g., accuracy, clarity, focus). However, most alignment paradigms treat these goals as monolithic, aggregating preferences into single reward that conflates attributes and obscures their individual contributions. We formalize this challenge as follows:
Given complex goal G and a dataset with sparse labels D, we aim learn policy π that maximizes the composite reward a(s,a), where:
Current world state (e-g., information acquired far, conversation history).
4: Next action (e.g,, follow-up question asked by the clinician agent πT).
The key challenges lie in the complexity of the goal and the sparsity of labeled data. First, directly optimizing R(s,a) is infeasible because human preferences for R(s,a) are noisy and subjective. To address this, ALFA decomposes into K attributes {A,,..., Ak), each corresponding to a verifiable criterion with a reward function Rç. We constrain the selection of A such that each (5(s,a) a) more measurable compared to R(s,a).
Second, we cannot observe parallel follow-up questions in natural conversations to construct preference pairs. To this end, we synthesize counterfactual synthetic data D Dyynth synth for each
As, where
D synth
{a a Ra((t > R2(at }}.
Lastly, ALFA uses reward integration strategy f, where R(s,a) to combine {RI,.., Rx} into a policy π,, such that:
π = arg max (sa)>>
optimizes performance on the complex goal G, aligning models to be better question-askers.
What Makes a "Good" Question?
To systematically improve LLM question asking, we define six key attributes grounded in cognitive science, psychology, and clinical communication research (Heritage & Maynard, 2006; Roter & Hall, 1987; Chouinard, 2007; Freed, 1994; Searle, 1969; Levinson, 2012). Unlike prior work that relies on implicit heuristics, explicitly decompose question quality into interpretable, tangible attributes that enhance clinical reasoning.
General Anestton-Quality Attributes. Effective questions must be clear, targeted, and answerable to drive meaningful interactions. We select three core attributes:
1. Clarity, aiming to avoid ambiguity and unnecessary complexity (e.g., jargons), ensur- ing precise communication (Heritage & Maynard, 2006; Roter & Hall, 1987; Burns et al., 2022);
Focus, directly addressing a specific information gap, yielding more informative re- sponses (Ronfard et al., 2018; Gopnik & Wellman, 2012; Chouinard, 2007; Freed, 1994). E.g., "Has anyone in your family had breast cancer?" is superior to "Has anyone in your family been sick?", and
3

---

Published as a conference paper at COLM 2025
3. Anstverability, ensuring the question is both within the respondent's knowledge domain and appropriate for them to answer (e.g, asking about their symptoms and experiences rather than expecting them to provide medical diagnoses or knowledge that falls within the clinician's 's expertise) (Levinson, 2012; Keil et al., 2008; Searle, 1969).
Domain-specific question-asking attributes. In clinical reasoning, question-asking is structured diagnostic skill. Drawing from clinical communication research (Richardson et al., 1995; Silverman et al., 2016; Heritage, 2010; Hall et al., 1995; West, 1984; Stivers & Majid, 2007; Ong et al., 1995; Proffit, 2013), we define three additional attributes essential for clinical decision-making:
4. Medical Accuracy requires alignment with established medical textbook knowledge & guidelines;
5. Diaguostic Relevance probes for symptoms, risk factors, or contextual details essential to refining differential diagnoses (DDX); and
Avoiding DDX Bias prevents suggestive or leading wording that could introduce cogni- tive biases and misguide diagnostic reasoning.
These six attributes form the foundation of ALFA, guiding question optimization improve LLM reliability in interactive clinical reasoning.
ALFA Framework Overview
We now introduce ALFA, a structured recipe that decomposes the overall question-asking objective ($4.1), generates attribute-specific preference data ($4.2), and trains policy that integrates the attributes (54.3) to ask better follow-up questions.
4.1 Grounded Attribute Decomposition
We first decompose the concept of "good" clinical questions into the six attributes A identi- fied in $3 rather than relying on implicit heuristics or coarse scoring. This decomposition enables two advantages: (1) altribule-specife training signals that isolate distinct aspects of question quality, and (2) a controlled preference structure for fine-grained alignment, guiding the next stages of data generation and model alignment.
4.2 Attribute-Specific Data Generation
Real-world clinical datasets are scarce, private, and rarely contain annotations distinguish- ing, for instance, clear vs. ambiguous questions (Mireshghallah et al., 2023; Ramesh et al., 2024). Therefore, we generate synthetic preference data by (1) collecting authentic clinical posts (see $5.3 for dataset curation details) and (2) using LLM to generate counterfactual variants along each attribute.
Counterfactual Perturbation. For each question a; in the dataset, we prompt an LLM to create "enhanced" and "corrupted" variants (a aa that explicitly alter only one attribute k at time (e.g., rewriting the original question to be more clear/ aamii/uuu while keeping others consistent. This enables us to create controlled preference pairs, where one version of question a.+ is better aligned with specific attribute than the other, a²-
Verification & Filtering. We use an LLM-judge (Zheng al., 2023) verify that the generated perturbations reflect their intended modification. With original question a;, we obtain an enhanced question at and a corrupted question at resulting in three preference pairs: tat (d)+ (a/ a ), and (2;, the L.M.-judge, and ask the judge to compare the pairs in the specified attribute dimension (e.g., which question is clearer). If the judge's decision matches the intended perturbation
Given each pair, we provide additional context to
direction-verifyinng that Re(att) > (a - retain the sample; otherwise, we discard it. This filtering step removes inconsistencies and ensures that our synthetic preference data provides reliable supervision for model alignment. See details in Appendix C.1.
Parsed conversation conclusions from future turns.
4

---

Published as a conference paper at COLM 2025
4.3 Attribute Integration Strategies
We now combine signals from the attribute-specific data so that the model produces questions that optimize for all attributes. Standard preference optimization algorithms DPO (Rafailov et al., 2024) and PPO (Schulman et al., 2017) allow distinct Points of Integration (POI) as described below*:
(1) Data Mixing pools all data into one training set and uses standard DPO/PPS This treats each attribute-specifer comparison as part of a larger set of "better vs. worse" pairs, enabling single policy to learn from all attributes at once (Wang et al., 2023b; Lambert et al., 2024).
(2) Reward Fusion trains separate reward models (RM), then averages the RM weights in an overall RM which can then be used in PPO to align final policy (Ramé et al., 2024).
(3) Policy Fusion trains separate policies or preference models, each specialized for one at- tribute, and then combine the model weights by averaging or taking a linear combination (Jang et al., 2023). This strategy offers the highest degree of parallelism and maximally preserves attribute strengths and interpretability.
Comparing these integration strategies informs how best to reconcile multiple, sometimes competing, objectives ge.g.,oouus vs. avoiding DDX bins) and thereby produce consistently high-quality diagnostic questioning.
Experimental Setup
5.1 MediQ-AskDocs Dataset Curation
We obtain data from r/AskDocs, a public online health forum, and filter for conversation threads where (i) the patient posts a health inquiry and engage with another user to discuss the issue, and (ii) another user asks follow-up questions to acquire more information. The resulting dataset contains 13,496 unique posts, 17,425 threads, and 24,263 questions. For creating the counterfactual perturbations (54.2),
sample 4463 /3///33/660 620 questions for the train/ dev /est splits. More details on dataset curation and sampling are in Appendix B.
5.2 Experiments
With the goal of instilling question-asking ability in LLMs in the clinical reasoning domain, we structure our experiments to progressively address two key questions: (1) Does the entire ALFA pipeline improve clinical question-asking performance? and (2) To what exlent is every component of ALFA necessary for improving performance? Accordingly, we organize our evaluation as follows:
1. Overall Performance. We first confirm that ALFA meaningfully reduces diagnostic errors and elicits better questions in interactive clinical scenario (56.1).
2. Key Pipeline Components. We isolate the two core components of ALFA:
We compare the decomposed attributes (e.g., clarity, (simply " "good or bad") to examine the effect of answerability) to "coarse" attribute decomposition ($6.2). We compare preference tuning to supervised fine-tuning (SFT) on the same synthetic data, revealing how pairwise reward signals refine guestion-asking beyond what SFT alone can achieve ($6.3).
3. Ablation Studies. We dissect each design choice to identify its role in the observed improvements. We compare attribute-inte intlgrttitttgraatinn stralegies, apply quality filter on the synthetic data, examine data perturbation directions (corruption vs. enhancement), ablate individual attributes, and test nut-of-distribution generalization on a separate diagnostic task (56.4).
WWhil reward sum, which trains attribute-specific reward models then combines the reward scores (e.g., via learned linear combination or average) to produce final scalar reward (Wu et al., 2023; Wang et al., 2024c), remains an option, do not utilize this strategy due to high compute cost of loading all reward models during PPO without substantial performance improvements (Rame et al., 2024; Shi et al., 2024).
All ablations done with DPO due to lighter compute requirements unless otherwise specified.
5

---

Published as a conference paper at COLM 2025
Collectively, these analyses elucidate how components in ALFA-attributt decomposition, data synthesis, preference tuning-worts work together to improve question-asking in high-stakes clinical contexts. See implementation details in Appendix C.
5.3 Evaluation
We evaluate the aligned models along two fronts: Direct Question Quality, measured through expert human annotations and automatic LLM-judge comparison for the overall question quality, and Clinical Decision Impact, quantified by how well the model questions help reduce diagnostic errors.
Direct win-rate with LLM-judge. We create an LLM-judge to compare pairs of questions, adopting prompt structures from Li et al. (2023) and Dubois et al. (2023). Specifically, we measure the percentage of times our aligned models' questions are preferred over those of 1a-ma--.--3b-Inttruee
the baseline instruction-tuned models the judge and report win-rate. Each comparison is carried out by gpt-4o with rationales, permuting the presentation order in each pair to mitigate ordering bias. As an validity check, the LLM-judge assigns a higher win-rate to questions from verified experts than those from non-experts (Appendix D), suggesting our evaluation aligns with domain expertise.
and
1-0....-88-nnttucct by
Expert manual evaluation. To assess the quality of generated questions and further val- idate the LLM-judge, we conduct manual preference rankings with three medical experts from our research team and compute win-rate of select models based on majority vote. See Appendix FF for further details.
Interactive diagnostic accuracy. We use the MediQ interactive framework (Li et al., 2024) patient-clinicia simulator- holistically evaluate ALFA in a more realistic setting. MediQ presents some initial information x² (often the patient's chief complaint), a medical inquiry k, and tests an expert agent's ability to ask follow-up questions to the patient until it has enough information to make diagnosis y. We replace the question generator module in the MediQ expert agent with models trained with ALFA, while keeping all other modules (patient system and diagmosis generator) consistent. We quantify the utility of the question generator with the accuracy of the diagnosis y.
The MediQ-AskDocs task. MediQ is compatible with any QA task with contextual infor- mation. We introduce a novel healthcare QA task as part of the MediQ-AskDocs dataset: 302 consumer healthcare multiple choice questions manually annotated by medical experts. The task is automatically generated by o1 using the test split of MediQ-Ask ($5.1) and achieves 85.9% agreement with majority voted manual annotations from medical experts. See Appendix E for task construction and annotation details. Additionally, use MedQA (Jin et al., 2020) with MediQ to examine the models' ability to generalize out-of-domain.
Results & Analysis
6.1 ALFA Improves Overall Question Asking
We begin by comparing ALFA with two baselines: (1) the base instruction-tuned mod- els 11-ma-3..233B-In-t
and 1lama-3. 1-8B-Instruct)) and (2) models trained via su- pervised fine-tuning (SFT) the human-written questions. The Policy Fusion attribute integration strategy ($4.3) is reported for both ALFA-DPO and ALFA-PPO in Table
Our results confirm that fine-tuning the human-written questions in MediQ-AskDoos (SFT) already outperforms base models, establishing usefulness of the dataset. Notably, superior performance of the ALFA-aligned models shows explicitly modeling structured, theory--roundee attributes substantially boosts both question quality and diagnostic accuracy, reducing diagnostic errors by 56.62% (increasing accuracy by 21.5%)
6

---

Published as a conference paper at COLM 2025
Model
Size
Win-rate MediQ-AD
Model
Size
Win-rate
MediQ-AD
3B
50.00
73.51
Base Model
8B
50.00
72.52
Coarse
3B
65.89
83.77
3B
61.04
78,98
8B
66.05
85.76
SFT
8B
58.23
85.08
ALFA
3B 64.97
87.75
ALFA-DPO
3B
64.97
87.75
(Fine-Grained) 8B
65.13
88.08
8B
65.13
88.08
ALFA-PPO 3B
64.84
86.75
Table 2: Fine-grained (ALFA) vs. Coarse Attributes. Fine-grained attributes lead to better downstream diagnostic accuracy and similar win-rates pared to coarse-graiin coarse-grained objective. Expert evalu- ation for 3B coarse vs. fine-grained shows identical win-rate against the base model: 59.4%.
Table 1: Main results. ALFA models consistently outperform base instruct and SFT models, Note that win-rates for base models are set to 50%
to represent equal preference when comparing against themselves.
6.2 Fine-Grained Attributes Outperform Coarse Attribute
The core idea of ALFA to decompose complex goal into structured, theory-grounded attributes rather than treating it as one coarse objective. We examine the role of attribute decomposition by comparing alignment with fine-grained attributes (AL.FA) with a sim- pler approach that optimizes models on coarse "better" vs. "worse" distinction (Geng et al., 2025). Both models undergo the same counterfactual data generation and alignment processes, but the coarse model lacks explicit attribute separation.
Results in Table 2 show that ALFA, trained with fine-grained attributes, achieves higher performance compared to models trained on coarse attributes MediQ diagnostic accu- racy, while the two methods show compa- rable LLM-judge win-rates. Aligning mod- els based on structured attributes guides the reasoning process theory-grounded way, resulting in substantially better downstream performance. Additionally, qualitative anal- ysis (Figure 3) shows that the coarse-aligned model tends ask more superficial ques- tions such as waiting for 8 weeks and check back, while the Al.Felmodel' question, "Was this the only issue that was found?", indicates some form reasoning to rule out other factors that might have contributed to the patient's concern. Further, coarsely aligned models generalize poorly to
Patient Context:
what is the treatment for sever hyaovitaninosis D7 My 25(0H)D test result was 4.68 ng//ll. Ny doctor prescribed four IM injections of 388,000 IU vitanin within four weeks, and of supp lerents after that, isn't four injections of 380, IU this short time tan nuch? da really need this should get
vitanin
opinion?
Coarse Aligned LM Outpul:
Are you getting It checked again in 8 wenkss?
ALFA-Aligned LM Output:
Was this the only Issue was found?
Figure 3: Qualitative example contrasting models
aligned with fine-grained and coarse attributes. ALFA model tends to ask more logical questions.
tasks ($6.4).
6.3 Preference Tuning Outperforms SFT
Another core advantage of ALFA is its use of pairwise preference learning, which allows models to refine question-asking beyond what supervised fine-tuning (SFT) achieves. To test whether ALFA'S 's improvements come solely from exposure to diverse synthetic data or from learning structured prefer- ences, we compare ALFA with models fine- tuned on real data (SFT-Real), synthetic en- hanced data (SFT-Synthetic), and both (SFT- Combined).
Model
Size
Win-rate MediQ-AD
3B
61.04
78,98
SFT-Real
8B
58.23
85,08
3B
62.50
82.12
SFT-Synthetic
8B
60.32
85.76
3B
62.50
83.77
SFT-Combined
8B
54.68
85.76
3B
64.97
87.75
ALFA
65.13
88.08
As shown Table 3, ALFA outperforms SFT- Combined despite learning from the same data, confirming that learning directional differences from pairwise comparisons is key to better question-asking. These results highlight pairwise contrastive optimization as necessary step for models to learn how to ask better follow-up questions.
Table Preference tuning crucial in improving model performance. Supervised fine-tuning on the same synthetic data does not show as much performance gain.
7

---

Published as a conference paper at COLM 2025
*PO
POI
Size
Win-rate
MediQ-AD
Win-rate (human)
3B
68.55
85.01
52.00
Data
8B
68.23
88.74
DPO
3B
64.97
87.75
41.00
Policy
8B
65.13
88.08
Data
3B
97.34
84.77
75.00
PPO
Reward
3B
97.98
84.44
74.00
Policy
3B
64.84
86.75
40.00
Table 4: Attribute integration strategies.
Data Synthesis Enhancement vs. Corruption
Model
Model
Win
MediQ-AD
Medio Acc (%)
Variant
Size
Rate
Acc. (%)
ALFA
3B
64.19
85.43
Model Size
82
-Unfiltered
8B
63.31
86.09
80
SFT
3B
64.97
87.75
ALFA
Corruption
Enhancement (a+,a)
Synthetic (a+.a.)
All
8B
65.13
88.08
(a,a-)
aa+,a,,
Data Synthesis Direction
(a.a-)
Table 5: Synthetic data quality. Filtering slightly improves diagnostic accuracy.
Figure 4: All synthetic data directions are help- ful. Including corruptions, enhancements, and the original data shows the best performance.
6.4 Ablation Studies
When to integrate the attributes? We now examine the effect of various attribute integration strategies from $4.3: data-mixing, reward-fusion (PPO only), and policy-fusion Table 4 reveals three key findings. First, data-mixing achieves higher question win-rate but yields lower diagnostic accuracy than policy-fusion, and reward-fusion in PPO mirrors data-mixing patterns. This suggests that greater attribute separation leads to improved performance, consistent with prior observations in Yang al. (2025). Second, The LLM-judge scores had substantial agreement with human expert assessments for strategies with higher win-rate (Gwet's AC1 score of .68 and .72 each for ALFA-PPO-Reward and ALFA-PPO-Data), establishing the LLM-judge as reliable proxy for human assessment in this simulated inter- action environment. The high L.L.-judge score of ALFA-PPO-Rewart and ALFA-PPO-Data are echoed in the human evaluation with the highest win-rates of 74% and 75% respectively. See Appendix F and G for further details on annotations and qualitative analysis.
II. Gains from Synthetic Data Quality. ALFA relies on synthetic question perturbations to expose models to counterfactual scenarios. To ensure data quality, we filtered out 13.9% of generated pairs where LLM-judge ratings misalign with intended perturbation directions ($4.2). Filtering slightly improves both question quality and diagnostic accuracy, emphasizing the value of high-quality synthetic data (Table 5).
III. Synthetic Corruption vs. Enhancement. In the counterfactual pairwise data gener- ation stage of ALFA, each original sample is synthesized in two directions along each attribute dimension (e.g. more relevant and less relevant) to get a= and a- In this section, examine how the generation influence performance. Specifically, we compare models trained with corruption only pairs (a,a") enhancement only pairs (at,a), synthetic corruption and enhancement pairs (a+ (at,a ,,,,, and all the above. In Figure 4, we find while all directions are beneficial, combining all three pairs brings the most gains, especially apparent in the smaller 3B model.
(" ("more X") vs. corrupted ("less X")
8

---

Published as a conference paper at COLM 2025
Attribute
Win-rate
MediQ-AD
ALFA with General Attributes (AlFa-General):
Did they treat you for monо? she school?
No Accuracy
63.95
84.11
No Answerability
64.60
84.11
ALFA with Clinical Attributes [ALFa-Clinical):
No Avoid DDX Bias
62.58
81.79
pain worsen or improve NSATDs? what you think about the diagnosis of febrile convuls.ion?
No Clarity
62.74
85.10
No Focus
63.95
84.44
No Relevance
63.39
84.44
what your age, sex, nedical history, and medications?
General
62.90
81.79
Figure 5: Models aligned with general vs. clinical attributes show distinct behaviors. ALFA-General: clear and focused, but less relevant and contains DDX bias ("mono"); ALFA-Clinical: professional but uses medical terms hindering answerability.
Clinical
66.05
86.09
All
64.97
87.75
Table 6: Policy Fusion DPO w/ attribute groups.
IV. Attribute-Specific Influences. A key component of ALFA the explicit decomposition of question-asking into fine-grained, theory-grounded attributes. To assess their individual contributions, we conduct an ablation study where we remove one attribute at a time and evaluate the model's 's performance. We also compare general question-asking attributes (clarity, focus, answerability) with clinical attributes (medical accuracy, diagnostic relevance, avoiding DDX bias).
Table 6 shows that removing any attribute leads to performance drops, confirming their importance clinical question-asking. Clinical attributes have stronger impact on MediQ accuracy. Avoiding DDX flas-factors such as premature closure and availability bias leading to incomplete/ /incorrect differential diagnoses- is the most critical, suggesting that models need explicit training to counteract cognitive biases. Qualitatively, the questions generated by models aligned with general attributes vs. clinical domaii-specific attributes show distinct styles (Figure 5), highlighting the impact of feature selection. These results validate ALFA'S 's structured attribute alignment, demonstrating that both domain-specific capabilities and general question quality contribute to effective clinical decision-making.
V. ALFA Models Robustly Generalize to Out-of-Distribution Settings. We further assess ALEA'S ability to generalize beyond the MediQ-AskDocs task by evaluating on a more challenging clinical reasoning benchmark unseen during training, MedQA (Jin et al., 2021), in the same MediQ-style interactive setting. Across intergration strategies, ALFA ALFA-aligned models generally outperform match the coarse-alignment baseline when moving to MedQA (Figure 6). These suggests that learning from structured, theory-grounded attributes can enhance an LLM's robustness in new and more diverse clinical settings, highlighting ALFA'S potential for broader applicability in real-world clinical scenarios.
VI. ALFA Outperforms State-of-the-art General & Medical LLMs ALFA-aligned models to both closed-source general purpose models and mediccl-specific models by plugging these models into MediQ as the question- dression-generator.. Table shows that ALFA shows superior downstream task utility, substantially outperforming even much larger SOTA (general and medical) LLMs.
Lastly, we compare
Interactive MediQ-MedQA Diagnostic Accuracy
Model
Diagnostic Acc. (%)
55
51.73
53.14
53.33
52.20
52.48
52.75
ALFA-DPO-SB
88.1
50
GPT-40
79.8
Accuracy (%)
50.97
Gemini-2
79.8
45
03-mini
71.2
Alpacare-7B
74.8
40
Alpacare-13B
73.5
MedAlpaaa-133
68.5
35
ClinicalCamel-70B
70.5
30
Mcditron-7B
70.2
ALFA
ALFA
ALFA PPO Data
ALFA PPO Reward
ALFA PPO Policy
Meditron-70B
72.5
DPO
DPO
DPO
Coarse
Data
Palicy
PMC-LLaMA-7B
71.9
Figure 6: 3B Model performance on the interactive MediQQ-MeQAA task. Models aligned with ALFA
PMC-LLaMA-13B
69.9
are more robust out-of-distribution data.
Table 7: Diagnostic Accuracy of Various Models

---

Published as a conference paper at COLM 2025
7 Related Work
LLMs have the potential to significantly transform medicine by enhancing personalized care and accessibility (Shanmugam al., 2024). Models trained with medical data contain rich medical knowledge (Singhal et al., 2025; Lewis et al., 2020; Chen et al., 2023; Labrak et al., 2024; Singhal et al., 2023; Brin et al., 2023); however, systematic evaluations reveal persistent weaknesses in instruction-follewe pragmatics that arise in real clinical encounters (Hager et al., 2024; Arroyo et al., 2024; Nov et al., 2023; Zhang et al., 2014). These shortcomings are partly masked by the dominance of static, single-tur medical QA benchmarks, on which current models already achieve near-saturated scores (Jin et al., 2020; Pal et al., 2022). Recent work has moved away from the static single-turn paradigm and highlight proactive information-seeeingg as a prerequisite to reliable and effective clinical reasoning (Li et al., 2024; Hu et al., 2024b). CoAD (Wang et al., 2023a) is an early exploration toward interactivity, yet it operates in a markedly simplified symbolic environment: the agent selects from small closed set of symptoms and diseases under dense
multi-hop reasoning, and the nuanced
iimposiiiii to obtain in real-life interactions. Crucially, the setting eliminates the linguistic realization entirely as the "question" an index to a symptom list. ALFA tackles the more demanding challenge in the open-text regime: teaching an LLM to decide simultaneously what information remains most diagnostically valuable and how to ask for it in a way that is bias-sensitive and pragmatically appropriate.
Methodologically, our approach builds on recent data-centric alignment techniques that create synthetic preference signals for alignment (Li et al., 2025; Mishra et al., 2024a; Ding et al., 2024; Park et al., 2024). Inspired by prior work on multi-objective RLHF (Zhou et al., 2023b; Wu et al., 2023), we extend PPO (Ouyang et al., 2022; Christiano et al., 2017) and DPO (Rafailov et al., 2023) to align models with attribute-specific datasets, and uniquely compare different integration points of the fine-grained preference signals (Rame et al., 2024; Chronopoulou et al., 2023; Wang et al., 2024a).
Discussion
Effective question-asking is a fundamental yet underdeveloped capability in large lan- guage models, particularly in high-stakes domains like clinical reasoning. We proposed ALFFA, a framework that explicitly teaches models to ask better questions by decomposing question quality into theory-grounded, fine-grained attributes and aligning them through preference-based optimization, rather than treating such nuanced and complex goal as monolithic objective. We introduced MediQ-AskDocs, a comprehensive dataset of training data, preference data, and healthcare QA task, showing that models trained with ALFA substantially outperform baselines. While focused medicine case study, ALFA is general recipe adaptable to any field where clear, targeted questioning is essential, paving the way for interactive and reliable systems.
Future Directions. While ALFA demonstrates significant improvements in clinical question-asking within controlled scenarios, several important directions warrant explo- ration. First, incorporating contextual factors that shape real clinical reasoning-suce as care setting constraints (rural vs. urban hospitals), available diagnostic resources, and physician-patient relationship history- enhance the framework's real-world appli- cability. Future work could explore dynamic weighting mechanisms conditioned on these contexts to improve attribute integration. Additionally, integrating multimodal inputs beyond text, such as tone analysis, non-verbal cues, and existing electronic health record data, could better mirror human clinical interactions. The ALFA framework could also benefit from incorporating collaborative decision-making elements, where models learn to ask patients about their own hypotheses or concerns, and to leverage input from healthcare team members. Finally, extending ALFA to other high-stakes domains requiring system- atic information-gathering -uughgssuhhhee as legal discovery, investigative journalism, or financial risk aasessment-could validate its generalizability beyond healthcare while revealing domain-specific attribute requirements.
10

---

Published as a conference paper at COLM 2025
Limitations
Manual attribute selection. ALPA requires manual selection of attributes when adapting to new expert domains. While it offers structured framework, determining which attributes are essential still depends on domain expertise. However, ALFA can also help evaluate attribute necessity across different fields.
LLM dependence for counterfactual generation. The counterfactual perturbation step relies on LLMs to generate and evaluate counterfactual question variants (specifically, meta-llama/L Laaaammmmmaaaa--311 3. -ns--00-Intruuc-FFP9 assuming they correctly interpret attributes like clarity and relevance. Future work should incorporate human verification of counter- factuals and attribute-level rankings.
Subjectivity in human annotation. and scenarioddppendent Some annotators expressed difficulty in ranking questions, stating that "none of the questions were good" or that "all of them were acceptable." " To ensure annotation quality, we implemented a four-question screening test with known ground- truth answers, filtering out annotators who failed to meet predefined accuracy threshold. However, this approach does not fully eliminate the risk of variability in domain expertise.
Evaluating follow-up questions is inherently subjective
Data and scope. Our dataset is derived from online health forum discussions (r (r/AskDocs) rather than in-person clinician-patient dialogues in hospital setting. While this source provides diverse real-world medical inquiries, it does not fully capture the structured questioning strategies used in professional clinical settings. Thus, while ALFA offers strong technical foundation for studying medical cquestion-asking it should not viewed as direct replacement for physician training real clinical interactions. Expanding to EHR-based or in-hospital dialogue datasets would improve clinical applicability.
The other presumption underlying this work is that all of these medical problems have verifiable solutions. In actual medical practice-particuaaty in settings where problems are acute and undefined-iit is quite likely that clinicians will come slightly or entirely different solutions for the same problem (based on their own experiences or expertise).
Ethics Statement
ALFA aims to improve LLM-driven question-asking in clinical reasoning, but its develop- ment and potential deployment pose potential ethical risks related to misinformation, bias, privacy, and regulatory compliance.
A primary concern is misinformation and overreliance Al-generated questions. While ALFA improves question quality, it does not provide any sense of guarantee on factuality. If used without human oversight, it could generate misleading, irrelevant, or overly confident questions, potentially influencing clinical decision-making and leading to misdiagnosis or unnecessary medical interventions.
Bias in training data and evaluation is a key risk. ALFA, trained on r/AskDocs, may not represent diverse populations, conditions, or expert strategies, leading to systematic biases that reinforce healthcare disparities. U.S.-based, English-speaking annotators further limit generalizability. Reliance on LLM-judges may introduce automation bias, reinforcing subtle inaccuracies. Future work should expand datasets and evaluation to more diverse populations and implement bias mitigation strategies.
Privacy and data security are additional risks. Although ALFA does not process private medical records, future adaptations using clinical data or electronic health records (EHRs) must protect sensitive patient information. Transparent data governance frameworks and strict access controls are necessary for responsible use in healthcare applications.
ALFA is intended as technical contribution to the field of computer science rather than a standalone clinical tool. The framework must be integrated with human-in-the-loop supervision, where clinicians retain final decision-making authority. Future work should explore uncertainty calibration, ethical safeguards, and regulatory alignment to ensure safe, fair, and reliable Al-assisted clinical reasoning.
11

---

Published as a conference paper at COLM 2025
Acknowledgements
This research was developed with funding from the Defense Advanced Research Projects Agency's (DARPA) SciFy program (Agreement No. FRR00112503001 The views expressed are those of the author and do not reflect the official policy or position of the Department of Defense the U.S. Government. This material is based upon work supported by the De- fense Advanced Research Projects Agency and the Air Force Research Laboratory, contract recommenda-
number(s): FA8650-23-C-7316 Any opinions, findings and conclusions, tions expressed in this material are those of the author(s) and do not necessarily reflect the views of AFRL or DARPA. We would like to thank Dr. Saeed Hassanpour and the Dartmouth Center for Precision Health and Artificial Intelligence (CPHAT) for helping to facilitate our collaboration with the medical professionals team in partnership with Lavita Al.
References
Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D Good- man. Star-gate: Teaching language models to ask clarifying questions. ar Xiv preprint drXiv:2403.1915: 2024.
Alberto Mario Ceballos Arroyo, Monica Munnangi, Jiuding Sun, Karen Y. C. Zhang, De- nis Jered Melnerney, Byron C. Wallace, and Silvio Amir. Open (clinical) 1lms are sensitive to instruction phrasings, 2024. URL httpps://arki: www.v..rrg/bbbo argg 09429.
Tal August, Lucy Lu Wang, Jonathan Bragg, Marti A Hearst, Andrew Head, and Kyle Lo. Paper plain: Making medical research papers approachable to healthcare consumers with natural language processing. ACM Transactions Computer-Human Interaction, 30(5):1-38, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXio preprint arXiv:2204.05862 2022.
Dana Brin, Vera Sorin, Akhil Vaid, Ali Soroush, Benjamin S Glicksberg, Alexander W Charney, Girish Nadkarni, and Eyal Klang. Comparing chatgpt and gpt-4 performance in usmle soft skill assessments. Scientific Reports, 13(1):16492, 2023.
Shohei T Burns, Nwamaka Amobi, Joshua Vic Chen, Meghan O'Brien, and Lawrence A Haber. Readability of patient discharge instructions. Journal of General Internal Medicine, pp. 1-2, 2022.
Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, al. Meditron-70b: Scaling medical pretraining for large language models. ar Xiv preprint arXiv:2311.16079 2023.
Michelle M. Chouinard. Children's 's questions: A mechanism for cognitive development. Monographs of the Society for Research in Child Development, 72(1):1-112, 2007. doi: 10.1111/
j.1540-5834.2007 20t.2000777002...... 00412.x.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems,, 30, 2017.
Alexandra Chronopoulou, Matthew E Peters, Alexander Fraser, and Jesse Dodge, Adapter- soup: Weight averaging improve generalization of pretrained language models. Xiv preprint ar Xrx:: 07002, 2023.
Yang Deng, Lizi Liao, Zhonghua Zheng, Grace Hui Yang, and Tat-Seng Chua. Towards human-centered proactive conversational agents. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 807-818,
2024.
12

---

Published as a conference paper at COLM 2025
Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wen- han Xia, Junjie Hu, Luu Anh Tuan, and Shafiq Joty. Data augmentation using llms: Data perspectives, learning paradigms and challenges. In Findings of the Association for Computational Linguistics ACI. 2024, pp. 1679-1705, 2024.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
Arsene Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, and Joumana Ghosn. Ddxplus: A new dataset for automatic medical diagnosis. Advances in neural information processing systems, 35331303333,330333113333 2022.
Alice F. Freed. The form and function of questions in informal dyadic conversation. Journal of Progmatics, 21(6):621-644, 1994. doi: (0.1016/0037--216694)))00
Yi Fung, Anoop Kumar, Aram Galstyan, Heng Ji, and Prem Natarajan. Agenda-driven question generation: case study in the courtroom domain. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Ålessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 572-583, Torino, Italia, May wwrreeemain... 49/.
2024. ELRA and ICCL. URL https::/aclannthooogy.oooo
Scott Geng, Hamish Ivison, Chun-Liang Li, Maarten Sap, Jerry Li, Ranjay Krishna, and Pang Wei Koh. The delta learning hypothesis: Preference tuning on weak data can yield strong gains. In COLM, 2025. URL https:://axii: http://wwvvvv..gggg../ ottpg///////...../ 06187.
Alison Gopnik and Henry M Wellman. Reconstructing constructivism: causal models, bayesian learning mechanisms, and the theory theory. Psychological bulletin, 138(6):1085, 2012.
Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, et al. Evaluation and mitigation of the limitations of large language models in clinical decision- making. Nature medicine, (0(9))2613-262) 2024.
JA Hall, DL Roter, and Barbara Junghans. Doctors talking with with doctors: improving communication in medical visits, 1995.
talking
John Heritage. Questioning in medicine. Why do you ask, pp. 42-68, 2010.
John Heritage and Douglas W. Maynard (eds.). Communication in Medical Care: Interactions between Primary Care Physicians and Patients. Cambridge University Press, Cambridge, UK, 2006.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Open- rlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint ar Xix::2055.11443, 2024a.
Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng, Anh Tuan Luu, Junxian He, Pang Wei Koh, and Bryan Hooi. Uncertainty of thoughts: Uncertainty-awart planning enhances information seeking in large language models. arXiv preprint ar aroto:240..03271, 2024b.
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized la rgee language model alignment via post-hoc parameter merging. arXiv preprint ar Xiv:2210.1 11564, 2023.
Daniel P Jeong, Pranav Mani, Saurabh Garg, Zachary C Lipton, and Michael Oberst. The limited impact of medical adaptation of large language and vision-language models. ar Xiv preprint ar Xiv:24411 2et.22471.888 2024.
13

---

Published as a conference paper at COLM 2025
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have. A Large-scale Open Domain Question Answering Dataset from Medical Exams. ar Xiv [cs. CLI, 2020.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14))6421, 2021.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering, 2019. URL https://arxiv.org abs/1909.000 099990614...90000 06146.
ChreyaJohri Jaehwan Jeong, Benjamin A Tran, Daniel Schlessinger, Shannon Wongvibulsin, Leandra A Barnes, Hong-Yu Zhou, Zhuo Ran Cai, Eliezer M Van Allen, David Kim, et al. An evaluation framework for clinical use of large language models in patient interaction tasks. Nature Medicine, pp. 1-10, 2025.
Jaehw,
Praveen K Kanithi, Clément Christophe, Marco AF Pimentel, Tathagata Raha, Nada Saadi, Hamza Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, and Shadab Khan. Medic: Towards a comprehensive framework for evaluating llms in clinical applications, 2024. URL httts:://axxivoorg/bbssss
073144.
Frank C. Keil, Courtney Stein, Lisa Webb, Van Dyke Billings, and Leonid Rozenblit. Dis- cerning the division of cognitive labor: An emerging understanding of how knowledge is clustered in other minds. Cognitive science, 32(2):259-300, 2008.
Seungone Kim, Juyoung Suk, Xiang Yue, Vijay Viswanathan, Seongyun Lee, Yizhong Wang, Kiril Gashteovski, Carolin Lawrence, Sean Welleck, and Graham Neubig. Evaluating language models as synthetic data generators. arXiv preprint ar XrXi::2412.03679, 2024.
Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. Biomistral: collection of open-source pretrained large language models for medical domains. Xiv preprint ar Xrio:2402.10377, 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Tvison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T ulu 3: Pushing frontiers open language model post-training. arXiv preprint 2111:22411.15124, 2024.
Stephen C Levinson. Interrogative intimations: On possible social economics of interroga- tives. In Questions: Formal, functional and interactional perspectives, pp. 11-32. Cambridge University Press, 2012.
Patrick Lewis, Myle Ott, Jingfei Du, and Veselin Stoyanov. Pretrained language models for biomedical and clinical tasks: Understanding and extending the state-of-the-art. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pp. 146-157, Online, November 2020. Association for Computational Linguistics. doi: 10.18653 /v12020 clinicalnlp-1.17. URL https: https: /acls/thoolaaaggg...oommo org/2020 wwt......iicannl...... clinicalnlp-1.17 17.
Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan S Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov. Mediq: Question-asking ilms and benchmark for reliable interactive clinical reasoning. In The Thirly-eighth Annual Conference Neural
Information Processing Systems, 2024.
Shuyue Stella Li, Melanie Sclar, Hunter Lang, Ansong Ni, Jacqueline He, Puxin Xu, Andrew Cohen, Chan Young Park, Yulia Tsvetkov, and Asli Celikyilmaz. Prefpalette: Personalized preference modeling with latent attributes, 2025. URL https://arxxiv.rr//bbss 13541.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://gt wei.gitt..cccoc
eval, 2023.
14

---

Published as a conference paper at COLM 2025
Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv preprint ar Xio:2406.11116, 2024.
Fatemehsadat Mireshghallah, Yu Su, Tatsunori Hashimoto, Jason Eisner, and Richard Shin. Privacy-preserving domain adaptation of semantic parsers, 2023. URL org/abs/2212. 22220
Ashish Mishra, Gyanaranjan Nayak, Suparna Bhattacharya, Tarun Kumar, Arpit Shah, and Martin Foltin. LIm-guided counterfactual data generation for fairer ai. In Companion Proceedings of the ACM on Web Conference 2024, pp. 1538-1545, 2024a.
Prakamya Mishra, Zonghai Yao, Parth Vashisht, Feiyun Ouyang, Beining Wang, Vidhi Dhaval Mody, and Hong Yu. Synfac-edit: Synthetic imitation edit feedback for factual alignment in clinical summarization. arxiv preprint arXiv:2402.13911 2024b.
Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical 2023.
artificial intelligence. Nature, 616(7956)
Hajra Murtaza, Musharif Ahmed, Naurin Farooq Khan, Ghulam Murtaza, Saad Zafar, and Ambreen Bano. Synthetic data generation: State of the art in health care domain. Compuder Science Review, 48:100546, 2023.
Oded Nov, Nina Singh, and Devin Mann. Putting chatgpt's medical advice to the (turing) test: survey study. JMIR Medical Education, 9:e46939, 2023.
Byoung-Doo Oh, You Kim, Chulho Kim, and Yu-Seop Kim. How to use language models for synthetic text generation in aerebrovascular disease-specific medical reports. In Proceedings of the 1st Workshop on Personalization of Generative Al Systems (PERSONALIZE 2024), pp. 10-17, 2024.
Lucille ML Ong, Johanna CJM De Haes, Alaysia M Hoos, and Frits B Lammes. Doctor- patient communication: review of the literature. Social science & medicine, 40(():903-99) 1995.
OpenAT, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne,
15

---

Published as a conference paper at COLM 2025
Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Fran- cis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thomp- son, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sher- win Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL htttt::///axivv wrwpe/iii.or//bbbbbeeao http://.rrgabs//33333/ 4.77//033333000774
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Gerardo Flores, George H Chen, Tom Pollard, Joyce C Ho, and Tristan Naumann (eds.), Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pp. 440-260. PMLR, 07-08 Apr 2022. URL pwtpp/////////aa22..a.. html.
pptes///ppccediiggs.
Chan Young Park, Shuyue Stella Li, Hayoung Jung, Svitlana Volkova, Tanushree Mitra, David Jurgens, and Yulia Tsvetkov. Valuescope: Unveiling implicit norms and values via return potential model of social interactions, 2024. URL 02472.
William R Proffit. Evidence and clinical decisions: asking the right questions to obtain clinically useful answers. In Seminars in orthodontics, volume 19, pp. 130-136. Elsevier, 2013.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, (1153728-53741 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2024. URL httttp::/aaxvv..or///aaaa
18290.
Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024.
16

---

Published as a conference paper at COLM 2025
Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. arXiv preprint ar arXi::2401.12187, 2024.
Krithika Ramesh, Nupoor Gandhi, Pulkit Madaan, Lisa Bauer, Charith Peris, and Anjalie Field. Evaluating differentially private synthetic data generation in high-stakes domains, 2024. URL https: /arxiv. www.riiww.....aeeeeeeeeeeeeeee httgg/abs/21111 08327.
Sudha Rao and Hal Daumé III. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2737-2746, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18 1255. URL httpp:://ccllaieee https://acllatthooogg.. wwwp:/yy.rrg/////////g
Rajat Rawat, Hudson McBride, Rajarshi Ghosh, Dhiyaan Nirmal, Jong Moon, Dhruv Ala- muri, Sean O'Brien, and Kevin Zhu. DiversityMedQA: A benchmark for assessing demo- graphic biases in medical diagnosis using large language models. In Daryna Dementieva, Oana Ignat, Zhijing Jin, Rada Mihalcea, Giorgio Piatti, Joel Tetreault, Steven Wilson, and Jieyu Zhao (eds.), Proceedings of the Third Workshop on NLP for Positive Impact, pp. 334-348, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ /v1 /02.24...pp4pii1111 .29 URL https: httttp::/aacaanthol...gg
wrg///2// n1p4pi-1.22 29/.
W Scott Richardson, Mark C Wilson, Jim Nishikawa, and Robert S Hayward. The well-built clinical question: a key to evidence-based decisions. ACP journal club, 123(3): 1995.
Jorge A Rodriguez, Emily Alsentzer, and David W Bates. Leveraging large language models to foster equity in healthcare. Journal of the American Medical Informatics Association, pp.
ocae055, 2024.
Samuel Ronfard, Imac M Zambrana, Tone K Hermansen, and Deborah Kelemen. Question- asking in childhood: A review of the literature and a framework for understanding its development. Developmental Review, 49:101-120, 2018.
Debra L. Roter and Judith A. Hall. Studies of doctor-patient interaction. Annual Review of wttp://wpp.....001.0.0000000
Public Health, 8:163-180, 1987. doi: 10.1146/
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL //arxiv http://axxxvoooggb/b///// absg//bs//7// .63347.
John R. Searle. Speech Acts: An Essay in the Philosophy of Language. Cambridge University Press, Cambridge, UK, 1969.
Omar Shaikh, Kristina Gligorić, Ashna Khetan, Matthias Gerstgrasser, Diyi Yang, and Dan Jurafsky. Grounding gaps in language model generations. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 6279-6296, 2024.
Divya Shanmugam, Monica Agrawal, Rajiv Movva, Irene Y Chen, Marzyeh Ghassemi, and Emma Pierson. Generative ai in medicine. arXiv preprint urXio:2412.10337, 2024.
Ruizhe Shi, Yifang Chen, Yushi Hu, Alisa Liu, Hannaneh Hajishirzi, Noah Smith, and Simon S Du. Decoding-ting time language model alignment with multiple objectives. arx preprint XrXi::2406.1888 18853, 2024.
Jonathan Silverman, Suzanne Kurtz, and Juliet Draper. Skills for communicating with patients. crc press, 2016.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172-118, 2023.
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature Medicine, pp. 1-8, 2025.
17

---

Published as a conference paper at COLM 2025
Tanya Stivers and Asifa Majid. Questioning children: Interactional evidence of implicit bias in medical interviews. Social Psychology Quarterly, 70(4): 2007.
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):1930-1940) 2023.
Augustin Toma, Patrick R Lawler, Jimmy Ba, Rahul G Krishnan, Barry Rubin, and Bo Wang. Clinical camel: An open expert-level medical language model with dialogue- based knowledge encoding. arXiv preprint ar XrX:v:2305. 12031, 2023.
Vasudha Varadarajan, Sverker Sikström, Oscar Kjell, and H. Andrew Schwartz. ALBA: Adaptive language-based assessments for mental health. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1466-2478, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/ /v15/v1v/ mo024.naac--oon...1 136. URL https: https://aclannt hatass///laatnooogyy.y org/ 2024. naacl-long 136/.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts arXiv preprint MxXiv:2406.12845 2024a.
Huimin Wang, Wai Chung Kwan, Kam-Fai Wong, and Yefeng Zheng. CoAD: Automatic diagnosis through symptom and disease collaborative generation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6348-6361, Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10.18653/v1 /2023 acl-long.350 URL https::/ wrg. ac acl-long. 350/.
Junda Wang, Zonghai Yao, Zhichao Yang, Huixue Zhou, Rumeng Li, Xun Wang, Yucheng Xu, and Hong Yu. NoteChat: A dataset of synthetic patient-physician conversations conditioned on clinical notes. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, PP. 15183-15201, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10. 18653/v1 f002.ffindngg-aa ac1.901. URL httpss///alaatthooog/ org/2024, findings-acl. 901/.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah Smith, Iz Beltagy, al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36:74764-74786, 2023b.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.0867. 2024c.
Ziyu Wang, Hao Li, Di Huang, and Amir M. Rahmani. Healthq: Unveiling questioning capabilities of 11m chains in healthcare conversations, 2024d. URL /ttps:///rxxv..or org/ abs/2409. 10099.99488 19487.
Candace West. Routine complications: Troubles with talk between doctors and patients. 1984.
Ka Wong, Praveen Paritosh, and Lora Aroyo. Cross-replication reliability - an empirical approach to interpreting inter-rater reliability. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7053-7065, Online, August 2021. Association for Computational Linguistics. doi: 10.18653 / v1 //22222..cc acl-loog..548. URL https://acland https://aclanttholog.ooo wwwpthooooooooooo...222 acl- acl-long.548
18

---

Published as a conference paper at COLM 2025
Nahathai Wongpakaran, Tinakon Wongpakaran, Danny Wedding, et al. A comparison of cohen's kappa and gwet's acl when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Medical Research Methodology, 13:61, 2013. doi: 10.1186/ 1471 11777-2288-1---41 URL https:: .eroii..rg//// org/10, 1186/1471 2288--22288111 1112888-1---0 13-61.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36:59008
14159008-59033, 59033, 2023.
Yunfei Xie, Juncheng Wu, Haoqin Tu, Siwei Yang, Bingchen Zhao, Yongshuo Zong, Qiao Jin, Cihang Xie, and Yuyin Zhou. A preliminary study of ol in medicine: Are we closer to an ai doctor? Xrviv preprint ar arX:i:2209 2024.
Rui Xin, Niloofar Mireshghallah, Shuyue Stella Li, Michael Duan, Hyunwoo Kim, Yejin Choi, Yulia Tsvetkov, Sewoong Oh, and Pang Wei Koh. A false sense of privacy: Evaluating textual data sanitization beyond surface-level privacy leakage. In Neurips Safe Generative Al Workshop 2024.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang- Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2024.
Jinluan Yang, Dingnan Jin, Anke Tang, Li Shen, Didi Zhu, Zhengyu Chen, Daixin Wang, Qing Cui, Zhiqiang Zhang, Jun Zhou, Fei Wu, and Kun Kuang. Mix data or merge models? balancing the helpfulness, honesty, and harmlessness of large language model via model merging, 2025. URL https:://axxii..tt////// http::///aaaii...rr//////2222// 0687.
Zonghai Yao, Aditya Parashar, Huixue Zhou, Won Seok Jang, Feiyun Ouyang, Zhichao Yang, and Hong Yu. Mcqg-srefit srefine::ee Multiple choice question generation and evaluation with it- erative self-critique, correction, and comparison feedback. ar Xiv preprint arxi:22410.11111, 2024.
Zonghai Yao, Aditya Parashar, Huixue Zhou, Won Seok Jang, Feiyun Ouyang, Zhichao Yang, and Hong Yu. Mcqg-srefine: Multiple choice question generation and evaluation with iterative self-critique, correction, and comparison feedback, 2025. URL https: //arxiv. wwwpxii.orggabs/2220 2410 13191.
Michael JQ Zhang, W Bradley Knox, and Eunsol Choi. Modeling future conversation turns to teach 11ms to ask clarifying questions. ar Xiv preprint ar Xr12:2200013788 2024a.
Michael JQ Zhang, Zhilin Wang, Jena D. Hwang, Yi Dong, Olivier Delalleau, Yejin Choi, Eu- nsol Choi, Xiang Ren, and Valentina Pyatkin. Diverging preferences: When do annotators disagree and do models know?, 2024b. URL
14632.
Thomas Zhang, Jason HD Cho, and Chengxiang Zhai. Understanding user intents in online health forums. In Proceedings of the 5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics, pp. 220-229, 2014.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonza- lez, and Ion Stoica. Judging llm-as-a-judde with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 46595-46623. Curran Associates, Inc., 2023. URL https: httttt:/s/////ooeeeiiii..gg/s neur nwwr insurips.coo peppp seapper_filesseee www.s//appeeeee/////// 2aA2: Bew____canc__ecceseeeemmmmm pdf.
Hongjian Zhou, Fenglin Liu, Boyang Gu, Xinyu Zou, Jinfa Huang, Jinge Wu, Yiru Li, Sam S Chen, Peilin Zhou, Junling Liu, et al. A survey of large language models in medicine: Progress, application, and challenge. aixiv preprint arXiv:2311.00 2023a.
19

---

Published as a conference paper at COLM 2025
Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one-preference-for-ee Multi-objective direct preference optimization. arviv preprint ar Xrix:22300003788, 20236.
A Extended Related Works
Clinical LLMs. LLMs have potential to highly impact medicine (Moor et al., 2023; Thirunavukarasu et al., 2023) from personalizing care to improving accessibility (Rodriguez et al., 2024; August et al., 2023). Thus, many language models have focused on clinical knowledge and usage including closed-sourced Med-IaLLM 2 (Singhal et al., 2025) to open models such as BioGPT (Lewis et al., 2020), Meditron (Chen et al., 2023), and BioMistral (Labrak et al., 2024) to name few (Toma et al., 2023; Zhou et al., 2023a). More recently, many non-medical, general purpose models such as OpenAl's oll have outperformed medically adapted models (Xie et al., 2024; Jeong et al., 2024). These models have shown human-leva performance on MedQA (Jin et al., 2020) and other medical knowledge benchmarks (Singhal et al., 2023) and some have even shown to provide human-level soft skill such as empathy (Brin et al., 2023).
Clinical Reasoning and Question-asking of LLMs. Reasoning abilities of these systems, especially under complex, high-stakes demands of medical interaction fulfilling various intentions of users (Nov et al., 2023; Zhang et al., 2014) require further attention. More specif- ically, clinical reasoning requires ability to ask effective questions (Silverman et al., 2016), crucial for information gathering phase with iterative hypotheses testing and updating. Shaikh et al. (2024) highlighted general lack of question-asking by LLMs in various contexts. While prior works have focused on improving question-asking of LLMs (Andukuri et al., 2024; Rao & Daumé III, 2018) with some focused diagnostic conversations, these works have been limited to rule-based, toy scenarios (Hu et al., 2024b), or prompting-base -bsee tech- niques (Li et al., 2024). Further, ALBA (Varadarajan under mental health assessment setting. Thus, our work expand on such prior works to- wards more flexible medical dialogue system with effective question-asking under various real-world user queries.
al., 2024) examined question-asking
Alignment Methods. Methodologically, our work adopts various alignment algorithms, especially PPO (Ouyang et al., 2022; Christiano et al., 2017) and DPO (Rafailov et al., 2023). Moreover, to integrate complex and nuanced preferences, multi-objective settings have been explored including MODPO (Zhou et al., 2023b) and MORLHF (Wu al., 2023; Bai al., 2022). Additionally various works have highlighted efficient methods to integrate multiple objectives, for example, through combining reward model or adapter weights (Rame al., 2024; Chronopoulou et al., 2023; Ramé et al., 2024).
Synthetic Data. With the growing capabilities of LLMs and to supplement human data, which can be sparse, synthetic data generation with LLMs have become popular method (Long et al., 2024; Xu et al., 2024), especially to perform task-specific post-training (Kim et al., 2024). Synthetic data is especially appealing in healthcare domain as privacy issues can make data access prohibitive (Ramesh et al., 2024; Xin et al.; Murtaza et al., 2023). Thus, synthetic data generation through both rule-based methods (Fansi Tchango et al., 2022) and LLMs (Oh et al., 2024; Mishra et al., 2024b; Yao et al., 2024; Wang et al., 2024b) have been explored for various medical tasks. However, data to investigate question quality, especially in medicine, remains under under-explored and our data generation method addresses this gap.
Evaluation Frameworks. To assess LLMs in various medical tasks, many different evalua- tion frameworks have been developed, typically consisting of static, single-turn question- answering task based on multiple choice questions (Jin et al., 2020; Pal et al., 2022; Jin et al., 2019; Rawat et al., 2024). However, with the advancement of LLMs, there has been a growing need to evaluate LLM agents beyond simple demonstration of knowledge (Thirunavukarasuu et al., 2023). MediQ (Li et al., 2024) proposes to evaluate LLMs' information-seekine ability through interactive clinical reasoning tasks, and constructs benchmark based on MedQA
20

---

Published as a conference paper at COLM 2025
(Jin et al., 2020) leveraging information asymmetry at benchmark construction time and at inference time. MEDIC (Kanithi et al., 2024) explores comprehensive assessment of LLMs using methods such LLM-as-a-judge (Zheng et al., 2023). Concurrent work HealthQ (Wang et al., 2024d) analyzes attribute related factors in their evaluation, but lacks theory- grounding and does not propose methods to specifically improve the attributes. Our work builds on prior works to comprehensively assess LLM's ability to seek information towards effective clinical communication utilizing LLM-as-a-judge and adopting MediQ framework with newly generated set of task-specific multiple choice questions (Yao et al., 2025).
B Dataset Curation
HealthQ Dataset. To study clinical conversations, we utilize data from publicly available online health forum r/AskDocs", subreddit consisting of both lay-users and expert users? We parsed each subsequent comments as a single thread and consider such threads as conversations between users. Since we are interested in clinical followup questions, we first selected threads where the first followup comment from the community contained sentences ending with question marks and further decomposed each conversation into atomic questions, conclusions, and presence of positive feedback from the post author (e-g,, thank you) using GPT-4o The resulting dataset contained 17, 47,225 threads, 13,496 unique posts, and 24,263 questions.
Synthetic Data. We sampled questions from the above conversations to build a seed set for synthetic data generation. To ensure balanced quality for both corruption and enhancement in contrastive learning, we used proxy measures such as the expert verification status of the question author, the outcome of the conversation (e.g., final conclusions), and positive feedback from post authors. This resulted in 8 proxy quality groups. We evenly sampled from these groups, with the test set including only questions from threads with final conclusions. We created distinct train, validation, and test sets containing 4,463; 433; and 620 questions, respectively, ensuring no post overlap.
Data Quality. We employed three key strategies to ensure data quality: (1) rskkDoos has strict anti-misinformation policies and expert verification processes, (2) previous studies have validated the medical quality of responses in this subreddit, and (3) we filtered for samples with positive feedback and clear conclusions to ensure conversation quality.
e
C Implementation Details
C.1 Counterfactual Perturbations
To generate counterfactual perturbations, we use meta-1lama/Llame 1 4405--Instruct with VLLM on 8 A100 80GB GPUs with a temperature of 1.0 and max generation length of 512. See Appendix H.1 for an example prompt.
Counterfactual Verification & Filtering To verify the quality of generated counterfactual perturbations, we use LLM-judge rank the generated questions in the perturbed attribute (e.g., accuracy). Furthermore, we used LLM-judge result to filter generated data in 6.4. To avoid self- self-preference bias, we used GPT-4o". See Appendix H.2 for an example prompt.
In Table 8, we report the percentage of pairs kept after the filter in the Salanced-Coorupter (O-C) directions, well as the
(C-O), Enhanced-Orie Oriignal (E-O), and Original-Corrupe number of training and dev samples after filtering. Intuitively, since the distance between
20033-2021 data accessed
VVeiffeed by moderators with photo of self with credential documents. See ht
com/r/Ase com/r/AskDocs/.
8 2000-40-2224-8--00
2pt-44-2622--8-000
21

---

Published as a conference paper at COLM 2025
enhanced and corrupted is the largest, the LLM-judge is the most likely to accurately detect the intended perturbation direction, whereas for the Enhanced-Origimn and Original- Corrupted pairs, more samples fail the LLM-judge filter. We can also see that among all the attributes, clarity has the lowest data quality before filtering.
Attribute
E-C
E-O
O-C
# Train
Dev
Accuracy
99.3
98.3
70.9
11,994
1,155
Answerable
99.6
98.3
69.4
11,933
1,154
DDX Bias
99.8
86.9
94.5
12,548
1,223
Clarity
85.8
73.4
70.3
10,250
986
Focus
92.0
72.8
75.3
10,660
1,095
Relevance
99.2
73.0
91.2
11,756
1,142
All
96.0
83.8
78.6
69,141
6,755
Coarse
99.8
94.2
95.7
12,939
1,246
Table Policy Fusion DPO models on attribute groups.
C.2 Training Hyperparameters
We use Open-RLHF (Hu et al., 2024a) to train all models, and adopt the default hyperpa- rameters. All models were trained on one A100 GPU and we list the hyperparameters of the final models below. Additionally, we experiment with different hyperparameter values (as listed parentheses below) and find minimal differences evaluation results.
Supervised fine-tuning:
Epoch: 2
Learning Rate: 5e-6 (le-6, le-5, 5e-5)
Warm up ratio: 0.03
LR Schedule: cosine_with_min
Batch size: 256
DPO:
Epoch:
Learning Rate: 5e-7 (le-6)
Beta: 2 (0.1, 4)
Warm up ratio: 0.03
LR Schedule: cosine wi mi_
Batch size: 256
Reward modeling:
Epoch: 1
Learning Rate: 9e-6 (5e-7, 1e-6, le-5, le-4)
Beta: 2
Warm up ratio: 0.03
LR Schedule: cosine_we with_inn_li
Batch size: 256 (4, 16, 64)
PPO:10
Epoch: 1
Learning Rate: 5e-7
Warm up ratio: 0.03
LR Schedule: cosine th_min_lr
Batch size: 256
Note that while it's possible to use model parallelism to train 8B PPO models with qlora, we did not have the compute resources to train 8B PPO models with the same hyperparameters and settings the 3B counterpart, the experiments did not include comparisons with the 8B PPO models.
22

---

Published as a conference paper at COLM 2025
Training times single A100 GPU with batch size 256:
SFT: 4hr (3B model), 8.5hr (8B model)
DPO: 10hr (3B model), 42hr (8B model) PPO: 48hr (3B model)
C.3 MediQ Interactive Benchmark
We evaluate the downstream performance of the trained model by generating questions in multi-turn clinical reasoning task using MediQ (Li et al., 2024). In MediQ, there is a patient agent and an expert agent interacting with each other, where the expert agent is provided some initial information in the beginning, and is expected to decide whether it wants to continue the interaction to acquire more information terminate the interaction and provide final answer. In this framework, the expert agent consists of three modules: abstention, question generation, and decision making. We fix the abstention and decision making modules using aataallama/Llaaa with the model variants in our experiments. The goal of this evaluation is show the effect of question quality on the final diagnostic accuracy. For reproducibility, we list the hyperparameters used in the MediQ interactive framework below:
1-8B-Instruce and replace the question generator
Abstention strategy: Scale Rationale generation: True
Self-consistency: False
Maximum interaction length: 15 Temperature: 0.6
D Expert vs. Non-Expert Human-Written Questions
In r/AskDocs, members can upload credentials to acquire expert flair (tag). In order to validate the quality of the LLM-judge, we aim to observe differences in the reported win-rates concerning questions generated by experts vs. non-experts. Since the original data extremely scarce, there is limited samples where an expert and a non-expert respond to the same patient information, we design the following comparison scheme:
1. Starting with two sets of contexts, one with expert-written responses, one with non-expert written responses.
2. Using variety modell- ALFADODDDDMM.-DD ALEA--DPO-PolicyEussoo ALEA-PPO-DattaMixx ALEA--POO-Policyyuuiionn generate responses conditioned on each set of contexts.
3. Use our LLM-judge to compare the human written response the model generated responses for each set of contexts.
Compute the win-rates of expert vs. model and non-expert vs. model.
Following the above procedure, we find that the win-rate of non-expert responses is 35.87% while the win-rate of expert responses is 50.52%, suggesting that expert-written questions are of higher quality than the non-expert questions. Thus, this finding validates the relative accuracy of our LLM-judge. Additionally, this also shows that the responses generated by ALFA-aligned models are approaching human-expert quality.
E MediQ-AskDocs Task Construction
Task Construction. To construct the MediQ-AskDocs task for clinical reasoning and use it in the MediQ interactive doctor-patient simulator, we need to parse each patient's informa- tion, collected from Reddit conversation thread, into the following components: initial information, additional information, inquiry, options, correct answer.
Taking the entire conversation thread between the patient and the community member, including the initial post from the patient, as the patient's record, we prompt of to extract the initial information of the patient with few shot examples. Then, in separate call, we
23

---

Published as a conference paper at COLM 2025
prompt o1 to extract the patient's inquiry-post in r/AskDocs often in the form that the patient posts a paragraph of their information and asking a health question and the conclusion from the responder if any. Finally, we treat the parsed conclusion as the correct option, and prompt the model to generate three alternative wrong answers to the inquiry.
We generate multiple choice questions following the above approach for all 302 threads in the test split of the MediQ-AskDaa dataset to form the novel interactive healthcare QA task.
E.1 Expert Annotations
Task Setup. We collected human expert annotations validate the machine-generated multiple choice questions. To validate machine machine-generated questions, options, and the correct answer, the task included three questions: 1) plausibility of the generated question ("Yes" or "No"), 2) selecting correct option out of four candidates with an additional option to select "None of the above", and 3) adding a free-text option a plausible option is not listed. We randomly assigned maximum of 3 annotators per sample and paid 20 USD/hr //rr. Participant recruitment process detailed in Appendix F.
Annotation Details. Due recruitment constraints, we collected three expert annotation per sample over 295 samples, excluding samples annotated during recruitment.
Coefficient Name
Coeff
Interpretation
StdErr
Fleiss' Kappa
0.58
Moderate Substantial
0.029
AC1
0.65
0.027
Table 9: Inter-rater agreement on multiple-choice question correct answers.
Results. On 298 samples, excluding test questions, questions were considered plausi- ble and correct answers generated by 01 showed 85.9% accuracy based majority vote on 298 samples. Upon examining agreement on 295 samples, excluding samples used during recruitment, we moderate to substantial agreement (Wongpakaran et al., 2013; Wong et al., 2021) as shown Table
Expert Manual Evaluation
Expert manual evaluation was conducted by panel of three research team members with extensive medical training and qualifications: two MD-PhD specialists with expertise in radiology, pathology, gastroenterology, and oncology, and one MD resident specializing in pathology. The members were instructed to rank the proposed questions given context (patient post and previous conversation prrraationqqeestonns and answers between medical expert and patient-it any) in an online medical consultation setting similar to r/ AskDocs.
$PO
POI
Win-rate
Data
52
DPO
Policy
41
Data
75
PPO Reward
74
Policy
40
Human
47
Table 10: Win-rate of each model variation over baseline model for different attribute integration strategies based on majority vote.
E1 Ranking Annotation Task Setup
We randomly selected 100 samples and asked our panel to rank 7 different questions per sample from best to worst follow-up questions. The seven question variations included one
24

---

Published as a conference paper at COLM 2025
*PO
POI
AC1
CT Lower
CT Upper 0.349
p-value
PA
PE 0.449
Data
0.129
-0.090
0.245
0.520
DPO
Policy
0.240
0.047
0.434
0.016*
0.620
0.500
Data 0.721
0.594
0,848
<0.001 s**
0.780
0.211
PPO
Reward 0.680
0.543
0.817
<0.001 ***
0.750
0.219
Policy 0.224
0.028
0.419
0.025*
0.610
0.498
pp=0.05, "*p=0.01, p<00000
Table 11: Gwet's AC1 inter-rater reliability results between Human and LLM Judge win-rate over baseline by model variation. PA denotes percent agreement and PE denotes expected percent agreement by chance.
humanewritten question and six model generated questions under different experimental setup (Base Model, ALFA DPO Data, ALFA DPO Policy, ALFA PPO Data, ALFA PPO Policy, ALFA PPO Reward). The questions were presented in randomly shuffled order, and annotation platform was constructed to not allow ties.
Expert Human Preference Generated Questions
DIO Buse
Dats Model
23.00
$3.00
80
$1.00
50.00
60.00
23.00
61.00
26.00
52.00
70
Patient Context:
DPS
Polcy
43.00
39.00
50.00
10.00
$4.00
10.00
40.00
60
testicles \n5o basiclly Ithis mostly trying sleep) when I' sed or ovethink about something negative, feel VERY hand pain in my pents testicles
Data
75.00
76.00
90.00
50.00
$9.00
47.00
84.00
-50
ruch that this? overthinking
when
Human 191 PPO
orace Revirrd Polly
40.00
37.00
45.00
11.00
$0.00
8.00
37.00
40
solution L'ne experiencing such pain, uninedits I'm 17.
know
Lowest Ranked ALFA DPO Policy:
Do you have any Bood swings?
74.00
74.00
90.00
53.00
92.00
50.00
86.00
Highest Ranked ALFA
Reward:
Can you describe the pain more detail? Is st dull ache, a sharp stabbing pain, or burning sensation? Dees occur throughout your
48.00
48.00
60.00
16.00
63.00
14,00
50.00
penis, or sostly
spot?
Base Madel
ALSA
ALFA
ALFA PP0
ALFA PPD
ALFA
Human Oracle
DPO
PPC
Figure An example of majority annotated best question (PPO Reward) and worst ques- tion (DPO Policy).
Policy
Figure 7: Expert preference ranking results showing pair-
wise win-rates. Models on y-axis compared to the
models on x-axis (e.g,, ALFA PPO Data has 75% win-rate over Base Model).
F22 Results
Similar to the results discussed in $6.4, human annotation win-rate shows preference towards ALFA PPO Data and ALEA PPO Reward, trained on fine-grained reward fusion as shown in Table 10. Interestingly, while automatic evaluation medical diagnostic accuracy, ALFA DPO Policy outperformed other integration strategies, we observe that it only outperforms ALFA PPO Policy. This diverging outcome could indicate various factors, including stylistic preferences (Zhang et al., 2024b) that might influence annotation decisions which require further investigation. As noted in $6.4, our medical diagnostic accuracy evaluation relies on patient provided information; however, as shown in an example in Figure 8, the best quality question from ALFA PPO Reward more open-ended and targets multiple aspects, which might not have been answered by the patient in our data.
Inter-rater Agreement Medical experts had substantial pairwise ranking agreement of .811 Gwet's AC1 score. The experts also showed high agreement with LLM judge, especially for model variations with higher win-rate. Experts noted that questions with lower win- rate showed similar quality leading to random assignment of preferences ties were not allowed on our platform, which could explain the lower agreement for model variations with lower win-rate.
25

---

Published as a conference paper at COLM 2025
Context
Patient:
In Hospital Aggi Real Lile Dr. House 40m, white, non smoker casual drinker. am currently in a hospital bed on Heparin Drip, have Vascular, Hematology, oncology, and even Gastro all stumped. Basically, 1 am under attack by blood clots but no one has a clue why. A year of this. Long story short calf DVT 10/20. Found May Thurner, stent placed, clot resolved, moved on with life. Taken off thinners.
July spinal surgery. 3 weeks later, blood clots EVERYWHERE, spent a week in hospital had thrombectomy. Simple answer - surgery caused clots (even though ive had many surgeries before with no clots). Put on Xarelto.
November- found more clots even Xarelto. Do tPA lysis catheter puts me in hospital another week. Clears clogged stents, put on Lovenox.
2 weeks later more clots. Just had another thrombectomy. have tested negative for everything. Negative for all genetic factors, negative for rare clotting disorders. And yet, here am.
CT Scan of chest and pelvis no tumors. Lymph nodes swollen but subcentimeterin the area where all the clotting keeps happening.
Everyone is stumped. They won't let me out of the hospital. I feel fine except for my leg which has been through alot.
Only info to go off of I have been anemic off and on, Monocyte % slighly high, folate very low, absolute immature gran count high.
am here in the hospital. can ask anything. Give me your wildest ideas. Something to have them test. I want to go home but also desperate for diagnosis. Thank you!
PS: WOW- thank you all. Updates from what I've relayed off of this thread- hematology is now done with me unless something changes. They say they've tested for every clotting disorder and looked as much as they could for underlying cancers. They say definitively it is not APS. Update 2 (I don't think anyone is still checking this but just in case) - Tumor marker test was negative but they did find a precancerous polyp from the colonoscopy.
Doctor: Yup, was wondering for their clotting disorders if they tested for connective tissue disease and things like antiphospholipid, lupus anticoagulant etc
Patient: Yes. They said had a weak positive for APS. They dont think its that but said they were going to run some more tests but havent yet
ALFA-PPO-Data
Have you had any of the following symptoms the past 2 weeks before the clots occurred: pain or swelling of the chest, abdomen, or groin, difficulty breathing, coughing or fever, pale gums, yellowing of the skin eyes, nosebleeds or bleeding gums?
ALFA-PPO-Reward
Can you describe the anemia you've 've experienced? Was it constant, or did it only occur during periods of clotting, and if so, how often did it occur?
Was it painful or associated with any other symptoms? Did you ever get the LVT?
ALFA-PPO-Policy ALFA-DPO-Policy
And have you had your thyroid checked?
Base
So they ran the APŠ test but aren't I convinced it's the cause. In that case, here are some wild ideas to consider:
"AAutoimmune response to something else**: Despite having low fo-
late, maybe the issue isn't I with folate deficiency but with an autoimmune response to something else. Could it
ALFA-DPO-Data
weak positive is not conclusive, and they' re still awaiting more results or further testing to rule out or confirm it.
That's the problem-
Okay, wild ideas time!
Here are a few potential test ideas, not that they're re guaranteed to work, but might be considered:
1. "Carddillip
Human
Did you or your sister have any negative tests before being diagnosed
with lupus / APS?
Table 12: Qualitative example of generated questions. The questions are ordered from best (ALFA- PPO-Data) to worst (Human) quality based on human annotation according to majority vote.
26

---

Published as a conference paper at COLM 2025
G Qualitative Analysis
Table 12 shows an example of multi-turn conversation and generated follow-up questions ordered by ranking of best to worst quality as denoted by our team medical experts. ALFA- PPO-Data ranked best out of all options and asks comprehensive and targeted follow-up question. Moreover, the example shows that even while prompted ask questions, and trained to do so, asking up questions is difficult task as shown by the response by base model and ALFA-DPO-Data.
H Prompts
H.1 Counterfactual Perturbations
You are a medical assistant and your task posted to an online health forum to vary some of their properties. The goal is to generate these diverse counterfactual questions to study the properties of clinical questions. You will be given patient's post, and the original clinician response, and you should rewrite the clinician response according to the instructions below.
rewrite medical questions
***PATINNT POST*** title post
***CLINICIAN RESPONSE*** question
***INSTRUCTOON*** Rewrite the clinician response so that it is less clear/more
ambiguous for the patient, while keeping everything else constant. The definition of this property and what it means for this property at varying scales are given below:
Definition: The ease with which reader can understand the intent and meaning of the question. A clear question avoids ambiguity and vagueness, providing enough detail to prevent mi misunderstanding or overloading with jargon. Very ambiguous: The question is highly ambiguous, vague, disorganized, making it very difficult to understand what the asker is seeking. The question may lead to multiple interpretations and confusion. Somewhat ambiguous: The question is somewhat ambiguous or vague and may include overly complex phrasing. It requires significant effort to interpret. In-between: The question is mostly understandable but could benefit from rewording or simplification to remove partial ambiguity or excessive jargon. Somewhat clear: The question is generally clear, with minimal ambiguity, and can be understood by layperson. There is little chance of misunderstandingg Very clear: The question is entirely unambiguous, easy to understand, and structured in logical, concise manner. No jargon or unnecessary complexity. Additional Tips for Clear Questions Use specific time frames: Instead "lately," try "in the past week" "since your last visit." Break down complex questions: asking two separate questions. Avoid medical jargon: Use plain language that patients wiohuut a medical background can understand.
while avoiding excessive complexity ty
a question could be answered in multiple ways, consider
Please make the rewritten question more realistic something that clinicians would ask in an actual patient interaction.
Return the rewritten question ONLY and do not include any other text. ***REWRITTEN RESPONSE***
27

---

Published as a conference paper at COLM 2025
H.2 Automatic Synthetic Data Verification
SYSTEM: Please act as an impartial judge and evaluate the quality of the responses provided by three medically trained AI assistants in medical interaction. Carefully read the questions being asked by these expert systems as response to the medical interaction and rank them in the provided dimensions. Begin your evaluation by comparing the three responses and provide short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses possible spelling or grammar mistakes and focus only on the content of the text. Be as objective as possible. The only ranking choice is ">" (greater than). For each dimension listed, provide your answer in the following example JSON format: { "dimension_name", { "ranking": "A B C", "reasoning": "Provide a clear and concise explanation for your ranking decision here." }
influence your evaluation. Ignore
USER: Please carefully review the previous interaction below, which includes patient post, title, and subsequent responses if any. ***PREVIOUS MEDICAL INTERACTION*** prev_context ***MEDICAL AI QUESTIONS TO PATIENT**** **Question question_a **Question 8:** question_b **Question C:** question_c ***EVALUATION DIMENSIONS*** dimensions ***SUPPLEMENTARR INFORMATION To help you evaluate the questions, please refer the provided additional information regarding the final conclusion of this patient's 's case below: Final diagnosis: finno_di Conclusion: conclusion
28

---

Published as a conference paper at COLM 2025
H.3 MediQ-Askkoees Task Construction
SYSTEM: You are a experienced expert working in the field of medicine education. Based on your understanding of basic and clinical science, medical knowledge, and mechanisms underlying health, disease, patient care, and modes of therapy, you are given patient case and you tasked to parse the patient's s inquiry into a multiple choice question. The generated multiple choice should consist of question and 4 options, which could be answered by the given patient conversation. Base your response on the current and standard practices referenced in medical guidelines. The created question should be answerable only with the patient information, rather than testing some hardcore scientific foundational knowledge recall. The questions should be faithful to the original patient's inqiury in their post. The correct answer should be correct, and the distractors should be plausible. The correct answer should be evenly distributed among the available options to enhance the quality and reliability of the questions. The output should be json format.
USER: You could use some parsed auxiliary information such as the final diagnosis and conclusion. Make sure that the multiple choice question you generate is not too easy but also not impossible to answer. Based on this patient record, faithfully generate multiple choice questions according to the patient inquiry and store them in the following json format:
{
"question": Igenerated question 11,
"optionA": [option A],
"optionB": [option B],
"optionC": [option c],
"optionD": [option D],
coorrect_answer:: [A or B or C or D]
}
After you generate the question, do a round of revision. In your revision, you should:
1. Identify any medical inaccuracies in your first response, corrsect them
if any exists.
2. Make sure the question is what the patient is asking for or concerned about in their post.
3. The correct answer is indeed correct, if none of the options are correct or more than one options are correct, revise the options to improve the question.
4. Ensure that the correct answer is in a random position among the available options (shuffle if necessary) to enhance the quality and reliability of the questions.
5. Guarantee that the json output is parsable.
Respond with the final revised question in the json format and NOTHING ELSE.
29


Reviews:
[{"review_id": "GUEzF3E2Kk", "reviewer": "Reviewer_aB4k", "summary": "The main contribution of this paper is the ALFA framework, which aims to improve the question-asking ability of large language models in expert domains, focusing on clinical reasoning.\n\nThe paper also contributes the MediQ-AskDocs dataset, which includes 17k real-world clinical dialogues and 80k synthetic attribute-specific preference pairs.\n\nThe paper presents a comprehensive analysis and comparisons among integration strategies (DPO, PPO, reward vs. policy fusion).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It presents rigorous empirical validation.\n\nGeneralizable framework applicable beyond clinical domains.", "reasons_to_reject": "Reliance on synthetic data and LLM judgments could introduce bias or hallucinations.\n\nAttribute selection is manual and might not generalize without domain-specific expertise.", "questions_to_authors": "How does ALFA handle conflicting signals between attributes (e.g., clarity vs. medical accuracy)? Could there be dynamic weighting mechanisms in place?", "extra_scores": {}}, {"review_id": "oeQjdUagmM", "reviewer": "Reviewer_yWPb", "summary": "This paper presents an approach for better question generation in the context of clinical reasoning and LLM-assisted diagnosis. There are three key concepts in the proposed approach:\n\n(1) Provide a better, more clear definition of what better quality questions should be, both in general and in the context of clinical reasoning, based on a set of grounded attributes extracted from related work. Examples of these are clarity and focus (for the general context) and diagnostic for the medical context.\n\n(2) With these attributes in mind, utilize an LLM to synthesize questions by modifying existing seed questions making them more or less satisfactory for a given \"good question\" fine-grained attribute, e.g. making a question more or less \"clear\". This results on \"enhanced\" and \"corrupted\" versions of each seed question\n\n(3) Utilize these pairs of synthesized questions as preference data for aligning LLMs, and test several preference-based optimization techniques to train models to ask better question along these fine-grained attributes.\n\nTo test this, the paper proposes to leverage the `r/AskDocs` reddit forum, which contains interactions verified by medical experts. The empirical framework proposed on this data is sound and complete, with analysis of the impact of several key decisions in the including but not limited to:\n\n- Filtering applied to the data generated as a result of step (3), performed via using an LLM-as-a-judge\n- Comparison of different LLM alignment techniques, including SFT and preference-based training. Critically, these are applied to exactly the same data. \n- Different decomposition of \"good question\" attributes from fine-grained (clarity, medical accuracy, etc.) to  \"coarse\" (just good or bad)\n\nThe resulting models are tested both on in-domain data as well on out-domain data, the latter being the result of a held-out portion constructed using the source data. Several key insights are derived from the empirical results, indicating which data filtering and training techniques lead to more aligned and/or better quality output.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Overall, I think this paper is a solid contribution to COLM. It tackles a relevant research problem and proposes a sound and effective solution.\n\n- The insights derived from the experiments presented allow us to learn which parts of the proposed pipeline are critical and which training schemes lead to better output. I learned a significant amount from this paper, and the lessons learned here will sure inform future work, so I think this paper can have a significant impact in our community.\n\n- The paper is generally clear and well written.", "reasons_to_reject": "None, but please see my questions below.", "questions_to_authors": "- In Table 1, both 3B and 8B base models exhibit a win-rate performance of exactly 50%. I find this a bit suspicious, could you confirm that these are empirical values, and if so, how these numbers were obtained?\n\n- In line 245: where does the number 56.62% come from?\n\n- I wonder if it would be possible to have a small study on the impact of the LLM used for data curation, specially since the paper relies on GPT-4o a lot. At this point, do we require a cutting-edge model for these steps, or can we get by with a cheaper, perhaps local model?  \n\n- Finally, it would be helpful if the paper could provide estimates on the amount of compute and budget used for the experiments, to help inform future work on the trade-offs in this sense.", "extra_scores": {}}, {"review_id": "72jjL37YTy", "reviewer": "Reviewer_cdw7", "summary": "The paper proposes ALFA, which trains models that generate clinical questions with measurably better alignment on multiple assessment criteria. The work does this by breaking down the assessment of a question into multiple important measurable characteristics grounded in theory. Then generated a dataset by agumenting the existing dataset to get a preference dataset on all the selected attributes. The authors then show how to use this dataset to instruction-tunned models.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors use a theory grounded approach to creating assessment dimensions of clinical questions.\n- Strong performance gains when using ALFA\n- Synthetic Counterfactual Data Generation with Controlled Perturbations is an interesting and scalable approach to preference dataset creation.\n- Ablation studies to investigate data synthesis direction, attribute inclusion, integration strategy (data/reward/policy fusion), and model size\n- Modular design (decompose → synthesize → align) provides a replicable recipe for future work in other expert domains\n- Authors contribute a high-quality dataset combining real-world with synthetic preference pairs and expert annotations", "reasons_to_reject": "Major Concerns:\n- No direct comparison with LLMs\n    - The paper compares ALFA primarily against instruction-tuned LLaMA-based models and SFT baselines. It does not benchmark against existing medically-tuned models (e.g., MedPaLM, PMC-LLaMA, or BioMistral) or other general purpose strong models like GPT4 or Gemini, which could provide a stronger empirical context for the results.\n    - There is one line in the paper mentioned without any details that ALFA performs better than GPT4o in downstream utility. This statement is not useful for the readers without any details.\n- Potential Page Limit Violation because the Table 7 is present in the appendix, but referenced in the results.\n\nMinor concerns:\n- Lack of Real Clinical Dialogues in the evaluation or in the dataset\n   - The paper uses Reddit QA in the dataset, and the interactive clinical tool (using LLM-based agents to roleplay), but the real-world setting is not explored in the work.\n- ALFA requires manual, domain-expert-defined attributes, which may not scale easily to other domains or more complex tasks. The paper acknowledges this.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZRZ4agVY4Y", "reviewer": "Reviewer_3HyA", "summary": "This work introduces a framework for LLMs to ask meaningful questions by tuning the model with the preferences on which question is better than the other question.  \nSpecifically, the authors define 6 attributes, which are clarity, focus and answerability for general questions, as well as accuracy, relevance and DDX bias for clinical domain questions.  \nBased on these attributes, the authors prompt an LLM to generate pairs of enhanced (better) questions and corrupted (worse) questions regarding each individual attribute, and suggest to tune the model using these preference pairs with standard preference optimization algorithms such as DPO and PPO.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written.\n2. The work is well-motivated. Especially, since the public datasets are not largely available in healthcare domains due to the privacy issues, researchers to develop question answering systems usually rely on LLM-generated QA pairs. In such scenario, the LLMs tuned by the suggested framework will serve as a useful tool to generate clinically meaningful dialogues, which can help improve quality of the resulting QA datasets by grounding the content in medically appropriate knowledge and style.\n3. The results show that the model aligned with the generated preferences pairs outperforms baseline models (llama-3.2-3b-Instruct and llama-3.1-8b-Instruct), demonstrating the need of explicit framework for LLMs to produce good questions.\n4. The ablation studies are well-structured.", "reasons_to_reject": "1. The dataset is collected from an open internet ($\\texttt{r/AskDocs}$), which seems not convincing with regards to its conversation quality.", "questions_to_authors": "None.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes ALFA, a modular framework for improving question generation in clinical contexts using attribute-specific preference data and alignment techniques. Reviewers agree that the work is well-motivated, clearly written, and presents strong empirical results with rigorous ablations. The use of grounded attributes and synthetic counterfactual data is seen as a scalable and insightful approach, and the dataset contribution is appreciated. However, some reviewers raise concerns about the reliance on synthetic and Reddit-based data, and the lack of comparison with state-of-the-art models like MedPaLM or GPT-4. There are also questions about the scalability of manually defined attributes and the absence of real clinical dialogues in evaluation. Despite these issues, the paper is generally viewed as impactful, and reviewers believe it will inform future research in expert-domain LLM alignment. I recommend acceptance.
Decision: Accept