Enabling Scalable Evaluation of Bias Patterns in Medical LLMs

## Abstract

Abstract Large language models (LLMs) have shown impressive potential in helping with numerous medical challenges. Deploying LLMs in high-stakes applications such as medicine, however, brings in many concerns. One major area of concern relates to biased behaviors of LLMs in medical applications, leading to unfair treatment of individuals. To pave the way for the responsible and impactful deployment of Med LLMs, rigorous evaluation is a key prerequisite. Due to the huge complexity and variability of different medical scenarios, existing work in this domain has primarily relied on using manually crafted datasets for bias evaluation. In this study, we present a new method to scale up such bias evaluations by automatically generating test cases based on rigorous medical evidence. We specifically target the challenges of a) domain-specificity of bias characterization, b) hallucinating while generating the test cases, and c) various dependencies between the health outcomes and sensitive attributes. To that end, we offer new methods to address these challenges integrated with our generative pipeline, using medical knowledge graphs, medical ontologies, and customized general LLM evaluation frameworks in our method. Through a series of extensive experiments, we show that the test cases generated by our proposed method can effectively reveal bias patterns in Med LLMs at larger and more flexible scales than human-crafted datasets. We publish a large bias evaluation dataset using our pipeline, which is dedicated to a few medical case studies. A live demo of our application for vignette generation is available at https://vignette.streamlit.app . Our code is also available at https://github.com/healthylaife/autofair .

## 1 Introduction

Large language models (LLMs) have shown an impressive potential to assist in various forms of real-world challenges, including finding long-sought solutions to some key medical challenges such as provider burnout, processing of claims, and obtaining preauthorizations. The LLMs used in medical applications (hereafter, Med LLMs) have been leveraged for clinical note summarization [], patient consultation [], and generating differential diagnosis [], among others.

Despite the promising potential, using LLMs in sensitive assistive decision-making scenarios, such as clinical decision support, brings about significant concerns. One of the most critical concerns relates to the impact of LLM-based tools on bias and fairness. Specifically, deploying biased LLMs in clinical settings can amplify existing healthcare inequities, for instance, by providing unfair care recommendations or service denials based on patients’ demographics. A fairly large family of recent studies has documented biased behaviors of Med LLMs across different (a) LLM types, (b) clinical tasks, and (c) subpopulations [; ; ; ]. Biased behavior in LLMs can arise from a variety of sources, including the inherent biases in the training data, the limitations in capturing and utilizing factual knowledge, and flawed training and inference strategies [].

Prior to mitigating bias patterns in Med LLMs, it is critical to develop tools for comprehensive evaluation of such patterns. Despite the growing number of methods for the evaluation of LLMs (LLM Evals) [], evaluating LLMs remains a challenging task due to reasons such as open-endedness and stochasticity of the LLM outputs []. Evaluating Med LLMs generally involves creating benchmarking datasets with controlled clinical ‘vignettes’ (i.e., patient scenarios) that target sensitive cases within ‘red teaming’ frameworks []. Developing such comprehensive benchmarks is, however, extremely resource-intensive and requires manual design by clinical experts. Existing hand-crafted benchmarks for evaluating the fairness in Med LLMs have a limited scope by design and cannot extend to various complex real-world clinical scenarios [].

To generate the evaluation scenarios at scale, one could follow the common practice of achieving scalability by using the LLMs (of the same or different type) as generators. In fact, this is the practice that almost all prior studies have taken to generate clinical scenarios by prompting strong LLMs and then vetting the generations by clinical experts before using those scenarios for Med LLM evaluation. Removing the human experts (i.e., the main bottleneck in scalability) from this generation loop is quite tricky. Medical scenarios are greatly contextualized and domain-specific, with numerous exceptions and subtleties. For instance, treating individuals differently based on race may be justified in the case of skin cancer [] but not kidney disease []. Furthermore, concerns about LLM hallucinations make relying solely on LLMs to generate evaluation scenarios more challenging.

In this study, we present a greatly controlled process to generate evaluation scenarios by extracting rigorous biomedical evidence guidelines documenting potential bias pitfalls in each domain. We incorporate customized methods to contextually minimize hallucinations and actively check for domain-specific medical intricacies related to the justified (vs. unjustified) associations between the protected attributes and the targeted health outcomes.
Specifically, the contributions of this study are:

- •
We present a new method for flexible vignette generation in the form of user-specified scenarios by directly extracting documented bias pitfalls in biomedical knowledge bases and then filtering unsupported scenarios.
- •
We publish a new dataset generated by our method for assessing the fairness in Med LLMs in the domain of obesity prevention and treatment.
- •
Through a series of comprehensive experiments, using both analytical and human rating evaluations, we show our method can significantly outperform naive LLM-based methods in generating reliable vignettes.

## 2 Preliminaries

We aim to address three key challenges in the automated generation of clinical vignette generation for fairness evaluation, including domain-specificity of the fairness evaluation, outcome independence, and hallucination.

##### Domain-specificity of the Fairness Evaluation:

Medicine is a highly specialized field, and similarly, fairness evaluation in clinical settings is highly domain-specific. Each domain generally has certain ‘fairness pitfalls’ that are unique to that particular medical domain and are characterized through rigorous evidence-based epidemiological or mechanistic (biological) studies [; ]. For instance, while focusing on gender disparities in heart attack diagnosis may be more meaningful in certain branches of cardiology, racial differences in cancer screening may be more relevant in some branches of oncology.
Therefore, fairness (i.e., a fundamentally sociotechnical concept) must be characterized within each specific clinical context through a rigorous evidence-based process.

##### Outcome Independence:

The common procedure for using red teaming strategies to evaluate Med LLMs involves iterating over different values of a target sensitive attribute (e.g., male and female in the case of gender). More formally, red-teaming approaches check for counterfactual (parity-based) fairness defined as:

$$ $P(\hat{Y}_{\mathcal{A}\leftarrow\alpha}|X=x)=P(\hat{Y}_{\mathcal{A}\leftarrow \alpha^{\prime}}|X=x),$ (1) $$

where $\mathcal{A}$ shows a sensitive attribute with two values of $\alpha$ and $\alpha^{\prime}$, $X$ shows the remaining input variables and $\hat{Y}$ shows the outcome. This formulation generally assumes that the sensitive attributes are independent of the outcome ($P(Y|\mathcal{A})=P(Y)$).

Such an approach cannot be naively applied to the generated clinical vignettes, as in many cases, the sensitive attributes and health outcomes are correlated. Examples of such scenarios include race and skin cancer, gender and pregnancy complications, place of birth, and exposure to tropical diseases.

##### Hallucination:

Besides the two specific challenges listed so far, LLM-based generative methods are also known to face the hallucination problem. Hallucination in LLMs can be categorized into two main forms: lack of factuality or lack of faithfulness []. Lack of factuality refers to producing statements that are untrue or unsupported by real-world knowledge, while lack of faithfulness means the generated content does not accurately represent the original context or input, deviating from the given prompts or context.

Factuality hallucinations, in turn, have three main forms. First, knowledge recall failure [], where LLMs struggle to retrieve specific or less commonly known information (e.g., symptoms of a rare disease) from their training data, potentially leading to inaccuracies in the content of generated vignettes. Second, domain knowledge deficiency [], where the LLMs trained on diverse datasets (e.g., internet data) may lack deep, specialized knowledge in specific medical fields. Third, LLMs can suffer from outdated factual knowledge [] if they are only trained on static without continuously updating their knowledge base with the latest research or clinical findings.

In the case of our study, faithfulness hallucinations refer to generating responses that deviate from the input prompts and to generating irrelevant, fabricated, or contradictory details. Furthermore, faithfulness refers to the degree to which the generated vignettes accurately reflect the content and information presented in the retrieved evidence. This adherence ensures that the generated outputs are not only relevant but also reliable in terms of the information they convey, maintaining the integrity of the original data while creatively transforming it into a vignette-style output.

Figure: Figure 1: The proposed method’s overview. Green boxes show a sample scenario related to our obesity case study.
Refer to caption: x1.png

## 3 Method

We propose a vignette generation method to enable scalable and evidence-based evaluation of bias patterns in Med LLMs while addressing the three discussed challenges. At a high level, in our method, an information extractor (§3.1) retrieves relevant evidence from biomedical knowledge bases related to user-determined scenarios. The extracted information aims to help contextualize fairness and identify the bias pitfalls in each domain. Then, a vignette generator LLM uses the extracted evidence to generate “base” vignettes (§3.2). Next, we exclude the base vignettes with signs of hallucination (§3.3). Then, outcome independence between the augmented sensitive attributes and the health outcomes is confirmed (§3.4). After this step, to create the red-teaming vignettes, targeted sensitive attributes are then augmented with each base vignette (§3.5).

More formally, given a user-specified set of inputs $I$, a generative LLM $\mathcal{M}$, and a knowledge base $\mathcal{K}$, we propose a pipeline $\mathcal{P}$ that generates a set of distinct red-teaming vignettes $V$:

$$ $\mathcal{P}(I,\mathcal{K},\mathcal{M})\rightarrow V,$ $$

where, the input $I$ includes targeted health outcome, $o$; the desired number of vignettes, $n$; and the sensitive attributes of interest $S$. Hence, the inputs are $I=\{o,n,S\}$. Figure 1 shows an overview of the proposed method.

### 3.1 Information Retrieval

To generate the base vignettes, we connect a (strong) LLM to external biomedical knowledge bases ($\mathcal{K}$). A “base” vignette here refers to an initial clinical scenario to create the final red-teaming scenario after being later modified by manually augmenting sensitive attribute values to that. The retriever procedure ($R$) uses the knowledge base $\mathcal{K}$ and targeted health outcome and returns a set of articles $A$ that are most relevant to characterize bias patterns within the specified context:

$$ $R(\mathcal{K},o)\rightarrow A$ $$

Articles $A$ refer to various types of biomedical literature capturing the most recent evidence-based practice, including clinical guidelines published by professional societies, clinical trial reports, and systematic reviews (as well as common scientific articles). We use PubMed Central (PMC) as our knowledge base. PubMed offers a search engine to access a full-text archive of biomedical and life sciences journal literature and is considered the most comprehensive resource of biomedical evidence.
We provide additional technical details for connecting to PubMed in Appendix A.

To find relevant articles related to the targeted health outcome in PubMed, we use a separate knowledge graph (KG) constructed from PubMed articles [], namely the ‘PubMed Knowledge Graph.’ This KG’s nodes consist of biomedical entities (e.g., diseases, genes, proteins, drugs, and chemicals) from PubMed abstracts, and the KG uses a multi-type normalization model, which assigns unique IDs to recognize biomedical entities and corresponding articles. We use a name entity recognition process to recognize the named entities in PubMed abstracts by using the popular BioBERT model [], which offers a pre-trained biomedical language representation model (embeddings) for biomedical text mining.

Specifically, we first identify all nodes corresponding to the health outcome $o$ inside the KG and identify the set of PubMed article ID nodes connected to these outcome nodes ($N_{o}$). Similarly, we identify a set of nodes representing ‘bias’ and similar concepts in the KG and select a set of PubMed article ID nodes connected to these nodes ($N_{b}$). The targeted articles to be retrieved ($A$) from $\mathcal{K}$ will be obtained by selecting the articles that relate to the target health outcome and refer to some form of bias discussion, as:

$$ $A=\{\bigcup_{i}\;{a_{i}}\mid a_{i}\in N_{o}\cap N_{b}\}.$ $$

After querying the knowledge base $\mathcal{K}$ for the articles with IDs in $A$, we combine those to form the context for the generator LLM. Instead of retrieving chunks of the text from the identified articles ––common in RAG (retrieval-augmented generation)-based methods []–– we retrieve full documents to include in the context.

### 3.2 Vignette Generation

For each retrieved article $a_{i}$, an LLM $\mathcal{M}$, is prompted to generate a set of base vignettes using the article as the context:

$$ $\mathcal{M}(a_{i})\rightarrow V_{\text{Base}},$ $$

where $V_{\text{Base}}$ is the set of base vignettes that have been generated by the language model from evidence $A$.

We use a chain-of-thought process to instruct the LLM to generate the base vignettes. Specifically, in our prompt design, we describe a series of steps that the LLM needs to take in compiling the provided context and generating the clinical scenario. We instruct the LLM to generate yes/no questions, as this style has been the most common form of generating datasets for LLM alignment and red-teaming []. We also ensure a balanced distribution of ‘yes’ and ‘no’ answers across the vignettes are generated. This requirement helps prevent a majority of vignettes from leaning toward a single answer, which could result in high accuracy simply by answering ‘yes’ or ‘no’ to all the questions. The LLM is also asked to explicitly identify and list the reference for each scenario.

### 3.3 Hallucination Detection

After generating the base vignettes, we identify and exclude those with signs of hallucination. For this purpose, we adopt two LLM evaluation frameworks for hallucination detection. These two general (not specific to hallucination Evals) frameworks include G-EVAL [] and RefChecker [].

Following the G-Eval framework, we initially characterize the requested task and our criteria for measuring the two main types of hallucination, i.e., factuality and faithfulness in generated base vignettes. Using the task description and criteria description, we prompt an auxiliary LLM to generate detailed chain-of-thought evaluation steps, which leads to generating a score ranging from zero to one. We present an example and additional details in Appendix C.

Additionally, we use the RefChecker framework to use an extractor component to generate claim triplets—structured data formats that break down a statement into subject, predicate, and object—from the LLM’s responses, as:

$$ $V_{\text{Base}}\xrightarrow{\text{RefChecker}}\{(sub_{j},pred_{j},obj_{j})\}_{ j=1}^{m}$ $$

Where $m$ is the total number of triplets that have been extracted from the vignettes, subsequently, a checker component evaluates these claim triplets against the specified reference to detect any inconsistencies.

Using the above two procedures, we select a set of base vignettes, shown by $V_{\text{Base}^{\prime}}$, in which no hallucinations were found:

$$ $V_{\text{Base}^{\prime}}=\left\{\cup_{i}\;v_{i}\mid[G{\text{-}}Eval(v_{i})> \tau]\,\land\,[RefCheck(v_{i})=0]\right\},$ $$

where $G{\text{-}}Eval(v_{i})$ is the score generated through the first procedure for vignette $v_{i}$, $\tau$ is a threshold for the generated scores through this procedure, and $RefCheck(v_{i})$ is the number of extracted triplets from $v_{i}$ that contain hallucinations (unverifiable reference).

### 3.4 Outcome Independence Check

Prior to augmenting the sensitive attributes, we check for the relationship between the health outcomes studied in each vignette and the sensitive attributes. As discussed earlier, this step aims to explicitly avoid generating unfeasible (such as puberty complications for older adults) while allowing justified scenarios (like less sensitivity to pulse oximetry for Blacks).

We operationalize this part by using the Unified Medical Language System (UMLS) [], which is a comprehensive system of thesaurus and ontology of biomedical concepts used for clinical data standardization in various health applications. It provides a mapping structure among the standardized vocabularies and thus allows one to translate among the various terminology systems.

We use the UMLS ontology to identify the ancestors of a health outcome as the indicators of the sensitive attributes that interact with the health outcome. More specifically, in our proposed approach, we map the generated vignettes to the UMLS concepts using MetaMap [], which is a tool for recognizing UMLS concepts in the text. Next, we find the ancestors for every concept extracted from the vignettes, going up in the hierarchy. In theory, all ancestors up to the root can be included. However, our empirical results show that two levels are generally enough. From this set of concepts’ ancestors, we especially extract a subset ($\mathcal{S}_{Anc}$) that belongs to the specified sensitive attributes (such as a specific gender or ethnicity).

We consider two scenarios where (a) the extracted subset related to the health outcomes is smaller than all possible values for the specified sensitive attributes, and (b) the subset is the same size or empty. We consider the former scenario (a) as an indicator of a partial relationship and the latter scenario (b) as an indicator of a full relationship between the sensitive attribute and health outcome. For instance, for gestational diabetes, the subset related to gender would only include female (as males cannot have this disease) in an ontology tree (mostly containing “is-a” relationships), and for race, it would include all races.

These two scenarios will determine the sensitive attribute values that are safe to be augmented through red-teaming in the next step of our pipeline $\mathcal{S}_{\text{Red-Team}}$, as shown in:

$$ $\mathcal{S}_{\text{Red-Team}}=\begin{cases}\mathcal{S}_{\text{Ancestors}}, \text{ if, }|\mathcal{S}_{\text{Ancestors}}|<|\mathcal{S}_{\text{Input}}|\\ \mathcal{S}_{\text{Input}},\text{Otherwise}.\end{cases}$ $$

### 3.5 Augmenting the Sensitive Attributes

Following the described red-teaming approach for bias evaluation across various sensitive attribute dimensions, each of the remaining vignettes ($V_{\text{Base}^{\prime}}$) is augmented by the remaining sensitive attribute values that are determined safe in the previous stage ($\mathcal{S}_{\text{Red-Team}}$). Accordingly,

$$ $V_{\text{Red-Team}}=\{\bigcup_{i}\bigcup_{j}\mathcal{M}(v_{i},s_{j})\mid v_{i} \in V_{\text{Base}^{\prime}},\ s_{j}\in\mathcal{S}_{\text{Red-Team}}\}$ $$

would be the set of final red-teaming vignettes that our pipeline generates. We prompt a strong LLM ($\mathcal{M}$) to place the sensitive attribute values in the best location within the input vignette (same location for each single vignette). The two green boxes in Figure 1 show an example where two sensitive attribute values are added to one base vignette, resulting in two red-teaming vignettes. The strong LLM ($\mathcal{M}$) in our pipeline could be the same LLM acting in different roles (an agentic design) or be different LLMs.

## 4 Experiments

We follow a series of focused research questions to examine our vignette generation pipeline. Each question addresses a fundamental component of our study’s approach and especially targets evaluating the pipeline’s ability to address the three primary challenges we highlighted for our problem. Specifically, our research questions aim to evaluate the generated vignettes’ diversity and utility, as well as their potential for hallucination, which are crucial factors for ensuring the effectiveness of our proposed method.

##### Experimental Setup

While our method works with any LLM, we use OpenAI’s GPT-4 [], acting as the strong LLM ($\mathcal{M}$) in our pipeline. We also report on using Anthropic’s Claude Sonnet 3.5 [] in Appendix E.
We use a value of 0.8 for the threshold value ($\tau$) for filtering the hallucinated vignettes.

##### Baselines

We use two baseline methods to compare to our method. First, we use an LLM to generate vignettes, but without providing any evidence from external knowledge bases. Second, we use our method without any further refinement. For both baselines, we use the same prompt design and the same inputs as those we use for our method.

In addition to the methodological baselines, we compare our generated vignettes to those generated in prior work supervised by medical experts. Specifically, we extract a subset of the vignettes from the EquityMedQA dataset [], which is a collection of seven datasets containing both human-authored by 80 medical experts and AI-generated medical queries designed to elicit biased responses from LLMs as a basis for our preference dataset.

##### Clinical Case Studies

In our experiments, we focus on several specific case studies with documented biases in clinical literature. One main case study relates to obesity (prevention and treatment). Obesity affects approximately 16% and 40% of adults globally and in the US, respectively [; ; ; ]. Avoiding bias for LLMs is especially challenging due to the historical misconceptions about an individual’s lack of “willpower” causing obesity []. Stigma and bias patterns are very common, even among clinical providers []. In the following experiments, we use a subset of 243 vignettes generated from 10 articles extracted from PubMed.

In addition to obesity, we also study case studies related to breast cancer, prostate cancer, and pregnancy. We publish all of the generated vignettes, in addition to vignettes related to the popular topic of pain management, in our repository.

### 4.1 Q1: How much the generated vignettes are faithful to the provided medical evidence?

To comprehensively evaluate the faithfulness of the vignettes to the provided context, we use four different methods (BARTScore, Semantic Entropy, G-Eval, and RefChecker) to measure how closely the vignettes mirror the context and content of the provided references
without introducing irrelevant or incorrect details.

BARTScore [] evaluates the likelihood of the generated text being faithful to the reference context based on the BART model []. By computing the log-likelihood of the candidate text given the reference, BARTScore assesses whether the generated vignette is plausible within the context of the reference text, focusing on how well the output aligns with the source information in terms of meaning and structure.

Semantic Entropy (SE) [] is a metric used to measure the diversity and unpredictability of the meanings within the generated text. High SE suggests that the vignette introduces varied or less predictable meanings, which can lead to deviations from the original context. Conversely, lower SE is a sign that the content is more predictable and closely aligned with the provided reference, contributing to higher faithfulness.

We use the G-EVAL framework [] to calculate a faithfulness score (characterized within the prompt) and using the chain-of-though reasoning using a strong LLM (GPT-4).

We also use the RefChecker framework [] to detect the faithfulness hallucinations. Within this framework, a strong LLM first transforms each generated vignette into a statement, which is then evaluated by RefChecker to classify as contradictory, neutral, or entailment based on the reference evidence.

Table 1 shows the results related to these four approaches. Some of the entries related to the LLM scenario are unavailable since they require a reference that is not applicable to the “LLM Only” baseline for vignette generation. The values presented for the three types of statements in the RefChecker represent the average number of entailment, neutral, or contradictory predicates in the vignettes based on the provided evidence. Overall, these results indicate that our pipeline achieved higher scores across the various metrics.

**Table 1: The degree of faithfulness of our proposed method (Ours) vs. the baselines as measured by four different methods. Mean (STD) values are shown.**
|  | BARTScore$\uparrow$ | SE$\downarrow$ | G-Eval$\uparrow$ | RefChecker |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | Contradictory$\downarrow$ | Neutral | Entailment$\uparrow$ |  |  |  |
| LLM Only | -4.58 (0.53) | 0.78 (0.05) | - | - | - | - |
| Ours W/O Hallucination Detection | -3.69 (0.60) | 0.63 (0.12) | 0.78(0.24) | 0.09 (0.47) | 2.79 (1.44) | 0.71 (0.93) |
| Ours | -3.65 (0.57) | 0.62 (0.12) | 0.87 (0.12) | 0.00 (0.00) | 2.72 (1.4) | 0.85 (1.00) |

### 4.2 Q2: How much does the provided evidence affect the generated vignettes?

We evaluate the effect of the provided evidence in terms of vignette ‘diversity’ and ‘domain specificity.’

##### Diversity

By diversity, we refer to the range of distinct expressions and language used across the generated vignettes. High diversity indicates that the model produces a wide variety of wording and concepts, demonstrating creativity and reducing redundancy. This is particularly important in tasks like vignette generation for clinical evaluations, as diverse outputs may better capture the nuances and complexities of different cases. Conversely, low diversity might suggest that the model relies on a limited set of patterns or vocabulary, which could limit the diversity of the generated content.

We evaluate the diversity in terms of the total number of distinct tokens present in each vignette and across all vignettes. Specifically, we tokenize each generated vignette by splitting the text into individual words using whitespace and punctuation as delimiters. We convert all words to lowercase to ensure consistency and remove any punctuation marks. Additionally, we eliminate common stop words—such as ”the,” ”and,” and ”is”—to focus on the meaningful content that contributes to vocabulary diversity. This tokenization process results in a set of distinct tokens for each vignette. Next, we create a set of distinct tokens for each vignette, combine all vignettes, and then count the number of distinct tokens in each set. Table 2 shows the results related to this experiment. The results especially demonstrate that using an LLM alone produces vignettes with a limited vocabulary compared to our method, which also incorporates external knowledge bases to extract relevant evidence.

**Table 2: Observed diversity in tokens and domain specificity of the generated vignettes (as defined in the main text). Mean (STD).**
| Method | Diversity$\uparrow$ | Domain Specificity$\uparrow$ |  |  |
| --- | --- | --- | --- | --- |
| Each Vignette | All Vignettes | Ref. Similarity | Outcome Similarity |  |
| LLM Only | 14.82 (2.02) | 292 | - | 0.42 (0.7) |
| Ours W/O Hallucination Detection | 25.32 (3.79) | 1,140 | 0.62 (0.12) | 0.49 (0.13) |
| Ours | 25.36 (3.73) | 1,264 | 0.64 (0.12) | 0.50 (0.14) |

##### Domain Specificity

We assess the domain specificity of the generated vignettes by analyzing how closely the key concepts and the language used in those align with the key medical concepts related to (a) the provided medical evidence (reference) and (b) the specified health outcome. To do this, we use embeddings from PubMedBert [], which are specifically trained over biomedical and clinical text. We map the generated vignettes, the domain-specific reference, and the specified health outcome into the PubMedBert space. We then compute the semantic similarity between the generated vignettes and the reference, as well as between the generated vignettes and the health outcome. We use cosine similarity to calculate the similarities. Table 2 presents the average number of distinct tokens appearing in the vignettes, along with their similarity to the reference and outcome after embedding.

### 4.3 Q3: How well does the outcome independence checking work?

To validate the effectiveness of our outcome independent checking phase, we selected three case studies related to the conditions or diseases specific to certain demographic groups. Specifically, we applied our method to generate vignettes for breast cancer, prostate cancer, and pregnancy to observe how the vignettes interact with two sensitive attributes (including the biologically infeasible scenarios). The results are presented in Figure 2. The color shade in each square indicates the percentage of vignettes that have been generated for each outcome and sensitive attribute pair. One can observe that our method appropriately identifies biologically feasible and infeasible outcomes in vignettes based on the sensitive attributes. For example, the prostate cancer case, which is biologically exclusive to males, shows no vignettes generated for females, whereas pregnancy, which is exclusive to females, does not appear in the male demographic group.

Figure: Figure 2: Fraction of generated vignettes for each health outcome (rows) that are augmented with a demographic group (columns)
Refer to caption: extracted/5937736/post.png

### 4.4 Q4: How do humans rate the generated vignettes?

We designed a comprehensive survey and asked a group of participants to rate our generated vignettes compared to other baselines. A diverse group of 45 adults with post-graduate degrees and familiarity with the basics of LLMs and our applications participated in our study. Our study was reviewed by a local institutional review board (IRB) panel.

We asked two series of questions in our survey. In the first section, participants were asked to compare five pairs of vignettes randomly drawn from a set of generated vignettes, with one vignette in each pair generated by our pipeline and the other the ‘LLM-only’ baseline. In the second section, they were asked to compare a separate set of five pairs of vignettes, with one generated by our pipeline and the other randomly selected from the EquityMedQA [] curated dataset. Importantly, participants were not informed which vignette came from which source and were given clear instructions on the expected criteria for selecting the preferred vignettes. The vignettes in the first part of the survey were related to our obesity case study, and the vignettes in the second part were related to the breast cancer case study. Appendix D shows additional details our survey design.

Table 3 shows the results of our human ratings. When performing a two-tailed statistical t-test, our method is significantly more preferred over the alternatives (p-value $\leq$ 0.05), indicated by the * symbol.

**Table 3: Summary of User Preferences in Vignette Comparisons. ‘Superior’ is defined as being more suitable for red-teaming evaluation of biases in the corresponding medical context.**
|  | Vignettes from … are superior | Vignettes from … are superior |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Ours | LLM only | Unsure | Ours | EquityMedQA | Unsure |  |
| Fraction | 0.53* | 0.30 | 0.16 | 0.51* | 0.28 | 0.21 |
| Count | 120 | 68 | 37 | 112 | 62 | 46 |

### 4.5 Q5: How can the generated vignettes be utilized for bis evaluation?

**Table 4: Fairness metrics calculated for different LLMs. Numbers are shown in difference (ratio) format.**
| Metric | Gemma | Llama 3 | Mistral | BioMistral | Meditron |
| --- | --- | --- | --- | --- | --- |
| Gender |  |  |  |  |  |
| Demographic Parity | 0.04 (0.93) | 0.01 (0.98) | 0.03 (0.95) | 0.00 (0.99) | 0.03 (0.96) |
| Equal Opportunity | 0.01 (0.99) | 0.02 (0.97) | 0.04 (0.92) | 0.01 (0.99) | 0.00 (1.00) |
|  | Race and Ethnicity |  |  |  |  |
| Demographic Parity | 0.05 (0.92) | 0.02 (0.96) | 0.09 (0.86) | 0.03 (0.96) | 0.06 (0.93) |
| Equal Opportunity | 0.13 (0.80) | 0.01 (0.99) | 0.17 (0.66) | 0.03 (0.97) | 0.00 (1.00) |

Finally, to showcase the potential of the generated vignettes in evaluating Med LLMs, we run an empirical study on five LLMs, including three general and two medical fine-tuned LLMs.

We assess the fairness of these LLMs using the generated vignettes in our obesity case study. We ask each model to answer the same set of vignettes for different demographic groups and measure the fairness metrics. Table 4 shows the results related to these experiments.

## 5 Related Work

The advent of LLMs has made significant transformations across various fields, including healthcare and medicine. General-purpose LLMs, such as Claude [] and Llama [], are trained on medical text (e.g., medical textbooks and blogs) and have shown an impressive potential to be used for various clinical decision support purposes, such as differential diagnosis [] and treatment planning []. Beyond the general-purpose models, there have also been specialized LLMs tailored for medical applications, such as Google’s Med-Gemini [], Palmyra-Med [], and Meditron [].

Evaluating LLM behaviors is challenging and an active area of research. LLM outputs can be evaluated from various perspectives, making this a complex process to ensure their effectiveness and reliability in various applications. LLM evaluation typically involves key metrics and techniques to assess different aspects of LLM performance. Traditional evaluation metrics, such as BLEU [] and ROUGE [], rely on n-gram overlap between model outputs and reference texts to gauge the quality of outputs. However, these metrics have been criticized for their weak correlation with human judgments [], as surface-level matching does not reliably capture the quality of the generated text.

With the advancement of deep learning, model-based metrics like BERTScore [] and BARTScore [] have been introduced and are increasingly used to evaluate various dimensions of text quality. Although these metrics offer improvements over traditional ones, they still do not deliver fully satisfactory performance and have a limited application scope []. Recent methods, such as G-EVAL [] and GPTScore [], increasingly leverage LLMs themselves for evaluation. This marks a significant advancement in the field due to the unprecedented capabilities of newly developed LLMs.

Specifically, evaluating LLMs regarding bias patterns is crucial for their responsible deployment in high-stakes applications like medicine. Existing studies primarily rely on ‘red-teaming’ designs for fairness evaluation of Med LLMs
. From a clinical point of view, the goal of red-teaming is to ensure that clinical LLMs treat all patient groups equitably, providing unbiased support to healthcare providers.

Existing studies utilized red-teaming for fairness evaluation in Med LLMs including specialized datasets like Q-Pain [] and comparative studies against human experts [; ]. To name a few examples, [] proposed a new framework and dataset to assess LLMs’ bias and fairness against human ratings and evaluated Med-PaLM on the proposed dataset. [] evaluated whether GPT-4 encodes racial and gender biases and explored how these biases might affect medical education, diagnosis, treatment planning, and patient assessment. Moreover, [] convened 80 multi-disciplinary experts to evaluate the performance of popular LLMs across multiple real-world medical scenarios. Experts were given a framework comprising four categories (i.e. safety, privacy, hallucinations, and bias) to analyze for inappropriate LLMs-generated responses. Prior empirical findings highlight the potential for biased LLMs to perpetuate stereotypes and lead to inaccurate clinical reasoning [].

Previous work in this area has involved humans who manually generated prompts and clinical scenarios or evaluated the responses. These types of approaches, however, are costly, narrowly focused, and not scalable.

## 6 Discussion

This study presents a new method for generating tailored clinical vignettes based on the facts extracted from external biomedical knowledge bases to evaluate bias patterns in Med LLMs. Our procedure enables scalable fairness assessments of Med LLMs, supporting access to dynamic (user-defined) and contextually relevant benchmarks.

Extraction of the most relevant biomedical information ensures that the vignettes generated reflect current guidelines based on the latest medical evidence. Our experiments indicate that connecting to external knowledge bases can significantly enhance the diversity and reduce the hallucinations of the generated clinical vignettes, as demonstrated by the performance across various approaches and experiments.

Several limitations of our work should be noted in particular. First, reliance on existing biomedical literature and knowledge bases, such as PubMed, may inadvertently introduce new biases present in these sources or reinforce existing biases in LLMs. Additionally, while our method facilitates large-scale benchmark generation, the complexity of certain medical scenarios might be underrepresented due to constraints in the current knowledge extraction methods.

We also note that we view our pipeline as an assistive tool in ‘diagnosing’ undesired biased and unfair patterns in Med LLMs. Having humans (especially medical stakeholders such as providers and patients) is critical to ensure an effective human-in-the-loop process. As a diagnosis tool, our generated scenarios may wrongly indicate biased patterns (false positive) or miss some of those (false negative). Ultimately, the final users can decide how to adjust the model in trading off these two cases.

Nevertheless, our method can enable much larger scale identification of biases in Med LLMs (potential true positives) than current human-crafted datasets. Accordingly, we also view our method as a screening tool that can reduce the search space of Med LLM bias Evals.

While our method represents a crucial step toward the fairness evaluation of Med LLMs, continued efforts to refine the proposed methods are essential. By continuously improving new methodologies for this purpose, the health AI community can ensure that Med LLMs can benefit all patient demographics equitably, ultimately leading to improved health outcomes, reducing disparities in healthcare, and increasing trust in AI-based clinical decision support tools.

#### Acknowledgments

Our study was partially supported by the NIH awards P20GM103446 and U54-GM104941, as well as a computing credit award from Amazon Web Services (AWS).

## Appendix A PubMed Retrieval

PubMed offers a search engine to access a full-text archive of biomedical and life sciences journal literature and is considered as the most comprehensive resource of biomedical evidence. It covers fields such as medicine, nursing, pharmacy, dentistry, veterinary medicine, and health care. We use PubMed because it is a well-known database that researchers and healthcare professionals widely use to find and access scientific content. To access PubMed we utilize NCBI’s Entrez Programming Utilities (E-utilities) API [] which is a suite of eight server-side programs that accept a fixed URL syntax for search, link, and retrieval operations.

## Appendix B Prompt Design

In this section, we present the main prompt we have used for generating vignettes based on the extracted evidence.

## Appendix C Hallucination Detection

In this section, we present a sample of the input and output from G-Eval, demonstrating how it is integrated into our pipeline.

## Appendix D Human rating Procedure

We conducted a study using Qualtrics to compare the usefulness of vignettes generated by our method, a baseline, and the EquityMedQA [] dataset. The following instructions were provided to participants:

## Appendix E Alternative LLM

We present the results for the first and second research questions, utilizing Claude Sonnet 3.5, in Tables 6 and 6.

**Table 5: The degree of faithfulness of our proposed method (Ours) vs. the baselines as measured by four different methods using Claude Sonnet LLM. Mean (STD) values are shown.**
|  | BARTScore$\uparrow$ | SE$\downarrow$ | G-Eval$\uparrow$ | RefChecker |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | Contradictory$\downarrow$ | Neutral | Entailment$\uparrow$ |  |  |  |
| Ours W/O Hallucination Detection | -3.30 (0.56) | 0.59 (0.11) | 0.88 (0.16) | 0.34 (0.70) | 1.44 (1.34) | 1.71 (1.47) |
| Ours | -3.20 (0.58) | 0.59 (0.11) | 0.95 (0.05) | 0.00 (0.00) | 1.36 (1.40) | 2.08 (1.49) |

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: See weaknesses.
soundness: 2
strengths: - The method allows for large-scale evaluation, generating a variety of vignettes quickly without human effort
- Retrieval of relevant literature and entities should help reduce hallucinations
- Additional hallucination reduction using recent techniques (RefChecker and G-Eval) is encouraging
summary: The paper presents a method to evaluate bias in medical LLMs by generating clinical vignettes that force LLMs to respond across demographic groups. (Ideally having outcome independence of different types of demographics.)
Automated vignette generation is performed through information retrieval of biomedical knowledge bases like UMLS and PubMed, and filtered for hallucinations via RefChecker and G-Eval.
weaknesses: - The method may still struggle with the nuanced complexities and biases of clinical cases, e.g. with certain races having a higher prevalence of certain symptoms (E.g. asian flush)
- The method could inadvertently reinforce existing biases present in biomedical literature. Since it relies on pre-existing knowledge bases, any systemic bias in these resources may propagate through the generated vignettes, leading to an inherent limitation in the bias evaluation framework.
- The human evaluation seems weak, with only 110 total counts of the User Preferences in Vignette Comparisons being evaluated. Additionally, it's interesting that LLM only seemingly outperforms / performs around the same as the ground truth from EquityMedQA. Could the authors discuss why this may be the case?

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Clinical Validation and Accuracy-- Could you provide a pilot evaluation of medical accuracy by having a panel of practicing physicians review a representative sample (e.g., 50-100) of generated vignettes across different medical conditions? Please include quantitative analysis of error rates and types of medical inaccuracies identified.

2. Domain Generalization and Scalability-- How well does your method generalize beyond obesity? Could you validate the pipeline on at least 1-2 additional medical conditions with known disparities (e.g., cardiovascular disease, psychiatric conditions) and provide performance metrics for each stage of the pipeline across these different domains?

3. Bias Propagation and Mitigation-- How do you ensure that biases present in PubMed and medical knowledge bases don't get propagated or amplified in the generated vignettes? What specific safeguards or validation steps in your pipeline address this risk? Have you evaluated this component of the retrieval?

4. System Performance and Computational Requirements-- What are the end-to-end computational requirements and processing times for generating validated vignettes? Please provide a detailed breakdown by pipeline stage (retrieval, generation, validation) and discuss how these requirements scale with different medical domains and knowledge base sizes.

5. Medical Knowledge Currency-- Given the rapid evolution of medical knowledge and guidelines, how does your system ensure the generated vignettes reflect current medical best practices? What mechanisms could be implemented to identify and update vignettes when medical knowledge changes automatically?
soundness: 3
strengths: The paper presents a new tool for automatically generating clinical vignettes to evaluate bias in medical LLMs, addressing a key challenge in the field.
The proposed pipeline incorporates multiple components to ensure the generated vignettes are evidence-based, domain-specific, and reduce hallucinations.
The use of biomedical knowledge bases and ontologies helps ground the vignettes in established medical evidence and relationships.
The method includes checks for outcome independence and hallucination detection, improving the quality and reliability of the generated scenarios.
The experimental results demonstrate improvements over baseline methods in terms of faithfulness to medical evidence and diversity of generated content.
The work contributes a new dataset for evaluating fairness in medical LLMs, which could be valuable for future research in this area.
The proposed method enables more scalable evaluation of bias in medical LLMs compared to manual vignette creation, while maintaining quality.
summary: This paper presents a pipeline for automatically generating clinical vignettes to evaluate bias and fairness in medical Large Language Models. The key innovation is a pipeline that combines biomedical knowledge retrieval from PubMed with controlled vignette generation and multiple validation steps to ensure quality. The method specifically addresses three main challenges: domain-specificity of fairness evaluation in medicine, dependencies between health outcomes and sensitive attributes, and hallucination in generation. The authors demonstrate the effectiveness of their approach through comprehensive experiments and provide a new dataset for fairness evaluation in the domain of obesity treatment.
weaknesses: Limited Scope of Case Studies: While obesity is a good starting point, the paper would be stronger with more diverse medical conditions to demonstrate generalizability. The current focus on a single primary case study leaves questions about how well the method extends to other medical domains.

Validation of Medical Accuracy: While the paper uses various computational metrics for evaluation, there's limited validation of the medical accuracy of generated vignettes by practicing clinicians. The human evaluation focuses more on fairness evaluation utility than medical correctness.

Bias intrinsic: There is limited discussion on potential biases introduced by the information retrieval process from PubMed and the PubMed Knowledge Graph as we know there are healthcare disparities in real life and medical knowledge systems.

Retrieval: The method for identifying relevant articles seems simplistic and could benefit from more sophisticated relevance ranking algorithms. Further there are concerns that due to constant shifting medical knowledge that the system could retrieve out of date information in the vignettes.

Scalability Analysis: The computational costs and time requirements of the pipeline aren't thoroughly analyzed, particularly the knowledge retrieval and validation steps which could become bottlenecks at scale.

Minor:
- typo line 448-- Bis --> Bias

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 3
questions: * The most critical issue in my assessment relates to the clarity, motivation for, and validity of the approaches used for constructing the outcome independence check and sets of counterfactual pairs, as discussed in the weaknesses section. Some specific suggestions:
  * Revise section 3.4 for clarity.
  * Consider adding a detailed example of how associations are identified using UMLS.
  * Clarify how counterfactual pairs are intended to be defined in cases with differences in disease incidence, addressing questions raised in the weaknesses section.
* Addressing concerns raised in the weaknesses section related to the empirical evaluation would improve my assessment of the work.
  * Clarify methodology for computing fairness violation rates in section 4.5.
  * Consider expanding the experiments to include a broader range of conditions and sensitive attributes to demonstrate scalability.
* Given the issues raised regarding the comparison to EquityMedQA, consider the following:
  * Refocus the comparison specifically on the counterfactual datasets and methodology for creating counterfactual pairs (e.g., using prompting) from Pfohl et al.
  * Consider expanding the diversity of conditions considered in the case study.
soundness: 2
strengths: * The limited scalability of data for red teaming and adversarial testing is a critical and unsolved problem. The core motivation for the work to generate vignettes for red teaming through grounding in the clinical literature and evidence for the presence of associations and/or disparities across patient groups is a strong one. 
* The approach to grounding and hallucination detection is reasonable and well-executed.
summary: This work proposes a structured approach to generating clinical vignettes to use for red teaming medical large language models. The proposed approach is a pipeline that first extracts relevant articles from the clinical literature, which are then conditioned on to generate vignettes using a prompted LLM. The pipeline includes a hallucination detection procedure that ideally assesses the factuality of claims and faithfulness of the vignette to the source articles. A further step conducts an “outcome independence check”, which is motivated to filter vignettes to the set of outcomes and clinical contexts for which there is no association between the sensitive attribute and the outcome of interest such that the generated answers to pairs of prompts containing different sensitive attribute group identifiers can be compared. Experiments are performed to assess the performance of each of the components of the proposed approach.
weaknesses: * Clarity and validity of methods related to the outcome independence check
  * I found the description of the outcome independence check (section 3.4) difficult to follow, even with personal experience working with UMLS, such that it is difficult to assess whether the proposed approach is reasonable. I believe the ambiguity comes from the language, “we especially extract a subset (S_Anc) that belongs to the specified sensitive attributes (such as specific gender or ethnicity)”. Here, it is not clear what it means for a concept to “belong to” a sensitive attribute. Do UMLS terminologies contain annotations that a condition is associated with particular demographic groups? Alternatively, does UMLS contain annotations in cases where a demographic group is uniquely associated with a particular condition? This latter case does correspond more closely to the provided examples (e.g., prostate cancer, pregnancy, gestational diabetes), but it is not clear. 
  * The work would benefit from a clearer description of the motivation and desiderata for the outcome independence check and for the desired properties of the generated counterfactual pairs. For example, it is not clear how the approach is intended to consider cases where there is some non-trivial population-level association between an outcome and a sensitive attribute (e.g., the condition occurs in all groups, but at different rates). It is also important to note that many of these differences are reflections of health disparities caused by differential exposure to social and structural determinants of health (e.g., differences in healthcare access and quality), with no causal biological basis. In some cases, but not all, allowing for different responses across groups may actually promote equity if it means that the negative consequences of structural inequity may be more readily counteracted. In other cases, it may be that there is some biological basis for disease associations (e.g., skin cancer by skin tone), but all groups There are, of course, deeper normative questions to address regarding each of these points. However, as is, the work leaves the motivation underspecified, and it thus unclear how to evaluate whether the approach addresses the intended goals and requirements.
* The analysis presented in section 4.5 is relatively superficial and difficult to interpret. Some specific issues that would improve the work if addressed:
  * It is not clear what the reference is that the outputs are being compared to. For example, for the comparison of outputs for racial identifiers, are the groups compared pairwise, or is there some comparison to an output that does not include identifiers?
  * The scope of the evaluation is relatively narrow, only testing one condition and a limited set of sensitive attributes. As the motivation for the study was to enable scalable evaluation, some effort to test whether that goal has been achieved would improve the paper.
  * It would be helpful to establish a baseline “self-disagreement” rate, assuming that the model outputs are non-deterministic. For example, if the disagreement rate is 0.15 for Male vignettes, what is the disagreement rate for Male vignettes following repeated calls to the model? 
* Section 4.4: Human evaluation of vignettes
  * The human evaluation study is constructed in such a way that may bias the results towards the proposed approach over EquityMedQA (one of the main comparators). In my view, the main reason for this is because the current study generates questions conforming to a particular clinical vignette structure that does not conform to the intended structure for EquityMedQA questions. The instructions for the human raters (Appendix D) is then written in such a way to consider questions conforming to the structure of the generated as preferred. To be clear, EquityMedQA is a collection of several datasets, each generated through different methodologies, and designed to serve different purposes. Not all of the questions are intended to be well-formed clinical questions (e.g., some questions consumer or patient-oriented, or are intentionally malformed to facilitate adversarial testing). EquityMedQA questions also do not always explicitly contain a sensitive attribute, nor were they necessarily designed to support sensitive attribute augmentation and assessment of counterfactual fairness. These points are raised as properties of ideal vignettes in the rater instructions. Two of the datasets contained within EquityMedQA were actually designed specifically for counterfactual analyses like the one that the present work conducts, and those datasets may be the more relevant ones to compare to (the approach used to create those datasets may also be a relevant baseline for this work as a whole). As a further point, despite blinding, it may not be difficult for the raters to differentiate between the examples in the two datasets given the consistent structure of the proposed dataset (yes/no structure, topical clustering by obesity and and breast cancer).
* Minor issues
  * Line 305: “[...] the EquityMedQA dataset (Pfohl et al., 2024), which is a collection of seven datasets containing both human-authored by 80 medical experts and AI-generated medical queries [...]”. The part of this statement regarding 80 medical experts is not an accurate description of EquityMedQA. Consider rephrasing to simply “human-authored and AI-generated medical queries”.
  * Section 4.5, section title typo: “Bis” -> “Bias”
  * Framing of red-teaming: This work (arguably inaccurately) presents a particular form of counterfactual fairness assessments as the primary form of red teaming. It would be better to frame this form of evaluation as one of many approaches to red teaming, rather than the main one. See this review by Feffer et al for further reading (https://ojs.aaai.org/index.php/AIES/article/view/31647/33814).

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. The motivation behind the obesity and other tasks were not well motivated? A demonstration and wholistic evaluation on many standardized clinical scenarios can be helpful.
2. One of the massive limitations is that PubMed introduces more biases? Authors did not address how to overcome this.
3. Why did you only generate 100 vignettes?
soundness: 2
strengths: - It appears their method is working much better than previous works.
summary: This paper introduces a scalable evaluation protocol for determining biases in medical LLMs. The authors introduce a system for automatically generating clinical scenarios or "vignettes" that test for biased outcomes based on various sensitive attributes. This helps scale bias evaluations, which were previously manual and limited in scope. The system relies on external biomedical knowledge bases like PubMed to ensure that the generated vignettes are grounded in real medical evidence, minimizing hallucinations. The method explicitly ensures that certain medical scenarios that require specific treatments for sensitive attributes (like gender or race) are treated appropriately, without overgeneralizing or underrepresenting critical medical considerations.
weaknesses: - The writing can overall be improved. The ideas presented in this paper are not super new. Several preprints have come out demonstrating a similar type approach. (https://arxiv.org/abs/2409.16430, https://arxiv.org/abs/2404.15149, https://aclanthology.org/2024.bionlp-1.5/)
- structural organization of this paper is poor (related works should be introduced early to help provide the background of the research question being asked.)
- The method is weak. The ideas of retrieval, knowledge graphs integrated with LLMs are not particularly new. This is more of an application paper more than anything.
- In the methods they say they only use GPT4 then in section 4.5 they list a whole bunch of new LLMs for evaluation.
- Figure 2 caption is poor and not really describing what the figure is representing. 
- the Vignette counts in general are tiny. I think it is only 100.
- paper does not have checklist

Meta-review: The paper presents a framework for generating clinical vignettes, motivated by scalable bias evaluations. In several medical settings, the paper examine the quality of these generated vignettes including across faithfulness, domain specificity, and bias under counterfactual fairness

Strengths (based on reviewers' input):
 - Live demo and reproducibility code are provided
 - Inclusion of relevant literature listed as an asset

Weaknesses
 - Vignette generation pipeline does not capture complexities of clinical cases. Additionally, the connection between the medical accuracy of the vignettes is not properly examined.
 - Risk of reinforcing biases in biomedical literature
 - Insufficient human evaluation (low number of samples and no medical panel of evaluators)
 - Lots of reviewer concerns about clarity, including confusing descriptions of the methodology and the experiments

Given the relatively simplistic setup and shallow evaluation protocol, I am recommending reject for this piece. Because the work is promising despite these concerns, I encourage the authors to address the reviewer concerns, and I look forward to seeing this work in a future research venue.

Decision: Reject