Enabling Scalable Evaluation of Bias Patterns in Medical LLMs

Authors: Hamed Fayyaz, Raphael Poulain, Rahmatollah Beheshti
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Medical LLMs, Fairness Evaluation

Abstract:
Large language models (LLMs) have shown impressive potential in helping with numerous medical challenges. Deploying LLMs in high-stakes applications such as medicine, however, brings in many concerns. One major area of concern relates to biased behaviors of LLMs in medical applications, leading to unfair treatment of individuals. To pave the way for the responsible and impactful deployment of Med LLMs, rigorous evaluation is a key prerequisite. Due to the huge complexity and variability of different medical scenarios, existing work in this domain has primarily relied on using manually crafted datasets for bias evaluation. In this study, we present a new method to scale up such bias evaluations by automatically generating test cases based on rigorous medical evidence. We specifically target the challenges of domain-specificity of bias characterization, hallucinating while generating the test cases, and various dependencies between the health outcomes and sensitive attributes. To that end, we offer new methods to address these challenges integrated with our generative pipeline. Specifically, we use medical knowledge graphs and medical ontologies; and customize general LLM evaluation frameworks in our method. Through a series of extensive experiments, we show that the test cases generated by our proposed method are reliable and can effectively reveal bias patterns in LLMs. Additionally,  we publish a large bias evaluation dataset, which provides a comprehensive platform for testing and improving the fairness of clinical LLMs. A live demo of our application for vignette generation is available at https://vignette.streamlit.app. Our code is also available at https://anonymous.4open.science/r/vignette_llm-2853.

Decision: reject
Reviewer scores: 5, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1729636881448

thread_type:
official_review

<thread_object>
creator: reviewer_6586

<content>
summary:
This paper introduces a scalable evaluation protocol for determining biases in medical LLMs. The authors introduce a system for automatically generating clinical scenarios or "vignettes" that test for biased outcomes based on various sensitive attributes. This helps scale bias evaluations, which were previously manual and limited in scope. The system relies on external biomedical knowledge bases like PubMed to ensure that the generated vignettes are grounded in real medical evidence, minimizing hallucinations. The method explicitly ensures that certain medical scenarios that require specific treatments for sensitive attributes (like gender or race) are treated appropriately, without overgeneralizing or underrepresenting critical medical considerations.

strengths:
- It appears their method is working much better than previous works.

weaknesses:
- The writing can overall be improved. The ideas presented in this paper are not super new. Several preprints have come out demonstrating a similar type approach. (https://arxiv.org/abs/2409.16430, https://arxiv.org/abs/2404.15149, https://aclanthology.org/2024.bionlp-1.5/)
- structural organization of this paper is poor (related works should be introduced early to help provide the background of the research question being asked.)
- The method is weak. The ideas of retrieval, knowledge graphs integrated with LLMs are not particularly new. This is more of an application paper more than anything.
- In the methods they say they only use GPT4 then in section 4.5 they list a whole bunch of new LLMs for evaluation.
- Figure 2 caption is poor and not really describing what the figure is representing. 
- the Vignette counts in general are tiny. I think it is only 100.
- paper does not have checklist

questions:
1. The motivation behind the obesity and other tasks were not well motivated? A demonstration and wholistic evaluation on many standardized clinical scenarios can be helpful.
2. One of the massive limitations is that PubMed introduces more biases? Authors did not address how to overcome this.
3. Why did you only generate 100 vignettes?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730408003495

thread_type:
official_review

<thread_object>
creator: reviewer_smbu

<content>
summary:
This work proposes a structured approach to generating clinical vignettes to use for red teaming medical large language models. The proposed approach is a pipeline that first extracts relevant articles from the clinical literature, which are then conditioned on to generate vignettes using a prompted LLM. The pipeline includes a hallucination detection procedure that ideally assesses the factuality of claims and faithfulness of the vignette to the source articles. A further step conducts an “outcome independence check”, which is motivated to filter vignettes to the set of outcomes and clinical contexts for which there is no association between the sensitive attribute and the outcome of interest such that the generated answers to pairs of prompts containing different sensitive attribute group identifiers can be compared. Experiments are performed to assess the performance of each of the components of the proposed approach.

strengths:
* The limited scalability of data for red teaming and adversarial testing is a critical and unsolved problem. The core motivation for the work to generate vignettes for red teaming through grounding in the clinical literature and evidence for the presence of associations and/or disparities across patient groups is a strong one. 
* The approach to grounding and hallucination detection is reasonable and well-executed.

weaknesses:
* Clarity and validity of methods related to the outcome independence check
  * I found the description of the outcome independence check (section 3.4) difficult to follow, even with personal experience working with UMLS, such that it is difficult to assess whether the proposed approach is reasonable. I believe the ambiguity comes from the language, “we especially extract a subset (S_Anc) that belongs to the specified sensitive attributes (such as specific gender or ethnicity)”. Here, it is not clear what it means for a concept to “belong to” a sensitive attribute. Do UMLS terminologies contain annotations that a condition is associated with particular demographic groups? Alternatively, does UMLS contain annotations in cases where a demographic group is uniquely associated with a particular condition? This latter case does correspond more closely to the provided examples (e.g., prostate cancer, pregnancy, gestational diabetes), but it is not clear. 
  * The work would benefit from a clearer description of the motivation and desiderata for the outcome independence check and for the desired properties of the generated counterfactual pairs. For example, it is not clear how the approach is intended to consider cases where there is some non-trivial population-level association between an outcome and a sensitive attribute (e.g., the condition occurs in all groups, but at different rates). It is also important to note that many of these differences are reflections of health disparities caused by differential exposure to social and structural determinants of health (e.g., differences in healthcare access and quality), with no causal biological basis. In some cases, but not all, allowing for different responses across groups may actually promote equity if it means that the negative consequences of structural inequity may be more readily counteracted. In other cases, it may be that there is some biological basis for disease associations (e.g., skin cancer by skin tone), but all groups There are, of course, deeper normative questions to address regarding each of these points. However, as is, the work leaves the motivation underspecified, and it thus unclear how to evaluate whether the approach addresses the intended goals and requirements.
* The analysis presented in section 4.5 is relatively superficial and difficult to interpret. Some specific issues that would improve the work if addressed:
  * It is not clear what the reference is that the outputs are being compared to. For example, for the comparison of outputs for racial identifiers, are the groups compared pairwise, or is there some comparison to an output that does not include identifiers?
  * The scope of the evaluation is relatively narrow, only testing one condition and a limited set of sensitive attributes. As the motivation for the study was to enable scalable evaluation, some effort to test whether that goal has been achieved would improve the paper.
  * It would be helpful to establish a baseline “self-disagreement” rate, assuming that the model outputs are non-deterministic. For example, if the disagreement rate is 0.15 for Male vignettes, what is the disagreement rate for Male vignettes following repeated calls to the model? 
* Section 4.4: Human evaluation of vignettes
  * The human evaluation study is constructed in such a way that may bias the results towards the proposed approach over EquityMedQA (one of the main comparators). In my view, the main reason for this is because the current study generates questions conforming to a particular clinical vignette structure that does not conform to the intended structure for EquityMedQA questions. The instructions for the human raters (Appendix D) is then written in such a way to consider questions conforming to the structure of the generated as preferred. To be clear, EquityMedQA is a collection of several datasets, each generated through different methodologies, and designed to serve different purposes. Not all of the questions are intended to be well-formed clinical questions (e.g., some questions consumer or patient-oriented, or are intentionally malformed to facilitate adversarial testing). EquityMedQA questions also do not always explicitly contain a sensitive attribute, nor were they necessarily designed to support sensitive attribute augmentation and assessment of counterfactual fairness. These points are raised as properties of ideal vignettes in the rater instructions. Two of the datasets contained within EquityMedQA were actually designed specifically for counterfactual analyses like the one that the present work conducts, and those datasets may be the more relevant ones to compare to (the approach used to create those datasets may also be a relevant baseline for this work as a whole). As a further point, despite blinding, it may not be difficult for the raters to differentiate between the examples in the two datasets given the consistent structure of the proposed dataset (yes/no structure, topical clustering by obesity and and breast cancer).
* Minor issues
  * Line 305: “[...] the EquityMedQA dataset (Pfohl et al., 2024), which is a collection of seven datasets containing both human-authored by 80 medical experts and AI-generated medical queries [...]”. The part of this statement regarding 80 medical experts is not an accurate description of EquityMedQA. Consider rephrasing to simply “human-authored and AI-generated medical queries”.
  * Section 4.5, section title typo: “Bis” -> “Bias”
  * Framing of red-teaming: This work (arguably inaccurately) presents a particular form of counterfactual fairness assessments as the primary form of red teaming. It would be better to frame this form of evaluation as one of many approaches to red teaming, rather than the main one. See this review by Feffer et al for further reading (https://ojs.aaai.org/index.php/AIES/article/view/31647/33814).

questions:
* The most critical issue in my assessment relates to the clarity, motivation for, and validity of the approaches used for constructing the outcome independence check and sets of counterfactual pairs, as discussed in the weaknesses section. Some specific suggestions:
  * Revise section 3.4 for clarity.
  * Consider adding a detailed example of how associations are identified using UMLS.
  * Clarify how counterfactual pairs are intended to be defined in cases with differences in disease incidence, addressing questions raised in the weaknesses section.
* Addressing concerns raised in the weaknesses section related to the empirical evaluation would improve my assessment of the work.
  * Clarify methodology for computing fairness violation rates in section 4.5.
  * Consider expanding the experiments to include a broader range of conditions and sensitive attributes to demonstrate scalability.
* Given the issues raised regarding the comparison to EquityMedQA, consider the following:
  * Refocus the comparison specifically on the counterfactual datasets and methodology for creating counterfactual pairs (e.g., using prompting) from Pfohl et al.
  * Consider expanding the diversity of conditions considered in the case study.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730673907171

thread_type:
official_review

<thread_object>
creator: reviewer_sxia

<content>
summary:
This paper presents a pipeline for automatically generating clinical vignettes to evaluate bias and fairness in medical Large Language Models. The key innovation is a pipeline that combines biomedical knowledge retrieval from PubMed with controlled vignette generation and multiple validation steps to ensure quality. The method specifically addresses three main challenges: domain-specificity of fairness evaluation in medicine, dependencies between health outcomes and sensitive attributes, and hallucination in generation. The authors demonstrate the effectiveness of their approach through comprehensive experiments and provide a new dataset for fairness evaluation in the domain of obesity treatment.

strengths:
The paper presents a new tool for automatically generating clinical vignettes to evaluate bias in medical LLMs, addressing a key challenge in the field.
The proposed pipeline incorporates multiple components to ensure the generated vignettes are evidence-based, domain-specific, and reduce hallucinations.
The use of biomedical knowledge bases and ontologies helps ground the vignettes in established medical evidence and relationships.
The method includes checks for outcome independence and hallucination detection, improving the quality and reliability of the generated scenarios.
The experimental results demonstrate improvements over baseline methods in terms of faithfulness to medical evidence and diversity of generated content.
The work contributes a new dataset for evaluating fairness in medical LLMs, which could be valuable for future research in this area.
The proposed method enables more scalable evaluation of bias in medical LLMs compared to manual vignette creation, while maintaining quality.

weaknesses:
Limited Scope of Case Studies: While obesity is a good starting point, the paper would be stronger with more diverse medical conditions to demonstrate generalizability. The current focus on a single primary case study leaves questions about how well the method extends to other medical domains.

Validation of Medical Accuracy: While the paper uses various computational metrics for evaluation, there's limited validation of the medical accuracy of generated vignettes by practicing clinicians. The human evaluation focuses more on fairness evaluation utility than medical correctness.

Bias intrinsic: There is limited discussion on potential biases introduced by the information retrieval process from PubMed and the PubMed Knowledge Graph as we know there are healthcare disparities in real life and medical knowledge systems.

Retrieval: The method for identifying relevant articles seems simplistic and could benefit from more sophisticated relevance ranking algorithms. Further there are concerns that due to constant shifting medical knowledge that the system could retrieve out of date information in the vignettes.

Scalability Analysis: The computational costs and time requirements of the pipeline aren't thoroughly analyzed, particularly the knowledge retrieval and validation steps which could become bottlenecks at scale.

Minor:
- typo line 448-- Bis --> Bias

questions:
1. Clinical Validation and Accuracy-- Could you provide a pilot evaluation of medical accuracy by having a panel of practicing physicians review a representative sample (e.g., 50-100) of generated vignettes across different medical conditions? Please include quantitative analysis of error rates and types of medical inaccuracies identified.

2. Domain Generalization and Scalability-- How well does your method generalize beyond obesity? Could you validate the pipeline on at least 1-2 additional medical conditions with known disparities (e.g., cardiovascular disease, psychiatric conditions) and provide performance metrics for each stage of the pipeline across these different domains?

3. Bias Propagation and Mitigation-- How do you ensure that biases present in PubMed and medical knowledge bases don't get propagated or amplified in the generated vignettes? What specific safeguards or validation steps in your pipeline address this risk? Have you evaluated this component of the retrieval?

4. System Performance and Computational Requirements-- What are the end-to-end computational requirements and processing times for generating validated vignettes? Please provide a detailed breakdown by pipeline stage (retrieval, generation, validation) and discuss how these requirements scale with different medical domains and knowledge base sizes.

5. Medical Knowledge Currency-- Given the rapid evolution of medical knowledge and guidelines, how does your system ensure the generated vignettes reflect current medical best practices? What mechanisms could be implemented to identify and update vignettes when medical knowledge changes automatically?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710554983

thread_type:
official_review

<thread_object>
creator: reviewer_jmd5

<content>
summary:
The paper presents a method to evaluate bias in medical LLMs by generating clinical vignettes that force LLMs to respond across demographic groups. (Ideally having outcome independence of different types of demographics.)
Automated vignette generation is performed through information retrieval of biomedical knowledge bases like UMLS and PubMed, and filtered for hallucinations via RefChecker and G-Eval.

strengths:
- The method allows for large-scale evaluation, generating a variety of vignettes quickly without human effort
- Retrieval of relevant literature and entities should help reduce hallucinations
- Additional hallucination reduction using recent techniques (RefChecker and G-Eval) is encouraging

weaknesses:
- The method may still struggle with the nuanced complexities and biases of clinical cases, e.g. with certain races having a higher prevalence of certain symptoms (E.g. asian flush)
- The method could inadvertently reinforce existing biases present in biomedical literature. Since it relies on pre-existing knowledge bases, any systemic bias in these resources may propagate through the generated vignettes, leading to an inherent limitation in the bias evaluation framework.
- The human evaluation seems weak, with only 110 total counts of the User Preferences in Vignette Comparisons being evaluated. Additionally, it's interesting that LLM only seemingly outperforms / performs around the same as the ground truth from EquityMedQA. Could the authors discuss why this may be the case?

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732775711841

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their time and for sharing their suggestions and constructive feedback. We provide point-by-point responses to the individual concerns below.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a framework for generating clinical vignettes, motivated by scalable bias evaluations. In several medical settings, the paper examine the quality of these generated vignettes including across faithfulness, domain specificity, and bias under counterfactual fairness

Strengths (based on reviewers' input):
 - Live demo and reproducibility code are provided
 - Inclusion of relevant literature listed as an asset

Weaknesses
 - Vignette generation pipeline does not capture complexities of clinical cases. Additionally, the connection between the medical accuracy of the vignettes is not properly examined.
 - Risk of reinforcing biases in biomedical literature
 - Insufficient human evaluation (low number of samples and no medical panel of evaluators)
 - Lots of reviewer concerns about clarity, including confusing descriptions of the methodology and the experiments

Given the relatively simplistic setup and shallow evaluation protocol, I am recommending reject for this piece. Because the work is promising despite these concerns, I encourage the authors to address the reviewer concerns, and I look forward to seeing this work in a future research venue.