RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Motivation
Biomedical QA systems can be beneficial for both healthcare providers and patients for many reasons: 1) With traditional search engines, finding reliable medical information can take time and effort due to the vast amount of unfiltered content available online, while QA systems allow users to quickly find answers; 2) such systems can serve as powerful learning tools for students and residents seeking to deepen their understanding of complex medical topics; 3) in low-resource settings there may be limited access to qualified healthcare professionals, which leads to delayed or incorrect diagnoses that may worsen health outcomes over time. Fortunately, biomedical QA systems can bridge this gap and extend the reach of health services to vulnerable populations worldwide.

However, in order for such systems to be safely deployed, ensuring that they provide fair behavior towards patients is critical. For example, imagine that a White and an African-American patient present themselves with similar symptoms at a hospital and that none of their symptoms indicate a problem related to their ethnicity.
If one was treated with the correct medication while the other received an incorrect one, this would be highly problematic. Thus, it is important to understand if current biomedical QA systems could result in such an outcome.
MedQA-USMLE
The MedQA-USMLE dataset Jin et al. ( 2021 ) is an open-domain QA dataset, which covers three languages: English, traditional Chinese, and simplified Chinese. MedQA has medical questions which represent real-world scenarios and evaluate physicians on their clinical decision making skills. The questions are varied and require a significant understanding of medical concepts. Here,
we choose to only use the English version, which is composed of 12,723 multiple-choice prompts taken from the professional medical board exams. Each prompt consists of context and question , e.g., “An 18-year-old male presents to the emergency room smelling quite heavily of alcohol and is unconscious. A blood test reveals severe hypoglycemic and ketoacidemia. A previous medical history states that he does not have diabetes. The metabolism of ethanol in this patient’s hepatocytes resulted in an increase of the [NADH]/[NAD+] ratio. Which of the following reaction is favored under this condition?” . Each question comes with four answer choices. The options for the above example are: Pyruvate to acetyl-CoA, Citrate to isocitrate, Oxaloacetate to malate , and Oxaloacetate to phosphoenolpyruvate .

M

F

W

A-A

B

H

As

Hetero

Bi

Homo

M+W

M+A-A

M+B

M+H

M+As

F+W

F+A-A

F+B

F+H

F+As

M+Hetero

M+Bi

M+Homo

F+Hetero

F+Bi

F+Homo

M

F

W

A-A

B

H

As

Hetero

Bi

Homo

M

F

W

A-A

B

H

As

Hetero

Bi

Homo

M+W

M+A-A

M+B

M+H

M+As

F+W

F+A-A

F+B

F+H

F+As

M+Hetero

M+Bi

M+Homo

F+Hetero

F+Bi

F+Homo
Question Selection
Some
phenomena are more prevalent in certain populations, such as pregnancy Sogancioglu et al. ( 2022 ) or prostate cancer. For other diagnoses, patient demographic information is irrelevant and should accordingly not be taken into account. For our experiments we build a dataset consisting of only questions whose answers do not depend on sex, ethnicity, or sexual orientation. We do so by following Logé et al. ( 2021 ) ’s approach and extract 100 vignettes, which are designed to allow for the inclusion of diverse ethnics and gender “profiles” in order to assess potential biases.
These vignettes are verified by two medical experts to be demographics-independent, and after the
demographics-enhancing process, which will be discussed in the next section, result in 16,700 questions overall , which are used to evaluate the effect irrelevant demographic information has on QA systems.
Demographics-enhanced Dataset Creation
We experiment with the following types of modified questions: dimensionless (i.e., no demographic information), ethnicity, gender, names, sexual orientation, gender+ethnicity, gender+sexual orientation, and gender+ethnicity+names.

The reasoning for each chosen dimension are as follows: dimensionless shows no demographic information, and hence will be used as a baseline to compare how many of the answers change when we add irrelevant demographic information. Ethnicity, sexual orientation, and gender, while not always shown in medical text, are sometimes mentioned when the demographic information is relevant. Hence, we want to see if the models associate any medical conditions with them. We use two genders, but expect that our results will generalize to additional genders.
As for names, these are clearly not medically relevant ever and are rarely shown in medical text. Hence, we choose them to see if there are unexpected differences in answers change.

Ethnicities include White, Black, African-American, Hispanic, and Asian. Genders include male and female. Names include the 10 names for each ethnicity from the Q-Pain dataset, which originated from the Harvard Dataverse’s Demographic aspects of first names dataset Tzioumis ( 2018 ) . And while “Black” and “African American” are largely synonymous, we want to see if they are different from the models’ perspective. Notably, to medically-untrained users, all of these may seem relevant and hence potentially be added to queries when such users request medical assistance.

We follow a similar process as the creators of the Q-Pain dataset and make each context, question, and answer (CQA) as neutral as possible. Given a CQA, such as “A 23-year-old female presents to a psychiatrist…”, we first automatically mask any word that indicates gender (e.g., male, female, he, she, wife, boyfriend): “A 23-year-old [GENDER_MASK] presents to a psychiatrist…”.
Then, given a dimension (e.g., gender), we automatically replace each unique masking with their corresponding token replacement (e.g., replacing “[GENDER_MASK]” with “male”).

Overall, each of these dimensions and their variations augment each of the 100 vignettes and result in overall 16,700 questions. See Table 1 for examples. And while we only use the English version of the dataset,
this process can be easily applied to other languages. The data will be publicly available and have an MIT License.

W

A-A/B

H

As

M+W

M+A-A

M+B

M+H

M+As

F+W

F+A-A

F+B

F+H

F+As

M

F

W

A-A

B

H

As

Hetero

Bi

Homo

M+W

M+A-A

M+B

M+H

M+As

F+W

F+A-A

F+B

F+H

F+As

M+Hetero

M+Bi

M+Homo

F+Hetero

F+Bi

F+Homo
Random Change
We use a version of the questions with no demographic information, and, in each prompt’s first sentence, replace the word “patient” with “person”. With this we examine the effect of a small but insignificant textual variation on each model. We choose this change over others (e.g., adding random words, irrelevant demographics, or fictitious cities) as this reduces the possibility of models changing their answers due to the context such random words had in the training data (e.g., Africa is more prevalent to the sleeping sickness disease than the US). Moreover, neither “person” nor “patient” reveal information about the human.
Models
We compare two existing algorithms: QAGNN Yasunaga et al. ( 2021 ) and BioLinkBert Yasunaga et al. ( 2022 ) .
While better models exist for the USMLE dataset, many of them have billions of parameters and we are unable to test them for computational reasons. That being said, BioLinkBert is currently among the state of the art on the USMLE dataset, and QAGNN is the top (and, to the best of our knowledge, only) KG-grounded model. We use existing implementations and models and describe both systems in the following.

M

F

W

A-A

B

H

As

Hetero

Bi

Homo

M+W

M+A-A

M+B

M+H

M+As

F+W

F+A-A

F+B

F+H

F+As

M+Hetero

M+Bi

M+Homo

F+Hetero

F+Bi

F+Homo
QAGNN
The main component of QAGNN is its KG, which is based on the Disease Database portion of the Unified Medical Language System (UMLS) and DrugBank. The graph contains about 10k nodes and 44k edges, where the embeddings for each node are initialized using the biomedically trained language model SapBERT Liu et al. ( 2020 ) . SapBERT was trained using the UMLS vocabulary set 2020AA version, which contains biomedical synonyms
from more than 150 controlled vocabularies, such as Gene Ontology and MeSH. QAGNN has 360M parameters.

For each answer choice of a given question, QAGNN first retrieves a subgraph from its KG using entity linking. That is, it finds entity mentions in the question and retrieves any entity in the main KG that appears in any 2-hop paths between pairs of found entities. Then, it concatenates the answer choice and question, followed by encoding using a LM. Next, it connects the encoded representation to the graph as a node. It then performs relevance scoring on each node in the created subgraph by concatenating it to the encoded representation node and calculating the likelihood using a LM. Lastly, using an attention-based graph neural network (GNN) module, it reasons over the graph to get a score for the answer choice. During the training procedure, it optimizes both the LM and its GNN end-to-end using cross-entropy loss. On the MedQA-USMLE dataset, SapBERT-based QAGNN achieves 38% accuracy.
BioLinkBert
The defining features of BioLinkBert are its pretraining method that incorporates document links and its LM which has similar hyperparameters to PubmedBERT Gu et al. ( 2020 ) and is trained from scratch on the PubMed abstracts PubmedBERT is trained on. BioLinkBert has 340M parameters.

Given a corpus of text, BioLinkBert views it as a graph: it uses Pubmed Parser to extract citation links between documents and views the hyperlinks as edges.
Then, to use the links in its LM pretraining procedure it places two documents which share a link in the same context, in addition to placing two random documents in the same context or a single document (contiguous). Next, it uses two self-supervised objectives. The first, masked language modeling, is common in many of the large LMs such as BERT Devlin et al. ( 2019 ) . In the second, document relation prediction, it classifies the link between the two documents as random, linked, or contiguous. On the MedQA-USMLE dataset, the base version of BioLinkBert achieves 40% accuracy while the large version achieves 44.6%. Here, we work with the base version because of its lower compute requirements.
Results
We look at two different effects of providing the model with irrelevant demographic information: 1) the percentage of questions for each model that change and 2) the accuracy change for each model. Note that these are not necessarily correlated: for example, accuracy does not change when initially incorrect answers change to other incorrect answers, or if the same numbers of answers change from incorrect to correct. It is also worth mentioning that any change in model’s answers is problematic, as these questions were verified to be independent of demographics.
Changed Answers
Table 2 shows the percentage of questions for each model that change between each dimension’s attribute and the dimensionless variation (e.g., between male and genderless).

The first column of Table 2 , “Random”, shows the result of our random change (Sec. 3.5 ).
While the other values in the table are larger, and while the words “patient” and “person” may have different connotations for each model based on its training data, this suggests that, to some extent, random noise plays a role in the amount of change each model exhibits. Notably for gender, ethnicity, and sexual orientation, both models change around the same number of answers, except that BioLinkBert has a much higher number for Asian. Additionally, both models have almost double the amount of changed answers for homosexual than bisexual or heterosexual. For gender+ethnicity, QAGNN has an equivalent amount or more than BioLinkBert, though for gender+sexual orientation, BioLinkBert has more than double the amount for homosexuals, with a massive percentage of 23. We also examine the amount of answers for each model for gender, ethnicity, and sexual orientation, that change from being correct to incorrect, from incorrect to correct, and from incorrect to incorrect (Table 3 ). We can see that a model can have an increase in performance (see QAGNN males column which results in a 2% increase) while having the same number of answers change as a demographics which result in a decrease in performance (see QAGNN White column which result in a 4% decrease). This implies that accuracy alone is not sufficient to understand the effect irrelevant demographic information has on models’ answer, and that further examination of the answers can contribute. For example, we see that adding most ethnicities results in 0 answers changing from incorrect to correct for QAGNN.
Changed Accuracy
While the reported accuracy on the original test dataset is 38% for QAGNN and 40% for BioLinkBert, the accuracy on our 100 randomly selected demographic-independent questions use to construct the vignettes is 40% for QAGNN and 39% for BioLinkBert.
Table 4 shows our accuracy results for each dimension for each algorithm.

As noted, accuracy change does not always correlate with answer change. For example from Table 2 , while both models have about the same number of changed answers for gender, only QAGNN’s accuracy for males is affected (increased by 2%). For ethnicity, both models’ accuracy drops, with BioLinkBert’s accuracy by 3% for Asian and QAGNN’s accuracy by 4% for Black. Sexual orientation improves BioLinkBert performance on bisexual and decreases QAGNN’s on every variation. Gender+ethnicity decreases QAGNN performance the most (up to 6%), while gender+sexual orientation improves BioLinkBert’s performance on any variation except for homosexual.
Analysis: Names
Similarly to the above experiments, we also evaluate the effect names have on the two types of models. For names by themselves, for each ethnicity (Black, White, Hispanic, Asian) we use the corresponding 20 names (10 for males and 10 for females). For names+ethnicity+gender, we split the names into their ethnicity and gender.

Table 5 and 6 show our results: Tables 5 displays the number of changed answers, while Table 6 shows accuracy changes.
We can see that names alone have a moderate effect on the performance of both models, decreasing the performance in any variation by up to 1.65%. From our baseline experiment this may be due to random noise. However, by looking at the number of changed answers, we can see that both models have the most change for Hispanics, with QAGNN change of up to 12.6% and BioLinkBert by up to 8.5%. Interestingly, QAGNN has the same number of changed answers for White, Black, and Asian, but a different number for Hispanic. More results can be seen in the combination of gender, ethnicity, and names, in which the performance can decrease by up to 3.9% for BioLinkBert in African American males, and by up to 2.1% for QAGNN in Asian females. However, the amount of changed answers is up to 15% in QAGNN for African American females and up to 11.9% for BioLinkBert in African American males. This implies that even though both models were trained on PubMed data, irrelevant information like names affect them, which is highly problematic.
Medical vs. Generic LMs
In addition to our main results, we also compare how the performance of a biomedically-trained transformer differs from that of a generic one. In particular, we use the same code for the BioLinkBert QA system, but instead of using the medically-trained base (trained from scratch on PubMed abstracts), we use a transformer which is trained on generic English text.

Similar to our analysis between QAGNN and BioLinkBert above, our analysis between the biomedical and generic models can be split into the amount of answers and accuracy that changes when the dimensions change. From Table 7 it is visible that the generic transformer has more than double the amount of answers change for each gender. It also has an equivalent amount or more for almost any ethnicity, except for Asians. Notably, for sexual orientation, the generic transformer has almost double the amount of answers change for bisexuals, while the biomedical transformer has more for homosexuals. The generic transformer has significantly larger values than the biomedical transformer in any gender+ethnicity combination, while for gender+sexual orientation, the biomedical system has significantly larger values for homosexuals. From Table 8 it is clear that BioLinkBert significantly outperforms its generic LM variation. From the change in accuracy we can see that, while the biomedical transformer’s accuracy increases when gender is removed (“no info”), the generic transformer’s accuracy decreases. We can also see that the biomedical transformer’s accuracy changes more for ethnicity and sexual orientation, while the generic model changes more for gender.
Future Work
Finally, we discuss three potential approaches to alleviate the aforementioned effects, including model architectures, data, and regularization.
Model Architecture
While both the KG-grounded LM and the text-based one are susceptible to irrelevant demographic information, our initial assumption that the KG-based LM would be less susceptible still holds. In particular, KGs are a condensed representation of knowledge, which rarely holds such irrelevant information. Hence, models that use such representations have a significant potential to be less affected. That being said, a potential reason that the tested KG-based LM is still susceptible may be due to the fact that it grounds the text using the KG, and does not only uses the KG. Hence, the demographically irrelevant information may still leak into the final representation, which the model uses to answer the question.
Data
Generally, large LMs are trained using a massive corpus. This is problematic as it is almost impossible to ensure that every piece of data is demographically independent. To try to alleviate this issue, we select biomedical models that are trained only on biomedical data, which often does not contain demographically irrelevant information. However, we still find that these models are susceptible to such information. Hence, future work should examine methods to reduce such issues in the training data, especially for models intended to critical settings.
Regularization
While developing models with different architectures or ensuring that every piece of data is demographically independent is time consuming, a potentially simple method to alleviate such problem is to regularize the input itself. For example, by masking demographically-significant words. And while relatively simple to implement (e.g., using keywords search), in medicine it is sometimes the case where such demographically-significant words are in fact significant. Hence, simple masking might reduce bias, but will also reduce performance. Future work should examine potential masking approaches that consider times where such words are actually needed.
Conclusion
We examine the effect of irrelevant demographic information on purely text-based and KG-grounded biomedical QA systems as well as a generic QA system, using a subset of the USMLE questions whose answers do not depend on the patient’s demographics. Our results show that irrelevant demographic information results in changed answers for all systems. We also find that, while all systems are affected by irrelevant demographic information, they differ with regards to how different types of demographic information influence them. These results provide evidence that more work is needed in order to ensure fair treatment of all patients by biomedical QA systems.
Limitations
As expert annotation is expensive, we only use 100 unique vignettes to create the 16,700 questions. However, this is almost twice as many as other published datasets, such as Logé et al. ( 2021 ) . Additionally, we only analyze one KG-grounded and one purely text-based system. While our main point, that there are problems one should be aware of, can be made based on experiments with two models, evaluating more systems can potentially lead to more fine-grained insights.
Ethics Statement
The main reason for this paper is to point out potential problems regarding fair treatment of all patients by biomedical QA systems. Future work should improve existing biomedical QA systems to ensure equal and just patient care. Moreover, such systems can be problematic for both patients and health experts. For example, a patient could follow the recommendations of such a QA model at home without expert supervision and a system could recommend an incorrect treatment because of their name, or physicians could use such systems to improve their quality of care, but the system could cloud their judgment and direct them to an incorrect answer.
Acknowledgments
We thank Dr. Peter Pressman for his help with reviewing the data and the reviewers for their feedback. The authors acknowledge financial support from NIH grants OT2TR003422 and R01LM013400.
Figure 1: An undesired behavior from a biomedical QA system: the model changes its answers when provided with different biomedically irrelevant information (e.g., names).; An undesired behavior from a biomedical QA system: the model changes its answers when provided with different biomedically irrelevant information (e.g., names).
Dimensions example. Given a question, for each dimension, we demographically-enhance the question by adding relevant words (e.g., Black, bisexual, named X) and changing its gender tokens in order to create multiple datasets for the specific dimension. SOr=sexual orientation.
Percentage of questions with changed answers as compared to a question with no demographic information about the patient. M =male; F =female; W =White; B =Black; A-A =African-American; H =Hispanic; As =Asian; SOr =sexual orientation; Random=Random change as described in Section 3.5 .
Percentage of answers that changed from from correct to incorrect, incorrect to incorrect, and incorrect to correct for each model. M =male; F =female; W =White; B =Black; A-A =African-American; H =Hispanic; As =Asian; SOr =sexual orientation.
Accuracy (in percentages) of the two models on our demographically enhanced datasets. M =male; F =female; W =White; B =Black; A-A =African-American; H =Hispanic; As =Asian; SOr =sexual orientation; O*=original test dataset; O=the original, unmodified 100 vignettes; D=No demographic information; Gen=Gender; 1=QAGNN; 2=BioLinkBERT.
Percentage of questions with changed answers as compared to a question with no demographic information about the patient. M =male; F =female; W =White; B =Black; A-A =African-American; H =Hispanic; As =Asian; SOr =sexual orientation.
Accuracy when including names (rows 1 and 2) or names together with gender and ethnicity information (rows 3 and 4) for each model. W =White; B =Black; A-A =African-American; H =Hispanic; As =Asian;
Percentage of questions with changed answers between the biomedical and generic model as compared to a question with no demographic information about the patient. M =male; F =female; W =White; B =Black; A-A =African-American; H =Hispanic; As =Asian; SOr =sexual orientation.
Accuracy (in percentages) of the biomedical and generic models on our demographically enhanced datasets. M =male; F =female; W =White; B =Black; A-A =African-American; H =Hispanic; As =Asian; SOr =sexual orientation; O*=original test dataset; O=the original, unmodified 100 questions; D=No demographic information; 1=Generic; 2=Biomedical.