Introduction
Natural language processing (NLP) has long been used in health care and life sciences. However, NLP systems exhibit surprising behaviors that can be difficult to predict or control: problems with general-purpose NLP systems reflecting stereotyping and stigmatizing biases have been apparent since the Microsoft Taybot debacle in 2016 and remain a major issue to this day Nadeem et al. ( 2021 ); Rudinger et al. ( 2017 ); Blodgett et al. ( 2020 ); Savoldi et al. ( 2021 ); Zarrieß et al. ( 2022 ) .

The World Health Organization states that social determinants of health, including the experience of racism, sexism, and other forms of discrimination, “can be more important than health care or lifestyle choices in influencing health.” ♠♠\spadesuit ♠♠\spadesuit \spadesuit https://www.who.int/health-topics/social-determinants-of-health#tab=tab_1 Thus, for biomedical NLP systems it is of particular importance to not be affected by factors irrelevant to biology and medicine, and for researchers to ensure they serve their users fairly irrespective of irrelevant attributes, such as names, as shown in Figure 1 . Here, we test the effect irrelevant demographic information has on biomedical question answering (QA) systems.

As a test-bed, we choose a subset of questions from the US Medical Licensing Exam level 1 (USMLE1;  Jin et al., 2021 ) whose answers, according to two medical professionals, are independent of the patient’s demographics. Although the questions are multiple-choice, correct answers require broad medical knowledge, including diagnosis and treatment of all common diseases, as well as an understanding of the underlying molecular and physiological mechanisms, potential drug side effects, probabilistic reasoning, and more.

We add irrelevant demographic information in a controlled way to the USMLE questions in order to answer the following research questions: (RQ1) Do the models’ answers change when being provided with irrelevant demographic information? (RQ2) Is the answer to RQ1 different for knowledge graph (KG)-grounded and text-based QA systems? We experiment with two biomedical QA systems: BioLinkBERT Yasunaga et al. ( 2022 ) , a text-based model, and QAGNN Yasunaga et al. ( 2021 ) , which is the highest performing KG-based model on USMLE.

There are good reasons to believe that neither system should be affected by irrelevant patient information: both are trained solely on biomedical text, which is most often independent of irrelevant demographic information, and QAGNN is additionally grounded by a KG that does not contain any demographic representations. Unfortunately, we find that both systems change many of their answers when provided with irrelevant patient demographic information.
We also observe that the two systems differ in which demographic information affects them. Finally, we compare biomedical to generic systems (i.e., trained on generic English text) and find that, as expected, the generic system changes even more of its answers in most cases (up to 17\% for gender). However, for some demographics, such as sexual orientation, the biomedical system changes up to 23\% of its answers. We hope that shedding light on this problematic behavior will motivate future work to further investigate its impact as well as possible solutions.