Related Work
Medical QA Many medical QA datasets are drawn from a variety of medical settings. MLEC-QA Li et al. ( 2021 ) for example, is based on the National Medical Licensing Examination in China, while emrQA Pampari et al. ( 2018 ) is based on clinical notes. HEAD-QA Vilares and Gómez-Rodríguez ( 2019 ) uses exams from the Spanish healthcare system, and MedQuAD Abacha and Demner-Fushman ( 2019 ) is based on 12 NIH websites and has questions on drugs, diseases, and other medical entities. DiSCQ Lehman et al. ( 2022 ) has questions from MIMIC-III discharge summaries that were generated by medical experts, and the Q-Pain dataset Logé et al. ( 2021 ) focuses on pain management. MedQA Jin et al. ( 2021 ) has questions from the professional medical board exams and covers three languages. Recent datasets focus on specific challenges identified from previous efforts Niu et al. ( 2003 ); Kell et al. ( 2021 ) . We selected English language questions from MedQA for this study, based on the breadth and depth of medical knowledge required and the fact that students must pass an exam with similar questions to become a physician in the US.

Biases in NLP Models Social biases have been reported in widely divergent NLP training sets and models, ranging from gender bias in machine translation Cho et al. ( 2019 ) to racial bias in opioid misuse prediction Thompson et al. ( 2021 ) . Social biases in dialog systems were examined through the use of demographically indicative names Smith and Williams ( 2021 ) . Several studies of natural language generation systems, transformers, and related models have shown outputs influenced by a variety of demographic characteristics in prompts, e.g., Sheng et al. ( 2019 ) . Methods to measure stereotype bias in language models (LMs) have been proposed, such as StereoSet Nadeem et al. ( 2021 ) and the CrowS-Pairs dataset Nangia et al. ( 2020 ) which contains information on nine types of demographics, such as age and race.

Bias in Medical NLP Some also focus particularly on evaluating biases in the medical domain. One work Borgese et al. ( 2021 ) analyzes unhealthy alcohol use risk bias between classifiers on electronic health records in trauma patients, while another Logé et al. ( 2021 ) examines gender and ethnicity biases in a pain management setting between GPT-2 and GPT-3. However, getting unbiased data to investigate model bias in a controlled way is difficult for pain management, where there is extreme societal bias. Hence, we use other data in our study. Racial biases in clinical settings were also examined Huang et al. ( 2022 ) . Some also focus on using NLP to evaluate whether medical licensure exams contain language patterns that exhibit biased or stereotypical language Padhee et al. ( 2021 ) .

Lastly, there is also work on evaluating pretrained transformer models and examining whether they contain biased information towards different demographics Zhang et al. ( 2020 ) . In contrast to prior work, we 1) examine the effect irrelevant demographic information has on QA systems in a clinical setting, using questions which require broad medical knowledge and are used by US medical students; 2) focus on models that are trained on biomedical text;
3) compare the effect of KG grounding on biases in a transformer-based model; and 4) compare biomedically-trained systems to a generic one, trained on English text.