Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper describes a dataset to evaluate question answering (QA) systems for social biases. Specifically, the authors construct a set of vignettes related to pain management. The system must then determine whether to provide pain medication, and if so, a low or high dose. To evaluate bias in the system, the authors construct templates where they replace the name of the patient with names that are commonly associated with major racial groups in the United States for both men and women. This allows the authors to consider how the system performs by race, gender, and the intersections of these two categories. Consistent with other research on pain management, they find that certain groups tend to be denied pain medication at higher rates than others. The paper also compares GPT-2 and GPT-3, finding the the more advanced model tends to provide better answers but is not any less biased than its predecessor.
strengths: The paper is very clearly written and the research design is well executed. This dataset will be valuable both as a benchmark for large language models/QA models and for domain-specific applications. This proof-of-concept can easily be extend to other medical and non-medical settings.
weaknesses: It appears that the authors only include vignettes where pain medication was considered to be appropriate (the system should answer "Yes" to all vignettes). It would have been interesting to also consider contexts where pain medication is inappropriate, particularly given the link between the overprescription of pain medication and the opioid epidemic. 
correctness: The claims made by the authors appear to be correct and the study is sound. The authors provide benchmarks using two leading LLM-based Q&A systems.
clarity: Yes
relation to prior work: The previous literature is succinctly discussed at the beginning of the paper.
additional feedback: I do not have any additional feedback to provide.
review_title: An important benchmark for evaluating bias in medical QA systems

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: In “Q-Pain: A Question Answering Dataset to Measure Social Bias in Pain Management,” the authors audit two automated Question Answering systems, GPT-2 and GPT-3, for bias in suggesting denial of pain treatment. The authors feed the QA system patient profiles where names, genders, and races were substituted inside of otherwise identical clinical vignettes describing severe patient pain. They show that even though neutral, non-biased closed prompts were provided to the GPT algorithms for training, the algorithms recommended pain treatment decisions (regarding the provision or denial of an opioid for pain treatment) that were apparently biased along race and gender lines when patient demographics were included.

This paper is a fascinating contribution to several literatures: the NLP fairness literature, the AI/clinical decision making literature, and the literature on the fair treatment of pain by human decision makers. The bias documented by the authors by the GPT-2 and GPT-3 systems is not dissimilar to bias found when human doctors make pain management decisions. While in the pain management domain one might hope AI might reduce bias against protected classes, unfortunately the authors document that the same concerns arise with non-human decision-makers. Of particular concern was the lack of improvement in this area between GPT-2 and GPT-3.


strengths: The strengths of the paper include its contribution, as detailed in the summary. I believe the paper has relevance to a broad audience, including AI fairness researchers, clinicians, and the pain management community. I appreciate the clarity and accessibility of the core auditing exercise; if results are reported accurately, it is hard to contest that the use of GPT-3 and GPT-2 and similar algorithms will not present important bias concerns across important clinical domains. The topic of equitable access to pain management is a very important one to society, and the choice of pain management as the clinical topic for audit is a strength of the paper.
weaknesses: As a referee without direct experience with GPT algorithms, the authors seemed to assume too much knowledge of the inner workings and training/tuning of such models. I believe the biggest weakness of the paper arose from the assumption of deep familiarity with the ins and outs of GPT models. Specifically, I was left with questions around how the GPT models are constructing racial bias from non racially biased prompts. Some discussion of the mechanisms behind the documented bias results and how GPT models are constructed would have made the paper much stronger.

Along similar lines, it is also interesting that the GPT models seem to somewhat mirror human doctors in prescribing for cancer pain more than non-cancer pain the least. Given the training dataset of clinical vignettes was entirely constructed from vignettes where pain treatment was recommended, what is generating this mirroring of real-world decisions regarding cancer?

I am also uncertain that the presented statistical analysis is most appropriate for the question. A simple regression model for each of GPT-3 and GPT-2 with regressors for the patient’s race and gender and type of pain (chronic non-cancer etc) (and perhaps interaction terms between them for an intersectionality analysis), where the dependent variable is the machine’s recommendation, would have largely eliminated the need for so many t-tests. This would also allow for greater power and perhaps lead to more interpretable conclusions, as black male and female patients compared to white male and female patients can be jointly used to identify the effect of being “black” compared to “white,” etc. As it is, it is good that the authors did a Bonferroni correction.

Finally, I did not understand why the training data was entirely situations were pain treatment should be offered, ”with the exception of five vignettes used in our closed prompts.” Perhaps giving “yes” and “no” scenarios to the GPT algorithm would produce better results? Also, I could not find any further information on these “five vignettes” in Subsection 3.2; I found this discussion very confusing.
correctness: The dataset appears to have been constructed soundly, with appropriate construction and validation by clinical experts. The experimental framework/design is in my opinion somewhat weak, for the reasons discussed above in “weaknesses.” The overall conceptual idea of auditing GPT models for bias is strong, and I believe the results presented would likely hold up in a more rigorous empirical framework, which is why I think the paper is suitable for acceptance even though I would have preferred a more rigorous approach.
clarity: It is reasonably well-written; some concerns are noted above.
relation to prior work: Yes.
additional feedback: N/a
review_title: Review for Q-Pain: A Question Answering Dataset to Measure Social Bias in Pain Management

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper introduces a QA dataset in the context of pain management for evaluation of social bias in language models. The dataset contains 55 clinical vignettes drafted by clinical experts and verified by physicians. In this sense, the quality of the QA dataset is ensured.  The paper then introduces a framework on how to use the dataset to evaluate social bias on GTP-2 and GTP-3 models. The results are carefully analyzed and the findings well align with that reported in medical studies. 
strengths: - The dataset is created by clinical experts and verified by physicians, hence quality is ensured. 
- The framework on how to use the dataset for social bias evaluation is clearly presented 
- The authors demonstrate how to perform meaningful analysis of the results. 
weaknesses: - The dataset is relatively small as indicated by the authors in the limitation section. 
- The results obtained are based on one single setting (i.e., with 3 closed prompts). Do similar findings hold with a different number of closed prompts provided? 


correctness: -The dataset is created and then evaluated by domain experts. Because of the relative small number of questions, the quality of this dataset is believe to be very high. 
-The proposed framework is carefully designed and clearly presented. 

clarity: The paper is well structured and well written. The authors have done a good job to explain the dataset creation, evaluation framework, and result analysis in great detail. The limitations of the dataset are also clearly listed. 
relation to prior work: The paper gives a good survey on the related studies from three perspectives: QA datasets in medical domain, pain management, and language model bias. The differences from related studies are clearly listed. 
additional feedback: This is an interesting dataset paper where the authors demonstrate how to create a high-quality dataset, how to use it, and how to perform results analysis. I have two questions unanswered after reading this paper. First, are the results obtained by using 3 closed prompts representative? Second, to evaluate race and gender bias, the slots in open prompts are replaced by predefined keywords (or representative names based on frequency). What will the results be if we use random strings in these slots? 

Authors have well addressed my comments and hopefully we will see some interesting work based on this dataset soon.
review_title: A domain-specific dataset for social bias evaluation 

Meta-review: This paper presents a new dataset for evaluating medical QA systems (specifically for pain management decisions) for social biases. Following a counterfactual analysis framework, the authors provide templates where patient names are replaced with names commonly associated with different racial and gender groups. They evaluate the performance of several systems on the dataset finding that certain groups tend to be denied pain medication at higher rates than others, a finding that is consistent with other pain management research. 

There is consistent agreement amongst the reviewers that this is a strong contribution to the workshop. Reviewers found the paper to be well written, well contextualized with respect to related literature, and to have relevance to a broad audience. The pain point of confusion amongst reviewers relates to the decision to focus on the “yes” scenario (i.e. when pain management is given out) but not the “no” scenario. The authors appear to have adequately clarified this decision in their rebuttal and have revised the paper draft in light of this as well. 

Decision: Accept