SDOH-NLI: a Dataset for Inferring Social Determinants of Health from Clinical Notes
Authors: Adam D Lelkes
Eric Loreaux
Tal Schuster
Ming-Jun Chen
Alvin Rajkomar
Venue: EMNLP 2023 Findings (2023)
Keywords: social determinants of health
natural language inference
nli
clinical notes
dataset

Abstract: Social and behavioral determinants of health (SDOH) play a significant role in shaping health outcomes, and extracting these determinants from clinical notes is a first step to help healthcare providers systematically identify opportunities to provide appropriate care and address disparities. Progress on using NLP methods for this task has been hindered by the lack of high-quality publicly available labeled data, largely due to the privacy and regulatory constraints on the use of real patients' information. This paper introduces a new dataset, SDOH-NLI, that is based on publicly available notes and which we release publicly. We formulate SDOH extraction as a natural language inference task, and provide binary textual entailment labels obtained from human raters for a cross product of a set of social history snippets as premises and SDOH factors as hypotheses. Our dataset differs from standard NLI benchmarks in that our premises and hypotheses are obtained independently. We evaluate both "off-the-shelf" entailment models as well as models fine-tuned on our data, and highlight the ways in which our dataset appears more challenging than commonly used NLI datasets.

SDOH-NLI: a Dataset for Inferring Social Determinants of Health from Clinical Notes
Adam D. ;elkes's Eric Loreaux² Tal Schuster',
Ming-Jun Chen' Alvin Rajkomar",
Google Research, CCrai Health
Abstract
Social determinants of health are frequently recorded elinical notes as unstructured text, so natural language processing (NLP) can be a valu- able tool for extracting actionable insights for care teams. However, research in this area often uses patient records from private health systems' elec- tronic health records (EHRs), which makes it diffi- cult to compare results from other health systems or even replicate the studies. The development and release of righ-quality publicly available datasets could enable more reproducible research this area.
Social and behavioral determinants health (SDOH) play significant role in shaping health outcomes, and extracting these deter- minants from clinical notes is a first step to help healthcare providers systematically iden- tify opportunities to provide appropriate care and address disparities. Progress on using NLP methods for this task has been hindered by the lack high-quality publicly available labeled data, largely due to the privacy and regulatory constraints on the use of real pa- tients' information. This paper introduces new dataset, SDOH-NLL, that is based on pub- licly available notes and which we release pub- licly. We formulate SDOH extraction as nat- ural language inference (NLI) task, and pro- vide binary textual entailment labels obtained from human raters for cross product of set of social history snippets as premises and SDOH factors as hypotheses. Our dataset dif- fers from standard NLI benchmarks in that premises and hypotheses are obtained indepen- dently. We evaluate both "off-thh-sseef" entail- ment models well as models fine-tuned on our data, and highlight the ways in which our dataset appears more challenging than com- monly used NLI datasets.
In this work, introduce new, public SDOH dataset based on MTSamples. com, an online collec- tion of transcribed medical reports. Our setup is motivated by the use cases of slicing patient pop- ulations along social determinant dimensions for population analytics, and retrieving patients with certain social determinants of health to allow for more targeted outreach and intervention. Given large set of social determinant factors, our goal to make binary determinations for each patient about whether that patient's notes imply a partic- ular SDOH factor. In other words, for example, we want to be able to find all patients who lack access to transportation, as opposed to just tagging transportation-rellted spans in their notes, as done in some previous work.
1 Introduction
There has been growing recognition that social and behavioral determinants of health (SDOH) play a significant role in shaping health outcomes for individuals and populations. The ability to accu- rately identify and extract social and behavioral determinants of health from clinical notes can pro- vide valuable insights that can enable healthcare providers to better understand and address the un- derlying determinants of health that contribute to poor health outcomes and health disparitics.
To achieve this goal, we formulate the task as a textual entailment problem, with patient note snip- pets as the premises, SDOH factors as the hypothe- ses, and binary entailment labels. We use human annotators to label 1,398 social history snippets according
curated list of 60 SDOH statements, resulting in a dataset of 29,635 labeled premise- hypothesis examples after some filtering (see Sec- tion for details). We release this dataset publicly. We also evaluate state-of-the-art publicly available large language models on our data in range of different settings (see Section 4).
"ooresponniig author. lelkes @ google.co Work completed at Google.
Equal leadership.
'https://ee //g wwwchhu...cmmeeeeee
wtt.lcccm//ooaeeeee haam//gooleeeaee remm/googll--eeerrdhhe dattassss
SDOH-NLI
A notable feature of our entailment dataset is

---

that unlike other entailment datasets, our premises and hypotheses were obtained independently, and we label the full cross product of premises and hypotheses. In traditional entailment datasets, the hypotheses are constructed to be tied to particular premise; however, in our formulation, the hypothe- ses are drawn from large set of SDOH factors that may may not be discussed in particular premise (drawn from clinical note). Since all our text comes from the same domain, we still have a non-negligible fraction of positive entailment la- bels (albeit with a much larger label imbalance than in standard NLI benchmark datasets).
deidentified medical discharge records for smoking status only, and small number of datasets based on MIMIC-III (Johnson et al., 2016), a large publicly available database of deidentified health records of patients who stayed in critical care units of Beth Israel Deaconess Medical Center between 2001 and 2012. For example, Gehrmann et al. (2018) annotated MIMIC-III with binary labels for cer- tain "phenotypes" including alcohol and substance abuse, and Lybarger al. (2021) annotated note spans with SDOH information.
We are aware of four other previous SDOH- related papers which used MTSamples data (Wang et al., 2015; Winden et al., 2017; Yetisgen et al., 2016; Yetisgen and Vanderwende, 2017), all of which focused on extracting and tagging SDOH- related spans from social history sections.
This requires NLI methods to understand both the premise and the hypothesis, and defeats typi- cal shortcuts that have been observed to work for other entailment datasets, such guessing the la- bel based on the hypothesis alone, or relying on simple syntactic clues such as the presence of nega- tions (see Section 2.2). Indeed, even though our task does not require domain domainn-pecific knowledge, we observe that state-of-the-art models struggle to generalize from common NLI benchmarks to our dataset, and highlight typical failure cases (see Section 4).
Among previous papers, the one methodologi- cally closest to ours is Lituiev et al. (2023) which also formulated SDOH extraction as an entailment problem and evaluated RoBERTa (Liu et al., 2019) fine-tuned on ANLI (Nie et al., 2020) on clini- cal notes from UCSF, without any in-domain fine- tuning experiments.
We evaluate both off-the-shelf and fine-tuned models different setups: dual encoders, zero/few- shot prompting, and binary classification. We show that ttate-of-the-art off-the-shelf models, even if they were fine-tuned on various NLI datasets, do not reliably solve our problem; on the other hand, models fine-tuned on our training set robustly gen- eralize both to unseen notes and to unseen factors, providing evidence for the usefulness of our dataset for model fine-tuning and evaluation.
2.2 Natural Language Inference
Natural language inference (NLI), also called rec- ognizing textual entailment (RTE), has been very well-studied NLP task; see e.g. Storks et al. (2019); Poliak (2020) for recent surveys. Many standard NLI datasets, such as SNLI (Bowman et al., 2015) or MultiNLI (Williams et al., 2018), are obtained by automatically collecting premises and, for each premise and target entailment label, having human annotators write a hypothesis with the specified entailment relation to the premise. (Even some of the datasets specifically designed to address these datasets' shortcomings, such as ANLI (Nie et al., 2020), follow a similar setup.) has been observed that this leads to annotation artifacts that lets mod- els do well on these tasks without requiring true understanding of entailment, including by using simple syntactic heuristics (McCoy et al., 2019) or by completely ignoring the premise, and con- sidering only the hypothesis (Gururangan et al., 2018: Poliak et al., 2018). ContractNLI (Koreeda and Manning, 2021) an example of a previous dataset which used the same fixed set of hypotheses for all the premises.
2 Related Work
2.1 Social and Behavorial Determinants of Health
There has been lot of interest in using NLP tech- niques for SDOH extraction; see Patra et (2021) for a recent survey. The range and granularity of SDOH factors vary considerably across different papers. There has also been a range of methods used, from rule-based heuristics, to n-grams, to fine-tuning pretrained Transformer models.
Many previous research studies on SDOH ex- traction were performed on EHR data from partic- ular health systems and are not released publicly. Exceptions include the i2b2 NLP Smoking Chal- lenge (Uzuner et al., 2008), which classified 502

---

3 Dataset Construction
We split the dataset along the snippets into train- ing, validation, and sets with a 70:15:15 ratio, with the following modification: we remove a sin- gle pair of factors, "The person lives alone" and "The person does not live alone," from the training and validation sets, in order to evaluate fine-tuned models' ability to generalize to unseen factors. In other words, the training, validation, and test sets have disjoint note snippets but the same SDOH fac- tors, except for the test set which also contains an additional two factors that are not present in the training or validation sets.
We scraped all 5,003 medical reports from MTSam- ples. Within these reports, we obtained 1,030 note sections related to social history by searching for a collection of note section titles identified as syn- onyms of social history. We then split each note section into sentences, resulting in 3,281 sentences. Many sentences (such as "He is married") appear in multiple notes; after deduplication, we have 1,398 unique sentences.
We manually curated a collection of SDOH fac- tors primarily from two sources, the AHRQ So- cial Determinants of Health Database (for Health- care Research and Quality, 2020) and UCSF SIREN's Compendium of medical terminology codes for social risk factors (Arons et al., 2018). We rephrased each factor full English sentence stating a fact about a person; e.g. "The person employed." " For binary factors, we included state- ment and its negation; c.g. "The person currently drinks alcohol" and "The person currently does not drink alcohol." For factors with multiple potential values (such as housing status), we aimed to list all the common options. We grouped these 60 state- ments into 10 categories: smoking, alcohol use, drug use, employment, housing, food, transporta- tion, health insurance, social support, and financial situation. See the full list Appendix A.
Model Evaluation
Since the typical use case is retrieving patients with particular SDOH factor, and because of the heavy label imbalance in our dataset, we use precision, recall, and F1 score as our evaluation metrics.
We evaluate state-of-the-art public models in four different setups:
Treating the problem as a retrieval task with SDOH factors as queries and note snippets as documents. We evaluate Sentence-TSs (Ni et al., 2022a) and GTR (Ni et al., 2022b), two statc-of fatte-of-thh-are the-art dual encoder models. We se- lect the cosine similarity threshold which max- imizes Fl score the training set.
For each social history sentence, we asked hu- man raters to select all the relevant categories and, within each category, all the statements that are sup- ported by the social history snippet. Each snippet was rated by least three raters. For each (snippet, statement) pair, we took the majority vote of raters to get binary entailment labels. Rater agreement was high, with a Krippendorff's alpha of 0.97 (com- puted over the binary entailment labels provided by different raters for (premise, hypothesis) pairs).
general- NLI model. We evaluate the SENTLI model (Schuster et al., 2022), a statc-of-the-art NLI model (T5 large fine- tuned on the SNLI (Bowman et al., 2015), MNLI (Williams et al., 2018), ANLI (Nie et al., 2020), Fever (Thorne et al., 2018), and VitaminC (Schuster et al., 2021) datasets). To help the model adapt to the label imbalance in our dataset, we also evaluate with re-tuning the threshold for positive prediction to maxi- mize F1 score on our training set.
We removed all SDOH statements which were not entailed by any of the social history snippets as well as all note sentences that were not relevant to any SDOH categories. Finally, after inspecting the dataset, we removed three SDOH factors ("The person is stably housed.". "The person has social support.", "The person does not have social sup- port.") because raters weren't it able to consistently give correct ratings. That left us with 787 unique sentences and 38 SDOH factors. Since each snip- pet would typically only entail one factor or a small number of factors, the resulting dataset is heavily imbalanced: only 4.6% of the labels are positive.
Zero/few-shot prompting, We evaluate Flan- T5 XXL (Chung et al., 2022) and Flan- UL2 (Tay et al., 2023) in both a zero- and a few-shot setting. These models instruction tuned for a large set of tasks, including NLI tasks. (See Appendix B for details.)
Fine-tuning experiments. We fine-tune T5 and Flan-T5 on our dataset (SDOH-NLI), on ANLI, and on a mixture of both at various model sizes. (See Appendix C for details.)

---

Model
Precision
Recall
F1
Sentence-T3 11B
.2826
.223
3669
GTR XXL
.3148
.4885
3829
SENTLI
.4488
.8161
.5749
SENTLI, threshold tuned
.5721
.7999
.6414
Flan-T5 XXL O-shot
.6867
.6552
.6706
Flan-T5
.6297
.6839
.6556
Flan-UL2 O-shot
.5922
.7111
.6421
Flan-ULA UL2 5-shot
.5336
.7126
.6231
Flan-T5 XXL finetuned on ANLI
356
.8736
.5058
F5an-T5 XXL finetuned on SDOH-NLI
.295
.8333
.7888
Plan-T5 XXL finetumed on ANLI + SDOH-NLI .8765
.8563
.8663
Table 1: Scores of retrieval-based, general-purpose NLI, in-context, and fine-tuned models on the SDOH test set. See Appendix C for additional fine-tuning experiments.
4.1 Results
fine-tuning on this mixture enabled robust general- ization to unseen factors without sacrificing overall test performance.
Table shows selected results (see Appendix D for more model fine-tuning metrics). First, observe that while our problem can naturally be framed as an information retrieval problem, even state-of-the-art retrieval models perform poorly on it; formulating the problem as natural language inference yields dramatically better results.
4.2 Discussion
What makes our dataset challenging for models fine-tuned on standard NLI datasets? We empha- size that it is not that it requires any specialized (e.g.. medical) knowledge: most of our examples describe everyday situations in plain English (c.g.. "The patient retired disability due her knee replacements")> human raters without any domain- specific training had no difficulty understanding them. We conjecture that the main difference is that the hypotheses typical NLI datasets are writ- ten to satisfy given entailment label for a given premise, whereas ours are obtained independently from the premises, and therefore their entailment status be more subtle and ambiguous. These models also struggled with distinguishing between statements about the present and the past; e.g., Flan-UL2 erroneously predicting that "He not a smoker" entails "The person wasn't 't a smoker in the past." Also, our dataset contains a lot of irrele- vant hypotheses and requires the model to correctly classify all of those; we see off-the-shell models occasionally giving them positive labels (c.g. pre- dicting that "She living alone and is now living in assisted living." implies "The person drank alco- hol in the past."), "), hurting precision. As an example, Flan T5 fie-ttned on ANLI had comparable recall to our best models, but much worse precision.
However, even powerful models fine-tuned for NLI, either on NLI datasets alone or as part of the much larger Flan collection, do not reliably solve our problem, with an FI score of at most .67 on the test set. Even T5 small fine-tuned on dataset outperforms the largest off- uff-the-sheef models, highlighting the added value of dataset,
When prompting instruction-tuned models, in- cluding few-shot examples does not appear add any value compared to a zero-shot setup. We con- jecture that this is because the models are already familiar with the NLI task and the prompt format from instruction tuning, and presenting them with a small number of additional examples not suf- ficient for teaching them the difference between this task and the NLI benchmark datasets they have seen during fine-tuning.
For fine-tuned models below XXL size, when fine-tuning on the SDOH-NLI training set only, observed poor generalization to the held-out SDOH factors in the test set (see Appendix for detailed metrics). Because of this, why we exper- imented with fine-tuning on a combination of our data and ANLI RI, a challenging general- purposs NLI dataset of a similar size. For smaller models,

---

5 Conclusion
The use of public, non-identifiable data that is re- leased for the NLP community helps balance the need to have reproducible (i.e.. honest and trustwor- thy) data to enable technical advances while limit- ing the need for sensitive, private medical records for research. We acknowledge that the purpose of the work is to identify SDOH to provide additional help and services to patients, and we warn against any use to deny patients care services based on any factor that can be identified. Although we picked a large number of SDOH factors to test our method, we acknowledge that there may be addi- tional factors that are important for specific patients and populations, so we encourage researchers to reflect on those possible factors and create datasets to help others study them, as well.
In this work, we introduced SDOH-NLI, a new en- tailment dataset containing social history snippets premises and social and behavioral determinants of health as hypotheses. Our dataset was designed both to reflect realistic use cases of SDOH extrac- tion in clinical settings as well as to provide a high quality entailment dataset to support broader NLI research. We evaluated baseline methods using ofte-of-theeart public models in a variety of setups, and highlighted novel and challenging features of our dataset.
Acknowledgments
The authors would like to thank Jonas B. Kemp, Von Nguyen, Birju Patel, Martin G. Seneviratne. Andy Strunk, and Vinh Q Tran for their valuable feedback and discussions.
References
Abigail Arons. Sarah DeSilvey, Caroline Fichtenberg, and Laura Gottlib. 2018. Compendium of medical terminology codes for social risk factors.
Limitations
Our dataset is English-only, and reflects the Amer- ican hcaltheare system. While a lot of the social and behavioral determinants of health mentioned in the data could apply elsewhere, too, their dis- tribution and the language used to describe them could reflect U.S. norms. Also, since the dataset is based on transcription samples, the text can be cleaner than in some other settings (such as notes in EHRs), where the task could be more challenging due to typos and abbreviations that do not appear in our dataset. In such settings, performance could be improved by first using the methods of Rajko- mar et al. (2022) to decode abbreviations, but we have not included this in our evaluations. Finally, dataset only contains short note snippets, and we have not evaluated the models' ability recon- cile contradictory statements or reason about the chronology of information in longer patient records. For longer contexts, especially if the social history sections don't 't fit in the Transformer model's con- text window, we recommend evaluating the meth- ods of Schuster et al. (2022).
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632 642.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph. Yi Tay. William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani. Siddhartha Brahma, et al. 2022. Scaling instructionn-finetueed language mod- cls. arXiv preprint arXiv:2210.11411.
Agency for Healthcare Research and MD. Quality, Rockville. 2020. Social determinants of health database.
Sebastian Gehrmann, Franck Dernoncourt. Yeran Li, Eric T Carlson, Joy T Wu, Jonathan Welt, John Foote Jr, Edward T Moseley, David Grant, Patrick D Tyler, et al. 2018. Comparing deep leam- ing and concept extraction based methods for pa- tient phenotyping from clinical narratives. PloS one, 13(2):e0192360.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural lan- guage inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112.
Ethics Statement
Machine learning research in the clinical domain has many ethical considerations. Given the nature of this work is to identify opportunities for care teams to improve the health outcomes of patients due to factors commonly not addressed, we think contributes to human well-being and avoids harms.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li- wei H Lehman, Mengling Feng, Mohammad Ghas- semi, Benjamin Moody, Peter Szolovits. Leo An- thony Celi, and Roger G Mark. 2016. Mimic-iii,

---

a freely accessible critical care database. Scientific data, 3(1):1-9.
social determinants of health from electronic health records using natural language processing: a system- atic review. Journal of the American Medical Infor- matics Association, 28(12):2716-272
Yuta Koreeda and Christopher D Manning. 2021. Con- tractnli: A dataset for document-level natural lan- guage inference for contracts. In Findings of the Association for Computational Linguisties: EMNLP 2021, pages 1907-1919.
Adam Poliak. 2020. A survey on recognizing textual entailment as an nip evaluation. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 92-109.
Dmytro S Lituiev, Benjamin Lacar, Sang Pak, Pe- ter I. Abramowitsch, Emilia H De Marchis, and Thomas A Peterson. 2023. Automatic extraction of social determinants of health from medical notes of chronic lower back pain patients. J. Am. Med. In- form. Assoc.
Adam Poliak, Jason Naradowsky. Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language in- ference. In Proceedings of the Seventh Joint Con- ference Lexical and Computational Semantics, pages 180-191.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 1692.
Alvin Rajkomar, Eric Loreaux, Yuchen Liu, Jonas Kemp, Benny Li, Ming-Jun Chen, Yi Zhang, Afroz Mohiuddin, and Juraj Gottweis. 2022. Decipher- ing clinical abbreviations with a privacy protecting machine learning system. Nature Communications, 13(1):7456.
Roberta:
Shayne Longpre, Le Hou, Tu Vu, Albert Webson Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph. Jason Wei, et al. 2023. The flan col- Designing data and methods for effective in- struction tuning. arXiv preprint arXiv:2301. 13688.
lection:
Adam Roberts, Hyung Won Chung. Anselm Lev- skaya, Gaurav Mishra, James Bradbury. Daniel An- dor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, et al. 2022. Scaling up mod- els and data with t5x and seqio. arXiv preprint arXiv:2203.11 17189.
Kevin Lybarger, Mari Ostendorf, and Meliha Yetisgen. 2021. Annotating social determinants of health us- ing active learning, and characterizing determinants using neural event extraction. Journal of Biomedical Informatics, l 113: 1.
Tal Schuster, Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, and Donald Metzler. 2022. Stretching sentence-pair NLI models to reason over long doc- uments and clusters. Findings of the Association for Computational Linguistics: EMNIP 2022, pages 412, Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguistics.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019, Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 12888448.
Jianmo Ni, Gustavo Hernandez Abrego, Noah Con- stant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022a. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1864-188 1874.
Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin c! robust fact verification with con- trastive evidence. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational I Linguistics: Human Lan- guage Technologies, pages 624 643.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hemandez Abrego, Ji Ma. Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. 2022b. Large dual encoders are generalizable re- trievers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9844-9855, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Shane Storks, Qiaozi Gao, and Joyce Y Chai. 2019. Recent advances in natural language inference: A survey of benchmarks, resources, and approaches. arXiv preprint 1rxiv:1904.01122.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Gar- cia, Jason Wei, Xuezhi Wang, Hyung Won Chung. Dara Bahri, Tal Schuster, Steven Zheng, et al. 2023. U12: Unifying language learning paradigms. In The Eleventh International Conference on Learning Rep- resentations.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Ad- versarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 4885-4901.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long
Braja G Patra, Mohit M Sharma, Veer Vekaria, Prakash Adekkanattu, Olga V Patterson, Benjamin Glicks- berg, Lauren Lepow, Euijung Ryu, Joanna M Bier- nacka, Al' ona Furmanchuk, et al. 2021. Extracting

---

Papers), pages 809-819, New Orleans. Louisiana. Association for Computational Linguistics.
A Labeling Setup
Raters were given the following instructions:
Özlem Uzuner, Ira Goldstein, Yuan Luo, and Isaac Ko- hane. 2008. Identifying patient smoking status from medical discharge records. Journal of the American Medical Informatics Association, 14(1):14-24
"In this task, you will be given a list of snip- pets from transcribed medical records, describing person's 's social history. Your job is to select the categories that are relevant to the snippet and, in each category, select statements about the person that supported by the snippet.
Yan Wang, Elizabeth S Chen, Serguei Pakhomov, El- liot Arsoniadis, Elizabeth W Carter, Elizabeth Lin- demann. Indra Neil Sarkar. and Genevieve Melton. 2015. Automated extraction of substance use infor- mation from clinical texts. In AMIA Annual Sympo- sium Proceedings, volume 2015, page 2121. Ameri- can Medical Informatics Association.
Only select categories and statements relevant the subject of the sentence, NOT if they apply someone else (such as the subject's relatives).
The source of the medical transcript samples is public dataset (MTSamples.com) other raters' or users health transcript data."
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long Papers), pages 1112-1122.
and not your,
For each snippet, they were required select more of the following categories (including "None of the above" if none of the categories were relevant). If they selected a category, the statements within the category would be displayed, and they were required to select one of statements within that category, or "None of the above":
Tamara Winden, Elizabeth Chen, Yan Wang, Elizabeth Lindemann, and Genevieve В Melton. 2017. Residence, living situation, and living condi- tions information documentation in clinical practice. In AMIA Annual Symposium Proceedings, volume 2017, page 1783. American Medical Informatics As- sociation.
Smoking
The person currently a smoker.
The person is currently not smoker.
Meliha Yetisgen, Elena Pellicer, David R Crosslin, and Lucy Vanderwende. 2016. Automatic identification of lifestyle and environmental factors from social history in clinical text. In Proceedings of AMIA 2016 Joint Summits Translational Science.
The person was a smoker in the past.
The person wasn't a smoker in the past.
None of the above
Meliha Yetisgen and Lucy Vanderwende. 2017. Auto- matic identification substance abuse from social history in clinical text. In Artificial Intelligence in Medicine: 16th Conference Artificial Intelligence Medicine, AIME 2017, Vienna, Austria, June 21- 24, 2017, Proceedings 16, pages 171-181. Springer.
Alcohol use
The person currently drinks alcohol.
The person currently does drink alco- hol.
The person drank alcohol in the past.
The person did not drink alcohol in the past.
None of the above
Drug use
The person drug user.
The person is not drug user.
The person was a drug user in the past.
The person wasn't a drug user in the past.
The person uses cocaine.
The person does not use cocaine.
The person used cocaine in the past.
The person did not use cocaine in the past.
The person uses marijuana.

---

The person does not use marijuana.
The person has access transportation.
The person used marijuana in the past.
The person does not have access to trans- portation.
The person did not use marijuana in the past.
The person has access to a car.
The person uses opioids (e.g.. heroin, fentanyl, oxycodone).
The person does not have access car. The person has access public transit.
The person does not opioids (c.g., heroin, fentanyl, oxycodone).
The person does not have access to pub- lic transit.
The person used opioids in the past (c.g., heroin, fentanyl, oxycodone).
The person has issues with finding trans- portation.
The person did not use opioids the past (e.g., heroin, fentanyl, oxycodone).
None of the above
Health insurance
None of the above
The person has private health insurance.
Employment
The person is on Medicare.
The person is employed.
The person on Medicaid.
The person is not employed.
The person does have health insur- ance.
The person is employed part time.
The person is student.
None of the above
The person is a homemaker.
Social support
The person is retired due to age pref- erence.
The person has social support.
The person is retired due to disability.
The person does not have social support.
The person is retired due to an unknown reason.
The person lives alone.
The person does not live alone.
None of the above
None of the above
Housing
Financial situation
The person lives in their own or their family's home.
The person below the poverty line. The person is above the poverty line.
The person lives in a housing facility.
The person is able to afford medications. The person is not able afford medica- tions.
The person is homeless.
The person is stably housed.
The person's ' housing is unsuited to their needs.
None of the above
None of the above
None of the above
Food
B Prompting Setup
The person is able to obtain food on consistent basis.
Our goal with the design of our zero/few-shot ex- periments was to stay close to how the models were trained and evaluated on similar NLI tasks. Since the models we used were fine-tuned on the Flan collection (Longpre et al., 2023), which includes several NLI datasets, reused some of the Flan collections's 's prompt templates.
The person is not able to obtain food on a consistent basis.
The person has consistent fruit and veg- etable intake.
The person does not have consistent fruit and vegetable intake.
In particular, we used the templates for the SNLI and MNLI dataset which were the most relevant our data. (We excluded templates for other datasets such as RTE, ANLI, or WNLI because the wording
None of the above
Transportation

---

of some of the prompts could be slightly mislead- ing; c.g. by referring the premise "para- graph.") For each example, we chose prompt tem- plate uniformly at random from the 19 templates. For few-shot experiments, we first picked a ran- dom positive example from the training set, then picked the other few-shot examples uniformly at random from the training examples. (Without ex- plicitly forcing at least one example be positive, it would be very likely that all the few-shot exam- ples would be negative, given the label imbalance in the dataset. We tried dropping that requirement and sampling all examples at random, which re- sulted in slight drop in model performance, as we expected.)
Note that since Flan-T5 models between sizes large and XL fine-tuned on either ANLI or our SDOH dataset alone underperformed the same models fine-tuned on the combined dataset, we did not perform these dataset ablations on smaller Flan-T5 T5 models or on T5 below the XXL size.
Since SNLI and MNLI have three answer op- tions ("yes." "it is possible tell," "no"), kept all three of these options in the prompt tem- plate, even though our dataset is binary. We ex- perimented with dropping either the "no" or the "it is not possible to tell" option; both of these devia- tions from the original prompts resulted in slight decreases in model performance.
We use rank classification to obtain binary labels (as is customary): i.e., instead of decoding model prediction, we score the three answer options (for three copies of each input example) and consider the prediction positive yees" has the highest score. Similarly to our experiment with SENTLI, we also tried taking softmax over the three options and using fixed threshold for "yes" which maximizes Fl score the training set; this resulted in small drop in test Fl score (e.g., .5501 to .6222 for Flan- UL2).
Fine-tuning Setup
All models were fine-tuned using the T5X frame- work (Roberts al., 2022) on TPUv3 chips for 10k steps with batch size 32 and learning rate lc-4, with the exception of the T5 small size models which were finetuned for 50k steps. For each model, we picked the checkpoint with the highest Fl score on the validation set.
Additional Model Fine-tuning Results
See Table for the full set of results from our fine- tuning experiments. Here we also include metrics on the subset of the test set consisiting on the new SDOH factors included in the training and vali- dation sets to highlight the differences in models' ability to generalize to unseen factors.

---

Model
Size
Dataset
Test P
Test R
Test Fl
New factors P
New factors R
New factors Fl
TS
small
ANLI SDOH-NLI
.7877
.7699
.7500
.6667
.0833
.1481
T5
base
+ SDOH-NLI
.7366
8161
.760
.75
.375
.5
IS
large
+ SDOH-NLI
8247
.7299
.7744
.75
.125
.2143
T5
XI.
+ INLI+SDOH-NIL SDOH-NLI
8391
8391
.8991
.9311
.5
.6486
8391
.75
.871
.371
XXL
ANLI SDOH-NLI
.024
.8066
.8577
8889
.9122
Flan-15
ANLI SDOH-NLI
8075
.7761
.6667
.1881
Flan-T5
base
ANLI SDOH-NLI
7956
8276
.8133
.6216
.9833
.7541
Flan-T5
ANLI
.177
.506
.939
.2421
9583
.366
Flan-T5
large
SDOH-NLI
8041
.6399
.391
.2388
.125
.1622
Tlan-T5
lunge
A
SDOH-NLI IDOHHII
8212
8448
8329
8846
9583
.92
Flan-T5
XL.
ANLI
2868
8851
4332
.7667
9583
8519
Flan-T5
XL
SDOH
9071
.7299
8089
.155
.222
Flan-15
A SDOH-NLI
8647
8448
.8447
.8846
.9583
.92
Flan-T5 XXL
3560
8736
5058
.7931
8679
Flan-15
SLI
Flan-T5
XXL
A SDOH-NLI
.8665
.8533
.8663
.8466
Table 2: Precision, recall, and Fl scores of various models the full test and the subset of the test consisting of the two held-out factors.


Reviews:
[{"review_id": "O7XTrS1ahn", "reviewer": "Reviewer_YiJH", "summary": "", "questions": "A: In line 89, what is the relation between the fact that there is a non-negligible fraction of positive entailment labels and coming from the same domain? \nB: In line 226, it is mentioned that the pair of factors (\"The person lives alone\" / \"The person does not live alone\") was removed from the training and validation sets. How did the models perform on this factor? \nC: Was the F1 score macro-averaged? Do you maybe have a confusion matrix or results showing the difference between positive and negative class performance, considering the heavy imbalance?\nD: In line 323, it is said that the dataset contains a lot of irrelevant hypotheses. Why was the dataset constructed in this way if it led to irrelevant hypotheses and noise?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful review, and thank the reviewer for appreciating the importance of the problem and the effort that went into curating the dataset.\n\nWe also thank the reviewer for the detailed critique of our datasets. We respectfully disagree that these features of our dataset would be reasons to reject the paper; we like to think of these as unique features that make the dataset novel and which represent real-world use cases. We will try to address these critiques below.\n\n___\n\n> This makes the dataset not just heavily imbalanced (only 4% positive examples) but also heavily redundant with this huge amount of repeating sentences.\n\nWe would argue that label imbalance makes the task **more challenging** and also more reflective of the motivating application. Real-world data is even more heavily imbalanced: electronic health records can contain many notes for a given patient, and individual notes are often long and address multiple topics. The total amount of text for even a single patient can be orders of magnitude more than what fits in a typical Transformer model’s context window. Inferring which patients have a particular social determinant of health requires processing all that text, most of which will be irrelevant.\n\nWe would argue that this setup is also a **better test of how well models recognize neutral entailment relationships**. For standard NLI benchmark datasets, hypotheses are typically at least somewhat relevant to the corresponding premises. Our dataset tests whether models robustly generalize to cases when hypotheses and premises might be completely unrelated.\n\n___\n\n> This can lead the models to pick up statistical patterns and not really rely on deeper understanding.\n\nAs we briefly mention in Section 2.2, that models can pick up statistical patterns has actually been observed for NLI datasets where premises and hypotheses are *not* collected independently. When hypotheses are written specifically for each premise, there are often simple syntactic artifacts (such as negation) that let the model guess the entailment relationship without any real understanding. In fact, it has been observed that models trained on hypotheses only in such datasets, completely ignoring the premises, achieve decent performance [1,2]. These shortcuts become impossible when, as in our dataset, hypotheses and premises are obtained independently and the model has to correctly label the entire cross product. As an example, Flan-T5 XXL fine-tuned on hypotheses only in SDOH-NLI (with keeping the rest of the setup and hyperparameters the same as for our best-performing model) learns the constant 0 classifier, getting an F1 score of 0. (It achieves an AUC-PR of .2, compared to .95 for Flan-T5 XXL fine-tuned on both hypotheses and premises.)\n\n___\n\n> most of the snippets of social history are literally snippets and not even formed to full sentences (such as \"no alcohol use\", \"unemployed\", \"1-2 ppd Cigarettes\").\n\nAgain, we would argue that this makes our dataset **more realistic**. (If anything, the text in MTSample is cleaner than in many real-world EHRs.) If, as you write, models fine-tuned on standard NLI datasets such as ANLI are to be expected to struggle in such real-world scenarios, that seems to be an argument for introducing a new dataset that captures the messiness of real-world text!\n\n___\n\nTo answer the questions:\n\n> A: In line 89, what is the relation between the fact that there is a non-negligible fraction of positive entailment labels and coming from the same domain?\n\nWe meant to make the point that although labeling the full cross product of independently collected premises and hypotheses has desirable properties, it wouldn’t work if the sentences were e.g. just randomly sampled from some large diverse text corpus (since most sentence pairs would be unrelated to each other). Our dataset has enough diversity to make the task challenging, but not too much that the fraction of positive entailment labels would become negligible.\n\n___\n> B: In line 226, it is mentioned that the pair of factors (\"The person lives alone\" / \"The person does not live alone\") was removed from the training and validation sets. How did the models perform on this factor?\n\nPlease see the last three columns in Table 2, Appendix D; they show the fine-tuned models’ performance on these factors.\n\n___\n\n> C: Was the F1 score macro-averaged? Do you maybe have a confusion matrix or results showing the difference between positive and negative class performance, considering the heavy imbalance?\n\nNo. All the precision, recall, and F1 scores are for the positive class. Here is the confusion matrix on the test set for Flan-T5 XXL fine-tuned on SDOH-NLI:\n\n\n| | |\n|:---|---:|\n|3919|11|\n|29|145|\n\n\n___\n\n> D: In line 323, it is said that the dataset contains a lot of irrelevant hypotheses. Why was the dataset constructed in this way if it led to irrelevant hypotheses and noise?\n\nWe don’t view hypothesis-premise pairs with neutral relation as noise but as a way to improve the robustness of the model in distinguishing between neutral and non-neutral relations. A similar approach was also followed in ContractNLI [3]. Including both more closely related pairs and more distant pairs exposes the model to a more diverse set of pairs (both in training and evaluation).\n\n___\n\nReferences:\n\n[1] Gururangan, Suchin, et al. \"Annotation Artifacts in Natural Language Inference Data.\" Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). 2018.\n\n[2] Poliak, Adam, et al. \"Hypothesis Only Baselines in Natural Language Inference.\" Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics. 2018.\n\n[3] Koreeda, Yuta, and Christopher D. Manning. \"ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts.\" Findings of the Association for Computational Linguistics: EMNLP 2021. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper addresses an important issue of detection of social and behavioral determinants of health from clinical notes, which can improve opportunities of patients and access to appropriate healthcare options.  \nThe paper introduces a novel dataset, with lots of manual annotation effort, that could be useful for the NLP community and NLI research.\nThe dataset is evaluated with various models, in both zero-shot and fine-tuned settings, and the results are reported. \nThe paper is well-written and easy to follow.\n", "reasons_to_reject": "The usability of the dataset is questionable. Since it was constructed as a cross product of 787 unique sentences and 38 SDOH factors, this yielded around 30 thousand instances, where the same premise (snippet) is always repeated 38 times and the same hypothesis always repeated 787 times. This makes the dataset not just heavily imbalanced (only 4% positive examples) but also heavily redundant with this huge amount of repeating sentences. This can lead the models to pick up statistical patterns and not really rely on deeper understanding.\nFurthermore, most of the snippets of social history are literally snippets and not even formed to full sentences (such as \"no alcohol use\", \"unemployed\", \"1-2 ppd Cigarettes\"). It is not completely surprising that models fine-tuned on a dataset like ANLI, that has a whole paragraph as a premise and a long sentence as a hypothesis, would struggle on this dataset. The premise \"unemployed\" does not necessarily contradict \"This person is employed\" since the subject is not even specified. It is not clear how useful would an NLI model fine-tuned on SDOH-NLI be for entailing SDOHs from unseen clinical notes considering the data, and would it be better to frame the problem as pure information extraction or information retrieval (in a form of similarity search).\nThe authors observe that the problem can naturally be framed as an information retrieval. This is true in the case when the whole clinical notes document, or a corpus of these documents, is used to determine SDOHs from a given query. But how exactly were these retrieval models fine-tuned for this specific dataset? Retrieval models are not meant to work on pairs of two short sentences at a time.", "questions_to_authors": "A: In line 89, what is the relation between the fact that there is a non-negligible fraction of positive entailment labels and coming from the same domain? \nB: In line 226, it is mentioned that the pair of factors (\"The person lives alone\" / \"The person does not live alone\") was removed from the training and validation sets. How did the models perform on this factor? \nC: Was the F1 score macro-averaged? Do you maybe have a confusion matrix or results showing the difference between positive and negative class performance, considering the heavy imbalance?\nD: In line 323, it is said that the dataset contains a lot of irrelevant hypotheses. Why was the dataset constructed in this way if it led to irrelevant hypotheses and noise?", "paper_topic_and_main_contributions": "The paper is a resource paper that introduces a new dataset, SDOH-NLI, for the task of recognizing social and behavioral determinants of health (SDOH) from clinical notes. The authors frame the extraction of SDOH characteristics as an NLI task, where snippets of social history are premises and SDOH characteristics are hypotheses. The authors claim the dataset is unique and unlike other NLI datasets since premises and hypotheses were collected independently one of another. The paper also tests the performance of baseline models on the newly developed dataset, including retrieval models and NLI models, in both zero-shot and fine-tuned settings. The results show that the dataset is challenging in case when NLI models are fine-tuned on existing NLI datasets, showing a big change in performance as opposed to fine-tuning on the new dataset.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gzDfMVldS9", "reviewer": "Reviewer_sNnf", "summary": "", "questions": "The reliance on human raters for assigning binary textual entailment labels might introduce bias or errors into the dataset, affecting its quality. Who are the human annotators? Are they experts?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review and for appreciating the importance of the problem and the effect our dataset could have in accelerating progress in the field. Let us try to address the reviewer’s critiques and questions.\n\n___\n\n> The paper doesn't elaborate on how privacy and regulatory constraints were navigated in the creation of this public dataset, which might raise some ethical and legal concerns.\n\nWe tried to briefly address this in the introduction and in our ethics statement. As we pointed out, a lot of previous research in the area relied on protected health information (PHI) from the electronic health records of particular health systems, preventing researchers from sharing their data publicly and making their results non-reproducible. By using already **fully public, non-PHI medical transcription samples**, we are able to share our dataset with the research community without jeopardizing patient privacy. (All of the text we used in our research is already publicly available on MTSamples.com.) The creation, usage, and planned release of this data set was approved by a rigorous cross-functional review process at our institution.\n\n___\n\n> Being a subjective domain, the Inter annotator agreement is expected to be low. Rater agreement was high, with a Krippendorff’s alpha of 0.97, seems to be unrealistic.\n\nWe believe that raters overall did a great job at correctly labeling our examples. We carefully examined the ratings; as we mentioned in Section 3, after careful inspection, we dropped three SDOH factors from our datasets that appeared too ambiguous and for which we weren’t satisfied with the quality of the ratings. Note also that Krippendorff’s alpha was computed over the binary entailment labels provided by different raters for (premise, hypothesis) pairs. The label imbalance in our dataset (i.e., that the vast majority of premises very clearly did not entail the corresponding hypotheses) also contributed to the high alpha.\n\nIn case it is helpful, here is a more detailed breakdown of rater agreement, grouped by social determinant category. For each category, the listed number is the proportion of snippets for which all three raters fully agree on the ratings for *all* the statements in that category.\n\n| Category            | Rater Agreement |\n|---------------------|-----------------|\n| Food                | 99.42%          |\n| Health insurance    | 99.02%          |\n| Financial situation | 98.56%          |\n| Transportation      | 97.47%           |\n| Alcohol use         | 90.11%           |\n| Smoking             | 89.37%           |\n| Employment          | 88.76%           |\n| Drug use            | 88.31%           |\n| Housing             | 87.85%           |\n| Social support      | 74.5%           |\n\nNote that these metrics were computed *before* we dropped a few ambiguous factors from the two categories with the lowest rater agreement (housing and social support).\n\n___\n\n> This work need a lot more details about the annotation guidelines, the data collection and the description on the nature of the dataset. As detailed description on how is it useful for stakeholders and what measures are taken for explanation and trustworthiness of model development is required.\n\nWe agree with the reviewer that there could be more discussion of these topics; the page limit for short papers imposes tight constraints on how much detail we can go into on any particular question. For the annotation guidelines, please see Appendix A which has the full instructions given to raters.\n\n___\n\n> The reliance on human raters for assigning binary textual entailment labels might introduce bias or errors into the dataset, affecting its quality. Who are the human annotators? Are they experts?\n\nWe have carefully inspected the dataset to make sure the labels are high quality. All of our raters were proficient in English but they were not required to be experts in the area. As we mentioned in Section 4.2, most of our examples describe everyday situations in plain English (e.g.,\"The patient is retired on disability due to her knee replacements\"), and our human raters without any domain-specific training had no difficulty understanding them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper tackles the vital issue of extracting SDOH from clinical notes, which is a significant step towards identifying opportunities for proper care and addressing disparities in healthcare.\n\nIt introduces a new dataset, SDOH-NLI, overcoming the hurdle of the lack of high-quality publicly available labeled data. This could accelerate the progress of NLP methods in this field.\n\nThe authors' innovative approach of formulating SDOH extraction as a natural language inference task could open new ways of dealing with such problems.\n\nThey provide an assessment of both off-the-shelf entailment models and models fine-tuned on their data, offering a comprehensive evaluation.", "reasons_to_reject": "The paper doesn't elaborate on how privacy and regulatory constraints were navigated in the creation of this public dataset, which might raise some ethical and legal concerns.\n\nBeing a subjective domain, the Inter annotator agreement is expected to be low. Rater agreement was high, with a Krippendorff’s alpha of 0.97, seems to be unrealistic. \n\nThis work need a lot more details about the annotation guidelines, the data collection and the description on the nature of the dataset. Ad detailed description on how is it useful for stakeholders and what measures are taken for explanation and trustworthiness of model development is required.", "questions_to_authors": "The reliance on human raters for assigning binary textual entailment labels might introduce bias or errors into the dataset, affecting its quality. Who are the human annotators? Are they experts?", "paper_topic_and_main_contributions": "The paper presents a new dataset, SDOH-NLI, which is designed to enhance the extraction of Social and Behavioral Determinants of Health (SDOH) from clinical notes using Natural Language Processing (NLP). It reformulates SDOH extraction as a natural language inference task, with binary textual entailment labels assigned by human raters. The dataset, which is based on clinical notes, is also made publicly available.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lbBt4hrObL", "reviewer": "Reviewer_Yo1y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review. Unfortunately we are not quite sure what the reviewer meant by the statement that “The experiments are not enough” or why they gave us a low rating on soundness; we would have appreciated more specific feedback. Our paper reports experiments in four different setups (zero-shot information retrieval, off-the-shelf NLI model, zero and few-shot prompting, and fine-tuning). Due to the page limit for a short paper, we weren’t able to report all our experimental results in the main paper, but Appendix D shows additional experimental results on 18 different fine-tuned models. If there are any specific experiments that the reviewer would like us to include, we'll be happy to consider reporting them in the paper. \n\nWe would also like to address the reviewer’s reproducibility rating. We take pride in publicly releasing our full dataset; it was included in our submission. Also, we exclusively used publicly available, open-source models, which should make it possible for anyone to reproduce and verify our results. For the zero/few-shot prompting experiments, we gave a detailed description of our setup in Appendix B. For the fine-tuning experiments, all relevant hyperparameters are listed in Appendix C. If there are any details still missing for the reviewer to be able to reproduce our results, we would be very happy to add them to the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper include the creation of a new dataset, SDOH-NLI, which can help healthcare providers identify opportunities to provide appropriate care and address disparities. The dataset is based on publicly available notes and is formulated as a natural language inference task, which makes it a valuable resource for researchers working on natural language processing techniques for healthcare applications. The paper also evaluates several state-of-the-art models on this dataset and provides insights into the performance of these models, which can help guide future research in this area.", "reasons_to_reject": "The experiments are not enough.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about the SDOH-NLI dataset, which is a new dataset that can help healthcare providers identify opportunities to provide appropriate care and address disparities. The paper addresses the problem of identifying social and behavioral determinants of health (SDOH) from clinical notes, which can be challenging due to the unstructured nature of these notes. The main contribution of the paper is the creation of the SDOH-NLI dataset, which is based on publicly available notes and is formulated as a natural language inference task. The paper also evaluates several state-of-the-art models on this dataset and provides insights into the performance of these models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces the SDOH-NLI dataset for recognizing social and behavioral determinants of health (SDOH) from clinical notes. The authors frame the problem as a natural language inference (NLI) task and also provide a baseline performance using multiple models. Reviews point to the relevance and novelty of the work, but also highlight several key concerns such as data imbalance, redundancy, and issues related to the study's design and methodology. All reviewers have provided a "Good" soundness score but are "Ambivalent" regarding their excitement. All reviewers agree on the paper's relevance in addressing an important issue in healthcare, specifically the extraction of SDOH factors from clinical notes. This could potentially aid in improving healthcare equity and decision-making. Reviewers 2 and 3 indicate a lack of detail about the annotation guidelines, data collection, and the utility of the dataset to stakeholders. Reviewer 2 expresses concern about the lack of discussion on navigating privacy and regulatory constraints in data collection, which is crucial given the sensitive nature of healthcare data. Given the dataset's potential value and the paper's overall soundness, the paper is recommended to be accepted into Findings. Authors are encouraged to address the listed concerns to enhance the paper's quality and impact.
Decision: Accept-Findings