SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The prediction target for this dataset is to determine whether an individual will vote in the U.S presidential election, from a detailed questionnaire.
Data Source We use data from the American National Election Studies (ANES) $^{10}$ . Since 1948, ANES has conducted surveys, usually administered as in-person interviews, during most years of national elections. This series of studies, known as the ANES "Time Series," constitutes a pre-election interview and a post-election interview during years of Presidential elections, along with other data sources. Topics cover voting behavior and the elections, together with questions on public opinion and attitudes.
We use features derived from the ANES Time Series. From the pool of over 500 questions in the ANES Time Series, we extract a set of features related to Americans' voting behavior, including their social and political attitudes, opinions about elected leaders, and media consumption habits.
Domain Shift We introduce a domain split by geographic region. We use the ANES Census Region feature, where the out-of-domain region is the region representing the southern United States (AL, AR, DE, D.C., FL, GA, KY, LA, MD, MS, NC, OK, SC,TN, TX, VA, WV). This simulates a study in which voter data is collected in one part of the country, and the goal is to infer voting behavior in another geographic region; this is a common occurrence with polling data, particularly during the U.S. primaries, which occur over a period of several weeks at the state level.
<sup>9</sup>https://www.census.gov/library/stories/2021/04/record-high-turnout-in-2020-general-election.html
10 https://electionstudies.org/
B.8 Childhood Lead Exposure
In this task, the goal is to identify children 18 or younger with elevated lead blood levels.
Background: Lead is a known environmental toxin that has been shown to affect deleteriously the nervous, hematopoietic, endocrine, renal, and reproductive systems<sup>11</sup>. In young children, lead exposure is a particular hazard because children more readily absorb lead than adults, and children's developing nervous systems also make them more susceptible to the effects of lead. However, most children with any lead in their blood have no obvious immediate symptoms.<sup>12</sup> The risk for lead exposure is disproportionately higher for children who are poor, non-Hispanic black, living in large metropolitan areas, or living in older housing.
The CDC sets a national standard for blood lead levels in children. This value was established in 2012 to be 3.5 micrograms per decileter  $(\mu \mathrm{g} / \mathrm{dL})$  of blood.[13] This value, called the blood lead reference value (BLRV) for children, corresponds to the 97.5 percentile and is intended to identify lead exposure in order to allow parents, doctors, public health officials, and communities to act early to reduce harmful exposure to lead in children. Thus, early prediction of childhood lead exposure, as well as accurate just-in-time prediction for children where obtaining actual laboratory blood test results is too costly or infeasible, is of high utility to many stakeholders.
Early detection of lead exposure can trigger many potentially impactful interventions, including: environmental and home analysis for early identification of sources of lead; testing and treatment for nutritional factors influencing susceptibility to lead exposure (such as calcium and iron intake); developmental analysis and support; and additional medical diagnostic tests.[14]
Using the laboratory blood test results from the NHANES (see 'Data Source' below), the task is to identify whether a respondents' blood level exceeds the BLRV using only questionnaire data. We use respondents of age 18 or younger as the target population (note that respondent data for ages 1-5 is restricted and thus not available to our benchmarking study). This simulates the prediction of expensive and time-consuming laboratory testing using a quick and inexpensive questionnaire. Laboratory testing is conducted by the CDC at the National Center for Environmental Health, Centers for Disease Control and Prevention, Atlanta, GA<sup>15</sup>
Data Source: The data are drawn from the CDC National Health and Nutrition Examination Survey (NHANES) $^{16}$ , a program of the National Center for Health Statistics (NCHS) within the Centers for Disease Control and Prevention (CDC). NHANES is a program of studies designed to assess the health and nutritional status of adults and children in the United States. The survey is unique in that it combines extensive interviews with physical examinations and high-quality laboratory testing. The NHANES interview includes demographic, socioeconomic, dietary, and health-related questions. The survey examines a nationally representative sample of about 5,000 persons each year. The examination component consists of medical, dental, and physiological measurements, as well as laboratory tests administered by highly trained medical personnel.
Findings from NHANES are used to determine the prevalence of major diseases and risk factors for diseases; to assess nutritional status and its association with health promotion and disease prevention; and are the basis for national standards for such measurements as height, weight, and blood pressure. Data from this survey are widely used in epidemiological studies and health sciences research.
We use only questionnaire-based NHANES features as the predictors, but use a prediction target from the NHANES' lab-based component. This simulates the development of a screening questionnaire to predict blood lead levels.
Distribution Shift: We use poverty as a domain-splitting variable. Children from low-income households and those who live in housing built before 1978 are at the greatest risk of lead exposure<sup>17</sup>.
<sup>11</sup>https://www.cdc.gov/Nchs/Nhanes/2017-2018/P_PBCD.htm
12https://www.cdc.gov/nceh/lead/prevention/blood-lead-levels.htm
<sup>13</sup>https://www.cdc.gov/nceh/lead/data/blood-lead-reference-value.htm
$^{14}$ https://www.cdc.gov/nceh/lead/advisory/acclpp/actions-bills.htm
<sup>15</sup>A detailed description of the methods and procedures used for laboratory testing for lead in the 2017-2018 NHANES survey is given at https://www.cdc.gov/Nchs/Nhanes/2017-2018/P_PBCD.htm; similar descriptions are available for each year of data collection.
16 https://www.cdc.gov/Nchs/Nhanes/
17https://www.cdc.gov/nceh/lead/prevention/populations.htm
However, due to factors mentioned above, impoverished populations can be less likely to be included in medical studies, including those that may involve in-person visits for blood laboratory testing, which is the primary method for lead exposure detection. We use the poverty-income ratio (PIR) measurement in NHANES. The PIR is calculated by dividing total annual family (or individual) income by the poverty guidelines specific to the survey year. The Department of Health and Human Services (HHS) poverty guidelines are used as the poverty measure to calculate this ratio. These guidelines are issued each year, in the Federal Register, for determining financial eligibility for certain federal programs, such as Head Start, Supplemental Nutrition Assistance Program (SNAP), Special Supplemental Nutrition Program for Women, Infants, and Children (WIC), and the National School Lunch Program. The poverty guidelines vary by family size and geographic location (with different guidelines for the 48 contiguous states and the District of Columbia; Alaska; and Hawaii).
The training domain is composed of individuals with PIR of at least 1.3; persons with  $\mathrm{PIR} \leq 1.3$  are in the held-out domain. The threshold of 1.3 is selected based on the PIR categorization used in NHANES, where  $\mathrm{PIR} \leq 1.3$  is the lowest level.
B.9 Hospital Readmission
Background: Effective management and treatment of diabetic patients admitted to the hospital can have a significant impact on their health outcomes, both short-term and long-term [83]. Several factors can affect the quality of treatment patients receive [81]. One of the costliest and potentially most adverse outcomes after a patient is released from the hospital is for that patient to be readmitted soon after their initial release; this can both be a sign of a condition that is not improving, and, at times, ineffective initial treatment. Thus, predicting the readmission of patients is a priority from both a medical and economic perspective.
In this task, the goal is to predict whether a diabetic patient is readmitted to the hospital within 30 days of their initial release.
Data Source: We use the dataset provided by  $[81]^{18}$ . The dataset represents 10 years (1999-2008) of clinical care at 130 US medical facilities, including hospitals and other networks. It includes over 50 features representing patient and hospital outcomes. The dataset includes observations for records which meet the following criteria: (1) It is an inpatient encounter (a hospital admission). (2) It is a diabetic encounter, that is, one during which any kind of diabetes was entered to the system as a diagnosis. (3) The length of stay was at least 1 day and at most 14 days. (4) Laboratory tests were performed during the encounter. (5) Medications were administered during the encounter.
The data contains such attributes as patient number, race, gender, age, admission type, time in hospital, medical specialty of admitting physician, number of lab test performed, HbA1c test result, diagnosis, number of medication, diabetic medications, number of outpatient, inpatient, and emergency visits in the year before the hospitalization, etc. We use the full set of features in the initial dataset, which is described in [81].
Distribution Shift: Patients can be (re)admitted to hospitals from a variety of sources. The source of a patient admission can be correlated with many demographic and other risk factors known to be related to health outcomes (e.g. race, income level, etc.).
We use the "admission source" as the domain split for TableShift. There are 21 distinct admission sources in the dataset, including "transfer from a hospital", "physician referral", etc. After conducting a sweep over various held-out values, we use "emergency room" as the held-out domain split. This matches a potential scenario where a model is constructed using a variety of admission sources, but a patient from a novel source is added; it is also possible e.g. that data from emergent patients could not be collected when training a readmission model. We note that this domain split provides 20 unique training subdomains (the other admission sources), which is the largest  $|\mathcal{D}^{\mathrm{train}}|$  in TableShift.
B.10 Sepsis
Background: Sepsis is a life-threatening condition that arises when the body's response to infection causes injury to its own tissues and organs. Sepsis is a major public health concern with significant
18 https://archive.ics.uci.edu/ml/datasets/Diabetes+130-US+hospitals+for+years+1999-2008
morbidity, mortality, and healthcare expenses; each year, 1.7 million adults in America develop sepsis, of which at least 350,000 die during their hospitalization or are discharged to hospice. The CDC estimates that 1 in 3 people who dies in a hospital had sepsis during that hospitalization $^{19}$ .
Early detection and antibiotic treatment of sepsis improve patient outcomes. While advances have been made in early sepsis prediction, there is a fundamental unmet clinical need for improved prediction [74]. The goal in this task is to predict, from a set of fine-grained ICU data (including laboratory measurements, sensor data, and patient demographic information), whether a patient will experience sepsis onset within the next 6 hours.
Data Source: We use the data source from the PhysioNet/Computing in Cardiology Challenge [74], which was designed by clinicians and other healthcare experts to facilitate the development of automated, open-source algorithms for the early detection of sepsis from clinical data. The dataset is derived from ICU patient records for over 60,000 patients from two hospitals with up to 40 clinical variables collected during each hour of the patient's ICU stay.
Distribution Shift: We explored multiple domain shifts for this dataset; we note that, in particular, splitting domains by hospital did not lead to a shift gap for tuned baseline models (although there is a third, held-out hospital that was used in the original challenge for this dataset, it is not publicly available and is not part of the TableShift benchmark). Instead, we use "length of stay" as a domain shift variable. We bifurcate the dataset based on how long a patient has been in the ICU, with patients having been in ICU for  $\leq 47$  hours in the training domain, and patients having been in ICU more than 47 hours in the test domain. This matches a scenario where a medical model is trained only on observed stays of a fixed duration (no more than two full days), but then used beyond its initial observation window to predict sepsis in patients with longer stays. We note that length of stay of 47 hours corresponds to the 80th percentile of the data for that feature.
B.11 ICU Patient Length-of-Stay
Background: According to [72], length of hospital stay is, along with patient mortality, "the most important clinical outcome" for an ICU admission. Accurately predicting the length of stay of a patient can aid in assessment of the severity of a patient's condition. Of particular clinical relevance, making these predictions early and with a non-zero time gap between the prediction and the outcome is of real-world importance: predictions must be made sufficiently early such that a patient's treatment can be adjusted to potentially avoid a negative outcome. The importance of this prediction task for real-world clinical care is underscored by the many previous works in the medical literature addressing this prediction topic (see e.g. [40, 72, 87]).
In our benchmark, the specific task is to predict, from the first 24 hours of patient data, an ICU patient's stay will exceed 3 days (a binary indicator for whether length of stay  $>3$ ). We note that this is directly adopted from MIMIC-extract.
Data Source: We use the MIMIC-extract dataset [87]. MIMIC-extract is an open-source pipeline for transforming raw electronic health record (EHR) data from the Medical Information Mart for Intensive Care (MIMIC-III) dataset [45].
MIMIC-III, the underlying data source, captures over a decade of intensive care unit (ICU) patient stays at Beth Israel Deaconess Medical Center in Boston, USA. An individual patient might be admitted to the ICU at multiple times in the dataset; however, MIMIC-extract focuses on each subject's first UCI visit only, since those who make repeat visits typically require additional considerations with respect to modeling and care [87]. MIMIC-extract includes all patient ICU stays in the MIMIC-III database that where the following criteria are met:  $(i)$  the subject is an adult (age of at least 15 at time of admission),  $(ii)$  the stay is the first known ICU admission for the subject, and  $(iii)$  the total duration of the stay is at least 12 hours and less than 10 days.
MIMIC-extract is designed by EHR domain experts with clinical validity (of data) and relevance (of prediction tasks) in mind. In addition to the filtering described above, MIMIC-extract's pipeline includes steps to standardize units of measurement, detect and correct outliers, and select a curated set of features that reduce data missingness in the preprocessed data; for details on the steps taken by the original authors to achieve this, see [87]. We use the preprocessed version of MIMIC-extract made
$^{19}$ https://www.cdc.gov/sepsis/what-is-sepsis.html
available by the authors  $^{20}$ . This includes the static demographic variables, alongside the time-varying vitals and labs described in [45]. Because event he preprocessed data contains missing values, we use the authors' default methods for handling missing data.
The resulting dataset contains approximately 24,000 observations.
Distribution Shift: We split the domains by health insurance type. We train on patients with all insurance types except Medicare, and use patients with Medicare insurance as the target domain.
B.12 ICU Patient In-Hospital Mortality
Background: As discussed in the background of  $\S B.11$ , hospital mortality is considered to be one of the most important outcomes for ICU patients. The clinical relevance of hospital mortality is perhaps even more clear than for length-of-stay prediction, as preventing patient mortality is one of the primary goals for many patients. Again, as discussed in  $\S B.11$ , making this prediction early is of particular importance, as early predictions can provide a proxy for overall patient risk and can be used to intervene to avoid mortality.
We note that in this task, we are predicting hospital morality (that the patient dies at any point during this visit, even if they are discharged from the ICU to another unit in the hospital). Hospital mortality events are distinct from (and a superset of) ICU mortality events. As mentioned above, the importance of this prediction task for real-world clinical care is underscored by the many previous works addressing this prediction topic (see e.g. [40, 72, 87].
Data Source: This task uses the same data source and feature set from MIMIC-extract described above in §B.11.
Distribution Shift: We split the domains by health insurance type. We train on patients with all insurance types except {Medicare, Medicaid} and use patients with {Medicare, Medicaid} insurance as the target domain.
B.13 FICO Home Equity Line of Credit (HELOC)
Background: FICO (legal name: Fair Isaac Corporation) is a US-based company that provides credit scoring services. The FICO score, a measure of consumer credit risk, is a widely used risk assessment measure for consumer lending in the United States.
The Home Equity Line of Credit (HELOC) is a line of credit, secured by the applicant's home. A HELOC provides access to a revolving credit line to use for large expenses or to consolidate higher-interest rate debt on other loans such as credit cards. A HELOC often has a lower interest rate than some other common types of loans. To assess an applicant's suitability for a HELOC, a lender evaluates an applicant's financial background, including credit score and financial history. The lender's goal is to predict, using this historical customer information, whether a given applicant is likely to repay a line of credit and, if so, how much credit should be extended.
In addition to desiring accurate credit risk predictions for their overall utility for both lenders and borrowers, lending institutions are incentivized (and, in some cases, legally required) to use models which achieve some degree of robustness: institutions can face severe penalties when borrowers are not treated equitably (e.g. [84]).
Data Source: We use the dataset from the FICO Community Explainable AI Challenge $^{21}$ , an open-source dataset containing features derived from anonymized credit bureau data. The binary prediction target is an indicator for whether a consumer was 90 days past due or worse at least once over a period of 24 months from when the credit account was opened. The features represent various aspects of an applicant's existing financial profile, including recent financial activity, number of various transactions and credit inquiries, credit balance, and number of delinquent accounts.
Distribution Shift: It is widely acknowledged that the dominant approach to credit scoring using financial profiles can unintentionally discriminate against historically marginalized groups (credit bureau data do not include explicit information about race [58]). For example, since FICO scores are
20The publicly-accessible dataset (which requires credentialed MIMIC-III access through PhysioNet due to privacy restrictions) is described at https://github.com/MLforHealth/MIMIC_Extract
$^{21}$ https://community.fico.com/s/explainable-machine-learning-challenge
based on payment history and credit use and many marginalized groups in the United States have lower or less reliable incomes, these marginalized groups can suffer from systematically lower credit scores [60, 71, 8, 58]; this has been referred to as the "credit gap" [49, 22]. In particular, debt and savings level play a role in credit scores and can systematically disadvantage Black and Hispanic applicants, even when demographic data are not formally used in the credit rating process [60, 58].
For this task, we partition the dataset based on the 'External Risk Estimate', a feature in the dataset corresponding to the risk estimate assigned to an applicant by a third-party service. This estimate was identified in the original FICO explainable ML challenge  $^{22}$ . We use individuals with a high external risk estimate (where "high" estimate is defined as exceeding an external risk estimate of 63, a threshold identified in the original challenge-winning model linked above) as the training domain, and individuals with estimate  $\leq 63$  as the held-out domain.
B.14 College Scorecard Degree Completion Rate
Background: Higher education is increasingly critical to securing strong job and income opportunities for persons in the United States. At the same time, the cost of obtaining a four-year college degree is extremely high: The average cost of college* in the United States is \(35,551 per student per year, including books, supplies, and daily living expenses and this cost has more than doubled in the 21st century alone, with an annual growth rate of \(7.1\%\) [39].
However, not all institutions have similar outcomes for students. Graduation rates across institutions in the U.S. vary widely, and failure to complete a degree can leave a student with significant debt and a reduced ability to repay it. Understanding factors related to degree completion is an area of active research.
For this task, our goal is to predict whether an institution has a low completion rate, based on other characteristics of that institution. While the definition of a "low" completion rate is ultimately subjective and context-dependent, we use a threshold of  $50\%$ , which is approximately equivalent to the median graduate rate across the institutions in the dataset. We use the completion rate for first-time, full-time students at four-year institutions (150% of expected time to completion/6 years).
Data Source: We use the College Scorecard $^{23}$ . The College Scorecard is an institution-level dataset compiled by the U.S. Department of Education from 1996-present. The College scorecard includes detailed institutional factors, including information about each institutions' student population, course offerings, and outcomes.
Distribution Shift: Institutions vary widely in their profiles, student populations, educational approach, and target industries or student pathways. We partition universities according to the CCBASIC variable $^{24}$ , which gives the Carnegie Classification (Basic) $^{25}$ . This classification uses a framework developed by the Carnegie Commission on Higher Education in the early 1970s to support its research program. Partitioning our data according to this variable measures the robustness over institutional subpopulations, and is thus a form of subpopulation shift. We use the following set of institutions as the target domain (all other institutional types are in the training domain): 'Special Focus Institutions-Other special-focus institutions', 'Special Focus Institutions-Theological seminars, Bible colleges, and other faith-related institutions', "Associate's-Private For-profit 4-year Primarily Associate's", 'Baccalaureate Colleges-Diverse Fields', 'Special Focus Institutions-Schools of art, music, and design', "Associate's-Private Not-for-profit", "Baccalaureate/Associate's Colleges", "Master's Colleges and Universities (larger programs)". Exact definitions of each institution class are available via the Carnegie Commission on Higher Education $^{26}$ .