SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Finally, we evaluate the trained models on the test splits of each dataset. As recommended in [50], we use in-domain and out-of-domain test accuracy (not in-domain train accuracy) to evaluate the models. For all results shown, we use the best model selected according to (in-domain) validation accuracy; this follows the selection procedure used to study domain generalization in the image domain in [38].
ID and OOD Accuracy are Correlated. Our results show that, across all models and tasks, indistribution (ID) and out-of-distribution (OOD) accuracy are correlated: as ID performance improves, OOD performance also tends to improve (see Figure 1;  $\rho = 0.81$ ). This linear trend holds across datasets and model classes. We note that, while this is consistent with findings for image [62] and question answering [61] models, the relationship between ID accuracy and OOD accuracy on tabular data was previously unknown. This result suggests that, for a wide variety of tabular data tasks, improving models' ID performance is likely to improve their OOD performance.
No Model Consistently Outperforms Baselines. While many models have been proposed for both (a) improving general performance on tabular data tasks over established baselines such as XGBoost and LightGBM, and (b) improving robustness to distribution shift, our results show that no model consistently outperforms the standard tabular baselines of XGBoost, LightGBM, or CatBoost in either respect. Figure 4a shows that, on average across all datasets, no model consistently achieves better performance (as measured as a fraction of the maximum OOD accuracy achieved by any model) compared to baseline methods. This finding has not been previously demonstrated in tabular data due to the lack of an existing benchmark.
No Method Eliminates Gaps. We investigate the empirical performance of several methods designed to improve robustness to distribution shift (described in Section 4.1). Our results show that, on the datasets where multiple training subdomains are available (and thus where domain generalization is viable), there is weak evidence that several techniques reduce gaps due to distribution shift, but no technique eliminates these gaps. However, it is important to note that this gap reduction comes at the cost of in-distribution performance: as Figure 4b shows, all robustness-enhancing models tend to shrink gaps by reducing average ID performance, not by improving OOD performance. This is shown in Figure 4b by the two parallel lines: one set of blue points representing baselines + tabular NNs, and another, shifted left, representing robustness-engancing and domain generalization models. Furthermore, we note that all domain generalization and domain robustness methods evaluated (excluding DRO) require additional information that is only present for some datasets – namely, the discrete variable over which a shift will occur (e.g. “race” for diabetes task) and data from at least 2 categories of this variable.
Change in label distribution is correlated with shift gap. We investigate the degree to which the three factors mentioned previously  $(p(x), p(y|x), p(y))$  are related to model performance. Our results, in Figures 5 and 8, show that change in the label distribution  $\Delta_y$  is correlated with shift gap
Figure 2: Results for baselines, robust learning, and domain generalization models across the 15 TableShift benchmark tasks. The  $y = x$  line indicates a model with no shift gap,  $\Delta_{\mathrm{Acc}} = 0$  (see Equation 1). Clopper-Pearson confidence intervals at  $\alpha = 0.05$  shown for all points. Note that domain generalization models are only used on domain generalization tasks (cf. Table 1). Results for the remaining TABLESHIFT tasks are shown in Figure 3. For exact results see Section E.3.
Figure 3: Additional results (cf. Figure 2). For exact results see Section E.3.
(b)
Figure 4: (a): Percentage of Maximum OOD Accuracy (PMA-OOD) across tasks (see Table 14 for exact values). \*: domain generalization models and Group DRO can only be trained on the subset of 10 tasks with multiple training subdomains (see "Domain Generalization" column in Table 1). (b): Average ID and OOD accuracy by model across domain generalization tasks only. We only show domain generalization tasks in order to compare all models on the same set of tasks. See Figure 9 for results on all tasks. Exact values in Table 15.
$\Delta_{\mathrm{Acc}}$  (Pearson correlation  $\rho = 0.71$ ). This persists even after accounting for ID accuracy: a simple linear regression of OOD accuracy on [ID accuracy,  $\Delta_y$ ] obtains  $R^2 = 0.996$ . This suggests that the change in the label distribution is an important factor in understanding tabular shifts (for example, the outliers in Figure 1 are from the four tasks with largest label shift: Public Coverage, HELOC, ASSISTments, College Scorecard; see Figures 2, 3 and Table 3). Label shift robustness methods in our study did not eliminate performance gaps under shift; in fact, label shift robustness methods often degraded both ID and OOD accuracy (e.g. Figure 4b). We provide similar analyses relating shift gap to  $(i)$  covariate shift and  $(ii)$  concept shift in Figure 8, but find that they are not clearly related.
Changes in predictions are related to covariate shift. As an exploratory finding, we find some evidence that changes in the predictions for OOD data are correlated with changes in  $p(x)$ , shown in Figure 7a ( $\rho = 0.99$ ). This suggests that shift gaps in the benchmark datasets not explained by the combination of ID accuracy and  $\Delta_y$  may be driven primarily by covariate shift (changes in  $p(x)$ ) as opposed to concept shift (changes in  $p(y|x)$ ). Further analysis is needed to confirm this exploratory finding. We note that relationships between other forms of shift showed much weaker correlation, roughly  $\rho \approx -0.2$  (see Figure 7).
The conclusions in this study are limited to the specific datasets and models evaluated. While we intentionally selected a diverse suite of benchmark datasets along several axes (domain, distribution shift, dataset size, etc.), our conclusions can only be extended to other distribution shifts insofar as they are similar to the shifts in TABLESHIFT. More empirical validation is needed, including studies comparing our findings to other tabular shifts.
Figure 5: Label shift  $(\Delta_y$ , measured via Equation (3)) and absolute shift gap  $\Delta_{\mathrm{Acc}}$  show moderate correlation across datasets and models (Pearson correlation  $\rho = 0.70$ ). Exact  $\Delta_y$  values in Table 3.
Our work does not exhaustively cover the space of all possible tabular data classifiers. In particular, "hybrid" methods combining some of the loss-based robust-
ness interventions (i.e. Group DRO) with various tabular data-specific model architectures (e.g. FT-Transformer, ResNet) might lead to different results. Our initial exploratory evaluation of hybrid methods (see Section E.5), however, does not suggest that hybrid methods led to qualitative changes in our results, but these methods warrant a more extensive evaluation. Finally, our work does not establish theoretical connections between the factors analyzed (ID accuracy, OOD accuracy,  $\Delta_y$ ).
We introduce the TABLESHIFT benchmark for studying distribution shift in tabular data. TABLESHIFT presents a diverse set of tasks for reliable study and benchmarking of tabular data models under distribution shift. We provide a Python API to access the datasets, along with implementations of several models including baselines, distributionally robust learners, and domain generalization methods. Finally, we present empirical results which form the first large-scale study of tabular data modeling under distribution shift.
Our results suggest multiple potential avenues for future work: First, improvements to in-distribution accuracy are likely to drive OOD accuracy gains. Second, improved robustness to label shift may reduce shift gaps. Third, hybrid methods which combine robustness-enhancing losses (such as Group DRO) with improved neural network architectures may be able to further improve OOD performance. Beyond these proposed directions, we hope that TableShift opens new research frontiers for tabular machine learning research beyond those addressed in the current work.
A Acknowledgements
This work was supported by a Microsoft Grant for Customer Experience Innovation. This work was also in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML, CCF-2019844), Google, Open Philanthropy, and the Allen Institute for AI.
Our research utilized computational resources and services provided by the Hyak computing cluster at the University of Washington.
B Benchmark Task Details
This section provides details on each of the benchmark tasks in TableShift. While we describe the data source for each task, we emphasize that TableShift does not host or distribute the data; each data source is publicly available (some require training or authorization, but all are available to the public). Based on our review of the datasets, we believe that the datasets do not contain personally identifiable information, offensive content, or proprietary information. For data collected from human subjects, the conditions of collection and the ethics approval under which the data were collected are described in the documentation associated with each dataset.
B.1 Food Stamps
**Background:** Food insecurity is a problem affecting more than  $10\%$  of households (13.5 million) across the United States in 2021<sup>4</sup>. Various programs exist to provide families and individuals with supplemental income to reduce food insecurity. However, diminished social support services in many U.S. states limit the ability of outreach providers to ensure all eligible individuals are receiving available benefits. Low-cost, low-friction screening tools powered by machine learning models might provide useful information whether an individual is receiving food stamps in order to identify likely candidates for both food security programs ("food stamps") and as a proxy for eligibility and need for additional support services.
Data Source: We use person-level data from the American Community Survey (ACS) $^5$ . We filter the data for low-income adults aged 18-62 (i.e. selecting only adults below the social security eligibility age) in households with at least one child in the household. We use an income threshold of $30000 based on the U.S. poverty threshold for a family with one child.
Distribution Shift: In the United States, food stamps programs are managed at the state level. We apply domain shift over states, at the regional level. Specifically, we use the ACS census region as the split. The ACS includes 10 regions, which are: Puerto Rico; New England (Northeast region); Middle Atlantic (Northeast region); East North Central (Midwest region); West North Central (Midwest region); South Atlantic (South region); East South Central (South region); West South Central (South Region); Mountain (West region); Pacific (West region). We use East South Central (South region) as the holdout domain for this task.
This split parallels the case where a system is trained on a subset of states in a specific geographic area (perhaps in a localized study that draws participants or respondents from some geographic areas, but excludes other areas), and then applied to another. It also parallels the case where there is an interest in simulating the effect of a policy change. Finally, it mirrors the challenge of predicting an effect of a policy outcome (food stamps eligibility/recipiency) where differences in the underlying policy (different programs or eligibility across states) are a confounder.
B.2 Income
**Background:** Income is a widely-used measure of social stability. In addition, income is often used as a criteria for various social support programs. For example, in the United States, income is used to measure poverty, and can be used to determine eligibility for various social services such as food stamps and medicaid. Income prediction has obvious commercial utility. Finally, income prediction
<sup>4</sup>https://www.ers.usda.gov/topics/food-nutrition-assistance/food-security-in-the-us/skey-statistics-graphics/
<sup>5</sup>https://www.census.gov/programs-surveys/acs/about.html
has a rich and unique history in the machine learning community, dating back to the "adult income" census dataset [51, 24].
Data Source: We use person-level data from the American Community Survey (ACS), as described in Task B.1. However, for the income prediction task, we use different filtering. We use the filtering described in [24], which filters for adults aged at least 16 years old, who report working more than zero hours in the past month with reported income at least  $100.00. We use an income threshold of$ 56,000, which is the median income, as in [33].
Distribution Shift: Income patterns can vary in many ways. Here, we focus on domain shift at the regional level. We use the same splitting variable (US Census Region) described in Task B.1. However, for the income prediction task, we use New England (Northeast region) as the held-out domain.
B.3 Public Coverage
Background: People use health-care services to diagnose, cure, or treat disease or injury; to improve or maintain function; or to obtain information about their health status and prognosis [67]. In the United States, health insurance is a critical component of individuals' ability to access health care. Public health insurance exists, among other reasons, to provide affordable and accessible health insurance options for individuals not willing or able to purchase insurance through the private insurance market. However, not all individuals have health insurance; only  $88\%$  of individuals in the U.S had health insurance in 2019 according to the National Health Interview Survey (NHIS). Increasing the proportion of people in the United States with health insurance is one of the four healthcare objectives of the U.S. Department of Health and Human Services "Healthy People 2030" initiative<sup>6</sup>. In this task, the goal is to predict whether an individual is covered by public health insurance.
Data Source: We use person-level data from the American Community Survey (ACS), as described in Task B.1. However, for this task, we filter the data to include only low-income individuals (those with income less than \(30,000) who are below the age of 65 (at which age all persons in the United States are covered by Medicare). This is the same filtering used in [24, 33].
Distribution Shift: Many factors can influence individuals' ability to access or utilize health insurance and healthcare services. These include spoken language skills, mobility (whether an individual has recently relocated), education, ease of obtaining services, and discriminatory practices among providers [67]. We focus on disability status, as this is a widely-known factor in obtaining access to adequate health care [67]. Disability is also a particularly realistic factor in that disability status is likely to contribute to nonresponse to certain forms of data collection for many tabular data sources (including the four methods used to collect the ACS data: internet, mail, telephone, and in-person interviews) that can disadvantage persons with certain disabilities and decrease likelihood of participation or cause them to be excluded from study population.
For this task, the holdout domain  $\mathcal{D}^{\mathrm{test}}$  consists of persons with disabilities; the training domain  $\mathcal{D}^{\mathrm{train}}$  consists of persons who do not have disabilities. This simulates a situation where data collection practices excluded disabled persons, potentially through the factors described above.
B.4 ACS Unemployment
Background: Unemployment is a key macroeconomic indicator and a measure of individual wellbeing. Unemployment is also linked to a variety of adverse outcomes, including socioeconomic, psychological, and health impacts [10, 16, 14, 63].
Data Source: We use person-level data from the American Community Survey (ACS), as described in Task B.1. However, for this task, we filter the data to include only individuals over the age of 18 and below the age of 62 (at which age persons in the United States are eligible to receive Social Security income).
Distribution Shift: Many factors are known to be related to unemployment. We focus on a form of subpopulation shift, and use education level as the domain split. We use individuals with educational
<sup>6</sup>https://health.gov/healthypeople/objectives-and-data/browse-objectives/health-care-access-and-quality/increase-proportion-people-health-insurance-ahs-01
attainment of GED (high school diploma equivalent) or higher as the training population  $\mathcal{D}^{\mathrm{train}}$ , and individuals without high school-level education as  $\mathcal{D}^{\mathrm{test}}$ . This simulates a survey collection with a biased sample that systematically excludes such persons.
B.5 Diabetes
Background: Diabetes is a chronic disease that affects at least 37.7 million people in the United States (11.3% of the U.S population); it is estimated that an additional 96 million adults have prediabetes.[7] Diabetes increases the risk of a variety of other health conditions, including stroke, kidney failure, renal complications, peripheral vascular disease, heart disease, and death. The economic cost of diabetes is also significant: The total estimated cost of diagnosed diabetes in 2017 is $327 billion [7]. Care for people with diagnosed diabetes accounts for 1 in 4 health care dollars in the U.S. – more than half of that expenditure is directly attributable to diabetes [7].
Early detection of diabetes thus stands to have a significant impact, allowing for clinical intervention and potentially reducing the prevalence of diabetes. Further, even prediabetes is acknowledged to have significant impacts both on health outcomes and quality of life [7], and early detection if high diabetes risk could serve to identify prediabetic individuals. There exists a considerable prior literature on models for early diabetes prediction, e.g. [88, 66, 42]
Data Source: We use data provided by the Behavioral Risk Factors Surveillance System (BRFSS) $^{8}$ . BRFSS is a large-scale telephone survey conducted by the Centers of Disease Control and Prevention. BRFSS collects data about U.S. residents regarding their health-related risk behaviors, chronic health conditions, and use of preventive services. BRFSS collects data in all 50 states as well as the District of Columbia and three U.S. territories. BRFSS completes more than 400,000 adult interviews each year, making it the largest continuously conducted health survey system in the world. BRFSS annual survey data from 2017-2021 is currently available from the CDC.
The BRFSS is composed of three components: 'fixed core' questions, asked every year, 'rotating core', asked every other year, and 'emerging core'. Since some of our features come from the rotating core, we only use every-other-year data sources; otherwise many features would be empty for the intervening years.
For the Diabetes prediction task, we use a set of features related to several known indicators for diabetes derived from [88]. These risk factors are general physical health, high cholesterol, BMI/obesity, smoking, the presence of other chronic health conditions (stroke, coronary heart diseases), diet, alcohol consumption, exercise, household income, marital status, time since last checkup, education level, health care coverage, and mental health. For each risk factor, we extract a set of relevant features from the BRFSS foxed core and rotating core questionnaires. We also use a shared set of demographic indicators (race, sex, state, survey year, and a question related to income level). The prediction target is a binary indicator for whether the respondent has ever been told they have diabetes.
Distribution Shift: While diabetes affects a large fraction of the overall population, diabetes risk varies according to several demographic factors. One such factor is race/ethnicity [42, 17], with all other race-ethnicity groups reported in the 2022 CDC National Diabetes Statistics Report displaying higher risk than 'White non-Hispanic' individuals[17]. Compounding this issue, it has been widely acknowledged that health study populations tend to be biased toward white European-Americans [23, 68, 26, 44]. As a result, these studies have tended to focus on risk factors affecting white populations at the expense of identifying risk factors for nonwhite populations [44], despite distinct differences in how these populations are affected by various disease risk factors, differences in individuals' genetic factors, and differences in how they respond to medication across racial and ethnic populations. This disparity is a contributing factor to race-based disparities in treatment for diabetes [21].
In order to simulate the domain gap induced by these real-world differences in study vs. deployment populations, we partition the benchmark task by race/ethnicity. We use "White non-Hispanic"-identified individuals as the training domain, and all other race/ethnicity groups as the target domain.
<sup>7</sup>https://www.cdc.gov/diabetes/health-equity/diabetes-by-the-numbers.html
<sup>8</sup>https://www.cdc.gov/brfss/index.html
B.6 Hypertension
Background: Hypertension, or systolic blood pressure (typically systolic pressure  $130\mathrm{mmHg}$  or higher or diastolic 80 or higher) affects nearly half of Americans [3]. Hypertension is sometimes called a "silent killer" because in most cases, there are no obvious symptoms of hypertension [3]; this would make an accurate at-risk model of hypertension useful. When left untreated, hypertension is associated with the strongest evidence for causation of all risk factors for heart attack and other cardiovascular disease [32]. Hypertension also increases the risk of stroke, kidney damage, vision loss, insulin resistance, and other adverse outcomes [4]. While existing tools have attempted to predict blood pressure without the use of a cuff (the gold-standard measurement of blood pressure), these tools are still significantly less accurate (see e.g. [77, 28]), and there is an ongoing need for effective blood pressure measurement.
Data Source: We use BRFSS as the raw data source, as described in Task B.5 above. However, for the hypertension prediction task, we use features related to the following set of risk factors for hypertension via [64]: Age, family history and genetics, other medical conditions (e.g. diabetes, various forms of cancer), race/ethnicity, sex, and social and economic factors (income, employment status). We collect all survey questions related to these risk factors and use them as the predictors for this task, along with a shared set of demographic indicators (race, sex, state, survey year, and a question related to income level).
Distribution Shift: We use BMI category as the domain splitting variable. Individuals with BMI identified as "overweight" or "obese" are in the held-out domain, and those identified as "underweight" or "normal weight" are in the training domain. This simulates a model being deployed under subpopulation shift, where the target population has different (higher) BMI than the training population.
B.7 Voting
Background Understanding participation in elections is a critical task for policymakers, politicians, and those with an interest in democracy. In the 2020 United States presidential election, for example, voter turnout reached record levels, but it is estimated that only  $66.8\%$  of eligible individuals voted according to the U.S. Census<sup>9</sup>. Additionally, so-called "likely voter models," that predict which individuals will vote in an electio, are widely acknowledged as critical to polling and campaigning in U.S. politics. Predicting whether an individual will vote is notoriously difficult; one reason for this challenge is that domain shift is a fundamental reality of such modeling (presidential elections only occur every four years, after which significant political and demographic changes occur prior to the next presidential election).