SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 17: Hybrid method results. We compare Group DRO to standard ERM for the highest-performing Transformer, tree-based, and baseline models in our study (FT-Transformer, NODE, and ResNet, respectively) over five TABLESHIFT tasks, following our hyperparameter tuning procedure. There is little or no evidence that Group DRO reduces shift gaps for these models, indicating that Group DRO may primarily improve weak (MLP) models but does not improve robustness for stronger models. See Section E.5 for details on experimental design.
Task	Base Estimator	Method	ID Test Accuracy	OOD Test Accuracy
Value	95% CI	Value	95% CI
Childhood Lead	FT-Transformer	ERM	0.971	(0.961, 0.979)	0.920	(0.915, 0.925)
Group DRO	0.971	(0.961, 0.979)	0.920	(0.915, 0.925)
NODE	ERM	0.971	(0.961, 0.979)	0.920	(0.915, 0.925)
Group DRO	0.971	(0.961, 0.979)	0.920	(0.915, 0.925)
ResNet	ERM	0.971	(0.961, 0.979)	0.920	(0.915, 0.925)
Group DRO	0.971	(0.961, 0.979)	0.920	(0.915, 0.925)
College Scorecard	FT-Transformer	ERM	0.948	(0.944, 0.952)	0.859	(0.839, 0.877)
Group DRO	0.935	(0.93, 0.939)	0.815	(0.793, 0.835)
NODE	ERM	0.944	(0.939, 0.948)	0.844	(0.823, 0.863)
Group DRO	0.946	(0.942, 0.95)	0.835	(0.814, 0.854)
ResNet	ERM	0.947	(0.943, 0.951)	0.854	(0.834, 0.872)
Group DRO	0.947	(0.942, 0.95)	0.824	(0.803, 0.844)
Food Stamps	FT-Transformer	ERM	0.843	(0.841, 0.846)	0.816	(0.812, 0.819)
Group DRO	0.826	(0.823, 0.829)	0.795	(0.792, 0.799)
NODE	ERM	0.849	(0.847, 0.852)	0.822	(0.819, 0.825)
Group DRO	0.845	(0.842, 0.847)	0.822	(0.819, 0.825)
ResNet	ERM	0.843	(0.84, 0.845)	0.820	(0.817, 0.824)
Group DRO	0.848	(0.846, 0.851)	0.818	(0.815, 0.822)
Hypertension	FT-Transformer	ERM	0.666	(0.661, 0.672)	0.604	(0.603, 0.605)
Group DRO	0.665	(0.659, 0.67)	0.608	(0.607, 0.609)
NODE	ERM	0.670	(0.664, 0.676)	0.597	(0.596, 0.599)
Group DRO	0.671	(0.665, 0.676)	0.592	(0.591, 0.593)
ResNet	ERM	0.667	(0.661, 0.672)	0.608	(0.606, 0.609)
Group DRO	0.663	(0.658, 0.669)	0.590	(0.589, 0.592)
Voting	FT-Transformer	ERM	0.879	(0.848, 0.906)	0.855	(0.841, 0.868)
Group DRO	0.894	(0.865, 0.919)	0.858	(0.844, 0.87)
NODE	ERM	0.885	(0.854, 0.911)	0.851	(0.838, 0.864)
Group DRO	0.898	(0.869, 0.923)	0.860	(0.847, 0.873)
ResNet	ERM	0.887	(0.856, 0.912)	0.836	(0.822, 0.849)
Group DRO	0.898	(0.869, 0.923)	0.847	(0.833, 0.861)
F Model Details
This section describes the models used in our study. For the hyperparameters used in our experiments, see Section I.
Our implementations of these models, along with associated code to train models with fixed hyperparameters or to tune hyperparameters at scale via the Ray framework, are available at https://github.com/mlfoundations/tableshift.
F.1 Baseline Models
XGBoost: XGBoost is a popular library for learning gradient-boosted trees. We use the original XGBoost implementation [20]. XGBoost introduced column subsampling, weight regularization, and introduced major improvements in efficiency for training gradient boosted models on large or out-of-core datasets.
LightGBM: LightGBM is a library for learning gradient-boosted trees which extends the success of XGBoost in working fast and with large datasets [48]. LightGBM introduces novel techniques such as converting continuous features to histograms (for computational efficiency and for to reduce overfitting), combining certain features using Exclusive Feature Bundling (EFB), and through the use of Gradient-based One-Side Sampling (GOSS).
CatBoost: CatBoost [25] is a library for learning gradient-boosted trees which includes novel techniques for leveraging categorical features. This includes heuristics to replace numeric or one-hot encoding of categorical features with label-derived heuristics; "appearance" (count) features for categorical features; and efficient greedy feature recombination techniques.
MLP: We use standard multilayer perceptrons, via the implementation in RTDL $^{29}$ . MLPs have been shown to be highly effective models for tabuilar data, particularly when a large model search space is used and regularization is carefully tuned [46].
F.2 Tabular Neural Networks
FT-Transformer: FT-Transformer is a transformer-based model that learns separate feature tokenizers for numeric and categorical data, and applies a transformer model [85] to the tokenized features.
Tabular ResNet: We use the version of Tabular ResNet proposed in [36]. We note that, despite the fact that this approach is shown to have competitive performance with many existing tabular data models in [36], it has not been widely used in the literature.
NODE Neural Oblivious Decision Ensembles (NODE) [70] is a method that leverages oblivious ensembling methods to train "tree-like" neural networks.
TabTransformer: TabTransformers [43] is a model that uses learned embeddings of categorical features, which are then passed through standard Transformer layers, alongside layer normalization of continuous features.
SAINT: SAINT [79] uses an enhanced embedding method for categorical features, alongside (optional) attention over both rows and columns, in a Transformer architecture. We note that, due to its use of featurewise feedforward layers, SAINT was impractical to use for our datasets with the largest numbers of features (ICU Hospital Mortality, ICU Length of Stay; both contain over 1000 features which resulted in over 13B parameters for even a single-layer SAINT model).
F.3 Robustness Models
Distributionally Robust Optimization (DRO): We use two variants of DRO, both via [53]. For both methods, the model attempts to optimize a worst-care risk within a bounded distribution of the training data via a projected gradient descent procedure.
Group DRO: Originally introduced as a subgroup robustness method in [76], Group DRO is a DRO method which attempts to optimize the worst-group loss during training. Group DRO can also,
29https://github.com/Yura52/rtdl
however, be used as a domain robustness method by treating the domains as "group labels", which is how we use it in our study. We note that this use of Group DRO has been applied previously; e.g. [38].
F.4 Domain Generalization Models
Invariant Risk Minimization (IRM): IRM [6] uses a modified training objective to learn models which a feature representation such that the optimal linear classifier on top of that representation matches across domains.
MixUp: Inter-Domain MixUp [90, 89] uses combinations of data points from random pairs of domains and their labels during training.
Domain-Adversarial Neural Networks (DANN): DANN [1] uses adversarial training to achieve domain robustness, where a discriminator attempts to predict the domain of a training example in order to match feature distributions across domains.
Risk Extrapolation (REx): REx [52] attempts to reduce differences in risk across training domains, in order to reduce a model's sensitivity to distributional shifts.
CORAL: CORAL (CORrelation Alignment) [82] attempts to ensure that feature activations are similar across domains; this can be used as either a domain generalization method or a domain adaptation method.
MMD: Similar to CORAL using a different kernel, MMD attempts to minimize the Maximum Mean Discrepancy (MMD) between domains.
F.5 Label Shift Robustness Models
Group DRO: here, we use Group DRO [76] with class labels as the grouping attribute.
Adversarial Label DRO: This method, proposed in [92], uses a distributionally robust objective to optimize for the worst-case weighting over label groupings. We note that this approach is computationally expensive, requiring sample-level gradients even following the authors' original implementation, and so was not practical for our datasets with very large  $n$  (ASSISTments, Public Health Insurance).
G Comparison To Other Benchmarking Toolkits
In this section, we provide a brief comparison of TableShift to other relevant benchmarking toolkits. We note that our goal in this section is not to fully characterize the functionality of other benchmarking platforms; it is only to compare and contrast their relevant attributes with TableShift and to motivate the creation of a novel benchmark and API for TableShift (as opposed to incorporating TableShift into an existing toolkit).
As noted above, there is no existing benchmark for domain shift in tabular data. However, in this section we compare to three main categories of relevant related toolkits: (1) domain shift benchmarks for non-tabular data (DomainBed, WILDS); (2) IID (non-domain-shift) benchmarks for tabular data ([37], OpenML); and (3) generic data-hosting platforms (Huggingface Datasets, TensorFlow Datasets. We briefly introduce and compare to each of these below.
G.1 Domain Shift Benchmarks for Non-Tabular Data
WILDS: WILDS<sup>30</sup> is perhaps the closest benchmark to TableShift, but only uses non-tabular data. WILDS demonstrates a lightweight, useful set of programming abstractions for benchmarking models and sharing results across a diverse set of datasets for domain shift. WILDS interfaces with image and text datasets, and includes a rich variety of datasets with real-world sensitive attributes, carefully selected domain shifts, and has wide adoption in the robustness community. WILDS includes a high-quality Python API, which has led to wide integration with researchers' open-source code and widespread adoption. However, WILDS is currently not compatible with tabular datasets and does not
30See https://wilds.stanford.edu/ and [50].
include any tabular datasets in its benchmark suite. The needs for tabular datasets are different than the datasets currently used in WILDS (i.e. non-Torch models must be supported by the benchmark; subgroup and domain-shift information are handled differently for our use cases; data preprocessing is also different for tabular data as noted above).
DomainBed:  $\mathsf{DomainBed}^{31}$  is a benchmark that contains several reference implementations, including some that have been adapted for use in TableShift. In addition to these model implementations, DomainBed serves as an interface to several existing datasets through a Python API. However, DomainBed is specifically adapted to image data. It only supports image datasets with a specific folder structure (which would make extending to tabular datasets nontrivial) and includes many image-specific augmentation components of its pipelines. It also uses ResNet50/ResNet18 networks designed specifically for image classification, and therefore does not currently support either deep learning models suited to image data, nor (more importantly) the effective non-DL baselines described above such as XGBoost and LightGBM.
Shift Happens: The "shift happens" benchmark<sup>32</sup> is a community-built benchmark suite for image models. It specifically aims to feature datasets with domain shift, for tasks including image classification under domain shift, and out-of-distribution detection. The benchmark includes a Python API. This benchmark does not support tabular datasets, and is much less widely used, perhaps due to the community-driven effort (as opposed to benchmarks such as WILDS and DomainBed, which come packaged with preselected datasets and domain splits).
Shifts 2.0: Shifts ([57], recently upgraded to Shifts 2.0 [57]) is a collection of multimodal tasks with domain shifts. The Shifts benchmark is a part of the Shifts Project, an international collaboration of academic and industrial researchers dedicated to studying distributional shift.[33] Shifts 2.0, the current version, includes five tasks: tabular weather prediction, tabular marine cargo vessel power consumption prediction, machine translation, self-driving car vehicle motion prediction, and segmentation of white matter Multiple Sclerosis lesions in 3D magnetic resonance brain images. While shifts does contain two tabular data tasks, its relatively small number of tasks makes it a less reliable benchmark compared to the rich set of tabular datasets comprising TableShift. Shifts also does not include any tasks with real-world sensitive subgroups (such as age, race, or gender) which are of particular interest in many tabular classification tasks. Additionally, the domains represented in the tabular tasks of Shifts do not cover many critical domains widely recognized as using tabular data (e.g. finance, medicine, etc.; see Section D.2).
G.2 IID Benchmarks for Tabular Data
Benchmark of [37]: The unnamed benchmark proposed in this work is intended to provide a consistent benchmarking suite for tabular dataset classification tasks, and was motivated by some of the same gaps described in this work. However, the datasets comprising the benchmark of [37] do not meet our specifications, for several reasons. First, the datasets are limited to be of maximum size  $10k$  observations; this is too small for reliable and repeated benchmarking comparisons. Additionally, the datasets are label-balanced; in contrast, we use the naturally-occurring label distributions for all datasets (and we show that these label distributions are importantly related to shift gap). Most critically, the benchmark datasets in [37] do not contain domain shifts; for most or all of the datasets, it is does not appear that a domain shift could be induced from splitting the existing data on an existing feature.
OpenML: OpenML has some overlap with the proposed functionality. However, OpenML both lacks functionality we seek to provide (subgroup robustness and domain shift utilities; a curated set of benchmarking datasets; lightweight and standardized control over common tabular preprocessing methods) and also provides extraneous functionality not needed for a lightweight tabular benchmarking library (tools for OpenML-hosted model/pipeline/evaluation sharing and collaboration; API/utilities for model training). Additionally, OpenML is not yet widely used in the tabular data community, as demonstrated by the wide calls for effective tabular data benchmarking tools ([15], [37], [78]) and the lack of usage of OpenML in most robustness works, even recent works (e.g. [91]), which largely focus on canonical tabular datasets such as COMPAS and Adult.
31See https://github.com/facebookresearch/DomainBed and [38].
32https://shift-happens-benchmark.github.io/index.html
33https://shifts.ai/
Table 18: Comparison of relevant benchmarks. DB: DomainBed. OML: OpenML. GOV22: [37]. SH: Shift Happens. HFDS: Hugging Face Datasets. TFDS: TensorFlow Datasets.
DB	OML	GOV22	WILDS	SH	Shifts 2.0	HFDS	TFDS	TableShift
Output	Supports tabular data input formats (e.g. .csv)	✓	✓	✓	✓	✓
Supports tabular output formats (e.g. pd.DataFrame)	✓	✓	✓	✓
Support for large/out-of-core datasets	✓	✓	✓	✓	✓	✓
Preprocessing	Supports tabular preprocessing: categorical encoding; missing value handling	✓
Provides shared utilities for user-defined preprocessing per dataset	✓	✓	✓	✓	✓	✓
Metadata	Feature-level metadata	✓
Dataset-level metadata	✓	✓	✓
Benchmark Tasks	Includes domain shift (non-IID) splits	✓	✓	✓	✓	some	✓
Meets criteria in §3.1	✓	✓	✓
Large test sets (≥10k)	✓	✓
Includes label-imbalanced datasets	✓	✓	✓	✓	✓	✓	✓	✓
Includes real-world sensitive attributes	some	✓	some	some	all
DataPerf: DataPerf is "a benchmark package for evaluating ML datasets and dataset working algorithms" [59]. Similar to WILDS and DomainBed, DataPerf covers many domains, not only tabular data. DataPerf has a much broader set of goals relative to TableShift, and includes a collection of tasks, metrics and rules that are intended to benchmark all stages of an ML pipeline, from raw data to test set selection and model selection. However, DataPerf does not offer domain shifts, and while it is possible domain shifts could be integrated into DataPerf, it does not natively support the kind of benchmarking that we intend to support with TableShift.
G.3 Other Data Hosting Platforms
Hugging Face Datasets (HFDS): HFDS is a generic dataset hosting utility provided by the company Hugging Face. It serves as a large, open dataset repository; however, these datasets are not curated for size, featurization, or quality. HFDS is a public platform where datasets can be contributed openly. However, of the tabular datasets on HFDS, few if any meet the specifications described in §3.1; in particular, most are not domain shift datasets.
TensorFlow Datasets (TFDS): TFDS is similar in many ways to HFDS. It is a public, open repository of datasets, and new datasets can be contributed via git. However, TFDS also has the same shortcomings as HFDS; in particular, its open format leads to a collection of datasets mostly not useful for tabular data benchmarking and almost no datasets with meaningful distribution shifts.
Neural network-based models were trained on GPU, either NVIDIA RTX 2080 Ti GPUs with 11GB of RAM, or NVIDIA Tesla M60 GPUs with 48 GB of RAM. We used a batch size of 4096 for training all models, except where this was not possible due to memory limitations (see code for details).
Where possible, gradient boosted tree models were trained using CPU (not GPU).
I Hyperparameter Grids
The hyperparameter tuning grid for each model is shown in Table 19. We make the full hyperparameter tuning code available as part of the release of this work, at https://github.com/mlfoundations/tableshift.
We made an effort to ensure our hyperparameter grids always included at least the full grid described in the original work(s) cited for each learning method used in our study. For some methods, our grid is a superset of the hyperparameter grid in the original study. This is to ensure, where possible, that we tune a similar range of certain parameters (i.e. learning rate) across all methods. For domain generalization methods, since we are not aware of any prior application to these methods to tabular data, we use the hyperparameter grids from [38].
Table 19: Hyperparameter grids used in all experiments.  $\clubsuit$ : all MLP parameters also tuned. Continued in Table 20.
Model	Hyperparameter	Values
Baseline Methods
MLP	Learning Rate	LogUniform(1e-5, 1e-1)
Weight Decay	LogUniform(1e-6, 1)
Num. Layers	{1,2,...,8}
Hidden Units	{64,128,256,512,1024}
Num Epochs	QRandInt(5,100,5)
Dropout	Unif(0,0.5)
Batch Size	{4096}
XGBoost	Learning Rate	LogUniform{1e-5,1}
Max. Depth	{3,...,10}
Min Child Weight	LogUniform{1e-8,1e5}
Row Subsample	Uniform{0.5,1}
Column Subsample (Tree)	Uniform{0.5,1}
Column Subsample (Level)	Uniform{0.5,1}
γ	LogUniform{1e-8,1e2}
λ	LogUniform{1e-8,1e2}
α	LogUniform{1e-8,1e2}
Max. Bins	{128,256,512}
LightGBM	Learning Rate	LogUniform{1e-5,1}
Min. Child Samples	{1,2,4,8,16,32,64}
Min. Child Weight	LogUniform(1e-8,1e5)
Row Subsample	Uniform{0.5,1}
Max. Depth	{None,1,2,...,31}
Column Subsample (Tree)	Uniform{0.5,1}
Column Subsample (Level)	Uniform{0.5,1}
λ	LogUniform{1e-8,1e2}
α	LogUniform{1e-8,1e2}
CatBoost	Learning Rate	LogUniform{1e-3,1}
Depth	QRandInt{3,10}
Bagging Temp.	LogUniform(1e-6,1)
L2 Leaf Reg.	LogUniform(1,100)
Leaf Estimation Iterations	QRandInt{1,10}
Domain Generalization Methods
DANN♣	LRG	LogUniform{1e-5,1e-1}
WeightDecayG	LogUniform{1e-6,1}
LRd	LogUniform{1e-5,1e-1}
WeightDecayD	LogUniform{1e-6,1}
D steps per G step	LogUniform{2,23}
Grad Penalty	LogUniform{1e-2,1e1}
Loss λ	LogUniform{1e-2,1e2}
IRM♣	IRMλ	LogUniform{1e-1,1e5}
IRM Penalty Anneal Iters	LogUniform{1,1e4}
MixUp♣	MixUp α	Uniform{1e-1,1e1}
VReX♣	VReXλ	LogUniform{1e-1,1e5}
VReX Penalty Anneal Iters	LogUniform{1,1e4}
CORAL♣	MMDγ	LogUniform{1e-1,1e1}
MMD♣	MMDγ	Uniform{1e-1,1e1}
Model	Hyperparameter	Values
Tabular Neural Networks
ResNet♣	Num. Blocks	{1,2,3,4}
d main	RandInt(64, 1024)
Hidden Factor	RandInt(1,4)
Dropout First	Uniform(0,0.5)
Dropout Second	Uniform(0,0.5)
FT-Transformer♣	Num. Blocks	{1,2,3,4}
Residual Dropout	Uniform(0,0.2)
Attention Dropout	Uniform(0,0.5)
FFN Dropout	Uniform(0,0.5)
FFN Factor	Uniform(2/3,8/3)
FFN Factor	{64, 128, 256, 512}
TabTransformer	Num. Blocks	{1,2,3,4}
Learning Rate	LogUniform(1e-5, 1e-1)
Weight Decay	LogUniform(1e-6, 1)
Num Epochs	QRandInt(5, 100, 5)
FFN Dropout	Uniform(0,0.5)
Attention Dropout	Uniform(0,0.5)
Model Dimension	32, 64, 128, 256
Depth	3, 4, 5, 6
Num. Heads	2, 4, 8
NODE	Num. Epochs	{1,2,3,4,5}
Num. Layers	2, 4, 8
Total Tree Count	1024, 2048
Tree Depth	6, 8
Tree Output Dim.	2, 3
FFN Factor	{64, 128, 256, 512}
Learning Rate	LogUniform(1e-5, 1e-1)
Weight Decay	LogUniform(1e-6, 1)
SAINT	Num. Epochs	{1,2,3,4,5}
Depth	4, 6
Model Dimension	4, 8, 12, 16, 32
Learning Rate	LogUniform(1e-5, 1e-1)
Weight Decay	LogUniform(1e-6, 1)
FFN Dropout	Uniform(0.1,0.8)
Heads	4, 8
Attention Type	Row, Col, RowCol
Domain Robustness Methods
DRO♣	Uncertainty set size	LogUniform{1e-4, 1.}
Geometry	{CVaR, χ2}
Group DRO♣	Group weights step size	LogUniform(1e-4, 1}
Label Shift Robustness Methods
Label Group DRO♣	Group weights step size	LogUniform(1e-4, 1}
Adversarial Label DRO♣	Adv. Learning Rate ηπ	LogUniform{1e-4, 1e-1}
Adv. radius r	LogUniform{1e-5,0.5}
Clip max r	LogUniform{1e-1,10}
ε	LogUniform{1e-4,1e-1}
Table 20: Hyperparameter grids used in all experiments.  $\clubsuit$ : all MLP parameters also tuned. Continued from Table 19.