SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 6: Best (In-Distribution and Out-Of-Distribution) accuracy pair observed on each benchmark task. Note that domain generalization models can only be trained on datasets with more than one training subdomain (see Table 1 for domain generalization datasets and Section 4.1 for a list of domain generalization models).  $\star$ : domain generalization models cannot be trained when only one training subdomain is present. See also Figures 1, 2,3.
Estimator	FICO HELOC	Food Stamps
ID Acc. (95% CI)	OOD Acc. (95% CI)	ID Acc. (95% CI)	OOD Acc. (95% CI)
Adv. Label DRO	0.745	(0.689, 0.795)	0.431	(0.419, 0.443)	0.843	(0.84, 0.846)	0.812	(0.808, 0.815)
CatBoost	0.727	(0.67, 0.778)	0.582	(0.57, 0.594)	0.849	(0.847, 0.852)	0.825	(0.821, 0.828)
DRO	0.745	(0.689, 0.795)	0.431	(0.419, 0.443)	0.844	(0.841, 0.846)	0.819	(0.815, 0.822)
FT-Transformer	0.745	(0.689, 0.795)	0.431	(0.419, 0.443)	0.843	(0.841, 0.846)	0.816	(0.812, 0.819)
Label Group DRO	0.745	(0.689, 0.795)	0.431	(0.419, 0.443)	0.771	(0.768, 0.774)	0.752	(0.748, 0.756)
LightGBM	0.647	(0.584, 0.7)	0.421	(0.409, 0.433)	0.836	(0.833, 0.838)	0.808	(0.805, 0.812)
MLP	0.734	(0.678, 0.785)	0.538	(0.526, 0.55)	0.841	(0.838, 0.844)	0.815	(0.812, 0.819)
NODE	0.745	(0.689, 0.795)	0.431	(0.419, 0.443)	0.849	(0.847, 0.852)	0.822	(0.819, 0.825)
ResNet	0.748	(0.693, 0.798)	0.431	(0.42, 0.443)	0.843	(0.84, 0.845)	0.82	(0.817, 0.824)
SAINT	0.745	(0.689, 0.795)	0.431	(0.419, 0.443)	0.849	(0.846, 0.851)	0.821	(0.818, 0.825)
TabTransformer	0.745	(0.689, 0.795)	0.431	(0.419, 0.443)	0.836	(0.834, 0.839)	0.807	(0.803, 0.81)
XGBoost	0.745	(0.689, 0.795)	0.431	(0.419, 0.443)	0.844	(0.842, 0.847)	0.82	(0.817, 0.824)
CORAL	★	★	★	★	0.818	(0.815, 0.82)	0.793	(0.79, 0.797)
DANN	★	★	★	★	0.809	(0.806, 0.812)	0.78	(0.776, 0.784)
Group DRO	★	★	★	★	0.84	(0.838, 0.843)	0.817	(0.814, 0.821)
IRM	★	★	★	★	0.812	(0.81, 0.815)	0.795	(0.791, 0.798)
MMD	★	★	★	★	0.813	(0.81, 0.816)	0.786	(0.782, 0.789)
MixUp	★	★	★	★	0.819	(0.816, 0.821)	0.785	(0.782, 0.789)
VREX	★	★	★	★	0.809	(0.806, 0.812)	0.78	(0.776, 0.784)
Table 7: Best (In-Distribution and Out-Of-Distribution) accuracy pair observed on each benchmark task. Note that domain generalization models can only be trained on datasets with more than one training subdomain (see Table 1 for domain generalization datasets and Section 4.1 for a list of domain generalization models).  $\star$ : domain generalization models cannot be trained when only one training subdomain is present. See also Figures 1, 2,3.
Estimator	Hospital Readmission	Hypertension
ID Acc. (95% CI)	OD Acc. (95% CI)	ID Acc. (95% CI)	OD Acc. (95% CI)
Adv. Label DRO	0.655	(0.641, 0.669)	0.603	(0.599, 0.607)	0.666	(0.66, 0.672)	0.601	(0.6, 0.603)
CatBoost	0.659	(0.645, 0.674)	0.618	(0.614, 0.623)	0.67	(0.665, 0.676)	0.599	(0.597, 0.6)
DRO	0.628	(0.613, 0.642)	0.578	(0.574, 0.582)	0.598	(0.592, 0.604)	0.416	(0.414, 0.417)
FT-Transformer	0.648	(0.633, 0.662)	0.618	(0.614, 0.622)	0.666	(0.661, 0.672)	0.604	(0.603, 0.605)
Label Group DRO	0.652	(0.637, 0.666)	0.616	(0.612, 0.62)	0.665	(0.659, 0.671)	0.604	(0.603, 0.605)
LightGBM	0.658	(0.643, 0.672)	0.598	(0.594, 0.602)	0.678	(0.672, 0.683)	0.634	(0.633, 0.635)
MLP	0.648	(0.633, 0.662)	0.612	(0.608, 0.617)	0.664	(0.658, 0.67)	0.583	(0.582, 0.584)
NODE	0.659	(0.645, 0.673)	0.624	(0.62, 0.628)	0.67	(0.664, 0.676)	0.597	(0.596, 0.599)
ResNet	0.639	(0.624, 0.653)	0.581	(0.577, 0.586)	0.667	(0.661, 0.672)	0.608	(0.606, 0.609)
SAINT	0.654	(0.639, 0.668)	0.61	(0.606, 0.615)	0.669	(0.664, 0.675)	0.595	(0.594, 0.596)
TabTransformer	0.584	(0.569, 0.599)	0.507	(0.502, 0.511)	0.624	(0.618, 0.63)	0.499	(0.498, 0.501)
XGBoost	0.651	(0.636, 0.665)	0.605	(0.601, 0.61)	0.671	(0.665, 0.677)	0.588	(0.587, 0.59)
CORAL	0.622	(0.607, 0.637)	0.571	(0.567, 0.576)	★	★	★	★
DANN	0.584	(0.569, 0.599)	0.506	(0.502, 0.51)	★	★	★	★
Group DRO	0.639	(0.624, 0.653)	0.6	(0.596, 0.605)	★	★	★	★
IRM	0.595	(0.58, 0.61)	0.55	(0.546, 0.555)	★	★	★	★
MMD	0.626	(0.611, 0.64)	0.57	(0.565, 0.574)	★	★	★	★
MixUp	0.589	(0.574, 0.604)	0.567	(0.563, 0.572)	★	★	★	★
VREX	0.584	(0.569, 0.599)	0.506	(0.502, 0.51)	★	★	★	★
Table 8: Best (In-Distribution and Out-Of-Distribution) accuracy pair observed on each benchmark task. Note that domain generalization models can only be trained on datasets with more than one training subdomain (see Table 1 for domain generalization datasets and Section 4.1 for a list of domain generalization models).  $\star$ : domain generalization models cannot be trained when only one training subdomain is present. See also Figures 1, 2,3.  $\heartsuit$ : the large feature dimensionality of both ICU datasets makes training Transformer-based models impractical (e.g. even a single-layer SAINT model requires  $>13\mathrm{B}$  trainable parameters on both ICU datasets)
Estimator	ICU Hospital Mortality	ICU Length of Stay
ID Acc. (95% CI)	OOD Acc. (95% CI)	ID Acc. (95% CI)	OOD Acc. (95% CI)
Adv. Label DRO	0.915	(0.893, 0.931)	0.876	(0.87, 0.882)	0.602	(0.572, 0.631)	0.544	(0.535, 0.553)
CatBoost	0.934	(0.914, 0.948)	0.892	(0.887, 0.897)	0.71	(0.682, 0.737)	0.674	(0.665, 0.682)
DRO	0.915	(0.893, 0.931)	0.876	(0.87, 0.882)	0.601	(0.571, 0.63)	0.544	(0.535, 0.553)
FT-Transformer	♥	♥	♥	♥	♥	♥	♥	♥
Label Group DRO	0.915	(0.893, 0.931)	0.876	(0.87, 0.882)	0.59	(0.56, 0.619)	0.542	(0.533, 0.551)
LightGBM	0.946	(0.928, 0.959)	0.883	(0.877, 0.888)	0.689	(0.66, 0.716)	0.655	(0.646, 0.663)
MLP	0.912	(0.891, 0.929)	0.877	(0.871, 0.882)	0.599	(0.569, 0.628)	0.544	(0.535, 0.553)
NODE	0.915	(0.893, 0.931)	0.876	(0.87, 0.882)	0.661	(0.632, 0.689)	0.609	(0.6, 0.618)
ResNet	0.915	(0.893, 0.931)	0.876	(0.87, 0.882)	0.606	(0.576, 0.635)	0.577	(0.568, 0.586)
SAINT	♥	♥	♥	♥	♥	♥	♥	♥
TabTransformer	0.915	(0.893, 0.931)	0.876	(0.87, 0.882)	0.604	(0.574, 0.633)	0.549	(0.54, 0.558)
XGBoost	0.927	(0.908, 0.943)	0.882	(0.876, 0.887)	0.71	(0.682, 0.737)	0.669	(0.66, 0.677)
CORAL	0.915	(0.893, 0.931)	0.875	(0.869, 0.881)	0.603	(0.573, 0.632)	0.544	(0.535, 0.553)
DANN	0.915	(0.893, 0.931)	0.876	(0.871, 0.882)	0.594	(0.564, 0.624)	0.545	(0.536, 0.554)
Group DRO	0.915	(0.893, 0.931)	0.876	(0.87, 0.882)	0.602	(0.572, 0.631)	0.544	(0.535, 0.553)
IRM	0.915	(0.893, 0.931)	0.876	(0.87, 0.882)	0.601	(0.571, 0.63)	0.544	(0.535, 0.553)
MMD	0.915	(0.893, 0.931)	0.876	(0.87, 0.882)	0.602	(0.572, 0.631)	0.544	(0.535, 0.553)
MixUp	0.915	(0.893, 0.931)	0.876	(0.87, 0.882)	0.602	(0.572, 0.631)	0.544	(0.535, 0.553)
VREX	0.913	(0.893, 0.931)	0.876	(0.87, 0.882)	0.597	(0.567, 0.627)	0.545	(0.536, 0.554)
Table 9: Best (In-Distribution and Out-Of-Distribution) accuracy pair observed on each benchmark task. Note that domain generalization models can only be trained on datasets with more than one training subdomain (see Table 1 for domain generalization datasets and Section 4.1 for a list of domain generalization models).  $\star$ : domain generalization models cannot be trained when only one training subdomain is present.  $\square$ : the large dataset size makes training adversarial label DRO models impractical (since per-example gradients must be computed). We leave these experiments to future work. See also Figures 1, 2,3.
Estimator	Income	Public Health Ins.
ID Acc. (95% CI)	OOD Acc. (95% CI)	ID Acc. (95% CI)	OOD Acc. (95% CI)
Adv. Label DRO	0.829	(0.827, 0.83)	0.819	(0.816, 0.822)	□	□	□	□
CatBoost	0.832	(0.83, 0.834)	0.814	(0.811, 0.817)	0.814	(0.812, 0.815)	0.69	(0.689, 0.691)
DRO	0.828	(0.826, 0.83)	0.818	(0.816, 0.821)	0.809	(0.808, 0.81)	0.647	(0.646, 0.648)
FT-Transformer	0.825	(0.823, 0.827)	0.818	(0.815, 0.821)	0.807	(0.806, 0.808)	0.662	(0.661, 0.663)
Label Group DRO	0.819	(0.817, 0.821)	0.818	(0.815, 0.821)	0.776	(0.775, 0.777)	0.364	(0.363, 0.365)
LightGBM	0.822	(0.82, 0.824)	0.809	(0.806, 0.812)	0.803	(0.802, 0.804)	0.639	(0.638, 0.64)
MLP	0.828	(0.826, 0.829)	0.813	(0.81, 0.816)	0.808	(0.806, 0.809)	0.612	(0.611, 0.613)
NODE	0.831	(0.829, 0.833)	0.81	(0.807, 0.813)	0.811	(0.81, 0.812)	0.662	(0.661, 0.663)
ResNet	0.826	(0.824, 0.828)	0.815	(0.812, 0.818)	0.81	(0.809, 0.811)	0.672	(0.671, 0.673)
SAINT	0.829	(0.827, 0.831)	0.81	(0.807, 0.812)	0.811	(0.81, 0.812)	0.68	(0.679, 0.681)
TabTransformer	0.818	(0.816, 0.82)	0.801	(0.798, 0.804)	0.803	(0.802, 0.804)	0.588	(0.587, 0.589)
XGBoost	0.821	(0.819, 0.823)	0.792	(0.789, 0.795)	0.805	(0.804, 0.806)	0.661	(0.66, 0.662)
CORAL	0.817	(0.815, 0.819)	0.791	(0.788, 0.793)	★	★	★	★
DANN	0.815	(0.813, 0.817)	0.812	(0.809, 0.815)	★	★	★	★
Group DRO	0.827	(0.826, 0.829)	0.813	(0.81, 0.815)	★	★	★	★
IRM	0.756	(0.754, 0.758)	0.699	(0.696, 0.702)	★	★	★	★
MMD	0.816	(0.814, 0.818)	0.768	(0.765, 0.771)	★	★	★	★
MixUp	0.821	(0.819, 0.823)	0.794	(0.791, 0.797)	★	★	★	★
VREX	0.714	(0.712, 0.716)	0.64	(0.637, 0.644)	★	★	★	★
Table 10: Best (In-Distribution and Out-Of-Distribution) accuracy pair observed on each benchmark task. Note that domain generalization models can only be trained on datasets with more than one training subdomain (see Table 1 for domain generalization datasets and Section 4.1 for a list of domain generalization models).  $\star$ : domain generalization models cannot be trained when only one training subdomain is present. See also Figures 1, 2,3.
Estimator	Sepsis	Unemployment
ID Acc. (95% CI)	OOD Acc. (95% CI)	ID Acc. (95% CI)	OOD Acc. (95% CI)
Adv. Label DRO	0.988	(0.987, 0.989)	0.925	(0.924, 0.926)	0.972	(0.971, 0.973)	0.96	(0.959, 0.961)
CatBoost	0.988	(0.987, 0.989)	0.925	(0.923, 0.926)	0.973	(0.973, 0.974)	0.962	(0.961, 0.963)
DRO	0.988	(0.987, 0.989)	0.925	(0.924, 0.926)	0.973	(0.972, 0.973)	0.961	(0.96, 0.962)
FT-Transformer	0.988	(0.987, 0.989)	0.925	(0.924, 0.926)	0.973	(0.972, 0.974)	0.962	(0.961, 0.962)
Label Group DRO	0.988	(0.987, 0.989)	0.925	(0.924, 0.926)	0.947	(0.946, 0.948)	0.926	(0.925, 0.927)
LightGBM	0.988	(0.987, 0.989)	0.928	(0.926, 0.929)	0.973	(0.972, 0.974)	0.96	(0.96, 0.961)
MLP	0.988	(0.987, 0.989)	0.925	(0.923, 0.926)	0.973	(0.972, 0.973)	0.96	(0.959, 0.961)
NODE	0.988	(0.987, 0.989)	0.925	(0.924, 0.926)	0.973	(0.972, 0.974)	0.962	(0.961, 0.963)
ResNet	0.988	(0.987, 0.989)	0.925	(0.924, 0.926)	0.972	(0.971, 0.972)	0.959	(0.958, 0.96)
SAINT	0.988	(0.987, 0.989)	0.925	(0.924, 0.926)	0.973	(0.972, 0.974)	0.962	(0.961, 0.963)
TabTransformer	0.988	(0.987, 0.989)	0.925	(0.924, 0.926)	0.972	(0.971, 0.973)	0.961	(0.96, 0.962)
XGBoost	0.988	(0.987, 0.989)	0.925	(0.923, 0.926)	0.973	(0.972, 0.973)	0.961	(0.961, 0.962)
CORAL	★	★	★	★	0.964	(0.963, 0.965)	0.95	(0.949, 0.951)
DANN	★	★	★	★	0.966	(0.965, 0.967)	0.948	(0.947, 0.95)
Group DRO	★	★	★	★	0.971	(0.97, 0.972)	0.958	(0.957, 0.959)
IRM	★	★	★	★	0.966	(0.965, 0.967)	0.948	(0.947, 0.95)
MMD	★	★	★	★	0.966	(0.966, 0.967)	0.953	(0.952, 0.954)
MixUp	★	★	★	★	0.844	(0.842, 0.846)	0.776	(0.774, 0.778)
VREX	★	★	★	★	0.873	(0.871, 0.874)	0.8	(0.798, 0.802)
Table 11: Best (In-Distribution and Out-Of-Distribution) accuracy pair observed on each benchmark task. See also Figures 1, 2,3.
Estimator	Voting
ID Acc. (95% CI)	OOD Acc. (95% CI)
Adv. Label DRO	0.875	(0.843, 0.902)	0.852	(0.839, 0.865)
CatBoost	0.883	(0.852, 0.909)	0.855	(0.842, 0.868)
DRO	0.881	(0.85, 0.907)	0.853	(0.839, 0.866)
FT-Transformer	0.879	(0.848, 0.906)	0.855	(0.841, 0.868)
Label Group DRO	0.862	(0.829, 0.89)	0.839	(0.825, 0.852)
LightGBM	0.881	(0.85, 0.907)	0.855	(0.841, 0.868)
MLP	0.892	(0.862, 0.918)	0.86	(0.847, 0.873)
NODE	0.885	(0.854, 0.911)	0.851	(0.838, 0.864)
ResNet	0.887	(0.856, 0.912)	0.836	(0.822, 0.849)
SAINT	0.888	(0.858, 0.914)	0.858	(0.845, 0.871)
TabTransformer	0.877	(0.846, 0.904)	0.859	(0.845, 0.872)
XGBoost	0.898	(0.869, 0.923)	0.851	(0.838, 0.864)
CORAL	0.883	(0.852, 0.909)	0.846	(0.832, 0.859)
DANN	0.892	(0.862, 0.918)	0.852	(0.838, 0.865)
Group DRO	0.877	(0.846, 0.904)	0.852	(0.839, 0.865)
IRM	0.804	(0.767, 0.837)	0.758	(0.742, 0.774)
MMD	0.892	(0.862, 0.918)	0.849	(0.835, 0.862)
MixUp	0.892	(0.862, 0.918)	0.851	(0.837, 0.864)
VREX	0.804	(0.767, 0.837)	0.754	(0.737, 0.77)
Table 12: Sample sizes by split. In particular, large test sizes are desirable for benchmarking, as they reduce the statistical uncertainty of comparing model performance.
Task	ID Test	OOD Test	OOD Validation	Train	Validation	Total
Food Stamps	78,628	48,878	5431	629,018	78,627	840,582
Income	158,016	75,911	8435	1,264,123	158,015	1,664,500
Public Coverage	500,782	817,877	90,876	4,006,249	500,781	5,916,565
Unemployment	161,365	163,611	18,180	1,290,914	161,364	1,795,434
Voting	520	2772	309	4159	520	8280
Hypertension	27,052	518,622	57,625	216,411	27,051	846,761
Diabetes	121,154	209,375	23,264	969,229	121,154	1,444,176
Readmission	4287	50,968	5664	34,288	4286	99,493
HELOC	278	6914	769	2220	278	10,459
ICU Length of Stay	1080	11,835	1316	8634	1079	23,944
ICU Hospital Mortality	890	13,544	1505	7116	889	23,944
Sepsis	140,288	134,402	14,934	1,122,299	140,287	1,552,210
Childhood Lead	1476	11,466	1274	11,807	1476	27,499
ASSISTments	266,566	1906	212	2,132,526	266,566	2,667,776
College Scorecard	12,320	1352	151	98,556	12,320	124,699
ASSISTments
Hospital Mortality
Hospital Readmission
ICU Length of Stay
Income
Figure 10: Alternate version of Figure 2 with adjusted scaling for increased detail.
Public Coverage
Figure 11: Alternate version of Figure 3 with adjusted scaling for increased detail.
Table 13: Label summary statistics for test sets in TABLESHIFT.  $\bar{Y}$  gives the proportion of positive labels for the respective split, and  $\mathrm{Var}(\bar{Y})$  the variance of the sample proportion.
Task	ID	OOD
\(\bar{Y}_{\text{ID}}\)	\(\text{Var}(\bar{Y}_{\text{ID}})\)	\(\bar{Y}_{\text{OOD}}\)	\(\text{Var}(\bar{Y}_{\text{OOD}})\)
Voting	0.804	0.017	0.754	0.008
ASSISTments	0.695	0.001	0.437	0.011
Childhood Lead	0.029	0.004	0.080	0.003
College Scorecard	0.127	0.003	0.311	0.013
Diabetes	0.127	0.001	0.174	0.001
FICO HELOC	0.255	0.026	0.569	0.006
Food Stamps	0.191	0.001	0.220	0.002
Hospital Readmission	0.416	0.008	0.494	0.002
Hypertension	0.402	0.003	0.584	0.001
ICU Hospital Mortality	0.085	0.009	0.124	0.003
ICU Length of Stay	0.398	0.015	0.456	0.005
Income	0.321	0.001	0.398	0.002
Public Health Ins.	0.224	0.001	0.636	0.001
Sepsis	0.012	0.000	0.075	0.001
Unemployment	0.034	0.000	0.052	0.001
Table 14: PMA-OOD results (cf. Figure ) and standard deviation of PMA-OOD over benchmark tasks. (Cf. Figure 4a.)
Estimator	PMA-OOD Mean	PMA-OOD Std.
VREX	0.876	0.076
IRM	0.910	0.069
Label Group DRO	0.915	0.129
MixUp	0.917	0.076
TabTransformer	0.922	0.091
DANN	0.932	0.075
DRO	0.932	0.107
MMD	0.940	0.059
CORAL	0.944	0.059
Adv. Label DRO	0.950	0.080
ResNet	0.956	0.069
MLP	0.961	0.054
Group DRO	0.962	0.059
NODE	0.963	0.066
SAINT	0.964	0.070
LightGBM	0.964	0.070
XGBoost	0.964	0.065
FT-Transformer	0.968	0.069
CatBoost	0.994	0.014
Table 15: Label summary statistics for test sets in TABLESHIFT.  $\bar{Y}$  gives the proportion of positive labels for the respective split, and  $\operatorname{Var}(\bar{Y})$  the variance of the sample proportion. (Cf. Figure 4b.)
Method	ID	OOD
ID Accuracy	Std.	OOD Accuracy	Std.
Adv. Label DRO	0.834	0.125	0.792	0.132
CatBoost	0.862	0.105	0.794	0.126
DANN	0.816	0.137	0.770	0.148
CORAL	0.824	0.129	0.777	0.134
DRO	0.843	0.129	0.773	0.147
FT-Transformer	0.866	0.103	0.794	0.126
Group DRO	0.832	0.129	0.791	0.133
IRM	0.800	0.130	0.749	0.136
Label Group DRO	0.829	0.123	0.759	0.134
LightGBM	0.856	0.108	0.781	0.124
MixUp	0.807	0.125	0.752	0.118
MLP	0.845	0.126	0.774	0.141
MMD	0.825	0.130	0.774	0.135
NODE	0.853	0.110	0.781	0.129
ResNet	0.844	0.126	0.773	0.139
SAINT	0.868	0.099	0.787	0.127
TabTransformer	0.835	0.136	0.759	0.160
VREX	0.786	0.127	0.720	0.128
XGBoost	0.857	0.105	0.783	0.122
E.4 Results with Additional Random Seeds
Our experiments on each model-dataset pair comprise a single run of 100 rounds of our hyperparameter tuning protocol described in Section 4.2. Here, we provide the results of additional experiments conducted using different random seeds, in order to evaluate the sensitivity of our results to the random variation inherent in the training and hyperparameter tuning process.
For these experiments, we conduct an identical procedure to the experiments described in the main text of our paper, but only change the random seed. This process affects the random initialization of model weights, random initialization of hyperparameter tuning, and training data shuffling, among other procedures. We note that it does not affect the train/test splitting in our datasets, as the train/test splits are defined by distribution shifts and are fixed to ensure comparability of the benchmark across experiments.
The results are shown in Table 16. Table 16 shows that, across the five models and three datasets evaluated, there is minimal variation in performance due to random seeds. Of the 90 measurements covering 45 trials represented in Table 16, the  $95\%$  Clopper-Pearson CIs for both ID and OOD accuracy overlap in all cases, with only four exceptions (LightGBM, iteration 0, Food Stamps ID and OOD accuracy; LightGBM, iteration 2, Hypertension OOD accuracy; MLP, Hypertension, iteration 0, OOD accuracy; FT-Transformer, iteration 0, OOD accuracy). These results provide evidence that our results are robust to variation due to random seed.
E.5 Results with Hybrid Methods
Our main study design is focused on benchmarking existing previously-proposed methods for tabular modeling. The methods we evaluate span models, which prescribe the functional form of a predictor  $f_{\theta}$ , and also objective functions, which describe the loss  $\mathcal{L}$  to be minimized while learning the parameters  $\theta$  of a fixed predictor  $f$ . Concretely, for example, FT-Transformer or MLP specify the form of  $f$ , while some robustness interventions, such as Group DRO, specify an objective that can be minimized over any smooth continuous function.
Our study does not explore potential combinations of different models and objective functions from the preexisting literature. In this section, we conduct an exploratory investigation into whether "hybrid methods" - combinations of different models and objective functions explored in our study - might improve robustness, for the best-performing compatible combinations of models and objective functions in our study.
In particular, our hybrid model study explores the use of the Group DRO objective function, in combination with three models from our study: FT-Transformer, NODE, and ResNet. Group DRO was selected as it is the highest-performing objective-based technique in our study (see Figure 4a), and the three models were selected as they are the highest-performing Transformer-based model, tree-based model, and baseline supervised model, respectively, in our study. We note that Group DRO cannot be easily combined with CatBoost, XGBoost, or LightGBM, as these are not smooth differentiable continuous functions, which is a requirement for the use of the Group DRO objective.
Our methodology in this section is as follows: for each estimator (FT-Transformer, NODE, ResNet), we train the model with both ERM (the standard procedure used in our main experiments above) and Group DRO. We follow the same hyperparameter tuning procedure as described in Section 4.2) above. We use the same hyperparameter grid defined in Section I for each model, but also include a full sweep over the Group DRO step size parameter, using the Group DRO grid described in Section I (thus, for model X, we take the union of the two hyperparameter grids:  $\{\mathrm{grid}(\mathrm{X})\cup \mathrm{grid}(\mathrm{Group}\mathrm{DRO})\}$ . We conduct this procedure for five benchmark datasets: Childhood Lead, College Scorecard, Food Stamps, Hypertension, and Voting.
The results of our hybrid model experiments are shown in Table 17. The results show little or no evidence that Group DRO reduces shift gaps for the models evaluated, as indicated by the fact that OOD test accuracy intervals tend to be overlapping, or higher, for ERM relative to Group DRO. Keeping in mind that Group DRO was parameterized over MLP models in our main experiments (as all prior works only use Group DRO with MLP), the results in Table 17 suggest that Group DRO may primarily improve weak (MLP) models but does not improve robustness for stronger models, explaining the improvements for Group DRO over vanilla MLP models in the main text.
Table 16: Results with additional random seeds. Varying random seeds has a minimal impact on the final results of our hyperparameter tuning procedure, indicating that our findings are robust to variation due to random seeds. See Section E.4 for details on experimental design.
Task	Base Estimator	Iteration	ID Test Accuracy	OOD Test Accuracy
Value	95% CI	Value	95% CI
College Scorecard	CatBoost	0	0.957	(0.954, 0.961)	0.885	(0.866, 0.901)
1	0.959	(0.955, 0.962)	0.879	(0.861, 0.896)
2	0.959	(0.956, 0.963)	0.882	(0.863, 0.898)
FT-Transformer	0	0.948	(0.944, 0.952)	0.859	(0.839, 0.877)
1	0.946	(0.942, 0.95)	0.850	(0.83, 0.868)
2	0.940	(0.936, 0.945)	0.830	(0.809, 0.85)
LightGBM	0	0.939	(0.935, 0.943)	0.822	(0.8, 0.841)
1	0.943	(0.938, 0.947)	0.839	(0.819, 0.859)
2	0.943	(0.939, 0.947)	0.837	(0.816, 0.856)
MLP	0	0.947	(0.942, 0.95)	0.845	(0.825, 0.864)
1	0.949	(0.944, 0.952)	0.859	(0.84, 0.878)
2	0.945	(0.941, 0.949)	0.859	(0.839, 0.877)
XGBoost	0	0.942	(0.938, 0.946)	0.830	(0.809, 0.85)
1	0.946	(0.942, 0.95)	0.842	(0.821, 0.861)
2	0.947	(0.943, 0.951)	0.845	(0.824, 0.864)
Food Stamps	CatBoost	0	0.849	(0.847, 0.852)	0.825	(0.821, 0.828)
1	0.850	(0.847, 0.852)	0.824	(0.821, 0.827)
2	0.849	(0.847, 0.852)	0.824	(0.82, 0.827)
FT-Transformer	0	0.843	(0.841, 0.846)	0.816	(0.812, 0.819)
1	0.848	(0.846, 0.851)	0.824	(0.82, 0.827)
2	0.844	(0.842, 0.847)	0.817	(0.814, 0.82)
LightGBM	0	0.836	(0.833, 0.838)	0.808	(0.805, 0.812)
1	0.844	(0.841, 0.846)	0.818	(0.814, 0.821)
2	0.843	(0.84, 0.846)	0.817	(0.814, 0.821)
MLP	0	0.841	(0.838, 0.844)	0.815	(0.812, 0.819)
1	0.845	(0.842, 0.847)	0.817	(0.814, 0.821)
2	0.844	(0.841, 0.846)	0.811	(0.808, 0.815)
XGBoost	0	0.844	(0.842, 0.847)	0.820	(0.817, 0.824)
1	0.843	(0.84, 0.845)	0.819	(0.815, 0.822)
2	0.845	(0.842, 0.847)	0.820	(0.816, 0.823)
Hypertension	CatBoost	0	0.670	(0.665, 0.676)	0.599	(0.597, 0.6)
1	0.671	(0.665, 0.676)	0.599	(0.597, 0.6)
2	0.671	(0.666, 0.677)	0.600	(0.598, 0.601)
FT-Transformer	0	0.666	(0.661, 0.672)	0.604	(0.603, 0.605)
1	0.670	(0.665, 0.676)	0.594	(0.593, 0.596)
2	0.672	(0.666, 0.677)	0.595	(0.594, 0.596)
LightGBM	0	0.678	(0.672, 0.683)	0.634	(0.633, 0.635)
1	0.672	(0.666, 0.677)	0.636	(0.635, 0.637)
2	0.672	(0.667, 0.678)	0.628	(0.627, 0.629)
MLP	0	0.664	(0.658, 0.67)	0.583	(0.582, 0.584)
1	0.669	(0.663, 0.674)	0.597	(0.596, 0.599)
2	0.668	(0.662, 0.673)	0.598	(0.597, 0.599)
XGBoost	0	0.671	(0.665, 0.677)	0.588	(0.587, 0.59)
1	0.669	(0.664, 0.675)	0.586	(0.584, 0.587)
2	0.669	(0.664, 0.675)	0.586	(0.584, 0.587)