SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(g) ACS Pub. Cov (2010→2017)
Figure 11. Target vs. source accuracies for 22 algorithms and datasets in our benchmark. Each point represents one hyperparameter configuration. (a)-(b): two examples of ACS Income dataset with California (CA) as the source state, and Puerto Rico (PR) and South Dakota (SD) as targets. (c)-(g): five examples of ACS Mobility, Taxi, ACS Pub. Cov, US Accident datasets. (h): simulated covariate shifts on on sub-sampled ACS Income dataset.
(h) ACS Income (Young  $\rightarrow$  Old)
Analysis From the numbers of numerical results in that Table, we give a more detailed analysis corresponding with our main body.
- Different algorithms do not exhibit consistent rankings over different distribution shift patterns. In Table 5, we boldface the best target performance within each class of methods. And we show the top-3 classes at the bottom. The results show that the algorithmic rankings across different settings and tasks are quite different. And even the rankings of algorithms within the same class vary a lot. This further demonstrates the complexity of  $Y|X$  shifts. Also, Figure 10 shows that even for the fixed source state, algorithmic rankings vary a lot across different target states.
- Tree ensemble methods show competitive performances but do not significantly eliminate the generalization error between source and target data, characterized by the difference
between o.o.d. and i.i.d. model performance. From the bottom, we could see that tree ensemble methods achieve top-3 performances in most of the selected settings (6 out of 7), exhibiting their superiority on tabular data. However, the performance degradation between source and target is still large.
- Imbalance methods and fairness methods show similar performance with the base learner (XGBoost). We find these methods do not have a significant improvement over the base learner (XGBoost). This may be due to that they are not designed to incorporate tree ensemble methods.
D.6 Full Results of 22 Settings
In this section, we provide the full results of our 22 settings in Table 3.
D.6.1 Average Accuracy
In Table 6, Table 7 and Table 8, we report the source accuracy and the target accuracy of all algorithms. Since in these settings, we have multiple target domains, as for the target accuracy of each algorithm, we report the average accuracy as well as the standard deviation calculated on all target domains. Here the standard deviation reflects the stability of out-of-distribution generalization performances across different target domains different instead of random seeds.
From the average result over, Table 6, Table 7 and Table 8, we can still obtain similar findings compared to that from the 7 selected settings in the main body. Especially, while tree ensemble methods serve as a strong in-sample benchmark illustrated by their competitive i.i.d. performance, these methods do not yield better performance than other methods in the target distribution uniformly. And when we average across different target domains, DRO methods do not perform better than their empirical counterpart, indicating that the worst-case optimization from the uncertainty sets constructed by existing DRO methods can rarely occur in practical setups and does not help much compared with standard ERM cases. This tricky part of worst-case intervention also holds when we compare imbalanced learning and fairness methods with the XGB base learner. These methods can lead to better o.o.d. results than the base learner sometimes indeed, while this may not hold true when we average over different target domains. In general, existing methods cannot generalize well averaging over different target domains, and it would be interesting to develop methods to close that gap.
D.7 Worst-Domain Accuracy
Another widely-used metric in evaluating generalization performance is worst-case accuracy. In settings built on ACS Income, ACS Pub.Cov, ACS Mobility, and US Accident, for the target accuracy of each algorithm, we report the worst target domain accuracy in Table 9 and Table 10. For other settings, we do not report the worst-case accuracy because there are not many target domains, and the average& standard deviation results could reflect the population generalization performance well. In terms of the worst target accuracy for each setting, we usually observe a large performance degradation (usually over a 10 percent performance drop) in all of the existing methods. And algorithms do not show a consistent and stable ranking performance in the o.o.d. performance. DRO / Imbalanced learning methods can perform quite well in one o.o.d. setup but are dominated by other methods in some other cases. This demonstrates a need to carefully understand the difference between different domains first and develop corresponding methods then.
D.8 Average Macro-F1 Score
In consideration of the label imbalance in real-world tabular datasets, we also calculate the Macro-F1 Score for all algorithms under our settings. In this section, we report the average results in Table 11, Table 12, and Table 13. Here the standard deviation reflects the stability of out-of-distribution generalization performances (not the randomness).
D.9 Worst-Domain Macro-F1 Score
In settings built on ACS Income, ACS Pub.Cov, ACS Mobility, and US Accident, for the target accuracy of each algorithm, we report the worst target domain Macro-F1 score in Table 14 and
Table 8. Average target domain accuracy of all algorithms on the Taxi, ACS Pub. Cov (Temporal) and ACS Income (Synthetic) datasets. In Taxi and ACS Pub. Cov (Temporal), we test all algorithms on the 3 target domains and report the mean and standard deviation across all 3 target domains. The standard deviation here reflects the stability of performances across different target states. In ACS Income (Synthetic), we simulate strong covariate shifts according to the "Age" feature, where  $10\%$  and  $20\%$  means the minor group ratio in training, respectively.
Dataset Source	Taxi	ACS Pub. Cov (Temporal)	ACS Income (Synthetic)
BOG	NYC	NY	CA	20%	10%
i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d
Base Methods	LR	75.6	71.2 ±2.0	83.6	74.2 ±1.2	76.3	68.5 ±2.3	80.8	68.2 ±4.0	92.2	81.6	94.0	79.3
SVM	75.7	76.1 ±3.1	83.4	74.4 ±1.4	77.4	69.1 ±2.0	80.7	67.7 ±3.9	92.0	80.0	94.1	79.5
MLP	77.7	70.0 ±1.3	84.5	73.7 ±0.6	78.0	70.6 ±2.4	82.5	70.3 ±3.8	92.5	82.1	94.3	79.5
Tree Ensemble Methods	Random Forest	76.1	77.5 ±4.7	85.7	71.6 ±0.9	79.1	71.3 ±2.5	83.4	70.4 ±4.1	92.7	81.7	94.4	78.5
LightGBM	86.3	73.2 ±4.3	87.0	72.0 ±0.9	79.7	72.1 ±2.4	84.0	70.7 ±3.9	92.6	83.0	94.2	80.4
XGBoost	85.0	71.5 ±2.7	86.7	72.5 ±1.0	79.3	71.9 ±2.6	83.6	71.1 ±3.7	92.7	82.2	94.4	79.7
DRO Methods	χ2-DRO	77.4	72.2 ±2.4	84.6	73.0 ±0.8	77.3	69.0 ±2.5	81.9	70.0 ±3.8	92.5	81.3	94.2	79.7
CVaR-DRO	77.0	73.4 ±2.5	85.9	72.8 ±0.6	76.6	69.1 ±2.4	82.2	69.1 ±4.5	92.5	81.2	94.3	79.0
χ2-DORO	75.8	73.3 ±2.9	84.9	73.9 ±0.8	71.7	61.2 ±3.5	79.7	64.8 ±3.9	92.3	81.4	93.8	77.4
CVaR-DORO	75.6	74.3 ±3.0	85.1	73.8 ±0.8	76.0	67.9 ±2.6	79.5	64.9 ±4.0	92.5	81.4	93.9	76.7
Group DRO	N/A	N/A	N/A	N/A	76.8	70.5 ±2.1	81.9	70.0 ±3.9	92.5	81.4	94.2	79.8
Imbalanced Learning Methods	SUBY	87.2	67.1 ±1.6	85.9	70.8 ±0.5	73.9	69.8 ±1.3	81.4	68.3 ±3.8	87.0	69.9	88.5	64.2
RWY	81.2	73.0 ±1.4	86.7	72.4 ±0.7	75.1	70.9 ±1.6	79.4	65.8 ±4.1	90.3	78.6	92.1	76.3
SUBG	86.3	72.5 ±4.4	85.6	71.4 ±0.9	79.4	71.8 ±2.3	83.7	70.7 ±3.6	92.6	81.7	94.4	79.5
RWG	81.1	69.1 ±3.6	86.5	71.3 ±0.9	79.4	72.8 ±2.2	83.8	70.8 ±3.7	92.6	82.7	94.5	80.1
JTT	84.6	69.5 ±0.9	85.4	71.2 ±0.8	73.8	66.7 ±2.3	79.5	67.4 ±3.5	91.5	78.9	93.5	76.5
DWR	82.9	68.2 ±1.9	83.7	70.7 ±0.9	79.2	71.3 ±2.4	83.8	70.7 ±3.9	92.5	82.6	94.2	80.3
Fairness Methods	DP	84.1	70.2 ±4.0	86.3	72.0 ±0.8	79.6	72.7 ±2.0	83.7	71.1 ±3.9	92.5	82.7	94.2	80.2
EO	81.1	71.9 ±3.0	87.1	71.8 ±0.8	79.5	72.3 ±2.5	83.4	70.1 ±4.0	92.6	81.7	94.3	79.5
EP	79.7	71.6 ±1.7	85.7	70.9 ±0.6	79.6	72.5 ±2.1	83.6	71.0 ±3.9	92.4	82.6	94.2	80.0
Exp	79.2	68.6 ±0.4	84.9	71.6 ±0.6	79.3	72.3 ±2.3	83.4	71.1 ±4.2	92.6	81.9	94.3	79.4
Threshold	80.8	69.8 ±1.7	84.3	59.7 ±3.9	79.2	71.9 ±2.2	83.4	70.4 ±3.9	91.8	78.3	93.5	74.0
Table 9. Worst target domain accuracy of all algorithms on the ACS Income and ACS Pub. Cov datasets. In each setting, we test all algorithms on the 50 American states and report the worst accuracy among all 50 target states.
DatasetSource State	ACS INCOME	ACS PUB. COV
CA	CT	MA	SD	FL	TX	NE	IN
i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d
Base Methods	LR	81.1	72.3	80.7	70.5	80.9	71.1	80.1	69.4	81.3	49.6	84.1	49.0	84.3	0.52	78.6	54.7
SVM	79.4	67.1	78.6	61.6	80.1	62.2	79.3	64.6	80.5	53.0	84.2	50.3	84.8	50.4	77.1	57.4
MLP	81.8	68.3	81.4	70.8	81.7	71.3	80.5	70.5	83.3	54.3	85.4	53.9	84.5	52.9	78.8	56.7
Tree Ensemble Methods	Random Forest	81.7	72.6	81.2	71.6	81.5	71.2	80.2	68.3	84.6	54.2	86.8	52.8	86.7	52.9	81.2	59.0
LightGBM	81.8	71.1	82.1	69.5	82.3	70.1	80.2	69.6	84.6	56.9	86.9	53.9	86.9	52.3	81.2	60.4
XGBoost	82.0	72.2	81.9	70.1	82.2	70.9	80.5	70.5	84.6	54.8	86.7	53.7	87.3	55.4	81.2	59.6
DRO Methods	\( \chi^2-DRO \)	81.8	69.7	81.2	69.7	81.6	67.9	80.2	70.3	82.9	52.5	85.4	50.5	83.4	52.9	78.4	62.4
CVaR-DRO	81.4	72.1	81.2	70.7	82.0	70.0	79.7	68.4	82.6	52.2	85.2	51.7	83.3	52.3	77.8	53.9
\( \chi^2-DORO \)	81.1	70.3	80.4	62.6	81.2	63.8	78.8	69.5	78.8	43.6	82.1	43.6	82.2	43.6	72.4	49.9
CVaR-DORO	80.9	68.7	80.5	64.9	81.1	64.3	79.6	69.1	80.7	49.5	84.3	52.4	81.9	43.6	76.4	56.2
Group DRO	81.9	72.6	81.3	71.8	81.7	69.2	79.8	0.68	83.3	53.2	85.4	53.2	83.5	51.3	72.2	43.6
Imbalanced Learning Methods	SUBY	81.0	65.2	81.5	68.0	82.0	66.3	74.8	72.5	79.8	51.3	82.3	45.4	77.8	64.5	77.8	68.3
RWY	81.5	65.2	81.8	68.3	82.1	67.6	77.7	73.2	80.7	53.3	82.5	45.6	83.7	57.1	78.7	65.8
SUBG	81.9	71.4	81.6	70.1	82.0	68.4	80.2	70.3	84.3	56.1	86.7	53.0	86.3	53.6	80.8	60.2
RWG	82.0	71.3	81.9	70.6	82.2	70.3	80.2	70.6	84.5	56.8	86.8	52.1	86.7	55.1	81.0	57.2
JTT	78.3	68.4	78.1	71.2	79.1	71.6	77.9	68.6	80.6	59.4	83.1	56.2	84.7	55.4	77.6	56.7
DWR	81.9	71.2	81.6	71.0	82.1	69.0	80.1	72.0	84.3	58.1	86.8	54.2	85.9	52.8	80.6	59.4
Fairness Methods	DP	81.3	70.3	81.0	68.8	81.8	70.7	79.3	68.4	84.3	57.4	86.6	53.7	86.4	52.9	80.6	57.8
EO	81.8	70.6	81.8	70.5	82.1	70.8	80.0	70.6	84.4	55.4	86.8	53.6	86.4	53.6	80.8	61.6
EP	81.8	70.7	81.9	69.8	82.1	70.9	80.3	69.2	84.4	55.5	86.8	54.4	86.4	53.6	80.9	60.1
Exp	81.4	70.6	81.0	69.4	81.8	70.4	79.1	68.8	84.5	55.4	86.8	52.8	86.4	52.5	80.9	61.5
Threshold	77.9	71.8	77.0	69.8	77.5	70.9	78.0	62.8	84.3	57.3	86.2	52.4	85.8	52.9	80.0	59.9
Table 10. Worst target domain accuracy of all algorithms on the ACS Mobility and US Accident datasets. In ACS Mobility, we test all algorithms on the 50 American states and report the worst accuracy among all 50 target states. And in US Accident, we choose 13 American states with relatively high sample sizes in testing.
DatasetSource State	ACS MOBILITY	US ACCIDENT
NY	CA	MS	PA	CA	FL	TX	MN
i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d
Base Methods	LR	78.6	64.7	77.1	64.6	77.3	64.8	75.4	66.3	83.2	44.9	90.7	48.3	95.1	57.4	86.4	52.8
SVM	79.8	64.6	77.7	66.8	79.4	65.3	77.1	64.7	83.2	49.2	88.7	58.3	93.7	55.7	88.6	56.1
MLP	78.7	64.7	77.1	64.8	78.1	66.3	75.9	67.0	84.7	58.4	92.7	61.1	94.9	57.2	88.9	57.1
Tree Ensemble Methods	Random Forest	80.4	68.0	79.0	68.2	80.6	67.5	78.6	68.7	86.6	66.6	93.5	58.2	95.2	58.5	91.6	68.0
LightGBM	80.6	68.7	78.7	69.0	80.0	67.9	78.6	69.8	87.1	65.3	93.3	59.0	95.4	58.6	92.2	68.1
XGBoost	80.4	68.6	79.0	69.0	80.9	68.3	79.0	69.4	87.1	65.6	94.6	59.0	95.3	58.8	92.5	68.5
DRO Methods	\( \chi^2-DRO \)	78.5	65.1	77.5	66.6	76.8	65.4	76.0	67.1	84.4	62.9	92.6	53.3	95.0	57.9	89.0	53.5
CVaR-DRO	78.6	65.0	77.2	65.6	77.7	64.4	75.7	67.6	84.7	57.2	92.7	57.5	95.0	58.4	89.5	56.3
\( \chi^2-DORO \)	78.6	64.3	77.8	66.9	77.1	64.7	75.3	64.7	84.7	53.9	92.3	61.1	94.3	57.0	87.6	59.8
CVaR-DORO	78.7	64.7	78.0	67.4	77.4	64.7	75.3	64.9	84.9	57.8	92.8	58.1	94.9	57.5	89.1	57.6
Group DRO	78.6	65.5	77.1	65.1	76.0	64.5	76.0	66.8	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
Imbalanced Learning Methods	SUBY	78.2	65.0	76.3	64.6	76.6	64.7	74.7	65.0	86.5	64.2	93.7	58.8	95.2	58.5	91.8	65.6
RWY	78.5	64.7	76.7	65.0	76.5	65.6	74.5	64.6	86.9	64.8	93.4	58.3	95.3	58.5	92.5	67.5
SUBG	80.2	68.8	79.2	68.9	79.5	67.0	78.6	69.2	86.3	66.4	93.0	58.1	95.1	58.4	91.1	67.4
RWG	80.3	68.4	79.1	69.4	80.1	68.3	78.7	69.4	87.5	65.4	93.4	59.4	95.2	58.4	92.4	68.6
JTT	78.3	64.7	76.5	64.7	77.6	66.6	75.1	65.6	86.5	65.9	92.7	58.6	94.9	58.6	91.8	68.2
DWR	79.7	67.4	78.3	68.5	79.1	67.4	77.5	69.0	87.2	65.8	93.4	58.1	95.3	58.5	92.3	67.6
Fairness Methods	DP	80.5	69.2	78.8	68.4	80.0	68.2	78.3	68.7	86.6	65.2	92.8	59.0	94.8	58.4	91.9	66.6
EO	80.3	69.3	78.9	68.9	79.8	0.68	78.5	68.5	86.6	65.6	93.2	58.8	95.2	58.5	92.2	68.1
EP	80.3	68.7	78.9	68.5	80.1	68.1	78.4	68.5	87.0	65.6	93.3	59.0	95.1	58.6	91.9	67.9
Exp	80.5	69.0	78.9	68.6	81.0	68.8	78.4	69.0	85.9	65.4	90.1	58.7	93.1	56.5	86.3	64.7
Threshold	80.5	69.0	78.6	69.2	78.2	66.7	78.5	68.7	84.3	64.5	88.1	59.5	91.2	55.7	82.0	64.0
Table 14. Worst target domain Macro-F1 Score of all algorithms on the ACS Income and ACS Pub. Cov datasets. In each setting, we test all algorithms on the 50 American states and report the worst Macro-F1 Score among all 50 target states.
DatasetSource State	ACS INCOME	ACS PUB. COV
CA	CT	MA	SD	FL	TX	NE	IN
i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d
Base Methods	LR	0.804	0.610	0.807	0.597	0.809	0.599	0.714	0.644	0.652	0.478	0.644	0.432	0.653	0.468	0.698	0.514
SVM	0.795	0.539	0.790	0.505	0.793	0.521	0.698	0.599	0.663	0.493	0.665	0.475	0.698	0.457	0.696	0.559
MLP	0.814	0.574	0.814	0.591	0.816	0.578	0.734	0.669	0.709	0.527	0.713	0.518	0.698	0.496	0.716	0.569
Tree Ensemble Methods	Random Forest	0.809	0.608	0.812	0.599	0.814	0.595	0.687	0.600	0.715	0.503	0.727	0.484	0.729	0.492	0.746	0.573
LightGBM	0.811	0.595	0.821	0.584	0.823	0.586	0.732	0.644	0.730	0.545	0.739	0.501	0.745	0.474	0.750	0.601
XGBoost	0.814	0.602	0.819	0.588	0.822	0.591	0.723	0.632	0.723	0.547	0.730	0.489	0.755	0.525	0.747	0.580
DRO Methods	\( \chi^2-DRO \)	0.813	0.583	0.812	0.584	0.816	0.570	0.732	0.669	0.704	0.536	0.697	0.514	0.685	0.544	0.715	0.588
CVaR-DRO	0.808	0.583	0.812	0.591	0.820	0.584	0.730	0.643	0.693	0.517	0.706	0.509	0.668	0.603	0.700	0.549
\( \chi^2-DORO \)	0.801	0.588	0.804	0.535	0.812	0.543	0.717	0.662	0.526	0.476	0.518	0.431	0.526	0.449	0.516	0.419
CVaR-DORO	0.802	0.560	0.804	0.543	0.811	0.539	0.720	0.655	0.660	0.481	0.675	0.474	0.559	0.444	0.649	0.562
Group DRO	0.812	0.605	0.813	0.596	0.817	0.580	0.741	0.677	0.706	0.511	0.717	0.513	0.700	0.504	0.419	0.304
Imbalanced Learning Methods	SUBY	0.807	0.562	0.815	0.574	0.820	0.559	0.719	0.606	0.710	0.642	0.710	0.609	0.704	0.616	0.748	0.660
RWY	0.812	0.553	0.818	0.576	0.821	0.569	0.733	0.645	0.725	0.646	0.729	0.602	0.736	0.555	0.749	0.654
SUBG	0.813	0.603	0.816	0.588	0.820	0.584	0.726	0.637	0.719	0.566	0.731	0.552	0.739	0.517	0.742	0.542
RWG	0.814	0.597	0.819	0.593	0.822	0.588	0.726	0.659	0.723	0.542	0.733	0.485	0.748	0.515	0.746	0.591
JTT	0.777	0.576	0.781	0.601	0.790	0.602	0.710	0.649	0.711	0.618	0.726	0.574	0.727	0.525	0.718	0.548
DWR	0.812	0.595	0.816	0.597	0.821	0.579	0.733	0.658	0.721	0.540	0.731	0.492	0.732	0.562	0.741	0.608
Fairness Methods	DP	0.807	0.585	0.810	0.582	0.817	0.593	0.716	0.658	0.727	0.569	0.729	0.507	0.750	0.488	0.742	0.587
EO	0.813	0.599	0.818	0.590	0.820	0.591	0.721	0.632	0.724	0.544	0.734	0.516	0.737	0.514	0.748	0.601
EP	0.811	0.596	0.819	0.588	0.821	0.580	0.723	0.646	0.724	0.561	0.731	0.487	0.747	0.539	0.748	0.589
Exp	0.808	0.586	0.810	0.579	0.818	0.585	0.706	0.626	0.724	0.551	0.729	0.484	0.738	0.476	0.741	0.611
Threshold	0.766	0.548	0.768	0.570	0.771	0.590	0.665	0.611	0.722	0.548	0.717	0.493	0.714	0.504	0.728	0.568
Table 15. Worst target domain Macro-F1 Score of all algorithms on the ACS Mobility and US Accident datasets. In ACS Mobility, we test all algorithms on the 50 American states and report the worst Macro-F1 Score among all 50 target states. And in US Accident, we choose 13 American states with relatively high sample sizes in testing.
Dataset Source State	ACS Mobility	US Accident
NY	CA	MS	PA	CA	FL	TX	MN
i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d	i.d.	o.o.d
Base Methods	LR	0.451	0.397	0.462	0.399	0.531	0.450	0.527	0.449	0.800	0.437	0.898	0.489	0.936	0.573	0.855	0.528
SVM	0.604	0.544	0.592	0.527	0.635	0.489	0.623	0.539	0.793	0.505	0.891	0.568	0.928	0.542	0.880	0.540
MLP	0.600	0.530	0.622	0.560	0.607	0.551	0.608	0.535	0.827	0.592	0.923	0.595	0.934	0.567	0.886	0.548
Tree Ensemble Methods	Random Forest	0.570	0.482	0.587	0.514	0.641	0.492	0.635	0.510	0.846	0.613	0.932	0.579	0.938	0.584	0.911	0.642
LightGBM	0.637	0.554	0.625	0.545	0.671	0.531	0.660	0.559	0.851	0.617	0.929	0.582	0.940	0.584	0.918	0.639
XGBoost	0.637	0.557	0.625	0.566	0.678	0.521	0.664	0.560	0.852	0.620	0.942	0.582	0.939	0.585	0.920	0.636
DRO Methods	\( \chi^2-DRO \)	0.613	0.571	0.599	0.539	0.540	0.374	0.606	0.539	0.824	0.596	0.922	0.534	0.933	0.573	0.883	0.534
CVaR-DRO	0.621	0.551	0.628	0.557	0.574	0.512	0.639	0.518	0.825	0.591	0.924	0.572	0.936	0.582	0.888	0.561
\( \chi^2-DORO \)	0.509	0.464	0.619	0.568	0.530	0.474	0.540	0.481	0.815	0.540	0.920	0.592	0.927	0.569	0.874	0.590
CVaR-DORO	0.533	0.500	0.625	0.531	0.521	0.479	0.510	0.443	0.825	0.607	0.922	0.569	0.934	0.575	0.887	0.571
Group DRO	0.588	0.516	0.566	0.480	0.564	0.491	0.595	0.508	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
Imbalanced Learning Methods	SUBY	0.657	0.590	0.649	0.548	0.674	0.565	0.678	0.568	0.854	0.623	0.933	0.585	0.938	0.582	0.915	0.635
RWY	0.657	0.574	0.647	0.573	0.678	0.557	0.677	0.576	0.856	0.618	0.930	0.581	0.940	0.582	0.920	0.637
SUBG	0.635	0.546	0.624	0.542	0.672	0.525	0.658	0.552	0.847	0.627	0.925	0.578	0.937	0.582	0.906	0.635
RWG	0.637	0.542	0.621	0.553	0.676	0.521	0.663	0.539	0.855	0.614	0.929	0.588	0.938	0.583	0.919	0.635
JTT	0.639	0.555	0.616	0.546	0.660	0.536	0.639	0.528	0.843	0.611	0.922	0.581	0.935	0.583	0.912	0.635
DWR	0.622	0.553	0.620	0.550	0.656	0.513	0.647	0.561	0.853	0.623	0.929	0.576	0.939	0.583	0.917	0.627
Fairness Methods	DP	0.630	0.555	0.621	0.556	0.663	0.536	0.668	0.557	0.847	0.618	0.922	0.590	0.933	0.582	0.915	0.630
EO	0.632	0.552	0.625	0.547	0.665	0.549	0.662	0.543	0.846	0.614	0.926	0.584	0.938	0.583	0.917	0.634
EP	0.633	0.550	0.620	0.552	0.671	0.529	0.660	0.563	0.852	0.621	0.928	0.587	0.937	0.583	0.914	0.636
Exp	0.633	0.544	0.622	0.551	0.669	0.531	0.663	0.544	0.837	0.613	0.893	0.580	0.909	0.564	0.854	0.619
Threshold	0.629	0.551	0.620	0.550	0.661	0.528	0.658	0.551	0.820	0.605	0.870	0.583	0.885	0.560	0.809	0.601
Table 15. For other settings, we do not report the worst-case results because there are not many target domains there and the mean/standard deviation results could reflect the generalization performance well.
For the results of Macro-f1 Score reported in Tables 11, 12, 13, 14 and 15, we find similar conclusion patterns to the accuracy score, while we usually witness a wider gap between o.o.d. and i.i.d. model performance.
D.10 Agreement and Maintenance Plan
Hosting Platform. We will use Github as the hosting platform of our code. We provide detailed preprocessing Python scripts to guide users to replicate and process data from scratch. We also illustrate methodologies to run full experiment results in the code. We also list the license for each dataset and user guidance in the Readme file in that Github.
Dependencies. The benchmark is built upon Python 3.8+ and depends on PyTorch, aif360, fairlearn, xgboost, lightgbm. Besides, it uses numpy, scipy, and pandas for basic data manipulation.
Maintenance Plan. The datasets provided here will be maintained by the authors of the paper, which can be contacted by raising an issue on GitHub or by contacting the first authors directly. Our benchmark and a simple open-sourced Python package based on that may be updated at the discretion of authors in the future, which includes more refined algorithm implementations, datasets, and framework with improved efficiency.
Author Statements. To the best of our knowledge, the proposed benchmark is based on existing datasets and does not violate any existing licenses non contain personally identifiable or privacy-related information. And we claim all the responsibility in case of a violation of rights if such a violation were to exist.