SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Performance Comparison with Baselines on Synthetic Dataset.  $(\alpha, \beta)$  control distribution shift intensity, and  $(0,1)$  represents no distribution shift. The best/second-best results are highlighted in boldface/underlined, respectively.
(α,β)	Methods	Adult	ACS-I	ACS-E
Acc (%)↑	ΔDP(%)↓	ΔEO(%)↓	Acc (%)↑	ΔDP(%)↓	ΔEO(%)↓	Acc (%)↑	ΔDP(%)↓	ΔEO(%)↓
(1.0,2.0)	MLP	82.09±0.05	15.11±0.04	14.33±0.05	77.95±0.52	3.51±0.59	3.77±0.55	80.95±0.10	1.10±0.06	1.43±0.06
REG	80.60±0.05	3.79±0.06	3.27±0.08	77.77±0.09	2.28±0.32	2.59±0.23	80.44±0.07	0.86±0.09	1.05±0.10
ADV	78.80±0.68	0.83±0.26	0.79±0.14	75.72±0.63	1.96±0.38	2.00±0.35	79.39±0.15	1.09±0.26	0.95±0.26
FCR	79.06±0.09	9.98±0.06	9.47±0.07	76.99±0.47	2.94±0.34	2.95±0.28	79.74±0.11	0.97±0.21	1.00±0.22
RFR	78.84±0.09	0.44±0.05	0.12±0.06	74.15±0.81	1.84±0.27	1.60±0.33	80.08±0.08	0.71±0.10	0.06±0.11
(1.5,3.0)	MLP	82.05±0.05	15.16±0.09	14.33±0.09	77.85±0.25	3.73±0.53	3.70±0.56	80.42±0.10	1.14±0.07	1.10±0.07
REG	80.64±0.08	3.74±0.11	3.23±0.10	77.87±0.18	2.25±0.28	2.37±0.27	80.21±0.13	0.72±0.04	0.75±0.03
ADV	78.71±0.41	1.07±0.87	0.87±0.96	75.79±0.68	2.22±0.53	2.44±0.48	79.58±0.13	1.07±0.19	1.26±0.18
FCR	79.05±0.12	10.01±0.07	9.51±0.06	77.06±0.68	3.39±0.33	3.10±0.36	79.59±0.26	1.17±0.24	1.08±0.23
RFR	78.91±0.03	0.46±0.10	0.16±0.09	74.19±0.58	1.82±0.29	2.17±0.32	80.47±0.03	0.72±0.04	0.71±0.05
(3.0,6.0)	MLP	82.07±0.05	15.23±0.14	14.45±0.15	77.89±0.45	3.35±0.36	3.47±0.41	80.30±0.04	1.17±0.04	1.13±0.04
REG	80.62±0.07	3.72±0.05	3.21±0.04	78.19±0.12	1.60±0.48	1.84±0.44	80.36±0.09	0.70±0.09	0.68±0.11
ADV	78.97±0.49	1.28±0.74	1.09±0.50	75.71±0.68	2.28±0.39	2.24±0.41	79.66±0.16	1.34±0.14	1.16±0.13
FCR	79.03±0.13	10.00±0.05	9.50±0.05	76.71±0.39	2.97±0.34	3.28±0.31	79.89±0.22	1.06±0.14	1.14±0.18
RFR	80.15±0.07	1.75±0.15	1.30±0.14	74.22±0.56	1.80±0.26	1.89±0.24	80.28±0.12	0.74±0.04	0.51±0.04
Figure 2: The fairness (DP) and prediction (Acc) trade-off performance on three datasets with different synthetic distribution shifts. The units for x- and y-axis are percentages  $(\%)$ .
Additionally, we also adopt  $\mathcal{L}_{RFR}$  as robust fairness regularization. The overall objective is given as follows:
$$
\mathcal {L} _ {a l l} = \mathcal {L} _ {C L F} + \lambda \cdot \left(\mathcal {L} _ {D P} + \mathcal {L} _ {R F R}\right), \tag {13}
$$
where  $\lambda$  is a hyperparameter to balance the model performance and fairness.  $\mathcal{L}_{CLF}$  is the loss function for the downstream task,  $\mathcal{L}_{DP}$  is to ensure the demographic parity on the source dataset, and  $\mathcal{L}_{RFR}$  is to ensure the transferability of fairness from the source dataset to the target dataset.
In the proposed RFR algorithm, we directly apply approximation to accelerate model training. Note that the optimal model weight perturbation is dependent on the current model weight  $\theta$ , two forward and backward propagations are required to calculate the final gradient.
In this section, we conduct experiments to evaluate the effectiveness of our proposed RFR, aiming to answer the three research questions. Q1: How effective is RFR to achieve fairness under distribution shift for synthetic distribution shift? Q2: How effective is RFR for real spatial and temporal distribution shift? Q3: How sensitive is RFR to the key hyperparameter  $\lambda$ ?
In this subsection, we present the experimental setting, including datasets, evaluation metrics, baseline methods, distribution shift generation, and implementation details.
Datasets. We adopt the following datasets in our experiments. UCI Adult [13] dataset contains information about 45,222 individuals with 15 attributes from the 1994 US Census. We consider
gender as the sensitive attribute and the task is to predict whether the income of the person is higher than  $50k or not. ACS-Income [14] is extracted from the American Community Survey (ACS) Public Use Microdata Sample (PUMS) with 3,236,107 samples. We choose gender as the sensitive attribute. Similar to the task in UCI Adult, the task is to predict whether the individual income is above$ 50k. ACS-Employment [14] also derives from ACS PUMS, and we also use gender as the sensitive attribute. The task is to predict whether an individual is employed.
Evaluation Metrics. We use accuracy to evaluate the prediction performance for the downstream task. For fairness metrics, we adopt two common-used quantitative group fairness metrics to measure the prediction bias [15, 16], i.e., demographic parity  $\Delta_{DP} = |\mathbb{P}(\hat{Y} = 1|A = 0) - \mathbb{P}(\hat{Y} = 1|A = 1)|$  and equal opportunity  $\Delta_{EO} = |\mathbb{P}(\hat{Y} = 1|A = 0,Y = 1) - \mathbb{P}(\hat{Y} = 1|A = 1,Y = 1)|$ , where  $A$  represents sensitive attribute,  $Y$  and  $\hat{Y}$  represent the ground-truth label and predicted label, respectively.
Baselines. In our experiments, we consider vanilla multi-layer perceptron (MLP) and two widely adopted in-processing debiasing methods, including fairness regularization (REG), adversarial debiasing (ADV), and fair consistency regularization (FCR). MLP directly uses vanilla 3-layer MLP with 50 hidden unit and ReLU activation function [17] to minimize cross-entropy loss with source dataset. In the experiments, we adopt the same model architecture for all other methods (i.e., REG and ADV). REG adds a fairness-related metric as a regularization term in the objective function to mitigate the prediction bias [18, 19]. Specifically, we directly adopt demographic parity as a regularization term, and the objective function is  $\mathcal{L}_{CLF} + \lambda \mathcal{L}_{DP}$ , where  $\mathcal{L}_{CLF}$  and  $\mathcal{L}_{DP}$  are defined in Eqs. (11) and (12). ADV [20] employs a two-player game to mitigate bias, in which a classification network is trained to predict labels based on input features, and an adversarial network takes the output of the classification network as input and aims to identify which sensitive attribute group the sample belongs to. FCR [21] aims to minimize and balance consistency loss across groups.
Figure 3: DP and Acc trade-off performance on three real-world datasets with temporal (Top) and spatial (Bottom) distribution shift. The trade-off curve close to the right bottom corner means better trade-off performance. The units for x- and y-axis are percentages (\%).
Synthetic and Real Distribution Shift. We adopt synthetic and real distribution shifts. For synthetic distribution shift, we follow work [22, 23] to generate distribution shift via biased sampling. Specifically, we adopt applying principal component analysis (PCA) [24] to retrieve the first principal component  $\mathcal{C}$  from input attributes. Subsequently, we estimate the mean  $\mu(\mathcal{C})$  and standard deviation  $\sigma(\mathcal{C})$ , and set Gaussian distribution  $\mathcal{N}_S(\mu(\mathcal{C}), \sigma(\mathcal{C}))$  to randomly sampling of target dataset. As for source dataset, we choose different parameters  $\alpha$  and  $\beta$  to generate distribution shift using another Gaussian distribution  $\mathcal{N}_{\mathcal{T}}(\mu(\mathcal{C}) + \alpha, \frac{\sigma(\mathcal{C})}{\beta})$  for randomly sampling. The source and target datasets are constructed by sampling without replacement. For real distribution shift, we adopt the sampling
and pre-processing approaches following Folktables [14] to generate spatial and temporal distribution shift via data partition based on different US states and year from 2014 to 2018.
Implementation Details. We run the experiments 5 times and report the average performance for each method. We adopt Adam optimizer with  $10^{-5}$  learning rate and 0.01 weight decay for all models. For baseline ADV, we alternatively train classification and adversarial networks with 70 and 30 epochs, respectively. The hyperparameters for ADV are set as  $\{0.0, 1.0, 10.0, 100.0, 500.0\}$ . For adding regularization, we adopt the hyperparameters set  $\{0.0, 0.5, 1.0, 10.0, 30.0, 50.0\}$ .
4.2 Experimental Results on Synthetic Distribution Shift
In this experiment, we evaluate the effectiveness of our proposed Robust Fairness Regularization (RFR) method with synthetic distribution shifts via biased sampling with different parameters  $(\alpha, \beta)$ . We compare the performance of RFR with several other baseline methods, including standard training, adversarial training, and fairness regularization methods. The results of performance value are presented in Table 1 and the results of fairness-accuracy tradeoff are presented in Figure 2. We have the following observations:
- The results in Table 1 demonstrate that RFR consistently outperforms the baselines in terms of fairness for small distribution shifts, achieving a better balance between fairness and accuracy. For example, RFR achieves an improvement of  $47.0\%$  on metric  $\Delta DP$  and  $84.8\%$  on metric  $\Delta EO$  compared to the second-best method in the Adult dataset with (1.0, 2.0)-synthetic distribution shift. Furthermore, we observed that the outperforms of RFR compared with baselines decrease as the distribution shift intensity increases. The reason is that the approximation RFR involved Taylor expansion over model perturbation and is effective with mild distribution shifts.
- The results in Figure [1] show that RFR achieved a better fairness-accuracy tradeoff compared to the baseline methods for mild distribution shift. We observed that our proposed method achieved a better Pareto frontier compared to the existing methods, and the bias can be mitigated with tiny accuracy drop.
The experimental results show that our method can effectively address the fairness problem under mild distribution shift while maintaining high accuracy, outperforming the existing state-of-the-art baseline methods.
4.3 Experimental Results on Real Distribution Shift
In this experiment, we evaluate the performance of RFR on multiple real-world datasets with real distribution shifts. We use ACS dataset in the experiment. The distribution of this dataset varies across different time periods or geographic locations, which also causes fairness performance degradation under the distribution shift. The results are presented in Table 2 and the results of the fairness-accuracy tradeoff are presented in Figure 3. The results show that our method consistently outperforms the baselines across all datasets, achieving a better balance between fairness and accuracy. Specifically, we have the following observations:
- Table 2 demonstrates that RFR consistently outperforms the baselines across all datasets in terms of fairness-accuracy tradeoff. This suggests that our method is effective in achieving robust fairness under real temporal and spatial distribution shifts. For example, RFR achieved an improvement of  $34.9\%$  on metric  $\Delta DP$  and  $34.4\%$  on metric  $\Delta EO$  compared to the second-best method in ACS-I dataset with temporal distribution shift.
- Figure 3 shows that RFR can achieve better fairness-accuracy tradeoff than baselines, i.e., RFR is particularly effective in addressing the fairness problem on the ACS dataset with source/target data varying across different time periods or geographic locations. This is important for many practical applications, where fairness is a critical requirement, such as in credit scoring, and loan approval.
- The variance for spatial shift on ACS-I is higher than that of temporal shift on all methods, which indicates neural networks easily converge to different local minima for spatial distribution shift. The proposed methods can achieve better fairness results for most cases and the tradeoff results clearly demonstrate the effectiveness.
Figure 4: The hyperparameters study for  $\lambda$ . The Left subfigure and Right subfigure show the results of accuracy and fairness performance, respectively.
Overall, the experimental results on temporal and spatial distribution shifts further support the effectiveness of our proposed method RFR and its potential for practical applications in diverse settings.
Table 2: Performance comparison with baselines on real temporal (the year 2016 to the year 2018) and spatial (Michigan State to California State) distribution shift. The best and second-best results are highlighted with hold and underline, respectively.
Real	Methods	ACS-I	ACS-E
Acc (%)↑	ΔDP(%)↓	ΔEO(%)↓	Acc (%)↑	ΔDP(%)↓	ΔEO(%)↓
2016 → 2018	MLP	77.75±0.44	3.26±0.38	3.48±0.41	80.46±0.05	1.07±0.10	1.02±0.10
REG	77.74±0.62	2.09±0.21	2.27±0.24	80.37±0.12	0.77±0.08	0.74±0.08
ADV	75.94±0.40	2.41±0.49	2.53±0.55	79.62±0.14	1.17±0.14	1.10±0.14
FCR	76.40±0.45	2.81±0.30	2.96±0.30	79.59±0.38	0.95±0.42	0.91±0.34
RFR	77.49±0.32	1.36±0.17	1.49±0.17	80.36±0.05	0.61±0.11	0.58±0.10
MI → CA	MLP	75.62±0.80	5.22±0.86	3.60±0.34	79.02±0.20	0.73±0.07	0.94±0.05
REG	75.52±0.78	2.88±0.44	2.17±0.22	75.34±1.11	0.42±0.09	0.61±0.11
ADV	73.38±1.07	1.04±0.58	0.54±0.38	77.56±0.41	0.61±0.18	0.80±0.13
FCR	74.28±0.35	5.06±0.62	3.67±0.51	77.96±0.22	0.44±0.14	0.67±0.38
RFR	74.63±0.45	1.35±0.39	1.30±0.24	78.84±0.21	0.44±0.09	0.65±0.07
4.4 Hyperparameter Study
In this experiment, we investigate the sensitivity of the hyperparameter  $\lambda$  in Equation  $\mathcal{L}_{all} = \mathcal{L}_{CLF} + \lambda (\mathcal{L}_{DP} + \mathcal{L}_{RFR})$  for spatial and temporal distribution shift across different datasets. Specifically, we tune the hyperparameter as  $\lambda = \{0.0, 0.1, 0.5, 1.0, 3.0, 5.0, 10.0\}$ . From the results in Figure 4, it is seen that the accuracy and demographic parity are both sensitive to hyperparameter  $\lambda$ , which implies the capability of accuracy-fairness control. With the increase of  $\lambda$ , accuracy decreases while demographic parity also decreases. When  $\lambda$  is smaller than 5, accuracy drops slowly while demographic parity drops faster. Such observation represents that an appropriate hyperparameter can mitigate prediction bias while preserving comparable prediction performance.
In this section, we present two lines of related work, including fairness in machine learning and distribution shift.
Fairness in Machine Learning. Fairness [25]; -31] is a legal requirement for machine learning models for various high-stake real-world predictions, such as healthcare [7 32 33], education [34 36], and job market [37 38]. Achieving fairness, either from a data or model perspective [39-42], in machine learning is a challenging problem. As such, there has been an increasing interest in both the industrial and research community to develop algorithms to mitigate these issues and ensure that machine learning models make fair and unbiased decisions. Extensive efforts led to the development of various techniques and metrics for fairness and proposed various definitions of fairness, such as group fairness [43-47, 40], individual fairness [48-53], and counterfactual fairness [54-56]. In this paper, we focus on group fairness, and the widely used methods to achieve group fairness are fair regularization and adversarial debias method. [19, 18] proposed to add a fairness regularization term
to the objective function to achieve group fairness, and [20] proposes to jointly train a classification network and an adversarial network to mitigate the bias for different demographic groups to achieve group fairness. Overall, ensuring fairness in machine learning is a critical and ongoing research area that will continue to be an important focus for the development of responsible machine learning.
Distribution Shift. Previous work [1-4,57] reveals that a classifier trained on a source distribution will perform worse on a given target distribution because of the distribution shift, and recently extensive works have explored the influence of distribution shift in model prediction. Moreover, distribution shift can significantly affect the fairness performance of machine learning models. The sensitivity of fairness to the distribution shift is notorious for the legal requirement. The degraded performance of fair models under a distribution shift would trigger new bias and discrimination issues. There are some works [23,58,21,59,60] that solve fairness under various distribution shifts. For example, [23] explore the fairness under covariate shift, where the inputs change with the in-distribution label. [60] proposes Shifty algorithms to hold fairness guarantees when the dataset in the deployment environment is out-of-distribution of the training datasets (distribution shift). Our work is different from those prior works from model weight perturbation perspective.
This paper aims the solve the fairness problem under the distribution shifts from the model weight perturbation perspective. We first establish a theoretical connection between distribution shift, data perturbation, and model weight perturbation, which allowed us to conclude that distribution shift and model perturbation are equivalent. We then propose a sufficient condition for ensuring fairness transference under distribution shift. To explicitly chase such sufficient conditions, we introduce Robust Fairness Regularization (RFR) method based on the established understanding, to achieve robust fairness. Our experiments on both synthetic and real distribution shifts demonstrate the effectiveness of RFR in achieving a better fairness-accuracy tradeoff compared to existing baselines. We believe that our understanding of distribution shift is valuable and intriguing to the development of robust machine learning models, and the proposed RFR approach can be of great practical value to build fair and robust machine learning models in real-world applications.
The authors thank the anonymous reviewers for their helpful comments. The work is in part supported by NSF grants NSF IIS-2224843, IIS-1939716, and IIS-1900990. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
[54] Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. NeurIPS, 30, 2017.
[55] Chirag Agarwal, Himabindu Lakkaraju, and Marinka Zitnik. Towards a unified framework for fair and stable graph representation learning. In UAI 2021: Uncertainty in Artificial Intelligence, 2021.
[56] Aooqi Zuo, Susan Wei, Tongliang Liu, Bo Han, Kun Zhang, and Mingming Gong. Counterfactual fairness with partially known causal graph. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[57] Chia-Yuan Chang, Yu-Neng Chuang, Guanchu Wang, Mengnan Du, and Zou Na. Dispel: Domain generalization via domain-specific liberating. arXiv preprint arXiv:2307.07181, 2023.
[58] Jessica Schrouff, Natalie Harris, Oluwasanmi Koyejo, Ibrahim Alabdulmohsin, Eva Schnider, Krista Opsahl-Ong, Alex Brown, Subhrajit Roy, Diana Mincu, Christina Chen, et al. Maintaining fairness across distribution shift: do we have viable solutions for real-world applications? arXiv preprint arXiv:2202.01034, 2022.
[59] Harvineet Singh, Rina Singh, Vishwali Mhasawade, and Rumi Chunara. Fairness violations and mitigation under covariate shift. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 3-13, 2021.
[60] Stephen Giguere, Blossom Metevier, Bruno Castro da Silva, Yuriy Brun, Philip Thomas, and Scott Niekum. Fairness guarantees under demographic shift. In International Conference on Learning Representations, 2022.
[61] Jiongli Zhu, Sainyam Galhotra, Nazanin Sabri, and Babak Salimi. Consistent range approximation for fair predictive modeling. Proceedings of the VLDB Endowment, 16(11):2925-2938, 2023.
[62] Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent YF Tan. Efficient sharpness-aware minimization for improved training of neural networks. International Conference on Learning Representations, 2022.
[63] Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware minimization. In International Conference on Machine Learning, pages 639-668. PMLR, 2022.
[64] Yuzhe Lu, Zhenlin Wang, Runtian Zhai, Soheil Kolouri, Joseph Campbell, and Katia Sycara. Predicting out-of-distribution error with confidence optimal transport. arXiv preprint arXiv:2302.05018, 2023.
[65] Yaodong Yu, Zitong Yang, Alexander Wei, Yi Ma, and Jacob Steinhardt. Predicting out-of-distribution error with the projection norm. In International Conference on Machine Learning, pages 25721-25746. PMLR, 2022.