SlowGuess/NAACL_2022_OCR huggingface dataset:
the drawback of dataset imbalance methods is that they lack the flexibility to control the performance-fairness tradeoff. Both Constrained and Fair-Batch also effectively reduce bias and achieve improved performance. Both of our methods,  $\mathsf{EO}_{\mathrm{CLA}}$  and  $\mathsf{EO}_{\mathrm{GLB}}$ , achieve competitive performance on the main task with the largest bias reduction. For all models except INLP, we can see that incorporating debiasing techniques leads to improved performance on the main task. We hypothesise that incorporating debiasing techniques (either in the form of adversarial training, data imbalance methods, or optimising towards equal opportunity) acts as a form of regularisation, thereby reducing the learned correlation between the protected attribute and main task label, and encouraging models to learn task-specific representations.
Performance-Fairness tradeoff. We plot the tradeoff between  $\mathrm{F}_{1}^{\mathrm{micro}}$  and GAP for all models on the Moji test set in Figure 1. In this, we vary the most-sensitive hyperparameter for each model: the number of iterations for INLP, the  $\lambda$  weight for adversarial loss for Adv, the step size of adjusting resampling probability for FairBatch, and the weight for minimising the loss difference for  $\mathsf{EO}_{\mathrm{CLA}}$  and  $\mathsf{EO}_{\mathrm{GLB}}$ . As we can see, INLP has limited capacity to reduce bias, and the performance for the main task is slightly worse than the other methods. Com
Figure 1:  $\mathrm{F}_1^{\mathrm{micro}}$  vs. GAP of different models on the Moji test set, as we vary the most sensitive hyperparameter for each model.
pared with Adv, Constrained, and FairBatch, our proposed methods  $\mathsf{EO}_{\mathrm{CLA}}$  and  $\mathsf{EO}_{\mathrm{GLB}}$  achieve fairer predictions while maintaining competitive performance (bottom right). Another advantage of our methods is that they allow for greater variability in the performance—fairness tradeoff, demonstrating the effectiveness and superiority of our proposed method. Note that only the pareto points for each model are plotted. For example, for Adv, we experimented with 7 values of  $\lambda$ , but the results are captured by only two pareto points.
4.5 Profession Classification
4.5.1 Task and Dataset
For our second dataset, the task is to predict a person's occupation given their biography (De-Arteaga et al., 2019), where each short online biography is labelled with one of 28 occupations (main task label) and binary gender (protected attribute). Following previous work (De-Arteaga et al., 2019; Ravfogel et al., 2020), the number of instances in the training, dev, and test sets are 257K, 40K, and 99K, respectively.3
4.5.2 Implementation Details
Following the work of Ravfogel et al. (2020), we use the "CLS" token representation of the pretrained uncased BERT-base (Devlin et al., 2019) to obtain text representations, and keep BERT fixed during training. Hyperparameter settings for all models are provided in Appendix A.3.
For CE, DS, and RW, there is no hyperparameter that controls the tradeoff between model performance and bias reduction.
3There are slight differences between our dataset and that used by previous studies (De-Arteaga et al., 2019; Ravfogel et al., 2020) as a small number of biographies were no longer available on the web when we crawled them.
Model	F1macro↑	F1micro↑	GAP↓
CE	75.95±0.10†	82.19±0.04†	16.68±0.46
INLP	71.44±0.40	79.54±0.18	13.52±1.54
Adv	70.88±2.31	79.72±1.02	16.78±0.87
DS	67.73±0.26	78.48±0.10	9.17±0.41
RW	69.21±0.36	76.18±0.32	8.58±0.49†
FairBatch	75.14±0.28	81.82±0.07	10.80±1.04
EOCLA	72.07±0.18	81.52±0.06	12.80±0.42
EOGLB	75.11±0.18	81.74±0.07	12.72±0.51
Table 2: Experimental results on the Bios test set (averaged over 10 runs). The best result is marked with “†” if the difference over the next-best method is statistically significant (based on a one-tailed Wilcoxon signed-rank test;  $p < 0.05$ ), noting that if the best method is one of our methods, we compare it to the next-best method which is not our own.
4.5.3 Experimental Results
Table 2 shows the results on the Bios test set. We can see that Adv is unable to reduce GAP even at the cost of performance in terms of  $\mathrm{F}_{1}^{\mathrm{micro}}$  and  $\mathrm{F}_{1}^{\mathrm{macro}}$ . Both DS and RW reduce bias in terms of GAP, at the cost of a drop in performance, in terms of  $\mathrm{F}_{1}^{\mathrm{micro}}$  and  $\mathrm{F}_{1}^{\mathrm{macro}}$ . We attribute this to the dramatic decrease in the number of training instances for DS, and the myopia of RW in only taking the ratio of occupation-gender combinations into consideration but not the difficulty of each target class. Among INLP, FairBatch, EOCLA, and EOGLB, we can see that FairBatch achieves a reasonable bias reduction with the least performance drop. This is due to it dynamically adjusting the resampling probability during training. Comparing EOCLA and EOGLB, we can see that EOGLB is better able to deal with the dataset class imbalance (reflected in  $\mathrm{F}_{1}^{\mathrm{macro}}$ ), while reducing bias.
Performance-Fairness tradeoff. Figure 2 shows the  $\mathrm{F}_1^{\mathrm{micro}}$ -GAP tradeoff plot for the Bios test set. We can see that INLP and Adv reduce bias at the cost of performance, as do DS and RW. Compared with FairBatch,  $\mathrm{EO}_{\mathrm{CLA}}$  and  $\mathrm{EO}_{\mathrm{GLB}}$  provide greater control in terms of performance-fairness tradeoff, such as achieving a smaller GAP with a slight decrease of  $\mathrm{F}_1^{\mathrm{micro}}$ . A similar trend is also observed for the  $\mathrm{F}_1^{\mathrm{macro}}$ -GAP tradeoff as shown in Figure 3. Although  $\mathrm{EO}_{\mathrm{CLA}}$  is outperformed by FairBatch,  $\mathrm{EO}_{\mathrm{GLB}}$  provides greater control in terms of performance-fairness
Figure 2:  $\mathrm{F}_1^{\mathrm{micro}}$  vs. GAP of different models on the Bios test set, as we vary the most sensitive hyperparameter for each model.
Figure 3:  $\mathrm{F}_1^{\mathrm{macro}}$  vs. GAP of different models on the Bios test set, as we vary the most sensitive hyperparameter for each model.
tradeoff, suggesting an advantage of  $\mathsf{EO}_{\mathrm{GLB}}$  in enforcing fairness across target classes, especially for the imbalanced dataset.
To better understand the effectiveness of our proposed methods, we perform two sets of experiments: (1) an ablation study, and (2) an analysis of training efficiency.
$\mathsf{EO}_{\mathrm{CLA}}$  can be reformulated as  $\mathcal{L}_{ce} + \lambda \sum_{y \in C} \{\max(\mathcal{L}_{ce}^{y,a}, \mathcal{L}_{ce}^{y, -a}) - \min(\mathcal{L}_{ce}^{y,a}, \mathcal{L}_{ce}^{y, -a})\}$ , effectively assigning more weight to worse-performing instances (argmax loss) and less weight to better-performing instances (argmin loss). To explore the impact of adjusting weights on model performance,
<sup>4</sup>We omit results for Constrained as it did not converge on this data set, presumably because of its brittleness over multi-class classification tasks.
we experiment with two versions: (1)  $\mathcal{L}_{ce} + \lambda \sum_{y\in C}\max (\mathcal{L}_{ce}^{y,a},\mathcal{L}_{ce}^{y,\neg a})$  denoted as EO max, where we assign higher weights to worse-performing instances without changing the weights assigned to better-performing instances; and (2)  $\mathcal{L}_{ce} - \lambda \sum_{y\in C}\min (\mathcal{L}_{ce}^{y,a},\mathcal{L}_{ce}^{y,\neg a})$  denoted as EO min, where we assign smaller weights to better-performing instances without changing the weights assigned to worse-performing instances. Correspondingly, for EO GLB, we have EO max and EO min. Hyperparameter settings for each model can be found in Appendix B.1.
Tables 3 and 4 show the results for the different models on Moji and Bios. We can see that the full  $\mathsf{EO}_{\mathrm{CLA}}$  and  $\mathsf{EO}_{\mathrm{GLB}}$  both achieve better bias reduction than ablated min and max counterparts on Moji, while maintaining similar levels of performance in terms of  $\mathsf{F}_1^{\mathrm{micro}}$ . On Bios, we can see that  $\mathsf{EO}_{\mathrm{CLA}}^{\mathrm{max}}$  outperforms  $\mathsf{EO}_{\mathrm{CLA}}$  in bias reduction and model performance except for  $\mathsf{F}_1^{\mathrm{micro}}$ , indicating that it is beneficial for bias reduction to increase the weight for worse-performing instances. On the other hand,  $\mathsf{EO}_{\mathrm{CLA}}^{\mathrm{min}}$  is inferior to  $\mathsf{EO}_{\mathrm{CLA}}$  in terms of both bias reduction and performance. We conjecture that reducing the weights for better-performing instances is harmful for model performance (especially for minority classes) over datasets with imbalanced distributions, as is the case for Bios. Among the three variants of  $\mathsf{EO}_{\mathrm{GLB}}$ ,  $\mathsf{EO}_{\mathrm{GLB}}^{\mathrm{max}}$  slightly improves performance on the main task and maintains the same level of bias reduction as  $\mathsf{EO}_{\mathrm{GLB}}$ , while  $\mathsf{EO}_{\mathrm{GLB}}^{\mathrm{min}}$  improves performance on the main task but does not reduce bias. Overall, these results show that our two methods perform best in their original formulations.
5.2 Training Efficiency
To understand the training efficiency of the different models, we perform experiments with varying training data sizes on both Moji and Bios. Based on results from Tables 1 and 2, we provide results for CE, FairBatch, EO<sub>CLA</sub>, and EO<sub>GLB</sub>.
Figure 4 presents the results for Moji. When the proportion of training data is no larger than 1K, FairBatch is unable to learn a decent model, while
Model	F1micro↑	GAP↓
EOCLA	75.03±0.25	10.83±1.40
EOmaxCLA	75.92±0.10	13.79±1.64
EOminCLA	75.33±0.19	14.50±1.78
EOGLB	75.20±0.20	11.49±1.07
EOmaxGLB	76.31±0.10	16.47±0.90
EOminGLB	76.27±0.13	18.01±0.40
Table 3: Ablation results over Moji test set (averaged over 10 runs).
Model	F1macro↑	F1micro	GAP↓
EOCLA	72.07±0.18	81.52±0.06	12.80±0.42
EOmaxCLA	72.09±0.19	79.95±0.12	8.98±0.43
EOminCLA	53.17±0.53	76.66±0.23	19.22±1.68
EOGLB	75.11±0.18	81.74±0.07	12.72±0.51
EOmaxGLB	75.37±0.06	81.89±0.03	12.47±0.51
EOminGLB	75.95±0.12	82.19±0.05	16.74±0.42
Table 4: Ablation results over Bios test set (averaged over 10 runs).
Figure 4:  $\mathrm{F}_1^{\mathrm{micro}}$  vs. GAP of different models on the Moji test set. The full training set is 100K instances.
both  $\mathsf{EO}_{\mathrm{CLA}}$  and  $\mathsf{EO}_{\mathrm{GLB}}$  are still effective. As we increase the number of training instances, improved performance on the main task can be observed for all models, and larger bias reduction is achieved for all models except CE. Overall,  $\mathsf{EO}_{\mathrm{CLA}}$  and  $\mathsf{EO}_{\mathrm{GLB}}$  perform well in low-resource settings and achieve better bias reduction for larger volumes of training instances, demonstrating their superiority.
Figure 5 presents the results for Bios. We see that FairBatch outperforms  $\mathsf{EO}_{\mathrm{CLA}}$  and  $\mathsf{EO}_{\mathrm{GLB}}$  especially in terms of  $\mathsf{F}_1^{\mathrm{macro}}$  and GAP. Our explanation is that FairBatch adopts a resampling strategy, while our method adopts a reweighting strategy. Although statistically equivalent, resampling outper
For the max and min versions of both  $\mathsf{EO}_{\mathrm{CLA}}$  and  $\mathsf{EO}_{\mathrm{GLB}}$ , we finetune with the corresponding best-performing  $\lambda$ , respectively. A smaller GAP value cannot be achieved by further adjusting/increasing the value of  $\lambda$ .
This is in line with previous research (Swayamdipta et al., 2020), which shows that easy-to-learn instances are important in optimising models.
Figure 5:  $\mathrm{F}_1^{\mathrm{micro}}$  and  $\mathrm{F}_1^{\mathrm{macro}}$  vs. GAP of different models on the Bios test set. The full training set is 257K.
Figure 6:  $\mathrm{F}_{1}^{\text{micro}}$ ,  $\mathrm{F}_{1}^{\text{macro}}$ , vs. GAP of different models on the subset of Bios data. Here, instances are from the top-8 most common classes, whose proportion is greater than  $4\%$  in the original dataset, resulting into a full training dataset size of 188K.
forms reweighting when combined with stochastic gradient algorithms (An et al., 2021). The data imbalance in Bios exacerbates this effect. To verify this, we generated a version of Bios with only instances belonging to the top-8 most common classes, whose ratio in the original training set is bigger than  $4\%$ . Figure 6 presents results with the subset of dataset consisting of the top-8 most common classes. The plots show a similar trend as observed for the Moji dataset on this relatively balanced dataset. Specifically, when the training dataset is small, FairBatch is unable to learn a decent model, while both  $\mathsf{EO}_{\mathrm{CLA}}$  and  $\mathsf{EO}_{\mathrm{GLB}}$  are still effective.
Consistent with previous work, we did not finetune the underlying pretrained models in obtaining document representations in this work. Finetuning may further remove biases encoded in the pretrained models, which we leave to future work. This work focused only on datasets with binary protected attributes, and future experiments should explore the methods' generalization to higher-arity attributes. For both INLP and Adv, we follow experimental setup from the original papers, noting that the fairlib (Han et al., 2022) debiasing framework which was developed after this work was done recently showed that both models can obtain better performance and fairness scores with a larger budget for hyperparameter finetuning.
We proposed to incorporate fairness criteria into model training, in explicitly optimising for equal opportunity by minimising the loss difference over different subgroups conditioned on the target label. To deal with data imbalance based on the target-label, we proposed a variant of our method which promotes fairness across all target labels. Experimental results over Twitter sentiment analysis and profession classification tasks show the effectiveness and flexibility of our proposed methods.
Our works aims to achieve fairer models, contributing to equal treatment for different demographic subgroups. However, its usage in the real world should be carefully calibrated/auditioned as debiasing for one projected attribute does not guarantee fairness for other protected attributes. In this work, due to the limitations of the dataset, we treat gender as binary, which is not perfectly aligned with the real world.
We thank anonymous reviewers for their helpful feedback. This research was supported by The University of Melbourne's Research Computing Services and the Petascale Campus Initiative. This work was funded in part by the Australian Research Council.
<sup>7</sup>https://pypi.org/project/fairlib/
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna M. Wallach. 2018. A reductions approach to fair classification. In Proceedings of the 35th International Conference on Machine Learning, pages 60-69.
Jing An, Lexing Ying, and Yuhua Zhu. 2021. Why resampling outperforms reweighting for correcting sampling bias with stochastic gradients. In Proceedings of the 9th International Conference on Learning Representations.
Su Lin Blodgett, Lisa Green, and Brendan O'Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119-1130.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Aréchiga, and Tengyu Ma. 2019. Learning imbalanced datasets with label-distribution-aware margin loss. In Advances in Neural Information Processing Systems, pages 1565-1576.
Jaewoong Cho, Gyeongjo Hwang, and Changho Suh. 2020. A fair classifier using kernel density estimation. In Advances in Neural Information Processing Systems.
Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge J. Belongie. 2019. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9268-9277.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnamaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 120-128.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Michele Donini, Luca Oneto, Shai Ben-David, John Shawe-Taylor, and Massimiliano Pontil. 2018. Empirical risk minimization under fairness constraints. In Advances in Neural Information Processing Systems, pages 2796-2806.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard S. Zemel. 2012. Fairness through awareness. In Innovations in Theoretical Computer Science 2012, pages 214-226.
Yanai Elazar and Yoav Goldberg. 2018. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 11-21.
Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1615-1625.
Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and removing disparate impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 259-268.
Xudong Han, Timothy Baldwin, and Trevor Cohn. 2021a. Balancing out bias: Achieving fairness through training reweighting. CoRR, abs/2109.08253.
Xudong Han, Timothy Baldwin, and Trevor Cohn. 2021b. Decoupling adversarial training for fair NLP. In Findings of the Association for Computational Linguistics, pages 471-477.
Xudong Han, Timothy Baldwin, and Trevor Cohn. 2021c. Diverse adversaries for mitigating bias in training. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2760-2765.
Xudong Han, Aili Shen, Yitong Li, Trevor Cohn, Timothy Baldwin, and Lea Frermann. 2022. fairlib: A unified framework for assessing and improving classification fairness. arXiv preprint.
Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems, pages 3315-3323.
Dirk Hovy and Anders Søgaard. 2015. Tagging performance correlates with author age. In Proceedings of the 53rd annual meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (volume 2: Short papers), pages 483-488.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations.
Miroslav Kubat and Stan Matwin. 1997. Addressing the curse of imbalanced training sets: One-Sided selection. In Proceedings of the Fourteenth International Conference on Machine Learning, pages 179-186.
Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed Chi. 2020. Fairness without demographics through adversarially reweighted learning. In Advances in Neural Information Processing Systems.
Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. 2020. Dice loss for data-imbalanced NLP tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 465-476.
Yitong Li, Timothy Baldwin, and Trevor Cohn. 2018. Towards robust and privacy-preserving text representations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 25-30.
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dólar. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988.
David Madras, Elliot Creager, Toniann Pitassi, and Richard S. Zemel. 2018. Learning adversarily fair and transferable representations. In Proceedings of the 35th International Conference on Machine Learning., pages 3381-3390.
Harikrishna Narasimhan. 2018. Learning with complex loss functions and constraints. In International Conference on Artificial Intelligence and Statistics, pages 1646-1654.
Flavio du Pin Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R. Varshney. 2017. Optimized pre-processing for discrimination prevention. In Advances in Neural Information Processing Systems, pages 3992-4001.
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon M. Kleinberg, and Kilian Q. Weinberger. 2017. On fairness and calibration. In Advances in Neural Information Processing Systems, pages 5680-5689.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256.
Yuji Roh, Kangwook Lee, Steven Whang, and Changho Suh. 2020. Fr-train: A mutual information-based approach to fair and robust training. In Proceedings of the 37th International Conference on Machine Learning, pages 8147-8157.
Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. 2021. Fairbatch: Batch selection for model fairness. In Proceedings of the 9th International Conference on Learning Representations.
Saeed Sharifi-Malvajerdi, Michael J. Kearns, and Aaron Roth. 2019. Average individual fairness: Algorithms,