Venue: NAACL 2025

# Inference-Time Selective Debiasing to Enhance Fairness in

Text Classification Models

Gleb Kuzmin\({}^{2,4}\) Neemesh Yadav\({}^{5}\) Ivan Smirnov\({}^{3,4}\) Timothy Baldwin\({}^{1,6}\) Artem Shelmanov\({}^{1}\)

###### Abstract

We propose selective debiasing - an inference-time safety mechanism designed to enhance the overall model quality in terms of prediction performance and fairness, especially in scenarios where retraining the model is impractical. The method draws inspiration from selective classification, where at inference time, predictions with low quality, as indicated by their uncertainty scores, are discarded. In our approach, we identify the potentially biased model predictions and, instead of discarding them, we remove bias from these predictions using LEACE - a post-processing debiasing method. To select problematic predictions, we propose a bias quantification approach based on KL divergence, which achieves better results than standard uncertainty quantification methods. Experiments on text classification datasets with encoder-based classification models demonstrate that selective debiasing helps to reduce the performance gap between post-processing methods and debiasing techniques from the at-training and pre-processing categories.1

Footnote 1: The code is available online at [https://github.com/glkuzi/selective-debiasing](https://github.com/glkuzi/selective-debiasing)

**Title:** _The University of Melbourne_

kuzmin@airi.net neemesh20529@iitd.ac.in ivs@isa.ru {timothy.baldwin, artem.shelmanov}@mbzuai.ac.ae

## 1 Introduction

Fairness is an important safety characteristic of a machine learning (ML) model, representing the model's ability to classify instances without discrimination based on various sensitive attributes, such as race, gender, and age [7]. For the past few years, numerous works have investigated and promoted fairness, and a variety of fairness definitions have been proposed [8, 10]. One prominent type of fairness is group fairness, also known as the equal opportunity criterion, which reflects the inequality of opportunities across different groups [11]. The inequality in the model predictions usually comes from inadequate or biased training data, and to address this problem and achieve better fairness, researchers have proposed various debiasing techniques [12, 13, 14, 15]. The majority of these techniques assume that one has access to the complete training data and the ability to retrain the model from scratch using some special loss function or reweighting the training instances. However, there are many situations when this assumption does not hold. There is a need for inference-time safety mechanisms that protect users from inadequate model behavior.

Inference-time safety mechanisms are primarily associated with uncertainty quantification (UQ) techniques [10] and selective classification [10, 11, 12, 13]. Selective classification aims to enhance the reliability of ML-based applications by abstaining from unreliable predictions with high uncertainty. We suggest that the same approach could be applied to increase fairness.

In this work, we propose an inference-time safety mechanism that aims to increase the overall quality of models in terms of prediction performance and fairness in situations when model retraining is prohibitive. We call this approach _selective debiasing_. Instead of rejecting predictions of selected instances as in selective classification, we apply to them inference-time debiasing using post-processing debiasing techniques. To the best of our knowledge, this style of approach is novel to the NLP community.

Our main contributions are as follows:

* We propose selective debiasing, an inference-time safety mechanism that aims to improve both the performance and fairness of model predictions by applying a post-processing debiasing method to only a selected subset of predictions.
* We suggest a scoring criterion that aims to se

lect the most unreliable and biased predictions. Experiments demonstrate that this scoring criterion is generally better than UQ techniques in selective debiasing.

## 2 Background

Debiasing techniques can be categorized into three groups: at-training, pre-processing, and post-processing Han et al. (2022).

At-training and pre-processing methods.One of the most popular at-training methods is adversarial training Adv Li et al. (2018). It aims to solve a minimax game between minimizing the loss for the primary task and maximizing the loss for predicting the protected attribute. The diverse adversaries method DAdv Han et al. (2021) extends Adv by using an ensemble of multiple diverse discriminators instead of just one. In the pre-processing category, one of the most remarkable methods is Balanced Training with Equal Opportunity BTEO Han et al. (2022). It rebalances the dataset to minimize the True Positive Rate TPR gap between two protected groups. In the same category, Balanced Training with Joint balance BTJ Lahoti et al. (2020) aims to improve the worst-case performance over all unobserved protected groups by focusing on the computationally identifiable regions of error.

Post-processing methods.There are two well-known approaches to post-processing debiasing: Iterative Null-space Projection INLP Ravfogel et al. (2020) and LEAst-squares Concept Erasure LEACE Belrose et al. (2023).

INLP is an iterative method that involves finding an orthogonal projection of a linear classifier matrix, which is initially learned to predict protected attributes from representations (e.g. hidden states of the standard model). This orthogonal projection is then iteratively used to remove all relevant information from these representations, which was used by the classifier to predict protected attributes.

LEACE is a concept erasure technique that renders representations impervious to the prediction of a specific concept while minimizing changes to the original representations. To construct a transformation matrix, it first whitens the data by equalizing the variance across all directions in the representation space. Next, the data is orthogonally projected onto the subspace that captures correlations between representations and protected attributes. Finally, the data is unwhitened using the same covariance matrix. This resulting transformation matrix is subtracted from the original representations (see the formal definition for LEACE in Appendix A).

At-training and pre-processing methods require retraining the model from scratch and access to the whole training set. They also cannot be selectively applied to a subset of predictions. Post-processing techniques do not involve changes to the model itself, can be trained on a subset of data, and can be applied to predictions selectively. However, their performance is usually worse.

In our work, we propose a method that combines the advantages of both post-processing and at-training / pre-processing methods. While it does not need access to the whole training dataset or retraining the model from scratch, it also has better performance than the standard post-processing techniques.

## 3 Proposed Method

We propose a selective approach, based on applying debiasing only to predictions with the highest bias score. This section introduces the general concept of selective debiasing and presents the bias quantification method underlying this approach.

Selective debiasing.Selective classification is a widely recognized safety mechanism that safeguards against using unreliable model predictions. In this approach, predictions flagged as unreliable due to high uncertainty scores are handled differently, e.g. they are rejected or are escalated to human operators for further review.

Instead of rejecting instances completely as in selective classification, we apply debiasing to selected predictions. In particular, we identify the potentially most biased instances using a bias quantification method \(\mathcal{B}(x_{i},p_{i})\) and replace the original prediction \(p_{i}=f(x_{i})\) with a prediction debiased using a post-processing method \(d\colon\hat{p_{i}}=d(f(x_{i}))\colon\)

\[\bar{p_{i}}=\begin{cases}p_{i}=f(x_{i}),~{}\text{if}~{}\mathcal{B}(x_{i},p_{ i})<h\\ \hat{p_{i}}=d(f(x_{i})),~{}\text{if}~{}\mathcal{B}(x_{i},p_{i})\geq h,\end{cases} \tag{1}\]

where \(h\) is a predefined threshold selected on a validation set.

We note that the proposed approach is different from the standard post-processing debiasing methods since we change predictions for only some instances. While debiasing all predictions might significantly reduce model performance, modifying only predictions likely to be of low quality or 

biased is less risky in terms of worsening outcomes and has the potential to correct errors. Such an approach also allows tuning the accuracy-fairness trade-off for debiasing methods Han et al. (2022); Kuzmin et al. (2023).

Bias quantification method.Selective classification is usually based on UQ methods. However, uncertainty on its own does not reflect the presence of bias; it simply highlights potentially erroneous predictions. Figure 1 presents a motivational example. It shows the rejection plots for oracle rejection strategies in selective classification for both accuracy and fairness (see the exact definition of fairness in Appendix E). We can see that the fairness oracle outperforms the UQ oracle in terms of fairness while keeping the same performance in terms of accuracy. These results illustrate that it is possible to improve fairness without penalty to accuracy by changing the order of instances being eliminated, i.e. using a different selection criterion.

Consider a multi-label classification model with classes \(c\in C\). To quantify how biased a model prediction is for a given instance, we suggest using the Kullback-Leibler (KL) divergence Kullback and Leibler (1951) between the originally predicted probability distribution \(p_{i}^{c}\) and distribution \(\hat{p_{i}}^{c}\) after debiasing:

\[\mathcal{B}_{KL}^{i}=\sum_{c\in C}p_{i}^{c}\log\left(\frac{\hat{p}_{i}^{c}}{ \hat{p}_{i}^{c}}\right). \tag{2}\]

KL divergence measures the difference in predictions between the standard and the debiased model. The greater the difference, the more information about the protected attribute is removed from the original representation of the instance. This approach could be used with various post-processing methods. In particular, we suggest using LEACE, but also present results with INLP.

Note that applying a post-processing method to a model is a matter of one or two matrix multiplications. An additional prediction step requires inferring only the last layer of a model, which is very fast. Therefore, the runtime overhead introduced by bias quantification is very small (see Appendix H).

## 4 Experiments

### Experimental Setup

Datasets.For our experiments, we use two English text classification datasets that, in addition to target variables, provide explicit protected attributes. The first is MOJI Blodgett et al. (2016), a dataset for sentiment analysis with a binary class ("happy" and "sad") and a binary protected attribute, which corresponds to the author's ethnicity African American English (AAE) vs. Standard American English (SAE). The second is a version of the widely used BIOS dataset De-Arteaga et al. (2019) for occupation classification with a binary gender as the protected attribute. BIOS-2 Subramanian et al. (2021) is a two-class subsample of the original BIOS dataset with a highly-skewed joint distribution of classes and protected attribute values. As it has been shown to be beneficial to report results for both "balanced" and "imbalanced" versions of datasets Kuzmin et al. (2023), we conduct experiments on both versions. Detailed information and statistics of the datasets are presented in Appendix C. Due to the limited availability of datasets with annotated protected attributes, most research on debiasing and fairness has been conducted on these few datasets Han et al. (2022).

Metrics.We employ several metrics to evaluate the predictive performance and fairness of the model. To evaluate the performance, we use accuracy. For fairness, we consider the widely used equal opportunity criterion Hardt et al. (2016); Han et al. (2022). We also use two aggregated metrics to evaluate the performance in terms of both accuracy and fairness. The first one is the distance to the optimal point (DTO) Han et al. (2021):

\[\text{DTO}=\sqrt{(1-\text{Accuracy})^{2}+(1-\text{Fairness})^{2}}. \tag{3}\]

The second one is the Fairness F-score (FF) - a smoothed minimum of accuracy and fairness:

\[\text{FF-score}=\frac{2\cdot\text{Accuracy-Fairness}}{\text{ Accuracy}+\text{Fairness}}. \tag{4}\]

Figure 1: Rejection results for fairness and accuracy with oracle scores on a synthetic dataset with a LogReg model; the FR-AUC and Acc-AUC are the areas under fairness– and accuracy–rejection curves correspondingly. The details are presented in Appendix B.

 

Details of the equal opportunity fairness calculation are presented in Appendix E.

ModelsFor the BIOS-2 dataset, we use BERT ("bert-base-cased") Devlin et al. (2019). For the MOJI dataset, we use the domain-specific BERTweet model Nguyen et al. (2020) which is good for processing data from social media sources. For both models, we add a three-layer MLP as a classification head, following Han et al. (2022b). Model hyperparameters are described in Appendix D.

BaselinesWe compare the proposed selective debiasing approach to inference-time debiasing of all predictions using LEACE and INLP, as well as to at-training and pre-processing debiasing techniques: Adv, DAdv, BTEO, BTJ. We also compare the proposed KL-based bias quantification score with a UQ baseline: Softmax Response (SR: Geifman and El-Yaniv (2017)), calculated as \({\cal B}_{SR}(x_{i})=1-\max_{c\in C}p^{c}_{i}\).

Details of debiasing methodsPre-processing and at-training debiasing methods were applied while training the model from scratch on the full dataset, whereas post-processing methods were trained using only 20% of the data. The optimal threshold for selective debiasing was chosen based on the first 15% of the validation set. "LEACE-last" in our experiments represents LEACE applied to the outputs of the last hidden layer of the classifier, while "LEACE-cls" is LEACE applied to each linear layer of the classification head of the model. The hyperparameters of debiasing methods are provided in Appendix D.

### Results

Table 1 presents results for various at-training and pre-processing debiasing methods, post-processing debiasing methods, selective debiasing based on LEACE with SR, and selective debiasing using the proposed KL-based bias quantification score. Here, we show results only for the threshold that gives an optimal selection percentage. The full results with various selection percentages are presented in Appendix F. The results for selective debiasing using INLP are provided in Appendix F.

In the majority of cases, the best results are unsurprisingly achieved by at-training and pre-processing debiasing techniques, as these methods retrain the models from scratch on the full training data. Nevertheless, the proposed selective debiasing approach based on LEACE substantially enhances the results of inference-time debiasing using post-processing techniques in terms of metrics that take into account both fairness and performance: FF-score and DTO. Inference time debiasing becomes competitive with at-training and pre-processing techniques. For LEACE-cls with KL selection, selective debiasing even outperforms these methods on MOJI-balanced. The results in Tables 15 to 17 also show that selective debiasing consistently outperforms standard inference-time debiasing in terms of FF-score.

LEACE-cls generally achieves better fairness than LEACE-last and slightly better joint fairness

\begin{table}
\begin{tabular}{l l||r|r r|r r|r r r|r r r} \hline \multicolumn{3}{l||}{**Debiasing method type**} & \multicolumn{2}{c|}{**No debiasing**} & \multicolumn{4}{c||}{**At-training**} & \multicolumn{4}{c||}{**Pre-processing**} & \multicolumn{6}{c}{**Post-processing \& Selective**} \\ \hline \multirow{3}{*}{**Dataset**} & \multirow{3}{*}{**Metric**} & \multirow{3}{*}{**Standard**} & \multirow{3}{*}{**Adv**} & \multirow{3}{*}{**DAdv**} & \multirow{3}{*}{**BTEO**} & \multirow{3}{*}{**BTJ**} & \multirow{3}{*}{**LELACE-last**} & \multicolumn{1}{c}{**LEACE-last+BL**} & \multicolumn{1}{c}{**LEACE-test+BL**} & \multicolumn{1}{c}{**LEACE-class**} & \multicolumn{1}{c}{**LEACE-class**} & \multicolumn{1}{c}{**LEACE-class**} \\  & & & & & & & & **opt. per. & opt. per. & perc. & opt. per. & & opt. per. \\ \cline{2-18}  & Fairness \(\uparrow\) & 61.8\({}_{\pm 1}\) & 73.7\({}_{\pm 1}\) & 73.4\({}_{\pm 1}\) & **75.2\({}_{\pm 1}\)** & 74.8\({}_{\pm 1}\) & 78.8\({}_{\pm 1}\) & 68.6\({}_{\pm 1}\) & 75.7\({}_{\pm 1}\) & 75.2\({}_{\pm 1}\) & 68.4\({}_{\pm 1}\) & 77.2\({}_{\pm 1}\) \\ MOJI & Accuracy \(\uparrow\) & **79.1\({}_{\pm 1}\)** & 72.0\({}_{\pm 1}\) & 72.4\({}_{\pm 1}\) & 73.6\({}_{\pm 1}\) & 73.2\({}_{\pm 1}\) & 68.3\({}_{\pm 1}\) & **77.6\({}_{\pm 1}\)** & 66.8\({}_{\pm 1}\) & **77.6\({}_{\pm 1}\)** & 71.8\({}_{\pm 1}\) \\ imbalanced & DTO \(\downarrow\) & 43.6\({}_{\pm 1}\) & 38.4\({}_{\pm 1}\) & 38.3\({}_{\pm 1}\) & **36.2\({}_{\pm 1}\)** & 36.7\({}_{\pm 1}\) & 39.9\({}_{\pm 1}\) & 38.6\({}_{\pm 1}\) & 43.6\({}_{\pm 1}\) & 41.4\({}_{\pm 1}\) & 83.8\({}_{\pm 1}\) & 36.2\({}_{\pm 1}\) \\ FF-socet \(\uparrow\) & 69.4\({}_{\pm 1}\) & 72.8\({}_{\pm 1}\) & 72.9\({}_{\pm 1}\) & 74.4\({}_{\pm 1}\) & **74.9\({}_{\pm 1}\)** & 71.8\({}_{\pm 1}\) & 72.8\({}_{\pm 1}\) & 72.8\({}_{\pm 1}\) & 74.4\({}_{\pm 1}\) & 70.8\({}_{\pm 1}\) & 73.2\({}_{\pm 1}\) & 72.7\({}_{\pm 1}\) & **74.4\({}_{\pm 1}\)** \\ \hline \multirow{3}{*}{MOJI balanced} & Fairness \(\uparrow\) & 69.5\({}_{\pm 1}\) & 83.8\({}_{\pm 1}\) & 84.7\({}_{\pm 1}\) & 85.5\({}_{\pm 1}\) & **85.6\({}_{\pm 1}\)** & 79.7\({}_{\pm 1}\) & 77.1\({}_{\pm 1}\) & **86.6\({}_{\pm 1}\)** & 77.6\({}_{\pm 1}\) & 77.0\({}_{\pm 1}\) & **87.5\({}_{\pm 1}\)** \\  & Accuracy \(\uparrow\) & 91.9\({}_{\pm 1}\) & 74.0\({}_{\pm 1}\) & 74.1\({}_{\pm 1}\) & **74.2\({}_{\pm 1}\)** & 74.5\({}_{\pm 1}\) & 73.6\({}_{\pm 1}\) & **74.0\({}_{\pm 1}\)** & 74.0\({}_{\pm 1}\) & 73.0\({}_{\pm 1}\) & 74.0\({}_{\pm 1}\) & 73.7\({}_{\pm 1}\) \\  & DFIO \(\downarrow\) & 41.5\({}_{\pm 1}\) & 30.7\({}_{\pm 1}\) & 30.1\({}_{\pm 1}\) & **73.0\({}_{\pm 1}\)** & **73.0\({}_{\pm 1}\)** & 33.3\({}_{\pm 1}\) & 34.4\({}_{\pm 1}\) & 34.7\({}_{\pm 1}\) & 29.3\({}_{\pm 1}\) & 35.2\({}_{\pm 1}\) & 34.7\({}_{\pm 1}\) & 29.1\({}_{\pm 1}\) \\  & FF-score \(\uparrow\) & 70.7\({}_{\pm 1}\) & 70.7\({}_{\pm 1}\) & 79.1\({}_{\pm 1}\) & **79.8\({}_{\pm 1}\)** & 79.8\({}_{\pm 1}\) & 76.5\({}_{\pm 1}\) & 75.5\({}_{\pm 1}\) & 75.5\({}_{\pm 1}\) & 75.2\({}_{\pm 1}\) & 75.5\({}_{\pm 1}\) & 75.5\({}_{\pm 1}\) & 75.5\({}_{\pm 1}\) & 75.5\({}_{\pm 1}\) & 75.5\({}_{\pm 1}\) & **80.0\({}_{\pm 1}\)** \\ \hline \multirow{3}{*}{BIOS-2 imbalanced} & Fairness \(\uparrow\) & 90.4\({}_{\pm 1}\) & **97.2\({}_{\pm 1}\)** & 96.4\({}_{\pm 1}\) & 95.8\({}_{\pm 1}\) & 96.1\({}_{\pm 1}\) & 96.6\({}_{\pm 1}\) & 92.8\({}_{\pm 1}\) & 93.0\({}_{\pm 1}\) & 94.5\({}_{\pm 1}\) & 97.3\({}_{\pm 1}\) & 94.5\({}_{\pm 1}\) & 94.8\({}_{\pm 1}\) & 95.7\({}_{\pm 1}\) \\  & DF 

performance in terms of DTO and FF-score.

When comparing the results of the proposed bias quantification method based on the KL distance with SR, we can see that our method notably outperforms SR on the MOJI datasets and is on par with SR on BIOS-2. We further explore other distance-based bias quantification methods (Euclidean and cosine distances) in Appendix G. Results in Tables 15 to 17 show that in most cases, selection by KL works comparably or better than other distance-based measures. Moreover, KL scores are easier to compute than distance-based scores.

## 5 Conclusion and Future Work

We proposed selective debiasing - a new simple inference-time safety mechanism for increasing model performance and fairness. We showed that it is helpful in the case when re-training a model from scratch for better fairness is prohibitive or there is no access to full training data. Additionally, for the selection of problematic predictions, we suggest a bias quantification approach based on KL divergence that achieves better results than the standard UQ method. The proposed mechanism fills the gap for efficient techniques that can be applied at inference time and opens the door for safer ML-based systems. In future work, we aim to investigate a deeper integration between UQ and debiasing methods.

### Limitations

In this work, we considered only group fairness (equal opportunity criterion), where there exist many other fairness definitions. However, this research is focused particularly on group fairness, and the equal opportunity criterion is the metric of choice in previous work on the same datasets. During all experiments, we assume that we have access to the protected attributes, which is not always the case. But this is a common assumption for any work on debiasing; moreover, it is necessary for the calculation of the fairness metric. Finally, all of the experiments were conducted on the English language, but the used methods are language-independent, so we do not expect significant differences in results for other languages.

## Ethical Considerations

In this work, we consider group fairness and instance-level bias quantification. We used only publicly available datasets and models, and only for the intended use. In our research, we used protected attributes to apply debiasing methods and to compute metrics; however, this is necessary for all debiasing methods. To avoid possible harm, we used only attributes that users self-disclosed for the experiments.

## Acknowledgments

We appreciate the anonymous reviewers for their valuable suggestions that helped enhance this paper. This research was supported in part through the computational resources of HPC facilities at HSE University.

## References

* N. Belrose, D. Schneider-Joseph, S. Ravfogel, R. Cotterell, E. Raff, and S. Biderman (2023)LeACE: perfect linear concept erasure in closed form. In Advances in Neural Information Processing Systems, Vol. 36, pp. 66044-66063. External Links: Link Cited by: SS1.
* S. L. Blodgett, X. Green, and B. O'Connor (2016)Demographic dialectal variation in social media: a case study of African-American english. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, pp. 1119-1130. External Links: Link Cited by: SS1.
* M. De-Arteaga, A. Romanov, H. Wallach, J. Chayes, C. Borgs, A. Chouldechova, S. Geyik, K. Kenthapadi, and A. T. Kalai (2019)Bias in bios: a case study of semantic representation bias in a high-stakes setting. In proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 120-128. Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186. External Links: Link Cited by: SS1.
* Y. Gal and Z. Ghahramani (2016)Dropout as a bayesian approximation: representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. External Links: Link Cited by: SS1.

 

* Geifman and El-Yaniv (2017) Yonatan Geifman and Ran El-Yaniv. 2017. Selective classification for deep neural networks. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 4878-4887.
* Han et al. (2021) Xudong Han, Timothy Baldwin, and Trevor Cohn. 2021. Diverse adversaries for mitigating bias in training. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 2760-2765, Online. Association for Computational Linguistics.
* Han et al. (2022a) Xudong Han, Timothy Baldwin, and Trevor Cohn. 2022a. Balancing out bias: Achieving fairness through balanced training. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 11335-11350, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Han et al. (2022b) Xudong Han, Aili Shen, Yitong Li, Lea Frermann, Timothy Baldwin, and Trevor Cohn. 2022b. FairLib: A unified framework for assessing and improving fairness. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 60-71, Abu Dhabi, UAE. Association for Computational Linguistics.
* Hardt et al. (2016) Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. In _Advances in Neural Information Processing Systems_, volume 29. Curran Associates, Inc.

* Kuzmin et al. (2023) Gleb Kuzmin, Artem Vazhentsev, Artem Shelmanov, Xudong Han, Simon Suster, Maxim Panov, Alexander Panchenko, and Timothy Baldwin. 2023. Uncertainty estimation for debiased models: Does fairness hurt reliability? In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 744-770, Nusa Dua, Bali. Association for Computational Linguistics.
* Lahoti et al. (2020) Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed Chi. 2020. Fairness without demographics through adversarially reweighted learning. In _Advances in Neural Information Processing Systems_, volume 33, pages 728-740. Curran Associates, Inc.
* Li et al. (2018) Yitong Li, Timothy Baldwin, and Trevor Cohn. 2018. Towards robust and privacy-preserving text representations. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 25-30, Melbourne, Australia. Association for Computational Linguistics.
* Nguyen et al. (2020) Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English tweets. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 9-14, Online. Association for Computational Linguistics.
* Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. _Journal of Machine Learning Research_, 12:2825-2830.
* Ravfogel et al. (2020) Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7237-7256, Online. Association for Computational Linguistics.
* Subramanian et al. (2021) Shivashankar Subramanian, Xudong Han, Timothy Baldwin, Trevor Cohn, and Lea Frermann. 2021. Evaluating debiasing techniques for intersectional biases. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 2492-2498, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Vazhentsev et al. (2022) Artem Vazhentsev, Gleb Kuzmin, Artem Shelmanov, Akim Tsvignu, Evgenii Tsymbalov, Kirill Fedyanin, Maxim Panov, Alexander Panchenko, Gleb Gusev, Mikhail Burtsev, Manvel Avetisian, and Leonid Zhukov. 2022. Uncertainty estimation of transformer predictions for misclassification detection. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8237-8252, Dublin, Ireland. Association for Computational Linguistics.
* Vazhentsev et al. (2023) Artem Vazhentsev, Gleb Kuzmin, Akim Tsvigun, Alexander Panchenko, Maxim Panov, Mikhail Burtsev, and Artem Shelmanov. 2023. Hybrid uncertainty quantification for selective text classification in ambiguous tasks. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 11659-11681, Toronto, Canada. Association for Computational Linguistics.
* Xin et al. (2021) Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. 2021. The art of abstention: Selective prediction and error regularization for natural language processing. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1040-1051, Online. Association for Computational Linguistics 

LEAST-Squares Concept Erasure

LEACE removes information about a concept \(Z\) from the representation space \(X\). To formally describe LEACE, we firstly introduce the following notions. Let \({\bf x}\in X\) be an instance from \(X\) (e.g. embedding from the last layer in the case of LEACE-last), \(\mathbf{\Sigma_{XX}}\) is the covariance matrix for \(X\), \(\mathbf{\Sigma_{XZ}}\) is the covariance matrix between \(X\) and \(Z\), and \(W_{\perp}\) stands for the pseudoinverse of the matrix \(W\). The **W** and \(\mathbf{P_{W\mathbf{\Sigma}_{XZ}}}\) defined as follows:

\[\textbf{W}=(\mathbf{\Sigma_{XX}}^{1/2})_{\perp}, \tag{5}\]

\[\mathbf{P_{W\mathbf{\Sigma}_{XZ}}}=(\mathbf{W\mathbf{\Sigma}_{XZ}})(\mathbf{W\mathbf{ \Sigma}_{XZ}})_{\perp}. \tag{6}\]

Then the final LEACE transformation is defined as follows:

\[\hat{\textbf{y}}(\textbf{x})=\textbf{x}-\textbf{W}_{\perp}\cdot\mathbf{P_{W\bm {\Sigma}_{XZ}}}\cdot\textbf{W}(\textbf{x}-\mathbb{E}[X]) \tag{7}\]

## Appendix B Fairness and UQ Oracles

In this section, we describe in detail oracle strategies for fairness and accuracy. For both strategies, we assume access to the ground-truth labels, while for fairness oracle we also use protected attributes. Accuracy oracle is built as follows - we find all erroneously classified instances and replace predictions on these instances with ground-truth labels while keeping all other predictions unchanged. This oracle shows the best possible UQ strategy that allows the detection of all erroneous predictions and gives the maximal increase in accuracy. The same idea is behind fairness oracle, but instead of accuracy, we use fairness as a target metric. For fairness, we first replace predictions for instances, which gives the maximal increase in fairness. These predictions are chosen greedily from the erroneous ones. To measure the quality of these oracle strategies and to compare them with other scores, we calculated several metrics: FR-AUC, Acc-AUC, and FF-score-AUC. Each corresponds to the area under the target metric-rejection curve, where the target metric is fairness, accuracy, or FF-score; the area under the curve is calculated on binarized over 100 points target metric values.

## Appendix C Datasets Statistics

The synthetic dataset was generated as a random 2 classes classification task using make_classification function from Scikit-learn library Pedregosa et al. (2011) with the following parameters: n_features=10, n_informative=5, n_clusters_per_class=2, random_state=42, n_redundant=2. The protected attribute for the synthetic dataset is designed as a condition over the first informative feature and equals 1 if this feature is greater than 0, and 0 otherwise. The overall statistics for each dataset are presented in Table 2. Tables 3 and 4 shows the joint distribution of the target variable and protected attributes.

 

Training Setup and Hyperparameters

To find an optimal set of hyperparameters, we conducted a grid search on the validation set. We used accuracy as an optimization target for standard models, and DTO for models with debiasing. The grid and optimal parameters for the standard models are described in Table 5. For each debiasing method, we tuned the method's parameters and kept the training parameters of the base model - the grid and optimal values for debiasing methods presented in Table 6. The training was conducted on a cluster with Nvidia V100 GPUs. An approximate number of GPU hours spent during the experiments is presented in Table 7.

[TABLE:A2.T4][E

\begin{table}
\begin{tabular}{l l c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & **Debiasing Method** & **Adv. Lambda** & **Adv. Diverse Lambda** & **INLP by Class** & **INLP Discriminator Reweighting** \\ \cline{3-8}  & Adv & 1.0 & - & - & - \\ Moji (imbalanced) & DAdv & 1.0 & 1.0 & - & - \\  & INLP & - & - & False & True \\ \hline \multirow{3}{*}{Moji (balanced)} & Adv & 1.0 & - & - & - \\  & DMdv & 1.0 & 1.0 & - & - \\  & INLP & - & - & False & False \\ \hline \multirow{3}{*}{BIOS-2 (imbalanced)} & Adv & 1.0 & - & - & - \\  & DMdv & 1.0 & 1.0 & - & - \\  & INLP & - & - & False & True \\ \hline \end{tabular}
\end{table}
Table 6: Optimal debiasing hyperparameters for BERTweet on MOJI and BERT on BIOS-2 for various debiasing methods. The base training parameters are the same as for the vanilla model. We use a grid search with the following grid values: Adv. Lambda/Adv. Diverse Lambda: [1e-4, 1e-3, 1e-2, 1e-1, 1, 1e2, 1e3], INLP by Class/INLP Discriminator Reweighting: [False, True]. The remaining parameters for each method used default values from Han et al. (2022b). For DAdv Adv. Lambda/Adv. Diverse Lambda parameters were tuned jointly, as in Han et al. (2022b).

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline \multirow{2}{*}{**Dataset**} & **Num. Epochs** & **Batch Size** & **Learning Rate** & **Weight Decay** & **Dropout Rate** \\ \hline MOI imbalanced & 20 & 32 & 1e-6 & 0 & 0.1 \\ MOJI balanced & 20 & 32 & 1e-6 & 0 & 0.1 \\ \hline BIOS-2 imbalanced & 20 & 16 & 1e-6 & 0 & 0.1 \\ BIOS-2 balanced & 20 & 32 & 1e-6 & 1e-4 & 0.1 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Overall computation statistics. GPU hours specify the approximate number of GPU hours spent for training and evaluating the corresponding model for all experiments on both imbalanced and balanced sets. The column Num. of Params contains the number of parameters of a single model.

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline \multirow{2}{*}{**Dataset**} & **Num. Epochs** & **Batch Size** & **Learning Rate** & **Weight Decay** & **Dropout Rate** \\ \hline MOJI imbalanced & 20 & 32 & 1e-6 & 0 & 0.1 \\ MOJI balanced & 20 & 32 & 1e-6 & 0 & 0.1 \\ \hline BIOS-2 imbalanced & 20 & 16 & 1e-6 & 0 & 0.1 \\ BIOS-2 balanced & 20 & 32 & 1e-6 & 1e-4 & 0.1 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Optimal training hyperparameters for BERTweet on MOJI and BERT on BIOS-2 for standard model. We use a grid search with the following grid values: batch size: [16, 32], learning rate: [1e-6, 5e-6, 1e-5, 3e-5, 5e-5], weight decay: [0, 1e-4]. The number of epochs is determined by early-stopping.

 

Equal Opportunity

There are a numerous amount of group fairness definitions; to avoid any mismatches, we are presenting the step-by-step process of equal opportunity criterion calculation. This criterion is based on recall values, or true positive rates (TPR) for each class and protected group.

* is true positives and false negatives for specific group.
* class index, \(\overline{TPR_{c}}\)
- \(TPR\) averaged across all groups for class \(c\).
* Finally, we calculate fairness with the following equation: \[Fairness=100\cdot(1-\delta).\] (10) 

Additional Experiments

To check how stable the proposed methods are, we compare selective debiasing results over 5%, 10%, and 15% of selection for random, SR, and KL scores. The results are presented in Tables9 to 11. The optimal percentage selected on the validation set from values from 1% to 15%; results for each dataset-method pair in Tables12 and 13. In general, optimal scores are better or comparable with results on various percentages, which allows us to use this approach to detect the optimal percentage of selection.

Table8 shows the performance of selective debiasing and post-processing debiasing methods trained on a full training set. As one can see, the performance on the full set is comparable with the results on only 20% from Table1.

The results for selective debiasing with INLP trained on 20% of data are presented in Table14. INLP-based selective debiasing improves the FF-score only on MOJI-balanced, while on other datasets, it is consistent with the base inference-time debiasing method. INLP-based approaches overall fall behind the corresponding LEACE-based techniques.

\begin{table}
\begin{tabular}{l l c c c c c c c c c c c c} \hline \hline
**Dataset** & **Standard** & **LEACE** & **Random** & **SR** & **KL** & **Random** & **SR** & **KX1** & **Random** & **SR** & **KL** \\  & & **S****\%** & **5\%** & **5\%** & **10\%** & **10\%** & **10\%** & **15\%** & **15\%** & **15\%** & **optimal** & **optimal** \\ \hline MOI imbalanced & 69.4,14 & 78.8,14 & 70.74 & 70.83 & 72.53 & 71.04 & 71.58 & 71.04 & 72.73 & 72.04 & 72.13 & 72.04 & 72.14 & 72.14 & 72.14 & 72.14 & **74.14** \\ MOH balanced & 70.7,17 & 76.52 & 71.88 & 72.44 & 75.54 & 72.73 & 74.04 & 78.22 & 72.33 & 75.54 & **79.83** & 77.33 & 75.54 & **79.83** \\ BIDS += unbalanced & 93.53 & 72.81 & 92.93 & 93.71 & 93.33 & 92.01 & 93.71 & 93.01 & 91.11 & 93.11 & 86.53 & 93.44 & 93.81 & 93.21 \\ BIDS += balanced & 91.11 & 63.04 & 90.69 & 91.69 & 92.01 & 89.33 & 92.22 & 89.61 & 88.73 & **92.61** & 88.43 & 90.93 & 92.31 & 91.93 \\ \hline \hline \end{tabular}
\end{table}
Table 10: FF-score of selective debiasing for LEACE on the last layer for various percentages.

\begin{table}
\begin{tabular}{l l|c c c c c c c c c c c c} \hline \hline
**Debiasing method type** & **No debiasing** & **At-training** & **Pre-processing** & & & & & & & & & & & & \\ \hline \multirow{2}{*}{**Dataset**} & \multirow{2}{*}{**Metric**} & \multirow{2}{*}{**Standard**} & \multirow{2}{*}{**Adv**} & \multirow{2}{*}{**DAdv**} & \multirow{2}{*}{**ETO**} & \multicolumn{2}{c|}{**RTJ**} & \multicolumn{2}{c|}{**LEXC:**} & \multicolumn{2}{c|}{**IEAAC:**} & \multicolumn{2}{c|}{**LEXC:**} & \multicolumn{2}{c|}{**IEAC:**} & \multicolumn{2}{c|}{**ELACE:**} & \multicolumn{2}{c|}{**ILEXC:**} & \multicolumn{2}{c|}{**BLACE:**} & \multicolumn{2}{c|}{**ENCAP:**} & \multicolumn{2}{c|}{**LRACE:**} & \multicolumn{2}{c|}{**LDECE:**} & \multicolumn{2}{c|}{**BDFAL**} \\  & 



\begin{table}
\begin{tabular}{l|c c|c c|c c|c c|c c} \hline \hline
**Debiasing method type** & \multicolumn{2}{c|}{**No debiasing**} & \multicolumn{3}{c||}{**At-training**} & \multicolumn{3}{c||}{**Pre-processing**} & \multicolumn{3}{c}{**Post-processing \& Selective**} \\ \hline \multirow{2}{*}{**Dataset**} & **Metric** & **Standard** & **Adv** & **DAdv** & **FTEO** & **BTJ** & **NLP** & **NLP+gR**, & **NLP+kL**, & **SPL+gR** & **NLP+gL**, & **opt-per.** & **opt-per.** \\ \hline  & Fairness \(\uparrow\) & 61.8\%, 73.7\%, 74.0\%, 73.4\%, 75.3\%, 74.0\%, **75.2\%,** 74.8\%, 74.8\%, **77.3\%,** 63.5\%, 79.0\%, **75.1\%** & 63.5\%, 79.0\%, **75.1\%** \\ MOII & Accuracy? & **79.1\%,** 87.2\%, 72.0\%, 72.4\%, 73.6\%, 73.6\%, 73.2\%, 68.4\%, 78.0\%, 78.8\%, 73.5\%, **74.4\%** & **79.8\%, 73.5\%** \\ imbalanced & DTO \(\downarrow\) & 43.6\%, 88.4\%, 88.3\%, 88.3\%, **86.2\%, 86.7\%, 74.0\%, 80.0\%, 74.6\%** & **40.0\%, 74.6\%** & **43.9\%, 79.1\%** & 79.0\%, **71.9\%** \\ FF-score \(\uparrow\) & 69.4\%, 72.8\%, 72.9\%, 72.9\%, 74.4\%, 74.0\%, 74.8\%, 74.0\%, 74.9\%, 71.9\%, 70.0\%, **71.9\%** & 70.0\%, **71.9\%** & **71.9\%, 71.9\%** \\ \hline  & Fairness \(\uparrow\) & 69.5\%, 83.8\%, 88.4\%, 84.7\%, 85.5\%, 85.0\%, 88.6\%, 88.0\%, 88.0\%, 71.7\%, 77.9\%, **68.5\%** & 68.5\%, 88.0\%, 71.7\%, 77.9\%, **68.5\%** \\ MOII & Accuracy? & 71.9\%, 70.4\%, 70.4\%, 74.0\%, 74.0\%, 74.0\%, 74.0\%, 74.0\%, 74.0\%, 74.0\%, 74.0\%, 63.0\%, 71.9\%, 68.5\%, 75.0\%, 71.9\%, 68.5\%, 75.0\%, 71.9\%, 68.5\%, 75.0\%, 71.9\%, 68.5\%, 75.0\%, 71.9\%, 68.5\%, 75.0\%, 71.9\%, 68.5\%, 75.0\%, 71.9\%, 68.5\%, 75.0\%, 71.9\%, 71.9\%, 71.8\%, 71.9\%, 71.9\%, 71.8\%, 72.8\%, 71.9\%, 72.9\%, 72.9\%, 73.0\%, 73.0\%, 73.0\%, 73.0\%, 73.0\%, 73.0\%, 72.9\%, 74.0\%, 75.0\%, 74.0\%, 76.6\%, 71.9\%, 71.9\%, 71.8\%, 72.8\%, 72.8\%, 74.0\%, 75.0\%, 75.0\%, 76.0\%, 72.9\%, 76.5\%, 72.9\%, 76.5\%, 75.0\%, 77.



Comparison with other Distances

We also conducted additional experiments to compare how proposed selection strategies differ from other similarity measures. Here, we consider several measures, calculated over the output from the last hidden layer of the model, and compare them with SR and KL strategies. The results are presented in Tables 15 to 17. In most cases, selection by KL works comparably or better than the best-performing distance-based measure. Moreover, KL scores are easier to compute than distance-based scores. However, in some cases, cosine distance could serve as a replacement for the KL score due to its similar performance.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c} \hline \hline
**Dataset** & **Standard** & **LEACK2** & **89.55** & **KL.5*** & **Euclidean** & **50.56** & **S0.195** & **KL.195** & **Euclidean** & **10.54** & **Cohen** & **10.195** & **SL.195** & **Exidation** & **10.55** \\ \hline Motif induced & 69.4x\({}_{13}\) & 71.8x\({}_{14}\) & 72.5x\({}_{14}\) & 71.0x\({}_{14}\) & 72.2x\({}_{14}\) & 71.5x\({}_{14}\) & 72.2x\({}_{13}\) & 73.5x\({}_{14}\) & 73.5x\({}_{14}\) & 72.5x\({}_{14}\) & 71.8x\({}_{14}\) & 73.0x\({}_{14}\) \\ Motif induced & 70.7x\({}_{13}\) & 72.5x\({}_{14}\) & 75.4x\({}_{14}\) & 74.2x\({}_{14}\) & 76.0x\({}_{14}\) & 76.0x\({}_{14}\) & 76.2x\({}_{14}\) & 76.2x\({}_{14}\) & 76.2x\({}_{14}\) & 76.5x\({}_{14}\) & 77.5x\({}_{14}\) & 77.5x\({}_{14}\) \\ MOOS-2 Influenced & 85.5x\({}_{14}\) & 87.0x\({}_{14}\) & 87.0x\({}_{14}\) & 89.1x\({}_{14}\) & 89.0x\({}_{14}\) & 89.0x\({}_{14}\) & 89.0x\({}_{14}\) & 89.0x\({}_{14}\) & 89.1x\({}_{14}\) & 89.1x\({}_{14}\) & 89.1x\({}_{14}\) & 89.1x\({}_{14}\) & 89.1x\({}_{14}\) \\ MOOS-2 Influenced & 91.1x\({}_{14}\) & 60.0x\({}_{14}\) & 91.6x\({}_{14}\) & 92.0x\({}_{14}\) & 90.1x\({}_{14}\) & 99.5x\({}_{14}\) & 92.2x\({}_{14}\) & 99.6x\({}_{14}\) & 79.1x\({}_{14}\) & 89.2x\({}_{14}\) & 85.4x\({}_{14}\) & 84.4x\({}_{14}\) & 82.8x\({}_{14}\) \\ \hline \hline \end{tabular}
\end{table}
Table 16: Comparison of FF-score of distance-based scores for LEACE-cls for various percentages.

\begin{table}
\begin{tabular}{l l l l l l l l l l l l l l l l l} \hline \hline
**Dataset** & **Standard** & **LEACK2** & **89.55** & **KL.5*** & **Euclidean** & **50.56** & **S0.195** & **KL.195** & **Euclidean** & **10.55** & **Cohan** & **10.195** & **Schin** & **10.195** & **Schin** & **10.195** & **Chon** & **10.195** & **Sk.195** & **KL.195** & **Enddition** & **10.195** & **Cohan** & **10.195** & **Cohan** & **10.195** & **Cohan** & **10.195** \\ \hline Motif induced & 69.4x\({}_{13}\) & 71.8x\({}_{14}\) & 69.5x\({}_{14}\) & 72.5x\({}_{14}\) & 71.0x\({}_{14}\) & 72.2x\({}_{14}\) & 71.5x\({}_{14}\) & 72.2x\({}_{14}\) & 71.5x\({}_{14}\) & 69.7x\({}_{14}\) & 71.8x\({}_{14}\) & 71.8x\({}_{14}\) & 71.8x\({}_{14}\) & 71.8x\({}_{14}\) \\ Motif induced & 70.7x\({}_{14}\) & 71.9x\({}_{14}\) & 72.4x\({}_{14}\) & 75.4x\({}_{14}\) & 74.0x\({}_{14}\) & 76.0x\({}_{14}\) & 76.2x\({}_{14}\) & 76.2x\({}_{14}\) & 76.2x\({}_{14}\) & 76.2x\({}_{14}\) & 76.5x\({}_{14}\) & 77.5x\({}_{14}\) & 77.5x\({}_{14}\) & 77.5x\({}_{14}\) \\ MOOS-2 Influenced & 85.5x\({}_{14}\) & 87.0x\({}_{14}\) & 87.0x\({}_{14}\) & 94.1x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) \\ MOOS-2 Influenced & 91.1x\({}_{14}\) & 89.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) & 99.0x\({}_{14}\) \\ \hline \hline \end{tabular}
\end{table}
Table 17: Comparison of FF-score of distance-based scores for INLP for various percentages.

 

Computational Efficiency

To estimate the computational efficiency of selective debiasing, we calculated the inference time of the standard model and the model with selective debiasing. The results are presented in Tables 18 and 19. Table 18 shows the inference time of models averaged for 10 runs, while Table 19 presents computational overhead for each debiasing method. The computational overhead is calculated as follows:

\[CompOverhead=100\cdot\left(\frac{T_{selective}}{T_{standard}}-1\right), \tag{11}\]

where \(T\) is the summary inference time of the debiasing method for all datasets. These experiments were conducted on one Nvidia H100 GPU. The proposed selective debiasing approach does not introduce much computational overhead - for LEACE-last and LEACE-cls it is less than 1%.

Table 20 shows a detailed comparison of debiasing methods. As one can see, at-training and pre-processing debiasing methods require a training model from scratch, while post-processing methods with selective debiasing do not require this. Hence, post-processing methods are especially beneficial when the full dataset or the model is unavailable, while selective debiasing allows for increasing the overall performance of these methods. On the other hand, there is some computational overhead for post-processing methods compared to other ones. However, this overhead is negligible in most cases.