InterFair: Debiasing with Natural Language Feedback for Fair Interpretable Predictions
Authors: Bodhisattwa Prasad Majumder
Zexue He
Julian McAuley
Venue: EMNLP 2023 Main (2023)
Keywords: Debiasing
Language Models
Rationale
Interactions
User Interventions

Abstract: Debiasing methods in NLP models traditionally focus on isolating information related to a sensitive attribute (e.g., gender or race).
We instead argue that a favorable debiasing method should use sensitive information 'fairly,' with explanations, rather than blindly eliminating it. This fair balance is often subjective and can be challenging to achieve algorithmically. We explore two interactive setups with a frozen predictive model and show that users able to provide feedback can achieve a better and \emph{fairer} balance between task performance and bias mitigation. In one setup, users, by interacting with test examples, further decreased bias in the explanations (5-8%) while maintaining the same prediction accuracy. In the other setup, human feedback was able to disentangle associated bias and predictive information from the input leading to superior bias mitigation and improved task performance (4-5%) simultaneously.

INTERFAIR:
Debiasing with Natural Language Feedback for Fair Interpretable Predictions
Bodhisattwa Prasad Majumder*1 Zexue He*2, and Julian McAuley²
Allen Institute for AI
University of San Diego
podhissttwammallenne org, {zehe, jmcauleyJ@ucsd.cco
contributes equally
Abstract
However, a user can potentially further tune the model's belief on the bias, leading to a correct pre- diction while minimally using biased information. While interactive NLP models recently focused on model debugging (Tandon et al.. 2021, 2022), improving explainability in QA (Li et al., 2022b), machine teaching (Dalvi et al.. 2022), critiquing for personalization (Li et al., 2022a), and dialog more expressive form of explanations (Lakkaraju et al., 2022; Slack et al., 2022), we focus on an under-explored paradigm of model debiasing us- ing user interactions. Objectively, we allow users to adjust prediction rationales at the test time to decrease bias in them, addressing the subjective aspect of fair and transparent debiasing.
Debiasing methods in NL. models traditionally focus on isolating information related to sen- sitive attribute (e.g. gender or race). We instead argue that favorable debiasing method should use sensitive information "fairly,' with explana- tions, rather than blindly eliminating it. This fair balance is often subjective and can he chal- lenging to achieve algorithmically. We explore two interactive setups with frozen predictive model and show that users able to provide feed- back can achieve a better and fairer balance he- tween task performance and bias mitigation. In one setup, users, by interacting with test exam- ples, further decreased bias in explanations (5-8%) while maintaining the same prediction accuracy. In the other setup, human feedback was able disentangle associated bias and pre- dictive information from the input leading to superior bias mitigation and improved task per- formance (4-5%) simultaneously.
In this paper, propose INTERFAIR, a modu- lar interactive framework that (1) enables users provide natural language feedback test time balance between task performance and bias mitiga- tion, (2) provides explanations of how a particular input token contributes to the task performance and exposing bias, and finally (3) achieves better perfor- mance than a trained model on full-text input when augmented with feedback obtained via interactions.
1 Introduction
Debiasing human written text is an important sci- entific and social problem that has been investi- gated by several recent works (Zhang et al., 2018; Jentzsch et al., 2019; Badjatiya et al., 2019; Hein- dorf et al., 2019; Ravfogel et al., 2020; Gonen and Goldberg, 2019; He et al., 2021). These methods primarily try to eliminate the biased information from the model's internal representations or from the input itself, disregarding the task performance during the process. However, in an ideal situation, a model should use only the necessary amount of information, irrespective of bias, to achieve an ac- ceptable task performance. This off between task performance and bias mitigation is subjective varies between users (Yaghini et al., 2021) and is often hard achieve via learning from data (Zhang et al., 2018: He et al., 2022). Figure shows the limit of an algorithmic approach where ignoring all gendered information can lead to a wrong result.
2 Background: Debiasing with Rationales
An interpretable debiasing algorithm produces a rationale along with prediction of the original task to expose the amount of bias or sensitive infor- mation used. Precisely, rationale is the minimal and sufficient part of the input responsible for the prediction. For text input, let the predictive input tokens for the task output be called task rationales and tokens revealing sensitive information be called bias rationales. Since the model solely uses the ra- tionales to predict the task output, these rationales are highly faithful (Jain et al., 2020).
According He et al. (2022), it is possible attach an importance score for each token to be included the respective task or bias rationales. Traditional debiasing algorithms face two failure

---

Input
Prediction
Task Rationales
Bise Rationales
Angela Lindvall model she has represented slmost every major fashion brand
Angela Linovali model
Angela Lindvall model
Classifier
Model r
Bile has represented almost every fashion brand
ehe ras represented almool every fashion brand
Don't use w: model Don't any
Update Prediction
Update Task Rationales
Reinstate Bias Delinition
Angela Lindvall model and represented almost every major fashion brand
(ocar)) Classifier
Annelaa Lincivall is a model and representod nimost every major tasnion brand
Angela Lindvall is model and she represented almoot every mejor fashion brand
Fashion Designer
Consider using w: model Don't any
Update Prediction
Update Task Rationales
Redefine
Blas Definition
Angela Lindvall she has represented almost every major fashion brand
Yozert Classifier
Angela Lindval is model she represented nimost
Angela Lindval model and she has represented aimont
Model
every major feahion brand
every major fashion brand
Pipeline. An algorithmically debiased model still suffers from generating biased rationales. Users interact with the final model states and perturb them using language feedback further to decrease bias and/or improve task performance.
Figure 1:
modes: 1) it produces a correct task prediction but with highly biased rationale and 2) it produces a wrong task prediction but a rationale with low bias. He et al. (2022) argue that weighing less on high- bias and high-task important tokens and promoting their low-bias replacements can simultancously ad- dress both of the failure modes. However, this technique is not perfect, and Figure shows a lim- iting case of this approach that opens up further room for improvement.
classification model, trained using the procedure described in He et al. (2022). The classification model generates a prediction and a pair of normal- ized scores (between O and 1) for each input token for its contribution toward task rationale and bias rationale. While large language models (LLMs) can be superior classifiers, the opaqueness of these models hinders faithful perturbation of rationales, which is one the goals of this work.
During operation, the user queries with a text input for the classification task (e.g.. predicting the profession from biography) and a known bias variable (e.g.. gender). After querying, the user receives the prediction, rationales (with importance scores) for the task prediction, and the bias vari- able. Since the goal is to potentially disentangle the bias from the predictive task, we restrict users to di- rectly modify the bias rationales only. A change in the bias rationales will trigger a change in the task rationales and, finally, in the prediction. Since ra- tionales are in natural language (tokens), we enable users interact in natural language (NL). INTER- FAIR converts the NL feedback to be actionable for the model to update its rationales.
3 INTERFAIR: Interactive Fair Debiasing
We highlight that even an algorithmically debiased model can have failure modes and one potential option is to fix the problem at the inference time. We argue that human users are better at fixing the failure cases that model is unable to learn from the training data. We also assume that the model parameters remain frozen during the fixing process, and users only interact with the final prediction and its associated hidden model states.
Task and Base Model We start with a frozen model that is algorithmically debiased and allow users interact and edit its rationale the infer- ence time towards lower bias. Since rationales are tied to task prediction, the user should edit them without lowering the task performance. Primarily, the users are encouraged to find better low-bias replacements for tokens highly important for both task performance and revealing bias. To this end, we hypothesize system, INTERFAIR, to achieve fair balance between task performance and bias. For the scope of this paper, we use classification as the predictive task and text only as the input modality. For the base model, we use an LSTM
3.1 Parsing NL Feedback
Rationales are presented to the users with impor- tance scores for each input token (see Figure I). To directly modify the bias rationales, users can increase or decrease the bias importance score for each token accordingly. For example, in the Fig- ure 1 example, it is prudent to decrease the bias importance for the word model and increase the bias importance for Agnela Lindvall.
The simplest form of feedback is to provide feed- back on the bias importance of certain input token

---

Example
k-shot
IID
Comp
Overall
[Input] Angela Lindvall is a model and she represented (..)
Model: GPT-3
[Bias] Gender
(ext-davvnci-003)
shot
58.7
34.2
46.5
[Feedback] Angela Lindvall is woman's name [Parse] High, High, NA, NA, NA, NA, NA, NA (...)
10 shot
74.2
45.8
60.0
20 shot
83.8
60.1
71.9
Table 1: NL feedback parser. Parse example; parsing accuracy on IID, compositional (Comp) splits, and overall test set.
by indicating whether they would be high or low. However, we expect users to have linguistic varia- tions in their queries. To generalize the process of parsing the NL feedback to actionable feedback for all input tokens, treat it as a sequence labeling task. Specifically, we build a parser that encodes the NL feedback, the bias variable (e.g., gender), and the original task input and produces a sequence of High / Low / NA labels for the complete input to- ken sequence. An example feedback and its parse are shown in Table 1. Such an approach allows to encode complex feedback multiple input tokens (see Figure 1).
if its updated bias importance is higher than a threshold. The new task rationales are used to generate the new prediction.
Gradient: Since changes in bias rationale scores affect task rationales scores (hence the task rationales), we can directly perturb the fi- nal hidden states h of the classification model that generate the task rationale scores for each token (Majumder et al., 2021). We compute a KL-divergence (AC) score between biasold and biasnew and and compute its gradient V w.r.to h. Finally, we update h by minimizing the K. via pack-propagation using the computed gradients. Note model parameters are up- dated in this process. The updated generates the new task rationales and a new prediction.
Since we do not have large annotated data for the parsing task, we instead adopt a few-shot frame- work, following (Slack et al., 2022). We use large language model (e.g. GPT-3; text-davinci-O00 as they have strong priors for language understand- ing (here, parsing) tasks from their pre-training phase. We use a few demonstrative parsing exam- ples for in-context learning of the parser. See the parsing task example in Table 1.
Experiments and Results
We break our experiments into two parts: 1) devel- oping the NL parser and 2) interactive debiasing with INTERFAIR. We use BiosBias (De-Arteaga et al., 2019), dataset made from large-scale user study of gender in various occupations. It contains short biographies labeled with gender and profes- sion information, and a possible confluence exists between gender and annotated profession labels.
3.2 Modifying Bias Rationales
After parsing the NL feedback, we use the parse labels to update the bias importance scores. First, we convert each parse label to numeric equivalent using the following map (parse label + important score): High 1; Low O; NA unchanged. Then we use linear combination update the bias importance scores:
Using INTERFATR, we would like to predict the profession from biographies without the influence of gender. Following (Ravfogel et al., 2020), we use 393,423 biographics with binary gender labels (male/female) and 28 professions labels (e.g. pro- fessor, model, etc.). We initially used 255,710 ex- amples for training and 39,369 for validation. We use 500 examples random sample from the rest 25%) as a test set for interactive debiasing.
biaSpew abiaspew + a biasuser
with a hyperparameter and biasuser being the nu- meric equivalent of the user feedback.
3.3 Modifying Task Rationales and Prediction
Change in bias importance scores should propagate the task rationale. We explored two strategies to update the task rationale.
For evaluation, we use accuracy for task perfor- mance (profession prediction) and use an off-the- shelf gender detector to measure the bias in the task rationales (Bias FI), following He et al. (2022).
Heuristic: Following (He et al., 2022), we pe- nalize current task importance for a token only

---

4.1 NL Feedback Parsing
Models
Acc.
Bias F1
Compre.
Suff.
Full Text
81.1
0.98
Following Slack et al. (2022), we use 5, 10, or 20 examples annotated by two independent annota- tors for the NL parser. We additionally obtain a set of 50 more annotations for testing the parser. While testing the performance the parser, we use the accuracy metric, i.e., if the parsed feedback matches with the gold parse. We also consider two splits for testing: an IID split where the gold parse contains non-NA labels for one or two contiguous input token sequences and compositional split where the gold parse has three or more contiguous token sequences. Table shows the parsing accu- racy, which reveals that the compositional split is harder than the IID due to its complexity. However, the few-shot parsing using LLMs is faster and eas- ier to adapt with newer user feedback instead of finetuning a supervised model (Slack et al., 2022).
Reranking
70.3
0.45
0.23
0.32
Adv
36.7
0.35
INTERFAIR-base
80.1
0.38
0.52
0.01
Constrained:
INTERFAIR-
80.1
0.33
0.51
0.01
SNFERFAIR-Graiiee 80.1
0.30
0.48
0.00
Unconstrained:
INTERFAIR-Heuride SFERFAiR-Grraiien 85.2 0.33
83,9
0.38
0.51
0.00
0.48
0.00
Table 2: Evaluation for task accuracy (Ace. (%) 1), bias (F1 J) and faithfulness for task rationales: Comprehensive- ness (Compre. f) and Sufficiency (Suff. 4)
INTERFAIR without feedback balances the task per- formance and bias very well.
In the constrained setup, the user locks in the task performance (by design) but are able to de- crease bias further the inference time just by perturbing model hidden states using NL feedback. In the unconstrained setup, users are able to mod- ify bias rationales in such a way that improves task performance while decreasing bias. Most importantly, even though 81% (Full Text perfor- mance) is the upper bound of accuracy for purely training-based frameworks, users achieve better task performance (4-5%) while keeping the bias in rationales minimal. In both setups, gradient- based changes in model states are superior the heuristic strategy to modify the final task rationales. Since unconstrained setup can also confuse users and may lead to failure modes, we see the lowest bias FI is achieved in the unconstrained setup; how- ever, users were able to keep the bias as low as the INTERFAIR-be -ase model in all interactive settings.
4.2 Interactive debiasing
We perform a user study with 10 subjects who inter- act with INTERFAIR and optionally provide feed- back to one of the two objectives- - 1) Constrained: Minimize bias in task rationales without changing the task prediction, and 2) Unconstrained: Min- imize bias task rationales a priority, however, can update task prediction if it seems wrong. The cohort was English-speaking and had an awareness of gender biases but did not have formal education in NLPM/L. The study included an initial training session with 10 instances from the BiosBias test set. Subsequently, participants engaged with 500 reserved examples designated for the interactive debiasing phase. The gender split of the subject pool was 1:1.
Test-time improvement of task performance and bias with frozen model indicates that 1) full-text- based training suffers from spurious correlation or noise that hampers task performance, and 2) interactive debiasing is superior to no feedback since it produces better quality human feedback to refine task performance while eliminating bias. This phenomenon can be seen as a proxy for data augmentation leading to a superior disentanglement of original task performance and bias.
To understand the change in model performance and bias, we consider two other debiasing models along with the base model (He et al., 2022) used in INTERFAIR: (1) Rerank, an inference-time debias- ing variant where the task rationale is considered based on ascending order of bias energy (He et al., 2022); (2) Adv, model trained with an adversarial objective (Zhang et al., 2018) to debias the model's latent space, but incapable of producing any ratio- nales.
Finally, since test-time interactions modify task rationales, check their faithfulness using com- prehensiveness and sufficiency scores, measured defined in (De et al., 2020). Sufficiency is defined as the degree to which a rationale ade- quate for making a prediction, while comprehen- siveness indicates whether all rationales selected
Table 2 shows that when task input, the bias in task rationales is very high. Reranking decreases the bias but also incurs a drop in task performance. The adversarial method does not produce any explanation and cannot use any ad- ditional feedback, leading to low task performance.
Full Text as

---

are necessary for making prediction. A higher comprehensiveness score and lower sufficiency indicate a high degree of faithfulness. We show that even after modification through interactions, the faithfulness metrics do not deviate significantly from the base models, and final task rationales from INTERFAIR remain faithful.
5 Conclusion
In summary, INTERFAIR shows the possibility of user-centric systems where users can improve model performances by interacting with it at the test time. Test-time user feedback can yield better disentanglement than what is achieved algorithmi- cally during training. Debiasing is subjective task, and users can take the higher agency guide model predictions without affecting model parame- ters. However, INTE does not memorize pre- vious feedback at a loss of generalization, which can be addressed via memory- interactions (Tandon et al., 2022), or persistent model editing (Mitchell ct al., 2021) as future work.
4.3 Discussion
Feedback format In our initial pilot study with sample size of N=5 (subjects with no background in NLP/ML), we investigated two feedback for- mats: 1) allowing participants to perturb weights through three options NA/High/Low, and 2) so- liciting natural language feedback. While it may seem more efficient to offer feedback by engaging with individual tokens and selecting a perturbation option, participants expressed confusion regarding how altering the significance of each token would effectively mitigate bias. Conversely, participants found more intuitive to provide natural language feedback such as "A person's 's name is unrelated to their profession." " To understand the possibility of this would change had our participants possessed a background in NLP/ML, we conducted a sup- plementary study involving another cohort of 5 participants, all of whom had completed at least relevant course in NLP/ML. These participants encountered difficulties directly manipulating token importance using the NA/High/Lov options and revealed a comparable trend to approaches em- ploying natural language feedback methods.
Acknowledgements We thank the anonymous reviewers and the members of the Aristo team at A12 for their insightful feedback. BPM was funded, in part, by Adobe Research Fellowship.
Limitations
Our framework does not persist user feedback which may make the debiasing process repetitive and tedious. Users may have to provide almost identical feedback different data points where the model is making a systemic error. It should be prudent store user feedback and apply it automat- ically and efficiently to minimize the user-in-the- loop effort. We also acknowledge that there can be multiple ways of debiasing a task, and it depends on the context of each example. Also, debiasing be- ing subjective task at the end, its evaluation rests the subjective evaluation of the experiments performed. We tried our best to make the subject sample as representative as possible; however, the sample can still suffer from vocio-cultural bias.
Beyond LSTMs LSTM-based base models en- joyed the gradient update during the interactive de- biasing, but to extend this to the model to no hidden states access (e.g., GPT-3), we have to restrict only to heuristic-based approach. We investigate mod- ular pipeline that uses GPT-3 (text-davinci-oee extract both the task and bias rationales and then followed by an LLST--baee predictor that predicts the task labels only using the task rationales. The rationale extractor and task predictor are not con- nected parametrically, another reason why we can only use heuristic-be bauret methods to update the task rationales. The final accuracy and Bias Fl were not significantly different than what was achieved in our LSTM-based setup despite GPT-3 based IN- TERFAIR-base having significantly better perfor- mance (acc. 84.0). This suggests the choice of the underlying base model may not be significant if the output can be fixed through iterative debiasing.
Ethics Statement
Our framework assumes that users will not pro- vide any adversarial feedback. We monitored user behavior during the user study and discarded any such feedback from the final evaluation of the sys- tem. However, in real-world environments, this assumption may not hold as users can direct the model to generate a more biased prediction than its base performance. However, since we do not have persistent user changes, an adversarial user cannot make negative impact on another user's session. Still, it prudent to have monitoring agen- cies restrict users from directing models to generate biased harmful content.

---

References
Bodhisattwa Prasad Majumder, Taylor Berg- Kirkpatrick, Julian J. McAuley, and Harsh Jhamtani. 2021. Unsupervised enrichment of
Pinkesh Badjatiya, Manish Gupta, and Vasudeva Varma. 2019. Stereotypical bias removal for hate speech de- tection task using imowledge-based generalizations. In WWW.
persona-grounded dialog with background stories. In ACL.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2021. Fast model editing at scale. arXiv preprint arXiv:2110.111
Bhavana Dalvi, Oyvind Tafjord, and Peter Clark. 2022. Towards teachable reasoning systems. arXiv preprint arXiv:2204. 13074.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null out: Guard- ing protected attributes by iterative nullspace projec- tion. In ACL.
Maria De-Arteaga, Alexey Romanov, Hanna Wal- lach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019, Bias in bios: case study of semantic representation bias in high-stakes setting. In FAT.
Dylan Slack, Satyapriya Krishna, Himabindu Lakkaraju. and Sameer Singh. 2022. Talktomodel: Understand- ing machine learning models with open ended dia- logues. arXiv preprint arXiv:2207. 04154.
Jay Yeunung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. 2020. Eraser: A benchmark to evaluate rationalized nlp models. In ACL.
Niket Tandon, Aman Madaan, Peter Clark, Keisuke Sakaguchi, and Yiming Yang. 2021. Interscript: A dataset for interactive learning of scripts through ror feedback. CoRR, abs/21 1207767.
Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do remove them. NAACI-HILT
Niket Tandon, Aman Madaan, Peter Clark, and Yiming Yang. 2022. Learning to repair: Repairing model out- put errors after deployment using dynamic memory of feedback. NAACL Findings. appear).
Zexue He, Bodhisattwa Prasad Majumder, and Julian McAuley, 2021. Detect and perturb: Neutral rewrit- ing of biased and sensitive text via gradient-based decoding. Findings of EMNLP, pages 4173-4181,
Mohammad Yaghini, Andreas Krause, and Hoda Hei- dari. 2021. A human-in iennan-i--the-ooo framework to construct context-aware mathematical notions of outcome fairness. In Proceedings of the 2021 AAAT/ACM Conference on AI, Ethics, and Society, pages 1023-1033.
Zexue He, Yu Wang, Julian McAuley, and Bod- hisattwa Prasad Majumder. 2022. Controlling bias exposure for fair interpretable predictions. Findings of EMNLP.
Stefan Heindorf, Yan Scholten, Gregor Engels, and Mar- tin Potthast. 2019. Debiasing vandalism detection models at wikidata. In WWW.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018. Mitigating unwanted biases with adversarial learning. In AIES.
Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, and By- C. Wallace. 2020. Learning to faithfully rational- ize by construction. ACI..
Sophie Jentzsch, Patrick Schramowski, Constantin Rothkopf, and Kristian Kersting. 2019. Semantics derived automatically from language corpora contain human like moral choices. In AIES.
Himabindu Lakkaraju, Dylan Slack, Yuxin Chen, Chen- hao Tan, and Sameer Singh. 2022. Rethinking ex- plainability dialogue: A practitioner's perspec- tive. arXiv preprint ar2iv:220..01875.
Shuyang Li, Bodhisatwa Prasad Majumder, and Ju- lian McAuley. 2022a. Self-supervised bot play for trancrrpt-ffree conversational recommendation with rationales. In Proceedings of the 16th ACM Confer- ence on Recommender Systems, pages 327-337.
Zichao Li, Prakhar Sharma, Xing Han Lu, Jackie CK Cheung, and Siva Reddy. 2022b. Using interactive feedback to improve the accuracy and explainabil- ity of question answering systems post-deployment. arXiv preprini 20X4v.2204.03025.


Reviews:
[{"review_id": "3Num6uSf5N", "reviewer": "Reviewer_beK8", "summary": "", "questions": "Question A: Could you please provide some potential explanations on why gradient methods outperform heuristic methods (Table 2)?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper! We were happy to read that you appreciated the work by stating that our framework for incorporating human feedback is 1) novel, 2) clearly motivated, and 3) showcases superior results as compared to competing baselines that strongly support our claims.\n\nWe addressed all of your questions with more details and additional experiments. Please see our response below. We would love to address additional questions during the discussion period if anything is unclear.\n\n-----------\n**Justification for using Natural Language Feedback**\n\n*Our pilot study revealed that users who do not have a background in NLP/ML found it easier to provide natural language feedback as opposed to changing the importance scores directly, which required some understanding of the underlying model.*\n\nDetails: In our initial pilot study with a sample size of N=5 (subjects with no background in NLP/ML), we investigated two feedback formats: 1) allowing participants to perturb weights through three options - NA/High/Low, and 2) soliciting natural language feedback. While it may seem more efficient to offer feedback by engaging with individual tokens and selecting a perturbation option, participants expressed confusion regarding how altering the significance of each token would effectively mitigate bias. Conversely, participants found it more intuitive to provide natural language feedback such as \"A person's name is unrelated to their profession.\" Notably, this situation might have evolved had our participants possessed a background in NLP/ML.\n\nTo corroborate this, we conducted a supplementary study involving another cohort of 5 participants, all of whom had completed at least one relevant course in NLP/ML. These participants encountered no difficulties in directly manipulating token importance using the NA/High/Low options. Despite the reasonable accuracy of our GPT-3 parser, directly adjusting token importance offers distinct advantages, i.e., zero parsing errors. Our findings reveal a comparable trend to approaches employing natural language feedback methods.\n\n| Models | Acc. $\\uparrow$ | Bias F1 $\\downarrow$ | Compre. $\\uparrow$ | Suff. $\\downarrow$ |\n|-|-|-|-|-|\n|Constrained|||||\n|InterFair Heuristic| 80.1 | 0.32 ($\\downarrow$0.02) | 0.50 | 0.01 |\n|InterFair Gradient| 80.1 | **0.29** ($\\downarrow$0.02) | 0.47 | 0.01|\n|Unconstrained|||||\n|InterFair Heuristic| 84.2 ($\\uparrow$0.3) | 0.37 ($\\downarrow$0.01) | 0.50 | 0.00|\n|InterFair Gradient| **85.6** ($\\uparrow$0.4) | 0.32 ($\\downarrow$0.01) | 0.47 | 0.00|\n\n(*numbers in parenthesis measure gains/losses from respective entries in Table 2, where all feedback was in natural language)\n\n---------\n**Q: Details about the human study**\n\n*Thank you for mentioning this. We will add more details in the final version.*\n\nDetails: In our human study (from the main paper), a cohort of 10 English-speaking individuals was selected. These participants had an awareness of gender biases but did not have formal education in NLP/ML. The study included an initial training session with 10 instances from the BiosBias test set. Subsequently, participants engaged with 500 reserved examples designated for the interactive debiasing phase. The gender split of the subject pool was 1:1.\n\nIn the newer study (with token-level importance perturbation), the setup was similar, but the subjects had a formal foundation in NLP/ML. Of these participants, 3 out of 5 were female, and the rest were male.\n\nWe did not collect both forms of feedback in a single study; however, our pilot (with inexperienced subjects in NLP/ML) revealed only 1 out of 5 subjects were comfortable with furnishing token-level feedback via direct manipulation of importance scores. \n\n-------------\n**Why is the gradient method better than the heuristic method**\n\n*The gradient method changes the task rationales through the learned non-linear activations in the base model, which is more accurate and better calibrated than the linear changes made by the heuristic methods.*\n\nDetails: As mentioned in Section 3.2, we follow He et al., 2022, wherein the task importance is linearly reduced by the quantity (updated bias importance for the token - A) if the difference is positive; otherwise, the task importance score remains unchanged. Here, A, the threshold, is a hyperparameter. However, a linear adjustment (a drawback in He et al., 2022) in task importance score may not necessarily yield the desired change in the outcome. \n\nTo accurately calibrate the updation of the task rationales, we employ gradient methods to modify the singular hidden state accountable for generating the task importance scores for tokens. This modification is based on minimizing the KL divergence between the previous and updated bias importance distributions. This approach is feasible due to the construction of both hidden vectors responsible for task and bias importance scores from the same encoder hidden states. \n\nBy definition, the gradient method produces smoother updates to task rationales based on user feedback, possibly leading to superior performance compared to heuristic methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. clear motivation on why human feedback matters\n2. novel methodology \n3. results show that their proposed method outperforms the no-feedback baseline", "reasons_to_reject": "1. I find it unclear on why users should provide natural language feedback to be parsed into High/Low/NA importance score per token, if the goal is just to adjust the weights of the already highlighted bias rationales? As I guess changing weights of a few tokens is easier/faster than writing a free-text response (to be later parsed by a GPT-3 LLM) anyway, why don't the authors only keep the option: ``To directly modify the bias rationales, users can increase or decrease the bias importance score for each token accordingly.'' Without more detailed motivation, the choice of natural language feedback may be an unnecessary hurdle for the experiment.\n2. Lacking details of human study (e.g. platform, subjects' background in NLP/ML bias, the distribution of feedback style in natural language form vs. in importance weight adjustment form)", "questions_to_authors": "Question A: Could you please provide some potential explanations on why gradient methods outperform heuristic methods (Table 2)?", "paper_topic_and_main_contributions": "Firstly, the paper proposes a new pipeline (InterFair) to collect human feedbacks about bias rationales and incorporate that feedback into how the model produces future task rationales and bias rationales.\nSecondly, the paper does an experiment and a human study to test their proposed pipeline.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1aTIsdj1P7", "reviewer": "Reviewer_Kj4w", "summary": "", "questions": "In line 121, the author mentioned that “the opaqueness of these models hinders faithful perturbation of reationales”, I wonder why LLms hinders perturbation of rational?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper! We were happy to read that you appreciated the work by stating that our framework is aptly motivated and showcases clear evidence of a decrease in bias and improvement in model performance.\n\nWe think that all your questions are addressable within the rebuttal period. Please see our response below. We would love to address additional questions during the discussion period if anything is unclear.\n\n-----------\n**Proposed method with LLM as a base model**\n\n*Similar trends are observed when InterFair uses LLMs as base models as compared to a no-feedback baseline. We will add additional results in the final version.*\n \nDetails: As a short paper, we initially aimed to keep our results self-contained, and hence we only used the LSTM model for the BiosBias dataset, extending He et al., 2022. However, our method can be applied to any base model (e.g., BERT) whose hidden states and gradients can be accessed. \n\nTo support our claim, we run two additional sets of experiments, with similar human study setup as the main paper: \n\n1) a BERT-based model with two separate heads for task and bias rationales, and a final layer that predicts the task prediction, only using selected task rationales as bottleneck. Since both task and bias rationale generators are integrated with the final predictor, we can use both heuristics and gradient-based methods to iteratively debias the outputs further. \n\n| Models | Acc. $\\uparrow$ | Bias F1 $\\downarrow$ | Compre. $\\uparrow$ | Suff. $\\downarrow$ |\n|-----|-----|-----|-----|-----|\n|InterFair Base| 86.2 | 0.37 | 0.48 | 0.01 |\n|Constrained|||||\n|InterFair Heuristic| 86.2 | 0.32 ($\\downarrow$0.05) | 0.50 | 0.01 |\n|InterFair Gradient| 86.2 | **0.29** ($\\downarrow$0.08) | 0.47 | 0.01|\n|Unconstrained|||||\n|InterFair Heuristic| 87.4 ($\\uparrow$1.2) | 0.37 (0.0) | 0.48 | 0.00|\n|InterFair Gradient| **87.8** ($\\uparrow$1.6) | 0.32 ($\\downarrow$0.05) | 0.48 | 0.00|\n\n(*numbers in parenthesis measure gains/losses from InterFair Base entries in the table)\n\n\n2) a modular pipeline that uses GPT-3 (text-davinci-003) to extract both the task and bias rationales and then followed by an LSTM-based predictor that predicts the task labels only using the task rationales. The rationale extractor and task predictor are not connected parametrically; hence we can only use heuristic-based methods to update the task rationales.\n\n| Models | Acc. $\\uparrow$ | Bias F1 $\\downarrow$ | Compre. $\\uparrow$ | Suff. $\\downarrow$ |\n|-----|-----|-----|-----|-----|\n|InterFair Base| 84.0 | 0.23 | 0.30 | 0.00 |\n|Constrained|||||\n|InterFair Heuristic| 84.0 | **0.21** ($\\downarrow$0.02) | 0.30 | 0.00 |\n|Unconstrained|||||\n|InterFair Heuristic| **84.8** ($\\uparrow$0.8) | 0.23 (0.0) | 0.31 | 0.01|\n\n(*numbers in parenthesis measure gains/losses from InterFair Base entries in the table)\n\nWe will add the additional results in the final version, as we concur with you that adding these would increase the accessibility of our work.\n\n-----------\n**Difference from He et al. 2022**\n\n*He et al. 2022 is our no-feedback baseline that does not include interactive updation. Our proposed methods outperform the baseline.*\n\nDetails: The key difference between He et al., 2022 and ours lies in the motivation. He et al., 2022 debias their models by a fixed definition: tokens that exhibit both a high bias and high task importance are discarded algorithmically. However, this solution may not be perfect due to the limit of learning statistical patterns from the data or in cases where the bias-performance trade-off is a subjective decision specific to an end-user. To achieve this, we propose InterFair, which allows interactive control (a key technical difference from He et al., 2022) over the task rationales by perturbing bias encoded in the model parameters, which shows superior performance in terms of both decreasing bias and improving task performance.\n\n------------\n**Opaqueness in LLM-based rationales**\n\nIn L121, our comment was a justification of not using black-box LLMs (GPT-3, GPT-3.5) since we do not have access to the hidden states in these LLMs responsible for generating extractive rationales leading to opaqueness, while we can clinically attribute to the responsible states for rationale generation in the LSTM architecture used in He et al., 2022 or BERT-based model we reported additional results on.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Clear motivation with a good showcase of how this work has done to better tradeoff between model performance and bias.", "reasons_to_reject": "Main experiments are conducted on LSTM, but have no idea on how the proposed methods will perform on LLMs, which in my view is more worthy of studying. Meanwhile, this work if quite similar to He et al. 2022, but the author doesn’t clearly explain the major shortcoming of He et al. 2022 that this work can better handle.", "questions_to_authors": "In line 121, the author mentioned that “the opaqueness of these models hinders faithful perturbation of reationales”, I wonder why LLms hinders perturbation of rational?", "paper_topic_and_main_contributions": "This work proposes an approach to better tradeoff between model debiasing and performance via user interaction on task and bias rationales. Experimental results show that the proposed methods can better mitigate model bias as well as achieving better task performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vkCzOhQPdw", "reviewer": "Reviewer_mchw", "summary": "", "questions": "A: In the Heuristic approach (L190), how are the new task rationales used to generate the new prediction?\n\nB: In the Gradient approach, you state that you do not update the model, but only the final hidden states (h) of LSTM. My question is how can you differentiate between per-token hidden states of dimensionality (hidden_state_dim, prediction_length) and rationales of (prediction_length, )? Am I missing something here?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper! We were happy to read that you appreciated our work by showing positive enthusiasm about our (novel) research direction and stating that our framework is well-motivated and the experiments sufficiently support our claims.\n\nWe acknowledge your clarification questions with more details and believe that we addressed them all. Please see our response below. We would love to address additional questions during the discussion period if anything remains unclear. \n\nAs a short paper, we originally aimed to keep our main paper to-the-point, but we would be happy to add the additional details, as requested by you, in the final version. We will also release the codes of the models with the final version for reproducibility. \n\n-----\n**Background for the algorithmically debiased models and details about baselines**\n\n*Thank you for the point. We will add the details in the final version of the main paper to improve readability.* \n\nDetails: L92 particularly refers to our no-feedback base model, as obtained from He et al. 2022, and indicates a failure mode, as depicted in Figure 1. This was in the context of L81-83 where we mentioned two possible failure modes for models that debias through rationales: 1) it produces a correct task prediction but with a highly biased rationale, and 2) it produces a wrong task prediction but a rationale with low bias.\n\n[He et al., 2022](https://aclanthology.org/2022.findings-emnlp.431.pdf) is the first attempt at debiasing through rationales, which we also consider as our no-feedback base model. Rerank is an ablation baseline from He et al. 2022, where the task rationale is selected based on a reversed order of bias importance (i.e., a less biased token with a high task energy will be selected). This is equivalent to when a user greedily selects all the tokens that have low bias importance. Hence, it is a competing baseline.\n\nAdv. refers to debiasing methods (such as [Zhang et al., 2018](https://arxiv.org/pdf/1801.07593.pdf) and [Ravfogel et al., 2020](https://aclanthology.org/2020.acl-main.647.pdf)) using adversarial training that debiases the latent space by projecting the hidden representations into the subspace where the bias is no longer predictable. This is one of the most common algorithmically debiased model architectures, hence a competing baseline. However, this model does not produce task rationales and hence cannot be improved through interactions. By comparing with Adv., we investigate if iteratively improving the rationale-based debiasing model’s output is better than that from an alternative black-box debiasing model.\n\n-----------\n**Details about the evaluation: faithfulness metrics**\n\n*Thank you for pointing this out. We will add more details in the main paper.*\n\nDetails: Since InferFair further changes the task rationales, it is prudent to make sure that the task rationales remain trustworthy towards the final prediction. Following the [ERASER benchmark ](https://arxiv.org/pdf/1911.03429.pdf)paper, we measure the faithfulness of extracted rationales using two scores: Sufficiency and Comprehensiveness. Sufficiency is defined as the degree to which a rationale is adequate for making a prediction, while comprehensiveness indicates whether all rationales selected are necessary for making a prediction. A higher comprehensiveness score and a lower sufficiency indicate a high degree of faithfulness. We show that even after modification through interactions, the faithfulness metrics do not deviate significantly from the base models, and final task rationales from InterFair remain faithful.\n\n----\n**Details about the mechanism for the incorporation of feedback**\n\n*More details and justifications will be added in the main paper. For a better understanding of InterFair’s working, we provide a brief pseudocode here.*\n\n```\n# Pseudocode for InterFair\n\nInitialize:\ndone = 0\nBase model generates output, task rationales (+ scores), and bias rationales (+ scores)\nfor each i in 1:N:\n\tFeedback, done <- user (to update the bias rationales)\n\tif done:\n            break\n            end if\t\n\tUpdated bias rationales <- feedback, old bias rationales (via linear transformation)\n\tUpdated task rationales <- updated bias rationales (via heuristic/gradient method)\n\tUpdated prediction, (bias rationales) <- updated task rationales\nend for \n```\n\n----\n**How are task rationales used to generate prediction in the heuristic method?**\n\nThe rationale selection score (i.e., importance score) is updated using heuristics and then used as soft binary decisions for each token to be selected in the task rationales or not. The soft selection (new task rationales) is then used for prediction.\n\n----------\n**How do we update the final hidden states in the gradient method?**\n\nA single hidden state, responsible for generating the soft selections of tokens to be in the rationales, is updated using the gradient method. First, we compute the KL divergence, $\\mathcal{K}$, between the old and updated bias importance scores (L198-199). $\\mathcal{K}$ is a function of the hidden state that generated the bias importance scores, which is a function of the input encoder hidden states of LSTM. The same input encoder hidden states also produce the hidden state accountable for generating the soft selections of tokens to be in the rationales, $h$, as mentioned in L200, which makes  $\\mathcal{K}$ a function of $h$. Hence, we can update $h$ through back-propagation by computing the gradient $\\nabla_h \\mathcal{K}(h)$. \n\nSimilar inference-time gradient-based updates have been executed in [Dathathri et al., 2020](https://arxiv.org/pdf/1912.02164.pdf), [Qin et al., 2021](https://arxiv.org/pdf/2010.05906.pdf).\n\n------\n**Compositional split for the parser data**\n\nCompositional split contains examples that are out-of-distribution from the few-shot examples for our GPT-3 based parser. The in-context examples for GPT-3 parser contained examples whose output had a maximum of two contiguous High or Low labels. However, in the test examples, there were parsing examples (compositional split) that contained more than two contiguous High or Low labels in the output. So the parser has to generalize in these out-of-distribution test examples, and hence we observe a lower performance as compared to when test examples were in-distribution. \n\n------\n**Justification of rationales selection in He et al., 2022**\n\nL81-83 mention two possible failure modes for models in He et al., 2022: 1) it produces a correct task prediction but with a highly biased rationale, and 2) it produces a wrong task prediction but a rationale with low bias. He et al., 2022 counter this by performing a trade-off, they deliberately drop tokens that have both high-bias and high-task importance by replacing them with low-bias but moderate-task important tokens (a bias-performance trade-off), anticipating that the moderate-task important can also generate a correct prediction. \n\n-----\n**Are hidden states part of the model?**\n\nThe hidden state ($h$), as mentioned in L195, is generated as an intermediate outcome during the forward pass of the model, which finally influences the final prediction. However, they are not the parameters of the model. InterFair works by updating the hidden states at the inference time but does not update any parameters of the model.\n\n**Presentation suggestions for L256**\n\nThank you for noting this. We will fix it in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well-written and intuitively structured\n* The motivation of the method's design is clear and well described\n* Incorporating human feedback into models' decision process is a potential, though challenging, research direction", "reasons_to_reject": "My main concern is the underspecification of multiple details throughout the work that are needed to understand the merit of the contribution. Below I list the more important ones, and I follow with assumably easier-to-address points in the \"Presentation Improvements\" section.\n\n* Background section does not contain an overview of the previous methods addressing debiasing, and specifically debiasing through rationales. This makes it difficult to assess the relatedness and novelty of the proposed approach.\n  * Subsequently, the important point of the motivation is that (L92) \"even an algorithmically debiased model can have failure modes\" probably refers to the previous methods, which, however, remain unknown to the reader at this point.\n  * The authors try to gap this bridge in the brief description of their baseline in the Results section, L256-261 (denoted as Rerank and Adv), but I have trouble grasping how these baselines work, not saying how they relate to InterFair, or whether they are sufficiently competitive.\n\n* Evaluation, including baselines, is difficult to interpret. The reader is prompted to understand the main metrics of comprehensiveness and sufficiency only from the cited work (L301). After following this order and checking the ERASER benchmark paper (https://aclanthology.org/2020.acl-main.408.pdf), I am convinced of the selection of these metrics in evaluation, but I believe their thorough description would be in place here.\n* I think the current description of the mechanism of incorporating feedback (Sec. 3.3) leaves a lot of space for ambiguity. After multiple reads, I do not feel confident in either the Heuristic nor the Gradient approach description. My questions address this, but the method description, as the main proposition of the paper, should be much more specific. Missing submission of the code or appendices with details currently makes it hardly possible to reproduce and possibly use the method.", "questions_to_authors": "A: In the Heuristic approach (L190), how are the new task rationales used to generate the new prediction?\n\nB: In the Gradient approach, you state that you do not update the model, but only the final hidden states (h) of LSTM. My question is how can you differentiate between per-token hidden states of dimensionality (hidden_state_dim, prediction_length) and rationales of (prediction_length, )? Am I missing something here?", "paper_topic_and_main_contributions": "This paper introduces the InterFair framework for iteratively incorporating user feedback to mitigate the model's reliance on biased features of the input. InterFair proposes first to parse the open-ended user feedback to a form of per-token bias weights with a large language model (GPT-3) used in a few-shot setup. Second, this feedback is used to update the model rationales through differentiation of the model's generated rationales. Results show that users' interactions with model rationales through InterFair can improve the comprehensiveness and sufficiency of models' rationales, without compromising prediction performance on named entity recognition tasks.", "excitement": 4, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* I do not understand the description of \"compositional split\" on L238: \"compositional split 238 where the gold parse has three or more contiguous 239 token sequences\". Subsequently, I can not relate to the conclusion on L240: \"the compositional split is harder than the IID due to its complexity\".\n* I believe the part on L084 would deserve further justification: \"He et al. (2022) argue that weighing less on high bias and high-task important tokens and promoting their low-bias replacements can simultaneously address both of the failure modes.\". How can weighting less on high-task important tokens address model failures?\n* L202: It seems to be arguable that hidden states (h) are not a part of the model, since you also state that they are used in prediction.\n* L256: perhaps missing \"(1)\"?"}}]

Author rebuttal:

Meta-review: The authors propose InterFair, a novel human-in-the-loop pipeline approach to model debiasing, where free-text feedback from humans is used to update importance scores for model rationales. The free-text feedback is parsed by a text-davinci-003 model and transformed into numeric scores, which are then used to update the bias importance of each individual token for a frozen model on the BiosBias classification task. Following the claim that it is challenging to fairly eliminate biases algorithmically, the authors evaluate their scenario at inference-time for two experiments, where users interact with test samples and inputs, showing that both scenarios further mitigate bias from explanations.

The reviewers agree that the paper is clearly motivated (beK8, KjGw, mchw), well written and inuitively structured (mchw). Reviewers note that the proposed methodology is novel (beK8) and has a lot of potential (mchw) — and that the presented experimental results support the authors’ approach (beK8, Kj4w).

The main criticisms presented in the reviews are lack of details regarding the human study (beK8) and more clarity with respect to the gap to previous work (beK8, Kj4w) as well as the experimental setup (mchw). Furthermore, the choice of a LSTM-based model as part of the pipeline and not a Transformer-based variant limits the scope of the work (Kj4w). Lastly, a question was raised with respect to the choice of natural language feedback instead of using plain token editing (beK8).

The authors responded to the questions raised in the reviews, committing to add clarifying details. The authors also presented additional results with a BERT-based model, motivated their choice of free-text input through a user study and clarified the gap with respect to previous work.

Upon reading the paper, reviews, as well as the discussion, it is my opinion that the paper is for the most part well written and easy to follow. Furthermore, it tackles a relevant problem in a novel manner. When reading the paper, my main concerns come from a significant reliance on work of He et al, where the reader is assumed to be familiar with the setup — something that can be improved in the manuscript. Another concern was the limited scope of experiments, which is addressed by the authors in the discussion period by adding experimental results on a Transformer-based model.
Decision: Accept-Main