cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: FairSwiRL: fair semi-supervised classification with representation learning
Authors: Shuyi Yang; Mattia Cerrato; Dino Ienco; Ruggero G. Pensa; Roberto Esposito
Affiliations: Intesa Sanpaolo (Italy); Johannes Gutenberg University Mainz; Institut National de Recherche pour l'Agriculture, l'Alimentation et l'Environnement; University of Turin; University of Turin
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Abstract Semi-supervised learning has shown its potential in many real-world applications where only few labeled examples are available. However, when some fairness constraints need to be satisfied, semi-supervised classification models often struggle as they are required to cope with the lack of sufficient information for predicting the target variable while forgetting its relationships with any sensitive and potentially discriminatory attribute. To address this issue, we propose a fair semi-supervised representation learning architecture that leads to fair and accurate classification results even in very challenging scenarios with few labeled (but biased) instances. We show experimentally that our model can be easily adopted in very general settings, as the learned representations may be employed to train any supervised classifier. Moreover, when applied to several synthetic and real-world datasets, our method is competitive with state-of-the-art fair semi-supervised approaches.
Title: Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization
Authors: Cennet Oğuz; Pascal Denis; Emmanuel Vincent; Simon Ostermann; Josef van Genabith
Affiliations: German Research Centre for Artificial Intelligence; Centre National de la Recherche Scientifique; Centre National de la Recherche Scientifique; German Research Centre for Artificial Intelligence; German Research Centre for Artificial Intelligence
Year: 2023
Domain: LLMs & NLP

Abstract: In multimodal understanding tasks, visual and linguistic ambiguities can arise. Visual ambiguity can occur when visual objects require a model to ground a referring expression in a video without strong supervision, while linguistic ambiguity can occur from changes in entities in action flows. As an example from the cooking domain, "oil" mixed with "salt" and "pepper" could later be referred to as a "mixture". Without a clear visual-linguistic alignment, we cannot know which among several objects shown is referred to by the language expression "mixture", and without resolved antecedents, we cannot pinpoint what the mixture is. We define this chicken-and-egg problem as visual-linguistic ambiguity. In this paper, we present Find2Find, a joint anaphora resolution and object localization dataset targeting the problem of visual-linguistic ambiguity, consisting of 500 anaphora-annotated recipes with corresponding videos. We present experimental results of a novel end-to-end joint multitask learning framework for Find2Find that fuses visual and textual information and shows improvements both for anaphora resolution and object localization as compared to a strong single-task baseline.
Title: Framework for bias evaluation in large language models in healthcare settings
Authors: Tara Templin; Sophia Fort; Prasanna Padmanabham; Pratyush Seshadri; Ram Rimal; Junier B. Oliva; Kristin Hassmiller Lich; Sean Sylvia; Nasa Sinnott-Armstrong
Affiliations: University of North Carolina at Chapel Hill; Fred Hutch Cancer Center; University of North Carolina at Chapel Hill; University of North Carolina at Chapel Hill; University of North Carolina at Chapel Hill; University of North Carolina at Chapel Hill; University of North Carolina at Chapel Hill
Year: 2025
Domain: LLMs & NLP

Abstract: A critical gap in the adoption of large language models for AI-assisted clinical decisions is the lack of a standardized audit framework to evaluate models for accuracy and bias. Our framework introduces a five-step framework that guides practitioners through stakeholder engagement, model calibration to specific patient populations, and rigorous testing through clinically relevant scenarios. We provide open-access tools for stakeholder engagement and an example of an audit. As the regulation of models becomes more critical, we believe adoption of an audit framework that tests model outputs, rather than regulating specific hyperparameters or inputs, will encourage the responsible use of AI in clinical settings.
Title: GATE X-E : A Challenge Set for Gender-Fair Translations from Weakly-Gendered Languages
Authors: Spencer Rarrick; Ranjita Naik; Sundar Poudel; Vishal Chowdhary
Affiliations: Microsoft; Microsoft; Microsoft; Microsoft
Year: 2024
Domain: LLMs & NLP

Abstract: Recommendation systems founded on graph neural networks (GNN) have been extensively employed because of their exceptional recommendation efficiency. Nevertheless, numerous recommendation biases also crop up, We have observed that delicate details such as gender and age are frequently implicitly apprehended by recommendation systems, culminating in unfair recommendations, and the associated algorithms of GNN will magnify this bias. To tackle these difficulties, this paper puts forth a method of introducing the notion of causal fairness into the issue of fairness in GNN-based recommendation systems, to accomplish counterfactual fairness of user-sensitive information and thereby attain unbiased recommendations. Specifically, given a GNN-based recommendation system model, which is implemented in our devised fairness framework, chiefly obtaining equitable effects through two facets: (1) attaining user embedding fairness through the counterfactual fairness technique; (2) mitigating the prejudiced impact caused by the GNN algorithm using the proposed central association subgraph method. The amalgamation of these two facets ultimately delivers unbiased recommendations. The effectiveness and sophistication of our proposed method for mitigating partiality problems in GNN recommendation systems from a causal perspective (MGRC) have been proven via experiments on four real-world datasets.
Title: General then Personal: Decoupling and Pre-training for Personalized Headline Generation
Authors: Yun-Zhu Song; Yi-Syuan Chen; Lu Wang; Hong-Han Shuai
Affiliations: National Yang Ming Chiao Tung University; National Yang Ming Chiao Tung University; University of Michigan; National Yang Ming Chiao Tung University
Year: 2023
Domain: LLMs & NLP

Abstract: Abstract Personalized Headline Generation aims to generate unique headlines tailored to users’ browsing history. In this task, understanding user preferences from click history and incorporating them into headline generation pose challenges. Existing approaches typically rely on predefined styles as control codes, but personal style lacks explicit definition or enumeration, making it difficult to leverage traditional techniques. To tackle these challenges, we propose General Then Personal (GTP), a novel framework comprising user modeling, headline generation, and customization. We train the framework using tailored designs that emphasize two central ideas: (a) task decoupling and (b) model pre-training. With the decoupling mechanism separating the task into generation and customization, two mechanisms, i.e., information self-boosting and mask user modeling, are further introduced to facilitate the training and text control. Additionally, we introduce a new evaluation metric to address existing limitations. Extensive experiments conducted on the PENS dataset, considering both zero-shot and few-shot scenarios, demonstrate that GTP outperforms state-of-the-art methods. Furthermore, ablation studies and analysis emphasize the significance of decoupling and pre-training. Finally, the human evaluation validates the effectiveness of our approaches.1
Title: Global Health Informatics: the state of research and lessons learned
Authors: Yuri Quintana; Theresa Cullen; John H. Holmes; Ashish Joshi; David Novillo-Ortiz; Siaw-Teng Liaw
Affiliations: Beth Israel Deaconess Medical Center; Pima County Health Department; University of Pennsylvania; University of Memphis; World Health Organization Regional Office for Europe; UNSW Sydney
Year: 2023
Domain: Health & Clinical AI

Abstract: The Journal of the American Medical Informatics Association (JAMIA) focus issue on Global Health Informatics (GHI) examines how health informatics can address global health issues. The selected papers discuss how informatics can improve health outcomes, especially in low- and middle-income countries (LMICs). The articles in this issue come from a broad range of researchers, practitioners, and policymakers, provide an overview of GHI initiatives and encourage researchers and practitioners to share ideas and best practices. GHI aims to harness digital health and informatics knowledge and tools to improve the care of individuals and populations in LMIC through achieving these 5 aims: effectiveness, efficiencies, patient satisfaction, provider well-being, and equity. A central consideration is the global north-south divide and the sociotechnical determinants of and impacts on the digital health maturity of LMIC. This issue seeks to document and understand how health informatics approaches have been undertaken in multiple LMIC scenarios and environments and how they can demonstrate the scientific impact of informatics-based best practices applied to COVID-19 and beyond.
Title: Group Fairness via Group Consensus
Authors: Eunice Chan; Zhining Liu; Ruizhong Qiu; Yuheng Zhang; Ross Maciejewski; Hanghang Tong
Affiliations: University of Illinois Urbana-Champaign; University of Illinois Urbana-Champaign; University of Illinois Urbana-Champaign; University of Illinois Urbana-Champaign; Arizona State University; University of Illinois Urbana-Champaign
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Ensuring equitable impact of machine learning models across different societal groups is of utmost importance for real-world machine learning applications. Prior research in fairness has predominantly focused on adjusting model outputs through pre-processing, in-processing, or post-processing techniques. These techniques focus on correcting bias in either the data or the model. However, we argue that the bias in the data and model should be addressed in conjunction. To achieve this, we propose an algorithm called GroupDebias to reduce unfairness in the data in a model-guided fashion, thereby enabling models to exhibit more equitable behavior. Even though it is model-aware, the core idea of GroupDebias is independent of the model architecture, making it a versatile and effective approach that can be broadly applied across various domains and model types. Our method focuses on systematically addressing biases present in the training data itself by adaptively dropping samples that increase the biases in the model. Theoretically, the proposed approach enjoys a guaranteed improvement in demographic parity at the expense of a bounded reduction in balanced accuracy. A comprehensive evaluation of the GroupDebias algorithm through extensive experiments on diverse datasets and machine learning models demonstrates that GroupDebias consistently and significantly outperforms existing fairness enhancement techniques, achieving a more substantial reduction in unfairness with minimal impact on model performance.
Title: GumbelSoft: Diversified Language Model Watermarking via the GumbelMax-trick
Authors: Jiayi Fu; Xuandong Zhao; Ruihan Yang; Yuansen Zhang; Jiangjie Chen; Yanghua Xiao
Affiliations: Fudan University; University of California, Santa Barbara; Fudan University; Fudan University; Fudan University; Fudan University
Year: 2024
Domain: LLMs & NLP

Abstract: Large language models (LLMs) excellently generate human-like text, but also raise concerns about misuse in fake news and academic dishonesty.Decoding-based watermark, particularly the GumbelMax-trick-based watermark (GM watermark), is a standout solution for safeguarding machine-generated texts due to its notable detectability.However, GM watermark encounters a major challenge with generation diversity, always yielding identical outputs for the same prompt, negatively impacting generation diversity and user experience.To overcome this limitation, we propose a new type of GM watermark, the Logits-Addition watermark, and its three variants, specifically designed to enhance diversity.Among these, the GumbelSoft watermark (a softmax variant of the Logits-Addition watermark) demonstrates superior performance in high diversity settings, with its AUROC score outperforming those of the two alternative variants by 0.1 to 0.3 and surpassing other decoding-based watermarking methods by a minimum of 0.1.
Title: Impact on bias mitigation algorithms to variations in inferred sensitive attribute uncertainty
Authors: Yanchen Wang; Lisa Singh
Affiliations: Georgetown University; Georgetown University
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: Concerns about the trustworthiness, fairness, and privacy of AI systems are growing, and strategies for mitigating these concerns are still in their infancy. One approach to improve trustworthiness and fairness in AI systems is to use bias mitigation algorithms. However, most bias mitigation algorithms require data sets that contain sensitive attribute values to assess the fairness of the algorithm. A growing number of real world data sets do not make sensitive attribute information readily available to researchers. One solution is to infer the missing sensitive attribute information and apply an existing bias mitigation algorithm using this inferred knowledge. While researchers are beginning to explore this question, it is still unclear how robust existing bias mitigation algorithms are to different levels of inference accuracy. This paper explores this question by investigating the impact of different levels of accuracy of the inferred sensitive attribute on the performance of different bias mitigation strategies. We generate variation in sensitive attribute accuracy using both simulation and construction of neural models for the inference task. We then assess the quality of six bias mitigation algorithms that are deployed across different parts of our learning life cycle: pre-processing, in-processing, and post-processing. We find that the disparate impact remover is the least sensitive bias mitigation strategy and that if we apply the bias mitigation algorithms using an inferred sensitive attribute with reasonable accuracy, the fairness scores are higher than the best standard model and the balanced accuracy is similar to that of the standard model. These findings open the door for improving fairness of black box AI systems using some bias mitigation strategies.
Title: Incorporating Attribution Importance for Improving Faithfulness Metrics
Authors: Zhixue Zhao; Νικόλαος Αλέτρας
Affiliations: University of Sheffield; University of Sheffield
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Feature attribution methods (FAs) are popular approaches for providing insights into the model reasoning process of making predictions. The more faithful a FA is, the more accurately it reflects which parts of the input are more important for the prediction. Widely used faithfulness metrics, such as sufficiency and comprehensiveness use a hard erasure criterion, i.e. entirely removing or retaining the top most important tokens ranked by a given FA and observing the changes in predictive likelihood. However, this hard criterion ignores the importance of each individual token, treating them all equally for computing sufficiency and comprehensiveness. In this paper, we propose a simple yet effective soft erasure criterion. Instead of entirely removing or retaining tokens from the input, we randomly mask parts of the token vector representations proportionately to their FA importance. Extensive experiments across various natural language processing tasks and different FAs show that our soft-sufficiency and soft-comprehensiveness metrics consistently prefer more faithful explanations compared to hard sufficiency and comprehensiveness.
Title: Intersectional Stereotypes in Large Language Models: Dataset and Analysis
Authors: Weicheng Ma; Brian Chiang; Tong Wu; Lili Wang; Soroush Vosoughi
Affiliations: Dartmouth Hospital; Dartmouth College; Dartmouth Hospital; Dartmouth College; Dartmouth College
Year: 2023
Domain: LLMs & NLP

Abstract: Despite many stereotypes targeting intersectional demographic groups, prior studies on stereotypes within Large Language Models (LLMs) primarily focus on broader, individual categories. This research bridges this gap by introducing a novel dataset of intersectional stereotypes, curated with the assistance of the ChatGPT model and manually validated. Moreover, this paper offers a comprehensive analysis of intersectional stereotype propagation in three contemporary LLMs by leveraging this dataset. The findings underscore the urgency of focusing on intersectional biases in ongoing efforts to reduce stereotype prevalence in LLMs.
Title: Language or rating scales based classifications of emotions: computational analysis of language and alexithymia
Authors: Sverker Sikström; Miriam Nicolai; Josephine Ahrendt; Suvi Nevanlinna; Lotta Stille
Affiliations: Lund University; Lund University; Lund University; Lund University; Lund University
Year: 2024
Domain: LLMs & NLP

Abstract: Rating scales are the dominating tool for the quantitative assessment of mental health. They are often believed to have a higher validity than language-based responses, which are the natural way of communicating mental states. Furthermore, it is unclear how difficulties articulating emotions-alexithymia-affect the accuracy of language-based communication of emotions. We investigated whether narratives describing emotional states are more accurately classified by questions-based computational analysis of language (QCLA) compared to commonly used rating scales. Additionally, we examined how this is affected by alexithymia. In Phase 1, participants (N = 348) generated narratives describing events related to depression, anxiety, satisfaction, and harmony. In Phase 2, another set of participants summarized the emotions described in the narratives of Phase 1 in five descriptive words and rating scales (PHQ-9, GAD-7, SWLS, and HILS). The words were quantified with a natural language processing model (i.e., LSA) and classified with machine learning (i.e., multinomial regression). The results showed that the language-based responses can be more accurate in classifying the emotional states compared to the rating scales. The degree of alexithymia did not influence the correctness of classification based on words or rating scales, suggesting that QCLA is not sensitive to alexithymia. However, narratives generated by people with high alexithymia were more difficult to classify than those generated by people with low alexithymia. These results suggest that the assessment of mental health may be improved by language-based responses analyzed by computational methods compared to currently used rating scales.
Title: Large Language Models are not Fair Evaluators
Authors: Peiyi Wang; Lei Li; Liang Chen; Zefan Cai; Dawei Zhu; Binghuai Lin; Yunbo Cao; Lingpeng Kong; Qi Liu; Tianyu Liu; Zhifang Sui
Affiliations: Peking University; Peking University; Peking University; Peking University; Peking University; Tencent Cloud; Tencent Cloud; The University of Hong Kong; The University of Hong Kong; Tencent Cloud; Peking University
Year: 2024
Domain: LLMs & NLP

Abstract: Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, Zhifang Sui. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
Title: LLatrieval: LLM-Verified Retrieval for Verifiable Generation
Authors: Xiaonan Li; Changtai Zhu; Linyang Li; Zhangyue Yin; Tianxiang Sun; Xipeng Qiu
Affiliations: Fudan University; Fudan University; Fudan University; Fudan University; Fudan University; Fudan University
Year: 2024
Domain: LLMs & NLP

Abstract: Xiaonan Li, Changtai Zhu, Linyang Li, Zhangyue Yin, Tianxiang Sun, Xipeng Qiu. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.
Title: Patient-centricity in digital measure development: co-evolution of best practice and regulatory guidance
Authors: Suvekshya Aryal; Jennifer M. Blankenship; Shelby L. Bachman; Soohyun Hwang; Yaya Zhai; Jennifer C. Richards; Ieuan Clay; Kate Lyden
Affiliations: Ipsen (France)
Year: 2024
Domain: Health & Clinical AI

Abstract: Digital health technologies (DHTs) have the potential to modernize drug development and clinical trial operations by remotely, passively, and continuously collecting ecologically valid evidence that is meaningful to patients' lived experiences. Such evidence holds potential for all drug development stakeholders, including regulatory agencies, as it will help create a stronger evidentiary link between approval of new therapeutics and the ultimate aim of improving patient lives. However, only a very small number of novel digital measures have matured from exploratory usage into regulatory qualification or efficacy endpoints. This shows that despite the clear potential, actually gaining regulatory agreement that a new measure is both fit-for-purpose and delivers value remains a serious challenge. One of the key stumbling blocks for developers has been the requirement to demonstrate that a digital measure is meaningful to patients. This viewpoint aims to examine the co-evolution of regulatory guidance in the United States (U.S.) and best practice for integration of DHTs into the development of clinical outcome assessments. Contextualizing guidance on meaningfulness within the larger shift towards a patient-centric drug development approach, this paper reviews the U.S. Food and Drug Administration (FDA) guidance and existing literature surrounding the development of meaningful digital measures and patient engagement, including the recent examples of rejections by the FDA that further emphasize patient-centricity in digital measures. Finally, this paper highlights remaining hurdles and provides insights into the established frameworks for development and adoption of digital measures in clinical research.
Title: PEEB: Part-based Image Classifiers with an Explainable and Editable Language Bottleneck
Authors: Thang M. Pham; Peijie Chen; Tin Nguyen; Seunghyun Yoon; Trung Bui; Anh‐Tu Nguyen
Affiliations: Auburn University; Auburn University; Auburn University; Adobe Research; Adobe Research; Auburn University
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: CLIP-based classifiers rely on the prompt containing a {class name} that is known to the text encoder.Therefore, they perform poorly with new classes or the classes whose names rarely appear on the Internet (e.g., scientific names of birds).For fine-grained classification, we propose PEEB-an explainable and editable classifier to (1) express the class name into a set of text descriptors that describe the visual parts of the class; and (2) match the embeddings of the detected parts with their textual descriptors in each class to compute a logit score for classification.In a zero-shot setting where the class names are unknown, PEEB significantly outperforms CLIP, achieving a 10-fold increase in top-1 accuracy.Compared to part-based classifiers, PEEB not only achieves state-of-the-art (SOTA) accuracy in the supervised-learning setting-88.80% and 92.20% accuracy on CUB-200 and Dogs-120 , respectively-but also the first to enable users to edit the text descriptors to form a new classifier without any re-training.Compared to concept bottleneck models, PEEB is also the SOTA in both zero-shot and supervised learning settings.