cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Algorithmic individual fairness and healthcare: a scoping review
Authors: J. W. Anderson; Shyam Visweswaran
Affiliations: University of Pittsburgh; University of Pittsburgh
Year: 2024
Domain: Health & Clinical AI

Abstract: Objectives: Statistical and artificial intelligence algorithms are increasingly being developed for use in healthcare. These algorithms may reflect biases that magnify disparities in clinical care, and there is a growing need for understanding how algorithmic biases can be mitigated in pursuit of algorithmic fairness. We conducted a scoping review on algorithmic individual fairness (IF) to understand the current state of research in the metrics and methods developed to achieve IF and their applications in healthcare. Materials and Methods: We searched four databases: PubMed, ACM Digital Library, IEEE Xplore, and medRxiv for algorithmic IF metrics, algorithmic bias mitigation, and healthcare applications. Our search was restricted to articles published between January 2013 and November 2024. We identified 2498 articles through database searches and seven additional articles, of which 32 articles were included in the review. Data from the selected articles were extracted, and the findings were synthesized. Results: Based on the 32 articles in the review, we identified several themes, including philosophical underpinnings of fairness, IF metrics, mitigation methods for achieving IF, implications of achieving IF on group fairness and vice versa, and applications of IF in healthcare. Discussion: We find that research of IF is still in their early stages, particularly in healthcare, as evidenced by the limited number of relevant articles published between 2013 and 2024. While healthcare applications of IF remain sparse, growth has been steady in number of publications since 2012. The limitations of group fairness further emphasize the need for alternative approaches like IF. However, IF itself is not without challenges, including subjective definitions of similarity and potential bias encoding from data-driven methods. These findings, coupled with the limitations of the review process, underscore the need for more comprehensive research on the evolution of IF metrics and definitions to advance this promising field. Conclusion: While significant work has been done on algorithmic IF in recent years, the definition, use, and study of IF remain in their infancy, especially in healthcare. Future research is needed to comprehensively apply and evaluate IF in healthcare.
Title: The Impact of Generative Artificial Intelligence on Socioeconomic Inequalities and Policy Making
Authors: Valerio Capraro; Austin Lentsch; Daron Acemoğlu; Selin Akgün; Aisel Akhmedova; Ennio Bilancini; Jean‐François Bonnefon; Pablo Brañas‐Garza; Luigi Butera; Karen M. Douglas; Jim A. C. Everett; Gerd Gigerenzer; Christine Greenhow; Daniel A. Hashimoto; Julianne Holt‐Lunstad; Jolanda Jetten; Simon Johnson; Chiara Longoni; Pete Lunn; Simone Natale; Iyad Rahwan; Neil Selwyn; Vivek Singh; Siddharth Suri; Jennifer Sutcliffe; Joe Tomlinson; Sander van der Linden; Paul A. M. Van Lange; Friederike Wall; Jay Joseph Van Bavel; Riccardo Viale
Affiliations: University of Milano-Bicocca; Massachusetts Institute of Technology; National Bureau of Economic Research; Michigan State University; Michigan State University; University of Modena and Reggio Emilia; Toulouse School of Economics; Universidad Loyola Andalucía; Copenhagen Business School; University of Kent; University of Oxford; Max Planck Society; Michigan State University; University of Pennsylvania; Brigham Young University; The University of Queensland; National Bureau of Economic Research; Bocconi University; Trinity College Dublin; University of Turin; Max Planck Society; Australian Regenerative Medicine Institute; University of Pennsylvania; Microsoft (United States); Michigan State University; New York Law School; University of Cambridge; University of Klagenfurt; New York University; Center for Interdisciplinary Studies
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Group recommendation has become an important task in information systems, which makes recommendations towards a group rather than an individual. This is due to the prevalence of group activities in people’s daily life. However, most group recommendation researchers focus on utility performance isolatedly, ignoring unfairness caused by popularity bias. In this paper, we propose a Mitigating Popularity Bias for Multi-View Group Recommendation (MPBGR). Specifically, we first detect the popularity bias using a discriminator and then enhance fairness on the item side. To avoid the utility decrease raised by enhancing fairness, we further consider the collaborative relationships between users and items across different groups as well as group-to-group interactions. We conduct extensive experiments and the experimental results show that the proposed model outperforms the state-of-the-art approaches in balancing utility and fairness.
Title: Neural Retrievers are Biased Towards LLM-Generated Content
Authors: Sunhao Dai; Yuqi Zhou; Liang Pang; Weihao Liu; Xiaolin Hu; Yong Liu; Xiao Zhang; Gang Wang; Jun Xu
Affiliations: Renmin University of China; Renmin University of China; Chinese Academy of Sciences; Renmin University of China; Renmin University of China; Renmin University of China; Renmin University of China; Huawei Technologies (China); Renmin University of China
Year: 2024
Domain: LLMs & NLP

Abstract: Recently, the emergence of large language models (LLMs) has revolutionized the paradigm of information retrieval (IR) applications, especially in web search, by generating vast amounts of human-like texts on the Internet. As a result, IR systems in the LLM era are facing a new challenge: the indexed documents are now not only written by human beings but also automatically generated by the LLMs. How these LLM-generated documents influence the IR systems is a pressing and still unexplored question. In this work, we conduct a quantitative evaluation of IR models in scenarios where both human-written and LLM-generated texts are involved. Surprisingly, our findings indicate that neural retrieval models tend to rank LLM-generated documents higher. We refer to this category of biases in neural retrievers towards the LLM-generated content as the source bias. Moreover, we discover that this bias is not confined to the first-stage neural retrievers, but extends to the second-stage neural re-rankers. Then, in-depth analyses from the perspective of text compression indicate that LLM-generated texts exhibit more focused semantics with less noise, making it easier for neural retrieval models to semantic match. To mitigate the source bias, we also propose a plug-and-play debiased constraint for the optimization objective, and experimental results show its effectiveness. Finally, we discuss the potential severe concerns stemming from the observed source bias and hope our findings can serve as a critical wake-up call to the IR community and beyond. To facilitate future explorations of IR in the LLM era, the constructed two new benchmarks are available at https://github.com/KID-22/Source-Bias.
Title: Invisible Relevance Bias: Text-Image Retrieval Models Prefer AI-Generated Images
Authors: Shicheng Xu; Danyang Hou; Liang Pang; Jingcheng Deng; Jun Xu; Huawei Shen; Xueqi Cheng
Affiliations: Institute of Computing Technology; Institute of Computing Technology; Chinese Academy of Sciences; Institute of Computing Technology; Renmin University of China; Chinese Academy of Sciences; Chinese Academy of Sciences
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: With the application of generation models, internet is increasingly inundated with AI-generated content (AIGC), causing both real and AI-generated content indexed in corpus for search. This paper explores the impact of AI-generated images on text-image search in this scenario. Firstly, we construct a benchmark consisting of both real and AI-generated images for this study. In this benchmark, AI-generated images possess visual semantics sufficiently similar to real images. Experiments on this benchmark reveal that text-image retrieval models tend to rank the AI-generated images higher than the real images, even though the AI-generated images do not exhibit more visually relevant semantics to the queries than real images. We call this bias as invisible relevance bias. This bias is detected across retrieval models with different training data and architectures. Further exploration reveals that mixing AI-generated images into the training data of retrieval models exacerbates the invisible relevance bias. These problems cause a vicious cycle in which AI-generated images have a higher chance of exposing from massive data, which makes them more likely to be mixed into the training of retrieval models and such training makes the invisible relevance bias more and more serious. To mitigate this bias and elucidate the potential causes of the bias, firstly, we propose an effective method to alleviate this bias. Subsequently, we apply our proposed debiasing method to retroactively identify the causes of this bias, revealing that the AI-generated images induce the image encoder to embed additional information into their representation. This information makes the retriever estimate a higher relevance score. We conduct experiments to support this assertion.
Title: The ethics of ChatGPT in medicine and healthcare: a systematic review on Large Language Models (LLMs)
Authors: Joschka Haltaufderheide; Robert Ranisch
Affiliations: University of Potsdam; University of Applied Sciences Potsdam
Year: 2024
Domain: Health & Clinical AI

Abstract: With the introduction of ChatGPT, Large Language Models (LLMs) have received enormous attention in healthcare. Despite potential benefits, researchers have underscored various ethical implications. While individual instances have garnered attention, a systematic and comprehensive overview of practical applications currently researched and ethical issues connected to them is lacking. Against this background, this work maps the ethical landscape surrounding the current deployment of LLMs in medicine and healthcare through a systematic review. Electronic databases and preprint servers were queried using a comprehensive search strategy which generated 796 records. Studies were screened and extracted following a modified rapid review approach. Methodological quality was assessed using a hybrid approach. For 53 records, a meta-aggregative synthesis was performed. Four general fields of applications emerged showcasing a dynamic exploration phase. Advantages of using LLMs are attributed to their capacity in data analysis, information provisioning, support in decision-making or mitigating information loss and enhancing information accessibility. However, our study also identifies recurrent ethical concerns connected to fairness, bias, non-maleficence, transparency, and privacy. A distinctive concern is the tendency to produce harmful or convincing but inaccurate content. Calls for ethical guidance and human oversight are recurrent. We suggest that the ethical guidance debate should be reframed to focus on defining what constitutes acceptable human oversight across the spectrum of applications. This involves considering the diversity of settings, varying potentials for harm, and different acceptable thresholds for performance and certainty in healthcare. Additionally, critical inquiry is needed to evaluate the necessity and justification of LLMs' current experimental use.
Title: Dynamic feedback loops in recommender systems: Analyzing fairness, popularity bias, and user group disparities
Authors: Yildiz Zoralioglu; Emre Yalcin
Affiliations: Sivas Cumhuriyet Üniversitesi; Sivas Cumhuriyet Üniversitesi
Year: 2026
Domain: Recommender Systems

Abstract: Abstract Ensuring equitable treatment of different user groups in recommender systems is a key challenge, and the issue of fairness has been widely explored in the literature. However, understanding fairness within a robust feedback loop, as it occurs in real-world settings, remains elusive. This study examines the interplay between popularity bias, calibration, accuracy, and beyond-accuracy performance of recommender systems using a novel dynamic feedback loop framework. The framework models iterative interactions between recommendation algorithms and user profiles, enabling the analysis of calibration, accuracy, and beyond-accuracy measures across user groups with varying preferences on popular items, i.e., Popular- , Diverse- , and Niche-focused . Empirical evaluations conducted on two benchmark datasets using three collaborative filtering algorithms reveal distinct disparities in how feedback loops affect different user groups. Niche-focused users, despite being the most active and information-rich, experience the steepest deterioration in system alignment over time, losing much of their initial calibration, long-tail exposure, and diversity advantages, along with proportional declines in accuracy. These results show that feedback dynamics progressively misalign the system with its most valuable users, making them the most disadvantaged over time. Popular-focused users remain most aligned with algorithmic tendencies, achieving steady accuracy gains but remaining confined to narrow, popularity-driven content with little to no long-tail exposure. Meanwhile, Diverse-focused users, initially balanced between popular and niche preferences, undergo gradual calibration drift and consistent reductions in diversity and long-tail representation, gradually converging toward recommendation patterns similar to Popular-focused users. Overall, the results demonstrate that feedback loops magnify structural inequalities, reinforcing popularity bias while reducing diversity and personalization across all user groups.
Title: Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews
Authors: Hyungyu Shin; Jingyu Tang; Yoonjoo Lee; Nayoung Kim; Hyunseung Lim; Ji Yong Cho; Hwajung Hong; Moontae Lee; Ju-ho Kim
Affiliations: KAIST; Huazhong University of Science and Technology; KAIST; KAIST; KAIST; LG AI Research; KAIST; LG AI Research; KAIST
Year: 2025
Domain: LLMs & NLP

Abstract: Peer review underpins scientific progress, but it is increasingly strained by reviewer shortages and growing workloads. Large Language Models (LLMs) can automatically draft reviews now, but determining whether LLM-generated reviews are trustworthy requires systematic evaluation. Researchers have evaluated LLM reviews at either surface-level (e.g., BLEU and ROUGE) or content-level (e.g., specificity and factual accuracy). Yet it remains uncertain whether LLM-generated reviews attend to the same critical facets that human experts weigh—the strengths and weaknesses that ultimately drive an accept-or-reject decision. We introduce a focus-level evaluation framework that operationalizes the focus as a normalized distribution of attention across predefined facets in paper reviews. Based on the framework, we developed an automatic focus-level evaluation pipeline based on two sets of facets: target (e.g., problem, method, and experiment) and aspect (e.g., validity, clarity, and novelty), leveraging 676 paper reviews from OpenReview that consists of 3,657 strengths and weaknesses identified from human experts. The comparison of focus distributions between LLMs and human experts showed that the off-the-shelf LLMs consistently have a more biased focus towards examining technical validity while significantly overlooking novelty assessment when criticizing papers.Dataset: https://figshare.com/s/d5adf26c802527dd0f62
Title: Measuring the Fairness Gap Between Retrieval and Generation in RAG Systems using a Cognitive Complexity Framework
Authors: Sandeep Avula; Chaehong Lee; Rongting Zhang; Vanessa Murdock
Affiliations: Amazon (United States); Amazon (United States); Amazon (United States); Amazon (United States)
Year: 2025
Domain: LLMs & NLP

Abstract: In this paper, we investigate the problem of quantifying fairness in Retrieval-Augmented Generation (RAG) systems, particularly for complex cognitive tasks that go beyond factual question-answering. While RAG systems have demonstrated effectiveness in information extraction tasks, their fairness implications for cognitively complex tasks - including ideation, content creation, and analytical reasoning - remain under-explored. We propose a novel evaluation framework that extends IR fairness metrics by incorporating centrality-based measures to account for influence of retrieved documents on generated output beyond ranking. Our framework evaluates RAG systems across various cognitive dimensions using two ranking approaches: lexical (BM25) and dense (BGE), and language models of varying sizes. Our findings provide insights into: (1) the propagation of fairness disparities from retrieval to generation phases, and (2) the variation in system performance across different cognitive dimensions.
Title: Popularity Bias in Recommender Systems: The Search for Fairness in the Long Tail
Authors: Filippo Carnovalini; Antonio Rodà; Geraínt A. Wiggins
Affiliations: Vrije Universiteit Brussel; University of Padua; Vrije Universiteit Brussel
Year: 2025
Domain: Recommender Systems

Abstract: The importance of recommender systems has grown in recent years, as these systems are becoming one of the primary ways in which we access content on the Internet. Along with their use, concerns about the fairness of the recommendations they propose have rightfully risen. Recommender systems are known to be affected by popularity bias, the disproportionate preference towards popular items. While this bias stems from human tendencies, algorithms used in recommender systems can amplify it, resulting in unfair treatment of end-users and/or content creators. This article proposes a narrative review of the relevant literature to characterize and understand this phenomenon, both in human and algorithmic terms. The analysis of the literature highlighted the main themes and underscored the need for a multi-disciplinary approach that examines the interplay between human cognition, algorithms, and socio-economic factors. In particular, the article discusses how the overall fairness of recommender systems is impacted by popularity bias. We then describe the approaches that have been used to mitigate the harmful effects of this bias and discuss their effectiveness in addressing the issue, finding that some of the current approaches fail to face the problem in its entirety. Finally, we identify some open problems and research opportunities to help the advancement of research in the fairness of recommender systems.
Title: Toward Human-Like Evaluation for Natural Language Generation with Error Analysis
Authors: Qingyu Lu; Liang Ding; Liping Xie; Kanjian Zhang; Derek F. Wong; Dacheng Tao
Affiliations: University of Macau; University of Macau; Southeast University; Southeast University; University of Sydney
Year: 2023
Domain: LLMs & NLP

Abstract: The pretrained language model (PLM) based metrics have been successfully used in evaluating language generation tasks. Recent studies of the human evaluation community show that considering both major errors (e.g. mistranslated tokens) and minor errors (e.g. imperfections in fluency) can produce high-quality judgments. This inspires us to approach the final goal of the automatic metrics (human-like evaluations) by fine-grained error analysis. In this paper, we argue that the ability to estimate sentence confidence is the tip of the iceberg for PLM-based metrics. And it can be used to refine the generated sentence toward higher confidence and more reference-grounded, where the costs of refining and approaching reference are used to determine the major and minor errors, respectively.To this end, we take BARTScore as the testbed and present an innovative solution to marry the unexploited sentence refining capacity of BARTScore and human-like error analysis, where the final score consists of both the evaluations of major and minor errors. Experiments show that our solution consistently and significantly improves BARTScore, and outperforms top-scoring metrics in 19/25 test settings. Analyses demonstrate our method robustly and efficiently approaches human-like evaluations, enjoying better interpretability. Our code and scripts will be publicly released in https://github.com/Coldmist-Lu/ErrorAnalysis_NLGEvaluation.
Title: FairVLM: Enhancing Fairness and Prompt Sensitivity in Vision Language Models for Medical Image Segmentation
Authors: Md Motiur Rahman; Saeka Rahman; Smriti Bhatt; Miad Faezipour
Affiliations: Purdue University West Lafayette; Purdue University West Lafayette; Purdue University West Lafayette; Purdue University West Lafayette
Year: 2026
Domain: Health & Clinical AI

Abstract: Vision-language models (VLMs) have demonstrated substantial promise in medical image segmentation by utilizing radiology reports as prompts to segment regions of interest. However, VLM deployment in clinical settings is challenged by two intertwined issues: i) demographic bias, where performance varies across demographic groups, and ii) prompt sensitivity, where semantically similar prompts yield inconsistent outputs. These challenges are interconnected; demographic underrepresentation can worsen a model’s sensitivity to prompts, and prompt instability can more heavily affect certain demographic groups. In this study, we present FairVLM, a unified framework that addresses both demographic disparity and prompt sensitivity in VLMs. FairVLM integrates three key components: (1) Semantic-Retaining Counterfactual Prompting (SRCP), which generates clinically consistent and diverse prompt variations via large language models; (2) Demographic-Aware Feature Normalization (DAFN), a lightweight module that mitigates latent representation bias across demographic groups; and (3) a Fairness-Calibrated Loss (FCL) that explicitly penalizes performance disparities while encouraging prompt consistency. Extensive evaluations on the Harvard-FairSeg dataset show that FairVLM significantly improves equity-scaled segmentation. It also reduces demographic disparity (DI) by over 65% and relative performance gap (RPG) by over 60%, while maintaining or boosting overall accuracy. FairVLM is robust to prompt changes, with less than 0.5% performance drop across varied prompts, and also generalizes well on unseen datasets. These findings present FairVLM as a new state-of-the-art, robust, and adaptable framework for a fair, prompt-invariant vision-language model. Code and data are available at https://github.com/Rahman-Motiur/FairVLM.
Title: Understanding the Effect of Model Compression on Social Bias in Large Language Models
Authors: Gustavo Gonçalves; Emma Strubell
Affiliations: Carnegie Mellon University; Carnegie Mellon University
Year: 2023
Domain: LLMs & NLP

Abstract: Large Language Models (LLMs) trained with self-supervision on vast corpora of web text fit to the social biases of that text. Without intervention, these social biases persist in the model's predictions in downstream tasks, leading to representational harm. Many strategies have been proposed to mitigate the effects of inappropriate social biases learned during pretraining. Simultaneously, methods for model compression have become increasingly popular to reduce the computational burden of LLMs. Despite the popularity and need for both approaches, little work has been done to explore the interplay between these two. We perform a carefully controlled study of the impact of model compression via quantization and knowledge distillation on measures of social bias in LLMs. Longer pretraining and larger models led to higher social bias, and quantization showed a regularizer effect with its best trade-off around 20% of the original pretraining time.