Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Prompts
Authors: Xuyang Wu, Yuan Wang, Hsin-Tai Wu, ZHIQIANG TAO, Yi Fang


Reviewer paper summaries: This paper presents a novel evaluation framework for assessing fairness in large vision-language models (LVLMs) across demographic attributes like gender, skin tone, and age. The authors utilize the FACET benchmark dataset and design custom prompts to test various open-source and closed-source LVLMs in zero-shot settings. Their methodology includes direct question and single-choice question prompts, focusing on person-related classifications. The results reveal persistent fairness issues across all tested models, with variations in performance depending on prompt type and demographic attribute.
Summary of strengths: - The research topic is important as bias in LLMs can potentially influence users.
- The evaluation models used are exhaustive and provide significant insights.
Summary of weaknesses: - Limited Novelty: The overall evaluation pipeline uses prompting to assess bias in LLMs. However, it focuses solely on prompt design, lacking sufficient innovation for debiasing tasks.
 - Shallow Analysis: What is the fundamental cause of gender bias? Is it due to the pretraining corpora or the training algorithm? What is the expected distribution for gender? Is a 50% male vs. 50% female ratio appropriate in all scenarios?
 - Dataset Choice: I encourage the authors to include more datasets to provide better and more generalizable insights.
Comments, suggestions and typos: N/A
Overall assessment scores: 2