Say My Name: a Model's Bias Discovery Framework

Authors: Massimiliano Ciranni, Luca Molinaro, Carlo Alberto Barbano, Attilio Fiandrotti, Vito Paolo Pastore, Vittorio Murino, Enzo Tartaglione
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: bias discovery, unsupervised debiasing

Abstract:
In the last few years, due to the broad applicability of deep learning to downstream tasks and end-to-end training capabilities, increasingly more concerns about potential biases to specific, non-representative patterns have been raised.
Many works focusing on unsupervised debiasing usually leverage the tendency of deep models to learn "easier'' samples, for example by clustering the latent space to obtain bias pseudo-labels. However, the interpretation of such pseudo-labels is not trivial, especially for a non-expert end user, as it does not provide semantic information about the bias features.
To address this issue, we introduce "Say My Name'' (SaMyNa), the first tool to identify biases within deep models semantically. Unlike existing methods, our approach focuses on biases learned by the model. Our text-based pipeline enhances explainability and supports debiasing efforts: applicable during either training or post-hoc validation, our method can disentangle task-related information and proposes itself as a tool to analyze biases. Evaluation on traditional benchmarks demonstrates its effectiveness in detecting biases and even disclaiming them, showcasing its broad applicability for model diagnosis.

Decision: reject
Reviewer scores: 6, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730648364066

thread_type:
official_review

<thread_object>
creator: reviewer_mtjk

<content>
summary:
The authors introduce the "Say My Name" pipeline, a method designed to identify and interpret biases learned by image classification models. The pipeline consists of five main steps:

1. Selection of Representative Subset: Selecting a subset of images that are representative of the biases learned by the model.
2. Captioning with Vision-Language Model: Using a vision-language model (VLM) to generate captions for the selected images.
3. Keyword Extraction: Selecting keywords that are common across captions within the same class.
4. Embedding Computation: Computing embeddings of the text descriptions for each class.
5. Keyword Ranking: Comparing the results from steps 3 and 4 to rank the top keywords associated with each class.

The authors apply their method to several datasets, including Waterbirds, CelebA, BAR, and ImageNet-A. They demonstrate that their approach can effectively identify meaningful textual descriptions of biases present in the models.

strengths:
1. **Clarity and Readability:** The paper is well-written and easy to understand. The motivation behind the work is clearly explained.
2. **Methodological Breakdown:** In Section 3.2, the authors provide a detailed breakdown of the five-step pipeline, with clear links to each subsection. This organization enhances the readability and comprehension of the methodology.
3. **Practical Utility:** Interpreting biases in machine learning models is crucial. The proposed method appears straightforward to implement and could be readily applied to various image classification tasks.

weaknesses:
1. **Lack of Novelty in Bias Identification:** The method for bias identification seems to be a straightforward application of existing techniques for extracting textual descriptors from images. Previous work has already explored the connection between classification errors and biases. The proposed pipeline essentially relies on using a vision-language model to caption images and then summarizing or ranking features. It's unclear how the authors' approach offers a significant technical contribution beyond existing methods or how it compares to simpler approaches, such as directly using a VLM followed by summarization with a large language model (LLM).
2. **Interpretability and Quantification of Results:** The extracted rankings of keywords are difficult to quantify and interpret. For example, in Figure 4, the top features for different classes in the BAR dataset have scores ranging from 0.4 to 0.55, associating "climbing" with terms like "cliff," "rock," "rocks," and "steep." However, it's unclear whether these terms represent biases or causal features. There is a lack of empirical analysis to validate whether humans agree that these are indeed biases and how these scores should be interpreted or used in practice.
3. **Focus and Relevance of Bias Mitigation Study:** The inclusion of bias mitigation using the identified descriptors seems somewhat tangential to the main focus of the paper. Previous work has shown that, for datasets like CelebA and Waterbirds, extracting spurious attributes can improve performance. As such, the contribution in this area appears limited and may distract from the primary contributions of the paper.

questions:
Please see the weaknesses section for the main questions.

One missing reference (https://arxiv.org/abs/2204.13749), where the authors learned the unbiased-biased split directly from training.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730673316575

thread_type:
official_review

<thread_object>
creator: reviewer_pg7a

<content>
summary:
This paper tackles the identification of hidden dataset bias within training data, which prevents model from learning intrinsic features that generalizable across distributions. With existing text-based models, it extracts and ranks the bias-related keywords out of data, and leveraging this as the pseudo bias-labels for supervised learning to debias the model. It shows effectiveness in various dataset bias benchmarks in both synthetic and real-world setups.

strengths:
This paper is well-written and addresses the critical research question of identifying unknown dataset bias (spurious correlation) within training. This would essentially enhance the explainability and reliability of models in real-world applications especially for safety critical purposes.

weaknesses:
**1. Lack of novelty and effectiveness**: Several key ideas of this paper already exist in previous paper [1]. These include 1) sampling keywords using pretrained captioning model, and 2) identifying bias key words. Despite of subtle technical difference, e.g., detecting bias keywords via contrasting true and false positives (this work) or true positive and false negative (Kim et al. [1]), but in overall this paper does not provide any scientific novelty for the same goal beyond the existing papers. Such resemblance in technical details is reflected in highly limited improvements in debiasing compared to Kim et al., as proposed in Table 1. Therefore, it would be helpful to further elaborate the novel contribution of this paper against existing baselines.

**2. Potential risk in identifying biased models**: Section 3.1 proposes to identify the biased models by looking at how confidently misclassify the training data. However, models might result in being overfitted to relatively small number of bias-conflicting data in training data, resulting in potential bias to be NOT detected in iteration $t^*$. Therefore, it is deemed required to further validate the effectiveness of utilizing train data for detecting bias against oracle, i.e., using held-out validation data. 

[1] Kim et al., Discovering and Mitigating Visual Biases through Keyword Explanation, CVPR 2024

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707811818

thread_type:
official_review

<thread_object>
creator: reviewer_udxg

<content>
summary:
This paper outlines a five-step process for identifying spurious bias of a model in natural language keywords. The steps are as follows: (1) sample selection, (2) captioning, (3) keyword selection, (4) classification embedding, and (5) keyword ranking. The experimental results showcase several sample outputs from this process. Furthermore, the evaluation highlights the utility of the identified keywords as pseudo-labels for groups, which can be leveraged by debiasing methods.

strengths:
This paper addresses the critical issue of model bias discovery through an interesting approach that utilizes natural language keyword descriptions.

weaknesses:
The proposed method lacks novelty since it shares many components with existing literature. For example, the iteration selection based on misclassification confidence outlined in Section 3.1 is a variant of the approach described by Nahon et al. (2023), while the keyword extraction from natural language captions is similar to that found in Kim et al. (2024). Although these references are cited, the paper does not clearly delineate which aspects are novel, making it challenging to assess its originality.

The authors claim contributions related to a text-based pipeline and the disentanglement of domain relevance and the usefulness of the extracted keywords in debiasing. However, only the latter contribution is substantiated by experimental results. The validity and utility of the first two claimed contributions remain unclear.

Furthermore, there is room for improvement in the presentation of the paper. Here are some suggestions:
1. **Clarify Equations**: The presentation of the equations can be enhanced. For instance, in Equation 2, the denominator just represents the number of misclassifications, but uses complemented Dirac delta unnecessarily. Equation 3 calculates the average of the mean embeddings for both correctly classified and misclassified instances, but is presented as complicated double sum.
2. **Refine Citations**: A more judicious selection of parenthetical and in-text citations would enhance clarity and reduce unnecessary repetition throughout the text.

[1] Remi Nahon, Van-Tam Nguyen, and Enzo Tartaglione. Mining bias-target alignment from voronoi cells. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4946–4955, 2023.
[2] Younghyun Kim, Sangwoo Mo, Minkyu Kim, Kyungmin Lee, Jaeho Lee, and Jinwoo Shin. Discovering and mitigating visual biases through keyword explanation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11082–11092, 2024.

questions:
1. Which portion is the most important contribution in the proposed pipeline?
2. Why is the model with the most confident misclassification useful in bias discovery? If the final model and the selected model are different by a lot, how would this selected model be useful in the final model bias discovery or mitigation?
3. The keywords are derived from the samples classified as the target class by the studied model. Although these keywords are correlated to the model classification, wouldn't this be not enough to indicate causation?
4. Is there any compounded bias effect since many off-the-shelf models participate in the pipeline? For example, captioning model may focus on specific aspect of image or text embedding model may be sensitive to specific keywords.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708919621

thread_type:
official_review

<thread_object>
creator: reviewer_mn1g

<content>
summary:
The authors introduce a framework for detecting and captioning semantic biases of deep learning vision models. The authors propose a tool that identifies biases learned by models and assigns human-interpretable semantic labels to these biases for explainability and debiasing.
The method operates by sample subset selection, sample captioning via MLLM, keywords selections via text encoder, extracting learned class embedding, and keyword ranking.
The authors test the framework on popular benchmark datasets. The proposed method successfully identified biases. Also, this discovery can be used with bias mitigation methods, effectively debiasing models.

strengths:
- The paper tackles an important problem in machine learning which is bias and spurious correlations, and propose an effective tool to analyse these biases from the endpoint of humans.

weaknesses:
- Experimental analysis on bias discovery is lackluster. I think correlation analyses between the proposed method and human annotations are needed.

- The efficacy of the method could depend heavily on the model type and alignment of the MLLM or text encoder. I believe there should be an experimental analysis to show the robustness of the method on this matter.

questions:
- The proposed method does not use a validation set. How are the hyperparameters of its various components tuned?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732313078791

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their useful comments. We would like to give a general answer regarding the critical point shared by most of the reviewers, which is novelty in comparison with Bias2Text (B2T) (Kim et al., 2024). B2T pursues the same objective of our paper- extracting human-interpretable descriptions of potential biases affecting a deep image classification model. 

**B2T and our method are fundamentally different** in the way they extract keywords. Our method provides a much more robust estimation of biases in a variety of contexts, in which B2T fails to produce useful information or is not applicable at all.

A key difference is that our method *does not require a validation set* for bias discovery, in contrast to B2T. This represents a key aspect in unsupervised bias discovery and mitigation, as in a realistic scenario a validation set comprising conflicting samples is rarely available (like in BAR or BFFHQ). Besides, B2T requires captioning the entire validation set in order to extract relevant biases from conflicting samples. On the other hand, our SaMyNa does not suffer from such limitations, as 

- we only need the training set; 
- we can extract candidate few exemplars thanks to Bias Mining. 

With this, SaMyNa is much more efficient than captioning the entire validation set and allows for the use of larger and more accurate captioners such as LLaVa-34B.

**Why B2T cannot leverage better captioners while SaMyNa can.** The quality of extracted keywords, as noted by Rev. Mn1G, directly depends on the quality of the captioner. B2T is forced to employ smaller and quicker captioners such as ClipCap, which, however, provides less accurate captions when compared to models such as LLaVa-34B. The table below shows the time required for SaMyNa and B2T on the CelebA dataset using LLaVa-34B on a NVIDIA A40 equipped with 48 GB, tested on batch size 5:

| Method        | Captioning time (LLaVA 34B) |
|-|-|
| SaMyNa (K=1)  | 17 minutes |
| SaMyNa (K=5)  | 86 minutes |
| SaMyNa (K=10) | 3 hours|
| SaMyNa (K=25) | 7 hours                     |
| SaMyNa (K=50) | 14 hours                    |
| B2T           | 60 days                     |

As we can see from the results, even when employing K=50 (200 images) in SaMyNa, we still achieve considerably lower times than B2T, which requires captioning the entire validation set (19,867 images).

We also evaluated the quality of ClipCap captioner for the sake of comparison with B2T: we found that ClipCap results in notably worse results as it is not able to fully capture the appearance of the images. This prevents its usage on many relevant datasets. An example of this is found in Table 14 of B2T’s supplementary material, in which the full ranking of the keywords for both CelebA’s classes is shown, as can be seen, the results for the “not blond” class are suboptimal. 


**Why B2T requires a full validation set and SaMyNa does not**. To further demonstrate that B2T requires the full validation set, we compare the extracted keywords of B2T and SaMyNa with varying sample sizes. In the table below, we report the gender-related keywords extracted by both methods on CelebA:

| | K=1|K=5|K=10|K=25|K=50|
|-|-|-|-|-|-|
|Blond (B2T). Expected: male.| **woman (6th)** |  N/A | N/A | N/A | N/A |
|Blond (SaMyNa). Expected: woman.| N/A | **woman (6th)** | **woman (1st)** | **woman (5th)** | **woman (1st)** | 
|Not Blond (B2T) Expected: woman.| N/A | woman (5th) | woman (3rd) | woman (5th) | woman (4th) |
|Not Blond (SaMyNa). Expected: male.| **male (1st)** | **male (1st)** | **man (1st)** | **man (1st)** | **man (1st)** |

We highlight cases in which the relevant keyword was ranked higher than the other method. 
The results clearly show that our method, which leverages the bias mining step, is consistently more accurate than B2T in extracting the right keywords. Furthermore, keep in mind that *B2T keywords need to be inverted as reported in the original paper* (e.g. woman -> man), thus for K=1, B2T actually predicts the opposite bias. This is straightforward for a binary attribute such as CelebA’s gender, but not obvious when more than two biases are present in the training set.

**To summarize, the key differences between our methods SaMyNa and B2T are:**

- B2T requires a large enough validation set containing conflicting samples, while SaMyNa leverages bias mining to find candidate exemplars directly from the training set.
- SaMyNa is less computationally heavy than B2T and for this reason, we can employ state-of-the-art captioners such as LLaVA-34B which greatly help in extracting meaningful and descriptive captions.
- SaMyNa scales better and more consistently than B2T when employing fewer samples for keyword extraction.

Please, find more detailed responses below each review and in the revised text. We hope that this response has clarified the difference between SaMyNa and B2T and addressed the concerns about the novelty by Reviewers UdxG, Pg7a, and mTJk.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary of the Paper:

The paper presents “Say My Name” (SaMyNa), a method to identify and name biases learned by deep models using a text-based pipeline. Unlike prior work that often requires a validation set with bias-conflicting samples, SaMyNa aims to operate solely on the training set. By extracting textual descriptions (via large-scale captioners), selecting representative samples, and ranking recurrent keywords, the method claims to offer human-interpretable insights into the model’s biased correlations. Additionally, the authors show that these discovered biases can aid in downstream debiasing, improving model robustness on known benchmarks.

Strengths:
	Human-Interpretable Approach: The method aims to provide a more human-readable interpretation of biases by extracting textual descriptions and ranking keywords.
Weaknesses:
	The approach does not offer contributions significantly distinct or differentiated from existing work, such as Bias2Text.

	Modest practical gains: While flexibility and the ability to operate without a dedicated validation set are claimed, the empirical improvements in debiasing performance are minor.