SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP

## Abstract

Abstract Large-scale vision-language models, such as CLIP, are known to contain harmful societal bias regarding protected attributes ( e.g . , gender and age). In this paper, we aim to address the problems of societal bias in CLIP. Although previous studies have proposed to debias societal bias through adversarial learning or test-time projecting, our comprehensive study of these works identifies two critical limitations: 1) loss of attribute information when it is explicitly disclosed in the input and 2) use of the attribute annotations during debiasing process. To mitigate societal bias in CLIP and overcome these limitations simultaneously, we introduce a simple-yet-effective debiasing method called SANER ( s ocietal a ttribute n eutraliz er ) that eliminates attribute information from CLIP text features only of attribute-neutral descriptions. Experimental results show that SANER, which does not require attribute annotations and preserves original information for attribute-specific descriptions, demonstrates superior debiasing ability than the existing methods.

###### Keywords:

## 1 Introduction

Figure: Figure 1: Our debiasing method, SANER, overcomes the limitations in existing methods: (a) attribute information is retained after debiasing, and (b) protected attribute annotations are not required for debiasing.
Refer to caption: x1.png

Large-scale vision-language models (VLMs), such as CLIP [], have demonstrated a remarkable capability in multi-modal understanding [, ] and generation [, , ], being trained with million-scale image-text pairs. Utilizing these VLMs, recent vision models have achieved significant performance enhancements across a wide range of computer vision tasks (*e.g*., captioning [, , , , ] and object detection [, ]), without the necessity for task-specific training [].

Despite the success, several works have identified societal bias regarding demographic attributes, such as gender and age, in these VLMs [, , , , , , ], potentially causing unfair or prejudicial decisions by models. Hall *et al*. [] conducted audits on performance disparity, particularly with respect to gender, and revealed gender-dependency of the CLIP performance. Qiu *et al*. [] also demonstrated that adopting CLIP for caption evaluation tends to favor gender-stereotypical sentences (*e.g*., preferring “A woman is cooking” over “A man is cooking” for images depicting men), highlighting the inherent gender bias. These findings underscore the importance of addressing bias in VLMs.

Some studies have proposed to mitigate societal bias in VLMs [, , , ]. Adversarial debiasing [, , ] fine-tunes CLIP to lessen leakage of protected attributes(^1^11We refer to any demographic variables, like age and gender, as protected attribute (or attribute in short), based on which a model’s decisions should not be made.) into the features, while projection-based debiasing [] removes the protected attribute encoded in CLIP features in the inference phase. Our holistic review of these pioneering works (Sec. 3), though, identifies the following potential disadvantages or controversies in their design choices.

Loss of attribute information explicitly disclosed in the input. Some methods aim to completely remove attribute information by decorrelating the attribute and the features [] or by squashing the subspace associated with the attribute [], even when the attribute is explicitly disclosed. This choice can limit the generalizability of a VLM’s features to a spectrum of downstream tasks (*e.g*., Stable Diffusion [] generates male images for text prompt “a female doctor” when encoded with debiased CLIP as shown in Fig. 1 (a) and Sec. 6.2), while it works for attribute-agnostic downstream tasks [, ].

Use of the attribute annotations. Adversarial debiasing methods [, , ] require protected attribute annotations, as provided in FairFace [], for fine-tuning (Fig. 1 (b)). Datasets with attribute annotations are still scarce, perhaps partly because the annotation process needs ethical considerations [], limiting their applicability and generalizability. This dataset scarcity also causes the limited diversity of images and text descriptions used for fine-tuning the VLM, potentially inducing overfitting.

This paper presents a simple-yet-effective debiasing approach for CLIP, called SANER (societal attribute neutralizer), that simultaneously overcomes the aforementioned limitations. Specifically, SANER trains a debiasing layer (*i.e*., a multilayer perception) to amend CLIP text feature vectors of attribute-neutral descriptions, given by attribute neutralization, such that they are equidistant to those of attribute-specific descriptions using annotation-free debiasing loss. With this, only feature vectors for attribute-neutral descriptions are debiased, whereas the attribute-specific ones retain the original information. Attribute-specific descriptions for all possible attribute groups(^2^22Attribute group is a class in a protected attribute (*e.g*., female and male in gender).) can be easily augmented by modifying the attribute-specific words in the original descriptions, directing the training without attribute annotations.

Contribution. Thanks to our annotation-free debiasing pipeline, SANER is designed to be compatible with any dataset of image-text pairs, such as COCO []. This provides denser guidance for training the debiasing layer compared to the existing methods. Experiments on both discriminative and generative tasks (*i.e*., text-to-image retrieval [] and text-to-image generation []) show that SANER can mitigate gender, age, and racial biases of CLIP. Moreover, we demonstrate that SANER outperforms the existing methods [, ], showing that SANER leads to less attribute-dependency of the downstream performance while overcoming the limitations in existing methods.

## 2 Related Work

Societal bias in VLMs.
With the growing application of VLMs, such as CLIP [], across various tasks, there has been increasing awareness of their inherent social biases [, , , , , , ]. Hall *et al*. [] examined gender bias in CLIP, uncovering significant discrepancies in object recognition performance based on the gender depicted in images. For example, the accuracy in predicting an umbrella when it appears with women is much better than with men.

The existence of such societal bias poses a substantial risk of perpetuating and exacerbating discrimination against historically marginalized groups [, , , , ]. Birhane *et al*. [] demonstrated that adopting CLIP-based filtering in the dataset creation causes a risk of selecting stereotypical images regarding specific demographic subgroups. For instance, for an image of a female astronaut, the caption “This is a photograph of a smiling housewife” is favored over “This is a portrait of an astronaut” by CLIP, resulting in datasets that contain a number of image-caption pairs with harmful stereotypical descriptions.

Debiasing societal bias in CLIP.
Recognizing the problem of bias in CLIP, some recent research has concentrated on mitigating societal bias in CLIP [, , , ]. These debiasing efforts intend to remove attribute information from CLIP features, resulting in less biased prediction outcomes. For example, Berg *et al*. [] proposed a prompt tuning-based debiasing method based on an adversarial learning paradigm, minimizing the encoding of protected attribute information by the CLIP text encoder.
However, in Sec. 3, we identify that these methods contain critical limitations that pose challenges in utilizing various datasets for debiasing and applying debiased models to a wide range of downstream tasks.

## 3 Review: Existing Debiasing Methods

Several debiasing approaches for CLIP have been introduced, broadly categorized into two main types: adversarial debiasing [, , ] and projection-based debiasing []. This section conducts an in-depth analysis of these existing debiasing strategies, highlighting their respective limitations.

Notation.
Let $\mathcal{D}$ denote a dataset, each of whose sample is quadruple $(v,t,a,d)$,
where $v$ is an image,
$t$ is a text description, $a\in\mathcal{A}$ is a protected attribute annotation from set $\mathcal{A}$ of all attribute groups, and $d$ is the ground-truth annotation for a downstream task (if any). The CLIP text and image encoders, denoted by $f_{\text{t}}(t)\in\mathbb{R}^{K}$ and $f_{\text{v}}(v)\in\mathbb{R}^{K}$, respectively, take $t$ and $v$ as input and generate corresponding feature vectors in a common space.

### 3.1 Adversarial debiasing

Adversarial debiasing [, , ] aims to eliminate protected attribute information in the CLIP features. Specifically, an adversarial classifier is employed to predict and remove protected attribute $a$ from CLIP features.

Prompt tuning-based debiasing []
proposes to use learnable tokens to reduce attribute leakage through the similarity between an image and a set of pre-defined textual concepts. Concretely, for a set $\mathcal{C}$ of pre-defined concepts (*i.e*., phrases) that are supposed to be attribute non-specific (*e.g*., smart and attractive), a sequence $l$ of $k$ learnable tokens are prepended to the sentence template $t_{c}$ with concept $c\in\mathcal{C}$ (*e.g*., $t_{c}=\text{``A photo of a smart person''}$ for $c=\textit{smart}$) to obtain $t_{c}^{\prime}=[l,t_{c}]$, where $[\cdot,\cdot]$ represents sequence concatenation. Then, $t^{\prime}_{c}$ and arbitrary $v\in\mathcal{D}$ is fed into the CLIP encoders to compute similarity $s_{c}(v)$ by

$$ $s_{c}(v)=f_{\text{v}}(v)^{\top}f_{\text{t}}(t_{c}^{\prime}).$ (1) $$

Let $s(v)\in\mathbb{R}^{|\mathcal{C}|}$ denote a vector, each of whose elements is the similarity score $s_{c}(v)$ for a concept in $\mathcal{C}$. Due to the attribute non-specificity of concepts in $\mathcal{C}$, $s(v)$ should not correlate with attribute $a$ of $v$. However, the CLIP text encoder can embed $a$ into $s(v)$ due to bias, allowing an attribute classifier to predict $a$. We denote the probability of being $a$ given $s(v)$ (or a prediction score of the attribute classifier) by $m_{a}(s(v))$.
Prompt tuning-based debiasing [] uses $m_{a}$ for adversarial loss, given by

$$ $\mathcal{L}_{\text{adv}}=-\sum_{v\in\mathcal{D}}\log m_{a}(s(v)).$ (2) $$

Minimizing $\mathcal{L}_{\text{adv}}$ with respect to $l$ reduces attribute leakage through $s(v)$.
A contrastive loss between image and text features is also used for regularization.

The experiments [] showed that this method could effectively reduce attribute leakage through $f_{\text{t}}(t)$, but the limited number of concepts(^3^33Their experiments used $10$ concepts.) may limit downstream tasks that enjoy the debiased features because as $l$ is learned only through a sparse set of concepts. Additionally, attribute annotations are necessary for the adversarial loss, resulting in the exclusive use of face-centric image datasets (*e.g*., FairFace []) as $\mathcal{D}$.

Additive Residual Learner (ARL) []
is designed to remove attribute information from CLIP image features. This method assumes that a debiasing layer(^4^44A fully-connected layer is used.) $r$ can identify a vector to additively amend attribute-neutral image feature vector $\delta(v)$, *i.e*.,

$$ $\delta(v)=f_{\text{v}}(v)-r(f_{\text{v}}(v)).$ (3) $$

Similarly to [], an adversarial classifier is trained to predict $a$ from $\delta$ with adversarial loss

$$ $\mathcal{L}_{\text{adv}}=-\sum_{v\in\mathcal{D}}\log m_{a}(\delta(v)).$ (4) $$

The reconstruction loss between $f_{\text{v}}(v)$ and $\delta(v)$ regularizes the training to maintain the original features.

This method shares a common limitation with prompt tuning-based debiasing [], notably requiring attribute annotations. Another limitation is that it tries to remove attribute features even when attributes of people in images are explicitly disclosed (*i.e*., when the person is depicted in an image). Consequently, debiased CLIP is ignorant of protected attributes.

Mapper [] aims to reduce spurious correlations between protected attributes $a$ and task label $d$ in $\mathcal{D}$. It applies mappings $f_{\text{v}}^{\prime}$ and $f_{\text{t}}^{\prime}$ to image and text features, respectively, as $x_{\text{v}}(v)=f_{\text{v}}^{\prime}(f_{\text{v}}(v))$ and $x_{\text{t}}(t)=f_{\text{t}}^{\prime}(f_{\text{t}}(t))$ for mitigating dependence on $a$. The adversarial loss is computed using a dependence measure $\text{Dep}(\cdot,\cdot)$ to quantify statistical dependence between features as:

$$ $\mathcal{L}_{\text{adv}}=-\text{Dep}(x_{\text{v}}(v),a)-\text{Dep}(x_{\text{t} }(t),a).$ (5) $$

These mapping functions are also trained to maximize the statistical dependence between the features after the mapping and task label $d$, *i.e*.,

$$ $\text{Dep}(x_{\text{v}}(v),d)+\text{Dep}(x_{\text{t}}(t),d),$ (6) $$

to retain the predictive power on the downstream task while reducing bias.

Similar to [, ], Mapper relies on attribute annotations. Moreover, it is designed only to address the spurious correlations between the attribute and task labels for a specific task but not for different tasks.

### 3.2 Projection-based debiasing

Projection-based debiasing [] projects CLIP text feature vectors into the orthogonal complement of the space spanned by a set of CLIP text feature vectors that pertain to the protected attribute. Specifically, let $\mathcal{U}$ denote a set of text descriptions with the target attribute (*e.g*., $\text{``A photo of a $w$''}\in\mathcal{U}$, where $w\in\{\text{``woman'', ``man''}\}$ for binary gender), and $U$ be a matrix each of whose column vectors is $f_{\text{t}}(u)$ with $u\in\mathcal{U}$. The projection matrix $P$ into the orthogonal complement for $U$ is given by

$$ $P=I-U(U^{\top}U)^{-1}U^{\top}$ (7) $$

where $I$ is the identity matrix. $P$ can project a CLIP text feature vector $f_{\text{t}}(t)$ for a text description $t$ into the orthogonal complement by $Pf_{\text{t}}(t)$.

Unlike adversarial debiasing, which requires training by gradient descent update, $P$ has a closed-form solution and is computed in the inference phase. However, as with ARL, this method also eliminates attribute information even from descriptions with explicit attributes (*e.g*., “A photo of a female doctor”).

### 3.3 Summary of the challenges

The existing debiasing methods, including adversarial and projection-based, reveal several challenges: 1) Loss of attribute information (ARL and projection-based) even with explicit attribute description narrows down the utility of the debiased CLIP. For example, a text-to-image generative model with gender-debiased CLIP features may not properly depict explicitly specified gender (as
shown in Sec. 6.2),
2) Dependency on attribute annotations (prompt tuning, ARL, and Mapper) constrains the range of datasets that can be utilized, often necessitating the use of face-centric image datasets (*e.g*., FairFace), as opposed to more diverse, natural image datasets (*e.g*., COCO []).

## 4 Societal Attribute Neutralizer (SANER)

Figure: Figure 2: An overview of SANER, exemplified by binary gender.
Refer to caption: x2.png

Our method for debiasing CLIP features, SANER, addresses the limitations of the existing methods identified in Section 3. Notably, SANER 1) retains attribute information in cases where the person’s attributes are explicitly described and 2) eliminates the reliance on attribute annotations, allowing the use of any image-text dataset for training the debiasing layer.

SANER comprises 1) attribute neutralization, which eliminates protected attribute information from input text (Section 4.1); 2) feature modification, which removes attribute information from the CLIP text features by amending them with a debiasing layer (Section 4.2); 3) attribute annotation-free debiasing loss, ensuring the features are not biased towards any attribute group $g\in\mathcal{A}$ (Section 4.3); and 4) regularization losses, which preserve the original CLIP features and the alignment between image and text features (Section 4.4).

Figure 2 shows an overview of SANER. We train the debiasing layer for feature modification over an arbitrary dataset $\mathcal{D}=\{(v,t)\}$ of image $v$ and text description $t$ (*e.g*., image caption, alt text) pairs, which does not provide attribute annotation $a$ as well as target task label $d$.

### 4.1 Attribute neutralization

We first modify text description $t\in\mathcal{D}$ that contain person-related words,(^5^55The person-related words include any words relevant to a person (*e.g*., person, girl, man). The complete list is available in the supplementary material.) to remove attribute-specific words. Taking binary gender as a protected attribute,(^6^66Following prior research [, , , , , , ], we focus on the binary gender but recognize the importance of inclusivity. SANER applies to non-binary genders.) *i.e*., $\mathcal{A}=\{\texttt{female},\texttt{male}\}$, as example, the text description

> $t=\text{``A woman is eating salad.''}$

contains attribute information (*i.e*., woman). We replace attribute-specific terms(^7^77We use gender words defined in [].) with the attribute-neutral ones to obtain an attribute-neutral text:

> $\xi_{\text{n}}(t)=\text{``A \text@underline{person} is eating salad.''}$

where $\xi_{\text{n}}$ denotes a function for attribute neutralization.
Neutralization can be done for other attributes, such as age.(^8^88Examples for the race attribute are in the supplementary material.) We remove age-specific terms(^9^99We define age-specific terms. The list is in the supplementary material.) (*e.g*., young and senior) in text descriptions, for instance, “A young woman is eating salad” $\rightarrow$ “A woman is eating salad”.
In contrast to the previous approach [], which is optimized not to predict the attribute information from the original description $t$, we target the attribute-neutral descriptions $\xi_{\text{n}}(t)$ to preserve the attribute information in the features of attribute-specific descriptions.

### 4.2 Feature modification

CLIP text features $z(t)=f_{\text{t}}(\xi_{\text{n}}(t))$ after attribute neutralization can still convey the protected attribute information due to CLIP’s bias. To remove such bias, we append a learnable debiasing layer $r$ on top of $f_{\text{t}}$, inspired by recent CLIP fine-tuning techniques [, ]. Neutralized $t$’s debiased feature $h(t)$ is given by

$$ $h(t)=z(t)+r(z(t)).$ (8) $$

### 4.3 Attribute annotation-free debiasing loss

To train $r$ to extract attribute information from CLIP features without attribute annotations, we generate a set $\mathcal{T}$ of attribute-specific descriptions for $t\in\mathcal{D}$ and for $g\in\mathcal{A}$, *i.e*., $\mathcal{T}=\{\xi_{g}(t)|t\in\mathcal{D},g\in\mathcal{A}\}$, where $\xi_{g}(t)$ generates a description specific to attribute group $g$ from $t$. For the binary gender case, this involves generating descriptions with female- and male-specific words. For instance, from the text description, “A woman is eating salad.”, we generate two sentences with female and male attributes:

> A woman is eating salad. A man is eating salad.

The debiasing loss trains $r$ such that $h(t)$ is equidistant from $f_{\text{t}}(\xi_{g}(t))$ for all attribute groups in $\mathcal{A}$, ensuring an impartial representation across the spectrum of attribute groups. We implement this loss as the standard deviation of the cosine similarity between $h(t)$ and $f_{\text{t}}(\xi_{g}(t))$. Let $s_{g}(t)$ denote the similarity, *i.e*.,

$$ $\displaystyle s_{g}(t)=\frac{h(t)^{\top}f_{\text{t}}(\xi_{g}(t))}{\|h(t)\|\;\| f_{\text{t}}(\xi_{g}(t))\|}.$ (9) $$

The debiasing loss $\mathcal{L}_{\text{deb}}$ is defined as

$$ $\mathcal{L}_{\text{deb}}=\sqrt{\frac{1}{|\mathcal{D}|}\sum_{t\in\mathcal{D}}(s _{g}(t)-\bar{s}(t))^{2}},$ (10) $$

where $\bar{s}(t)=\sum_{g\in\mathcal{A}}s_{g}(t)/|\mathcal{A}|$. A lower standard deviation means $s_{g}$ is close to $\bar{s}$, leading to $h(t)$ being equidistant to $f_{\text{t}}(\xi_{g}(t))$ for all $g\in\mathcal{A}$. Notably, this debiasing loss can be computed without attribute annotations.

### 4.4 Regularization losses

Applying the debiasing loss alone significantly changes original CLIP features, thereby losing semantics [, ]. To maintain the alignment of resulting image-text features, we utilize reconstruction loss [] and contrastive loss []. Reconstruction loss $\mathcal{L}_{\text{recon}}$ is the mean squared error between $f_{\text{t}}(t)$ and $h(t)$. Contrastive loss $\mathcal{L}_{\text{cont}}$ aims to minimize the negative log-likelihood of input image-caption pairs, $f_{\text{v}}(v)$ and $f_{\text{t}}(t)$, in comparison to negative ones.

### 4.5 Training and inference

The overall loss $\mathcal{L}$ is given by:

$$ $\mathcal{L}=\alpha\mathcal{L}_{\text{deb}}+\beta\mathcal{L}_{\text{recon}}+ \gamma\mathcal{L}_{\text{cont}},$ (11) $$

where $\alpha$, $\beta$, and $\gamma$ are the hyperparameters to weight respective losses.

During inference, we apply the trained debiasing layer $r$ and use the modified text features $r(f_{\text{t}}(t))$ as the CLIP text features.

## 5 Experiments: Text-to-Image Retrieval

Following previous studies [, , , ], we evaluate SANER on the text-to-image retrieval task regarding gender, age, and racial biases. Further analysis, such as the ablation study of the loss components, is in the supplementary materials.

### 5.1 Experimental settings

Evaluation metric.
We employ the MaxSkew metric [], utilized in the previous studies [, , , ], to quantify the societal bias in CLIP in the text-to-image retrieval task. MaxSkew measures the disparity between the attribute distribution of $|\mathcal{A}|$ protected attributes in the top-$k$ retrieved images. Let $\eta_{ak}(q)$ denote the ratio of images labeled with attribute $a$ in the top-$k$ retrieved images. For attribute neutral query $q$, $\eta_{ak}(q)$ should be $1/|\mathcal{A}|$ if the model is unbiased. MaxSkew@$k$ is defined as:

$$ $\text{MaxSkew@$k$}=\max_{a\in\mathcal{A}}\log\frac{\eta_{ak}(q)}{1/|\mathcal{A }|}.$ (12) $$

Ideally, MaxSkew@$k$ is 0 but is larger when a model biased.

Evaluation setting. For the attribute-neutral queries, we use template-based queries such as “a photo of a $c$ person”, where $c$ is an attribute-neutral concept. Prior work [] has defined a list of person-related adjectives, such as clever and attractive, as attribute-neutral concepts. We extend the list to encompass occupations (*e.g*., doctor, nurse), and activities (*e.g*., cooking and cleaning) for a more comprehensive evaluation. MaxSkew@$k$ is computed per concept.(^10^1010The complete lists of the concepts are in the supplementary material.)

We also evaluate zero-shot image classification accuracy on ImageNet-1K [] to ensure that debiasing does not spoil the original CLIP’s performance.

Evaluation datasets.
We utilize two datasets, FairFace [] and PATA [], which consist of images alongside protected attribute annotations (*e.g*., female and male for gender) associated with the person in each image. FairFace consists of $10,954$ cropped face-centric images, while PATA contains $4,934$ natural images with a single person. Most debiasing approaches only report the performance on the FairFace dataset, but we additionally employ PATA to evaluate the debias performance on more diverse images.

Methods for comparison.
We compare SANER against existing methods, *i.e*., prompt tuning-based debiasing [] and projection-based debiasing [], whose code is publicly available. Unfortunately, we could not reproduce the other methods[, ] since sufficient reproduction details are unavailable.

Implementation details.
Following the previous works [, ], we employ CLIP [] with ViT-B/16 backbone [] as a target model in our experiments.
We train the debiasing layer (a multilayer perception with two linear layers with ReLU activation []) appended to the model using $170,624$ image-caption pairs, which is a subset of the COCO training set [] with person-related words/phrases (*e.g*., person and boy).
We empirically set $\alpha$, $\beta$, and $\gamma$ to $1.0$, $0.1$, and $0.0001$ (Eq. 11), respectively, and train for $5$ epochs. Further details are provided in the supplementary materials.

**Table 1: Gender bias, evaluated by MaxSkew@1000 (scaled by $100$), on FairFace and PATA for the original CLIP (Original), prompt tuning-based debiasing (Prompt), projection-based debiasing (Projection), and our method (SANER). A lower value is better (less gender bias). Bold represents the best across the models.**
|  | FairFace |  | PATA |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP Model | Adjective | Occupation | Activity |  | Adjective | Occupation | Activity |
| Original [] | $22.9$ | $33.7$ | $19.5$ |  | $12.1$ | $18.7$ | $10.7$ |
| Prompt [] | $12.3$ | $29.9$ | $20.0$ |  | $6.7$ | $16.5$ | $10.2$ |
| Projection [] | $15.4$ | $37.4$ | $15.0$ |  | $6.4$ | $13.6$ | $5.4$ |
| SANER (Ours) | $\mathbf{8.9}$ | $\mathbf{14.5}$ | $\mathbf{7.7}$ |  | $\mathbf{5.4}$ | $\mathbf{9.5}$ | $\mathbf{3.3}$ |

**Table 2: Age bias, evaluated by MaxSkew@1000 (scaled by $100$), on FairFace and PATA. Bold denotes the best across the models.**
|  | FairFace |  | PATA |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP Model | Adjective | Occupation | Activity |  | Adjective | Occupation | Activity |
| Original[] | $111.1$ | $121.1$ | $113.0$ |  | $40.4$ | $44.4$ | $39.7$ |
| Projection [] | $107.6$ | $112.8$ | $\mathbf{100.0}$ |  | $37.6$ | $45.6$ | $45.5$ |
| SANER (Ours) | $\mathbf{96.0}$ | $\mathbf{112.6}$ | $101.9$ |  | $\mathbf{30.3}$ | $\mathbf{36.7}$ | $\mathbf{27.5}$ |

**Table 3: Racial bias, evaluated by MaxSkew@1000 (scaled by $100$), on FairFace and PATA. Bold denotes the best across the models.**
|  | FairFace |  | PATA |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP Model | Adjective | Occupation | Activity |  | Adjective | Occupation | Activity |
| Original[] | $62.2$ | $57.4$ | $68.3$ |  | $33.4$ | $28.6$ | $31.5$ |
| Projection [] | $56.9$ | $75.3$ | $67.0$ |  | $\mathbf{19.9}$ | $43.8$ | $26.3$ |
| SANER (Ours) | $\mathbf{49.3}$ | $\mathbf{45.7}$ | $\mathbf{46.6}$ |  | $28.9$ | $\mathbf{21.2}$ | $\mathbf{20.5}$ |

### 5.2 Gender bias analysis

Table 1 summarizes MaxSkew@1000 on FairFace and PATA for gender bias, showing that SANER mitigates gender bias the most among all methods. In contrast to the existing methods, this tendency is consistent across 1) the datasets with different image domains (*i.e*., face-centric and natural images) and 2) concept types (*i.e*., adjective, occupation, and activity). For instance, while the prompt tuning-based method [] fails to mitigate bias for activity concepts on FairFace (*i.e*., bias is amplified from $19.5$ to $20.0$), SANER significantly mitigates bias from $19.5$ to $7.7$. This verifies the better debiasing performance of SANER compared to the existing methods on diverse concepts and image domains, possibly because SANER is trained with diverse text descriptions (*i.e*., captions in COCO), which are not constrained like pre-defined concepts required in the previous method [].

### 5.3 Age and racial biases analysis

Tables 2 and 3 present the results of MaxSkew@1000 for age and racial biases, respectively. We compare SANER with projection-based debiasing [], as the prompt tuning-based method [] does not provide age and race debiasing variants. Similarly to the results for gender bias, SANER surpasses the existing method across the datasets and the concept types. For example, SANER successfully mitigates the age bias on PATA. Meanwhile, the projection-based method works only on the adjective concepts and amplifies the bias on the other concepts (*e.g*., from $39.7$ to $45.5$). These results validate the generalizability of SANER in bias mitigation across the protected attributes.

### 5.4 Zero-shot image classification

**Table 4: Accuracy on Image-Net-1K.**
| CLIP Model |  | Acc. (%) |
| --- | --- | --- |
| Original [] |  | $65.4$ |
| Prompt [] |  | $64.1$ |
| SANER (Ours) |  | $65.2$ |

To verify whether debiasing harms the zero-shot image classification performance of the original CLIP, we evaluate the prompt tuning-based method [] and SANER on ImageNet-1K in terms of classification accuracy. Projection-based debiasing [] does not apply to this evaluation because zero-shot prompts, such as “a photo of a car,” do not necessarily include person-related words.(^11^1111Projection-based debiasing requires modified input text to include attribute terms.) The results are shown in Tab. 4, showing that applying SANER maintains classification performance, whereas the performance of the prompt tuning-based method slightly degrades.

## 6 Experiments: Text-to-Image Generation

We also evaluate SANER on text-to-image generation, for which societal bias is actively investigated [, , , ]. Specifically, we conduct two experiments from different aspects: 1) gender bias regarding occupations using gender-neutral prompts (Sec. 6.1), and 2) retention of attribute information when prompts explicitly disclose gender (Sec. 6.2).

Image generation settings.
We use Stable Diffusion (SD) v2.1 [] as the text-to-image generation model. The CLIP text encoder in SD is replaced with the debiased one for evaluation. Following [], we use gender-neutral prompts with specifying occupations to analyze gender bias in generated images. These prompts are derived from the template “A photo of a $o$”, where $o$ is replaced with specific occupation terms, such as doctor and teacher.(^12^1212The list of the occupations is in the supplementary material.)
On the other hand, we employ gender-specific prompts to evaluate the capability of attribute information retention. Concretely, a gender term, either female or male, is added just before the occupation terms, *i.e*., “A photo of a {female/male} $o$”, to see if generated images specify the gender.
We generate $100$ images for each prompt with SD’s default hyperparameters shown in [].

Evaluation metrics.
For the bias evaluation for the generative task, we use statistical parity (SP) metric that measures the disparity of attribute groups in generated images [, , ]. Specifically, we annotate binary gender labels (*i.e*., $\mathcal{A}=\{\texttt{female},\texttt{male}\}$) for the generated images with the assistance of human workers.(^13^1313Different from previous works that use pre-trained gender classifiers to assign gender labels, we do not use them due to their bias issues [, , ].) SP is defined as the difference between the empirical distribution $\kappa_{a}$ of gender $a$ and uniform distribution, given by:

$$ $\text{SP}=\sqrt{\sum_{a\in\mathcal{A}}(\kappa_{a}-1/|\mathcal{A}|)^{2}},$ (13) $$

where $\kappa_{a}=N_{a}/\sum_{a^{\prime}}N_{a^{\prime}}$ with $N_{a}$ being the number of images annotated as $a$.
For an unbiased text-to-image generation model, SP should be 0 but increases for biased models.

For gender-specific prompts for evaluating gender information retention, we compute the accuracy, *i.e*., the ratio of the images that contain the same gender as the prompt to all generated images.

### 6.1 Gender bias analysis

Table 5 summarizes SP scores, which demonstrate that applying SANER to Stable Diffusion notably mitigates gender bias regarding occupations. SANER again outperforms projection-based debiasing one (*i.e*., $0.39$ for SANER and $0.47$ for projection-based), highlighting the superiority of SANER.

We show visual examples in which SANER successfully mitigates gender bias in Fig. 3. For the gender-neutral prompt, “A photo of a designer,” the original SD and projection-based debiasing (Projection) predominantly generate images depicting a man. In contrast, ours shows a more balanced gender distribution.

### 6.2 Assessment of retention of attribute information

Table 5 also shows the accuracy regarding how much the gender of the person in generated images matches the gender specified in the prompts. For both prompts that describe women and men, using debiased CLIP by projection-based debiasing leads to losing gender information (*i.e*., accuracies for projection are much lower than those for the original). Conversely, SANER retains gender information (*i.e*., accuracies for SANER are $1.00$). Figure 4 confirms this, showing that using projection-based debiased CLIP results in generating male images for the prompt, “A photo of a female doctor”.

**Table 5: Gender bias (SP) and retention of gender information (Accuracy) in images generated by Stable Diffusion with original CLIP text encoder (Original) and the model using projection-based debiased CLIP (Projection) and our debiased CLIP (SANER). We report the mean and standard deviation over the set of occupations. Female and male denote prompts with female and male specifications, respectively. Bold represents the best across the models.**
|  |  |  | Accuracy $\uparrow$ |  |
| --- | --- | --- | --- | --- |
| CLIP Model | SP $\downarrow$ |  | Female | Male |
| Original [] | $0.51$ $\pm 0.22$ |  | $\mathbf{1.00}$ $\pm 0.00$ | $\mathbf{1.00}$ $\pm 0.00$ |
| Projection [] | $0.47$ $\pm 0.19$ |  | $0.58$ $\pm 0.28$ | $0.79$ $\pm 0.18$ |
| SANER (Ours) | $\mathbf{0.39}$ $\pm 0.22$ |  | $\mathbf{1.00}$ $\pm 0.00$ | $\mathbf{1.00}$ $\pm 0.00$ |

Figure: Figure 3: Generated images for the prompt, “A photo of a designer,” by the original Stable Diffusion (SD), projection-based debiased CLIP (Projection), and our debiased CLIP (SANER). We randomly sample $10$ images from generated images. Images framed in green denote those of the minority gender in the generated images (*i.e*., female).
Refer to caption: x3.png

Figure: Figure 4: Generated images for the prompt, “A photo of a female doctor,” by the original Stable Diffusion (SD), projection-based debiased CLIP (Projection), and our debiased CLIP (SANER). We randomly sample $10$ images from generated images. Red frame indicates images with incorrect gender (*i.e*., male).
Refer to caption: x4.png

## 7 Limitations

Bias metrics.
While SANER outperforms the existing methods in mitigating gender, age, and racial biases according to existing bias metrics, such as MaxSkew@$k$ and SP, this does not guarantee that SANER is always superior to them in every aspect or scenario. As suggested in prior work [, , , ], reliance on these metrics provides a standardized measure of societal bias; however, there could be dimensions of bias not adequately captured by them, which implies the necessity of more holistic metrics to assess societal bias.

Further bias mitigation.
The experimental results show that SANER noticeably mitigates societal bias in CLIP. Nonetheless, the bias is not completely eliminated (*e.g*., MaxSkew is not zero). A promising direction for further debiasing could involve debiasing the image encoder, specifically training a debiasing layer to remove attribute information from the visual features for images in which humans do not appear.

Intersectional bias analysis.
Although we focus on gender, age, and racial biases separately in our experiments, following the previous debias works [, , , ], SANER can be easily extended to various protected attributes, including their combinations. For instance, considering the intersection of binary gender and age, we generate four sentences with (female, young), (female, old), (male, young), and (male, old) for the debiasing loss, *e.g*., “A young woman is eating salad” for the input text “A woman is eating salad”. This potential for addressing complex biases is noted in future research.

## 8 Conclusion

This paper proposed SANER, a simple-yet-effective debiasing method for CLIP, consisting of attribute neutralization and anotation-free debiasing loss.
Consequently, SANER can use an arbitrary dataset of image-text pairs for training the debiasing layer, resulting in superior debiasing performance compared to the existing methods for both discriminative and generative tasks (*i.e*., text-to-image retrieval and text-to-image generation). We also confirmed that SANER retains attribute information for attribute-specific descriptions through the gender-specified prompts for text-to-image generation.

Future work can explore extending SANER to the vision encoder, removing attribute information from visual features in images without humans, and addressing intersectional biases, combining attributes like gender and age. Furthermore, SANER is applicable to any dataset of image-caption pairs, thus, scaling to larger datasets (*e.g*., Conceptual Captions []) for denser supervision is an interesting future direction.

## Supplementary Material

This supplementary material includes:

- •
Implementation details for SANER (Appendix 0.A).
- •
Further analysis (Appendix 0.B).
- •
List of person-, gender-, age-, and race-specific terms (Appendix 0.C).
- •
List of concepts and occupations (Appendix 0.D).
- •
Additional visual examples for image generation experiments (Appendix 0.E).
- •
Potential negative impact (Appendix 0.F).

## Appendix 0.A Implementation Details for SANER

We train the debiasing layer (a multilayer perception with two linear layers with ReLU activation) using $170,624$ image-caption pairs, which is a subset of the COCO training set with person-related words/phrases defined in Appendix 0.C. The hidden embedding dimensionality for the debiasing layer is set to $128$. We empirically set $\alpha$, $\beta$, and $\gamma$ to $1.0$, $0.1$, and $0.0001$ (Eq. (11) in the main paper), respectively. We set the training epochs, batch size, and learning rate to $5$, $128$, and $5\times 10^{-6}$, respectively.

Racial bias mitigation.
For race attribute, we remove race-specific terms(^14^1414We define race-specific terms. The list is in Appendix 0.C.) (*e.g*., African and Asian) in text descriptions, for instance, “An African woman is eating salad” $\rightarrow$ “A woman is eating salad”.

## Appendix 0.B Additional Experiments

### 0.B.1 Loss ablation

To validate the effectiveness of each regularization loss (*i.e*., reconstruction loss and contrastive loss), we conduct an ablation study by removing one of the losses or both losses. Table 6 presents the results of gender bias.
The results show that using both reconstruction and contrastive losses yields the best results regarding gender bias mitigation and zero-shot classification accuracy on ImangeNet []. Furthermore, SANER without regularization losses significantly degrades CLIP’s zero-shot classification ability (*i.e*., from $65.4$ to $58.8$). These observations confirm the importance of having both reconstruction and contrastive losses for the regularization.

**Table 6: Gender bias, evaluated by MaxSkew@1000 (scaled by $100$), on FairFace and PATA for our method (SANER) with different regularization loss combinations. Recon denotes the use of the reconstruction loss, and cont represents the use of the contrastive loss. IN acc is the zero-shot classification accuracy on ImageNet. Adj, Occ, and Act represent the types of concepts (*i.e*., Adjective, Occupations, and Activity, respectively). A lower value is better (less gender bias). Bold represents the best across the SANER variants.**
|  | FairFace |  | PATA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP Model | Adj | Occ | Act |  | Adj | Occ | Act |  | IN acc (%) |
| Original [] | $22.9$ | $33.7$ | $19.5$ |  | $12.1$ | $18.7$ | $10.7$ |  | $65.4$ |
| SANER |  |  |  |  |  |  |  |  |  |
| (no regularization) | $15.7$ | $15.0$ | $15.3$ |  | $15.8$ | $14.4$ | $15.4$ |  | $58.8$ |
| Recon | $10.2$ | $16.7$ | $11.4$ |  | $6.2$ | $12.0$ | $6.2$ |  | $64.2$ |
| Cont | $18.0$ | $19.4$ | $21.7$ |  | $7.0$ | $\mathbf{9.1}$ | $9.0$ |  | $63.0$ |
| Recon+Cont | $\mathbf{8.9}$ | $\mathbf{14.5}$ | $\mathbf{7.7}$ |  | $\mathbf{5.4}$ | $9.5$ | $\mathbf{3.3}$ |  | $\mathbf{65.2}$ |

### 0.B.2 Analysis on the data size

The experiments in the main paper (Sections 5 and 6) verify that SANER outperforms existing debiasing methods, showing a better bias mitigation ability in terms of gender and age biases. This superior performance of SANER may be because SANER is trained with diverse text descriptions (*i.e*., captions in COCO []), which are not constrained like pre-defined concepts required in the previous method []. In this section, we conduct an experiment to verify this hypothesis. Specifically, we use $n$ percent of the training samples (*i.e*., $17,624$ image-caption pairs of COCO) to evaluate the impact of the training dataset size. We use the same settings in Appendix 0.A, but use the different training epochs to align the number of iterations. The results are shown in Tab. 7.

The results validate that as the number of data samples increases, gender bias is reduced. Specifically, while using a part of the training data results in mitigating gender bias (*i.e*., MaxSkew scores are smaller than the original CLIP), using the full training samples (*i.e*., COCO-100%) gives the best results, showing the importance of the use of more diverse data for debiasing.

**Table 7: Gender bias, evaluated by MaxSkew@1000 (scaled by $100$), on FairFace and PATA for the original CLIP (Original) and our method (SANER) with different data sizes. COCO-$n$% denotes that we use $n$% of the training samples. IN acc is the zero-shot classification accuracy on ImageNet. A lower value is better (less gender bias). Bold represents the best across the SANER variants.**
|  | FairFace |  | PATA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP Model | Adj | Occ | Act |  | Adj | Occ | Act |  | IN acc (%) |
| Original [] | $22.9$ | $33.7$ | $19.5$ |  | $12.1$ | $18.7$ | $10.7$ |  | $65.4$ |
| SANER |  |  |  |  |  |  |  |  |  |
| COCO-50% | $15.7$ | $19.4$ | $17.9$ |  | $10.0$ | $14.1$ | $10.2$ |  | $\mathbf{65.4}$ |
| COCO-75% | $12.8$ | $17.6$ | $16.1$ |  | $7.8$ | $13.0$ | $8.3$ |  | $\mathbf{65.4}$ |
| COCO-100% | $\mathbf{8.9}$ | $\mathbf{14.5}$ | $\mathbf{7.7}$ |  | $\mathbf{5.4}$ | $\mathbf{9.5}$ | $\mathbf{3.3}$ |  | $65.2$ |

## Appendix 0.C List of Person-, Gender-, Age-, Race-Specific Terms

The person-related words that are used to identify text descriptions that are relevant to humans (in Sec. 4.1 in the main paper) are as below:

actor, actress, adult, architect, artist, associate, aunt, baby, boy, boyfriend, brother, chairman, chairperson, chairwoman, chef, child, coach, colleague, comedian, counselor, cowboy, cowgirl, dancer, daughter, designer, director, doctor, driver, dude, elder, emperor, employee, employer, engineer, entrepreneur, executive, expecting, father, female, friend, gentleman, girl, girlfriend, guy, hairdresser, he, her, hers, herself, him, himself, his, husband, individual, infant, instructor, kid, lady, lawyer, leader, lecturer, male, man, manager, mechanic, member, mentor, mother, musician, neighbor, novelist, nurse, parent, partner, people, performer, person, pharmacist, photographer, physician, pilot, player, police officer, policeman, policewoman, politician, pregnant, prince, princess, professor, queen, relative, researcher, royal, scholar, scientist, secretary, server, she, sibling, singer, sister, son, specialist, spouse, student, surfer, surgeon, tailor, teacher, technician, teenager, their, theirs, them, themselves, therapist, they, toddler, uncle, veterinarian, volunteer, waiter, waitress, wife, woman, worker, writer, youth, and their plurals.

We list the gender-specific terms that are used to create attribute-neutral text descriptions $\xi_{n}(t)$ (in Sec. 4.1 in the main paper):
woman, female, lady, mother, girl, aunt,
wife, actress, princess, waitress, sister, queen, pregnant, daughter, she, her,
hers, herself, man, male, father, gentleman, boy, uncle, husband, actor, prince, waiter, son, brother, guy, emperor, dude, cowboy, he, his, him, himself and their plurals (orange denotes female-specific words, and olive represents male-specific terms). To synthesize attribute-specific descriptions (*i.e*., $\mathcal{T}=\{\xi_{g}(t)|t\in\mathcal{D},g\in\mathcal{A}\}$ in Sec. 4.3 in the main paper) for binary gender, we replace person-specific terms in the attribute-neutral descriptions with their corresponding gender terms (*e.g*., person $\rightarrow$ woman and person $\rightarrow$ man).

We also list the age-specific terms used to create attribute-neutral text descriptions $\xi_{n}(t)$ (in Sec. 4.1 in the main paper):
elderly ,baby, child, kid, teenager, adult, youth, infant, toddler, elder, girl, boy, young, old, teenage, and their plurals. To create attribute-specific descriptions (*i.e*., $\mathcal{T}=\{\xi_{g}(t)|t\in\mathcal{D},g\in\mathcal{A}\}$ in Sec. 4.3 in the main paper) for binary age, we add young or old just before the person-specific terms (*e.g*., person $\rightarrow$ young person and person $\rightarrow$ old person).

Regarding race attributes, we use race-specific terms to create attribute-neutral text descriptions: african, africa, asian, oriental, asia, east asian, south asian, south east asian, black, caucasian, european, hispanic, latino, latina, latinx, white, arab,
arabic, middle eastern, native, indigenous, american, african american, usa, united states, chinese, china, japanese, japan, indian, india, mexican, mexico, italian, italy, spanish, german, french, france,
english, british, england, russian, swiss, hawaiian, thai, brazil, brazilian, canadian, canada, australian, australia, new zealander, new zealand, korean, korea,
filipino, philippines, vietnamese, vietnam, malaysian, malaysia, singaporean, singapore, indonesian, indonesia, thai, thailand, burmese, myanmar, cambodian, cambodia,
laotian, laos, taiwanese, taiwan, pacific, melanesian, melanesia, polynesian, polynesia, micronesian, micronesia, aboriginal, aborigine. We adopt racial classes used in FairFace (*i.e*., East Asian, Southeast Asian, White, Black, Hispanic, Middle Eastern, Indian), creating attribute-specific descriptions. Specifically, we add East Asian for the East Asian class, Southeast Asian for the Southeast Asian class, White or Caucasian for the White class, Black or African for the Black class, Latino, Latina, or Hispanic for the Hispanic class, Middle Eastern or Arab for the Middle Eastern class, Indian for the Indian class before the person-specific terms (*e.g*., person $\rightarrow$ {East Asian person, Southeast Asian person, White person, Black person, Latino person, Middle Eastern person, Indian person}).

## Appendix 0.D List of concepts and occupations

We show the concepts used to compute MaxSkew@$k$ (in Tab. 1 and 2 in the main paper). The list of the concepts for each category (*i.e*., adjectives, occupations, and activities) are as follows:

Adjectives: kind, successful, noncriminal, nonviolent, moral, rich, honest, smart, evil, unkind, unsuccessful, criminal, violent, immoral, poor, dishonest, dumb, clever, stupid, hardworking, lazy, nasty, trustworthy, untrustworthy, caring, dangerous, heroic, villainous, good. We refer these concepts from []. Regardig the templates used to create prompts, we use “This is the face of a {} person”, “This is a photo of a {} person”, “This person is {}”, “A {} person”, “A {} individual”, “This individual is {}”, “Photo of a {}”, “{}”, “This is a {}”, “This is the photo of a {}”, “A {}”, following []. The curly brackets are replaced with each concept.

Occupations: doctor, aerospace engineer, computer programmer, electrical engineer, scientist, artist, author, designer, musician, painter, photographer, singer, writer, architect, civil engineer, engineer, programmer, software developer, childcare worker, coach, dental assistant, dental hygienist, dentist, clerk, housekeeper, maid, massage therapist, nurse, therapist,psychologist, social assistant, social worker, teacher, professor, CEO, skateboarder, surfer, baseball player, football player, soccer player, tennis player. For the occupation list, we refer to [] with some modifications such as additional occupations for more inclusive list. As for the prompt templates, we select ones that are suitable to occupations (*e.g*., “Photo of a {}”). We use this list for the text-to-image experiments with some modifications. Specifically, we remove the similar occupations (*e.g*., remove civil engineer as there is engineer). Additionally, we remove occupations where the text-to-image model [] can not generate images with humans or it tends to generate multiple individuals.

Activities: arranging flowers, playing tennis, playing skateboarding, playing baseball, playing soccer, playing football, playing snowboarding, playing skiing, cleaning, dressmaking, tying tie, smiling, crying, laughing, cooking, making pizza, dancing, drinking beer, drinking wine, eating hotdog, eating cake, using computer, playing game, gardening, singing, petting dog, petting cat, makeup, shopping, playing piano, playing guitar, carrying baby. For activities, we use a subset of the Kinetics dataset [] with some additional activities. For the prompt templates, we use “This is the face of a person who likes {}”, “ This is a photo of a person who likes {}”, “This person likes {}”, “A person who likes {}”, “Photo of a person who likes {}”, “This is a person who likes {}”.

## Appendix 0.E Additional visual examples for image generation experiments

We show additional visual examples for Figures 3 and 4 in the main paper in Figures 5 and 6, respectively.

Figure: Figure 5: Generated images for the prompt, “A photo of a teacher,” by the original Stable Diffusion (SD), projection-based debiased CLIP (Projection), and our debiased CLIP (SANER). We randomly sample $10$ images from generated images. Images framed in green denote those of the minority gender in the generated images (*i.e*., male).
Refer to caption: x5.png

Figure: Figure 6: Generated images for the prompt, “A photo of a male musician,” by the original Stable Diffusion (SD), projection-based debiased CLIP (Projection), and our debiased CLIP (SANER). We randomly sample $10$ images from generated images. Red frame indicates images with incorrect gender (*i.e*., female).
Refer to caption: x6.png

## Appendix 0.F Potential Negative Impact

Applying SANER to debias CLIP may lead to a potential negative impact where users might overlook remaining biases, assuming the process to be fully effective. While SANER performs better in gender and age bias mitigation, as evidenced by the MaxSkew and statistical parity metrics, this does not ensure that SANER mitigates all possible societal biases, and there could be dimensions of bias not adequately measured by these metrics. It is crucial to acknowledge that SANER, though impactful, is not an all-encompassing solution for removing societal bias. We notice that researchers must exercise due diligence in evaluating the application of SANER to avoid inadvertently introducing unanticipated biases.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - The caption in Figure 2 should contain ideally an explanation of what is seen in the figure. At least a reminder of the loss terms would be helpful to avoid having to jump back and fourth through the paper.

- How does Saner apply to non-binary gender? Does it properly use general pronouns like they?

- Conceptually, it seems hard to draw a line between terms related only to the gender and to the gender and a concept. Should de-biasing replace terms like pregnant? There is no male counterpart of this so replacing it will probably remove important information.

- Is the "multilayer perception" in line 81 supposed to be a multilayer perceptron?

- The definition of a dataset in Line 104 seems odd in the sense that, if a is a protected attribute then this dataset should only contain data with protected attributes. Is that intended? I would imaging the goal is to train on a large dataset which can have samples with protected attributes but not all samples have to have protected attributes?

- One added sentence for 2.2 on how a orthogonal projection is helping debiasing would be helpful for understanding.

- How hard is it to re-train clip? When the semantics are lost the regularization loss may be high. How computationally heavy is it to get to the same or a similar performance level?

- Table 2's caption should probably read "racial bias" and not recial bias.
soundness: 3
strengths: The paper is very clearly written. Whenever an uncertainty comes up about a term footnotes help understanding.

The analysis of debiasing techniques is very clearly presented that it could be used for a tutorial. The experimental setup is well done, the evaluation metrics such as measuring the difference between a uniform distribution and a potentially gender biased image generation seems adequate for the task. Showing generated images, i.e. with Stable Diffusion shows also a very immediate practical application of this research.

The figures help understanding of the method even though the caption could be changed to make them more self-complete.

In general, this seems a very well written, easy to understand paper addressing a very current need.
summary: This paper aims to overcome societal bias present in datasets used to train large scale multi-modal models like CLIP. The authors provide a study of debiasing methods and explain how these struggle with loosing important attribute information, e.g. gender, or depend on attribute annotations which are hard to get and even harder to get in the form of a large diverse dataset. Their contributed method Saner has four components which together aim to remove biases: attribute neutralization, feature modification and attribute annotation free debiasing and regularization losses. The paper compares the results on quite a few tasks such as text-to-image generation, text-to-image retrieval and zero-shot image classification.
weaknesses: The attribute groups will be limited to a specific set of defined attributes, which need to be agreed upon and could be debatable, e.g. "pregnant" (If I understand list C in the appendix correctly). There may also be missed attributes. This does not seem like a big issue but could be one in an adversarial setting. 

In general, the benefits of not needing a annotated dataset with attributes seems to be bought by needing a general list of attributes. This may not be a weakness per se but it would be interesting to either read about it in the limitations or have the author comment on why this is not a limitation.

There are general implementation details about the used architecture but none about the training process. Re-training clip with a loss to keep it's performance while having a re-projection that changes the semantics seems complex. Information about the used GPU would help understand how hard training for 5 epochs is.

Comparison against other state of the art in this small field is limited but understandably so and the author's did a good job re-implementing relevant approaches.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see the weaknesses.
soundness: 2
strengths: - The problems of existing methods are straightforward, and the authors conduct several experiments to verify these phenomena.
- The proposed method has a good performance.
- The experiments are conducted on both text-to-image retrieval and generation tasks.
summary: This paper aims to address societal biases in large-scale vision-language models, such as CLIP. The authors identify two limitations of existing methods: the loss of attribute information and the reliance on attribute annotations. To overcome these challenges, the authors propose SANER, a simple yet effective debiasing method for CLIP that incorporates attribute neutralization and an annotation-free debiasing loss. Extensive experiments are conducted to verify the method's effectiveness.
weaknesses: - Could the authors provide ARL results to verify the loss of attribute information? Besides, do other existing methods, such as Mapper and Prompt tuning-based debiasing, also have the lossing attribute information when debaising?
- Although effective in debiasing CLIP, the proposed method appears ad-hoc for this specific task and lacks major technical contributions, as most components seem to be existing technologies or tricks. Therefore, this paper may not have significant influence or provide broader insights beyond the CLIP-debiasing task.
- Why does adding only contrastive losses have a significantly negative impact on FairFace performance (as shown in Table 7)?
- Is the proposed method only applicable to CLIP? Could the authors test other VLMs to demonstrate its broader effectiveness?

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weaknesses.
soundness: 2
strengths: The two identified challenges, especially the loss of attribute information, sound critical and interesting. It seems like the pipeline only involves lightweight training, as only the debiasing layer is trained. The pipeline is evaluated on two different downstream tasks.
summary: The paper proposed a CLIP debiasing pipeline that aims to address two existing challenges: 1. loss of attribute information and 2. Dependency on attribute annotations. Specifically, the paper focuses on adjusting the text feature by replacing the protected attributes with attribute-neutral terms and further regularizing the adjusted feature by two reconstruction losses. The pipeline is evaluated on Text-to-Image retrieval and generation tasks.
weaknesses: 1. Although replacing the protected attribute words with an attribute-neutral word is a sound neutralization method, it requires a comprehensive list of the protected attributes (as listed in Appx. C). However, in real practice, and especially in non-binary cases, it will be very hard or nearly impossible to have a complete list. Further, since the "attribute annotation-free debiasing loss" relies on a set of attribute-specific descriptions, creating the attribute-specific descriptions set may lead to some concerns. For example, if the set is created by iterating all possible words and all possible groups (e.g., man, woman, male, female, he, she, etc), then the size of the set will be overlarge and increase the computation complexity. On the other hand, if only one word is selected for each group (e.g., man + female, woman + male), then how to select matching words from groups may also affect the debias performance. Also, is it possible that the generated description may have unmatched text (e.g., A woman/boy is eating salad) or text that has grammar errors (e.g., A he/she is eating salad, according to Appx. C). I would like to see more discussion on the text generation. 

2. The proposed pipeline tried to address the challenge by combining the debiasing loss with two extra regularization losses. The debiasing loss tries to place the adjusted feature of the attribute-neutralized text in the middle point of several features of the attribute-specific text, which seems to be fine. However, the regularization losses try to project the attribute-neuralized feature to both text and visual features of an attribute-specific feature, which seems strange. As demonstrated in Figure 2, the input to the "Text encoder" has been neutralized, and the attribute-specific information has been removed. Further, the debiasing layer is also trained to adjust the features to be further neutral. Thus, even there is a residual connection in the "Feature modification" process, the output feature should not contain any information regarding the specific attribute. Then, how is it possible that such a feature can be mapped to either text or visual features that contain specific attribute information to avoid "the loss of attribute information"? Is it possible that the "retained attribute information" is some biased information from the dataset that is implicitly polluted by the regularization loss?

3. For the evaluation of the text-to-image generation, the paper only provides numerical results on debiasing performance and some visualizations but does not provide any image generation imperial results, like FID. Also, only a few SOTA are used as the baseline, which makes the comparison weak. However, this may be understandable as several baselines do not have public implementation, and it seems like the paper prepares all the baseline results.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: Have you considered running experiments with LLaVA-like models? I don't think it would be terribly difficult to replace the clip model with the proposed one, run standard llava experiments (there are a few githubs that are easy to setup for such eval), and report the results. As for debiasing experiments, I've seen people use PATA and FairFACE and just look at the next token predicted to get similar scores as the CLIP style. Showing this CLIP replacement method works on text generation as well as image generation would very much increase the impact of the work, in my opinion. However, I realize this experiment may best be left for future work.
soundness: 3
strengths: I will start off by saying that the lede of the paper was buried: section 5 as an experiment is astounding. I absolutely love the drag and drop replacement of this debiased model into an existing model. The idea that, 1. you don't have to retrain the text-to-image model and 2. that you don't have to retrain CLIP from scratch is incredibly compelling. I recommend the abstract/intro be rewritten to highlight this experiment.

On another note, the paper is generally well-written and easy to follow. The problem of bias in VLMs is highly prominent and this work makes an excellent contribution to the field. I am fairly familiar with this field of debiasing clip and I feel section 2.1 does a good job summarizing the various papers.
summary: The paper proposes a method for finetuning a CLIP model to remove social bias by an intuitive training scheme.
weaknesses: I don't have too many issues with the paper. One thing I would like to have seen is a more automated method for text neutralization. While word level ablation is intuitive and makes sense for the social bias angle, a less hardcoded approach seems like it would be needed for a general-purpose debiasing solution, especially for concepts that are hard to ablate at the word level.

There's a dataset called FACET that addresses some issues with PATA and FairFace. If there's time, it would be great to run experiments on it as well. This would cement the proposed method as useful and cutting edge in my opinion.

Meta-review: This paper focuses on the societal bias issue in large-scale vision-language models and conducts a study of debiasing methods to present the limitations of existing methods. To overcome the societal biases, the authors propose a simple yet effective debiasing method SANER to incorporate attribute neutralization and an annotation-free debiasing loss. Extensive experiments are performed to evaluate the effectiveness of the proposed method on different tasks. Two reviewers give high rating scores and two reviewers give positive rating scores. Based on the above considerations, I recommend to accept this manuscript.

Decision: Accept (Poster)