Fair Federated Learning with Biased Vision-Language Models (2024)

Abstract: Existing literature that integrates CLIP into federated learning (FL) largely ignores the inherent group unfairness within CLIP and its ethical implications on FL applications. Furthermore, such CLIP bias may be amplified in FL, due to the unique issue of data heterogeneity across clients. However, in identity-sensitive FL applications, model fairness (i.e., group fairness) is imperative for model development. Therefore, this work explores a critical question ignored by the existing literature: how can we build a fair FL framework using biased pre-trained VLMs (e.g., CLIP)? To address this problem, we propose a fairness-aware adaptation framework tailored for VLM (e.g., CLIP) in the context of FL, named Fair Federated Deep Visiual Prompting or FF-DVP. As implied by its name, FF-DVP trains a fair FL model with fairness-aware deep visual prompting (DVP). Moreover, FF-DVP incorporates modality-fused classification heads to learn client-specific knowledge and fairness constraints. These modules explicitly address a unique kind of bias in FL, namely the bias triggered by data heterogeneity. We show that FF-DVP can be readily extended to prevailing parameter-efficient fine-tuning methods (e.g., adapter or LoRA) for debiasing purposes. To the best of our knowledge, FF-DVP is the first to leverage biased VLMs for building fair FL frameworks. Extensive results on human face attribute recognition (FAR) applications suggest that FF-DVP effectively improves model fairness and training convergence, outperforming state-of-the-art baselines.

heading: 1 Introduction
text: Federated learning (FL) emerges as a novel machine learning (ML) paradigm wherein ML models are trained from distributed data sources (McMahan et al., 2017). In FL, a central server stores a global model, while multiple local clients participate in the collaborative model training without sharing their private data. Such a decentralized
design of FL makes it a promising solution for privacy-sensitive applications, like online facial services or medical diagnosis (McMahan et al., 2017). However, FL models commonly suffer from scalability issues. That is, traditional FL models struggle to achieve training convergence under high data complexity (e.g., data amount, data dimensionality) and high data heterogeneity (e.g., non-i.i.d. data) (Zhou et al., 2022). Therefore, we aim to design a scalable fair FL framework that withstands high data complexity and heterogeneity.
On the other hand, large foundation models become increasingly popular for ML tasks involving complex and large-scale data. These models, with billions parameters, are pre-trained using Internetscale data (Chuang et al., 2023). They can extract domain-generalized features from inputs, and generalize to various downstream tasks across different domains (e.g., sentiment analysis, image classification (Bommasani et al., 2021)).
Recently, exploiting large vision-languagemodels (VLMs) for federated learning (FL) has gained increased attention (Lu et al., 2023; Yang et al., 2023; Guo et al., 2023a; Chen et al., 2023). The strong generalization ability of VLMs has empowered FL models to overcome the data complexity and data heterogeneity across clients, leading to better personalization and generalization (Lu et al., 2023). However, existing literature that combines VLMs (e.g., CLIP) and FL largely neglects the inherent bias within such VLMs and its ethical implications on FL applications. The unique issue of data heterogeneity in FL would exacerbate CLIP bias, severely compromising the fairness of the aggregated FL model (Chang and Shokri, 2023). On the other hand, in privacy- and identity-sensitive applications, it is imperative to develop FL models that are both accurate and fair. For instance, in an FAR-based criminal detection system, if the model consistently makes false-positive predictions on a specific demographic group, great ethical concerns
10002
would be raised (Najibi, 2022). Unfortunately, as shown in Figure 1, pre-trained VLMs indeed encode the bias of their pre-training data, and may perpetuate such bias in downstream tasks (Chuang et al., 2023). Additionally, in FL applications, to transfer a pre-trained VLM to the domain-specific data, it is common to fine-tune the model on the clients’ data and then update the global model via aggregation. In this case, if the local data distributions are biased and heterogeneous, fine-tuning a biased VLM on such local data would further amplify the bias of the global model. The bias triggered by data heterogeneity in FL represents a unique challenge, whereas there exist no fair methods tailored for VLM-based FL frameworks. Therefore, this work studies a critical question: how can we develop a fair FL framework using the biased pre-trained VLMs?
In this work, we focus on the fairness of the FL model w.r.t. the unknown global data distribution. Ideally, a fair FL model shall perform nondiscriminatively against any demographic group (i.g., group fairness) while achieving satisfactory overall performance. To achieve this, we propose a fairness-aware adaptation framework tailored for VLMs in the context of FL: Fair Federated Deep Visiual Prompting or FF-DVP. We highlight that FF-DVP is specifically designed to mitigate the bias in FL settings: 1) FF-DVP minimizes the communication burden between the server and clients: given the large size of CLIP, FF-DVP is lightweighted in terms of communication costs in FL; 2) FF-DVP performs demographic-agnostic and domain-generalized feature extraction: it removes sensitive demographic information from CLIP features while retaining domain-generalized ones to counter data-heterogeneity-triggered bias in FL;
3) FF-DVP learns client-specific knowledge and fairness constraints: FF-DVP adapts CLIP to learn client-specific knowledge for targeted tasks (e.g., FAR applications) and meets client-specific fairness constraints (e.g., demographic parity).
To this end, FF-DVP consists of two debiasing modules. Firstly, FF-DVP debiases the CLIP features through fairness-aware deep visual prompting (DVP). The fairness-aware DVP is a sequence of learnable parameters prepended to the visual tokens. This DVP is designed to remove the demographic-related information from CLIP features while preserving the domain-generalized information. The extracted fair, domain-generalized feature overcomes the unique data-heterogeneitytrigger bias in FL. Secondly, FF-DVP learns clientspecific knowledge and fairness constraints with modality-fused classification heads. The classification heads also contribute to the learning of fair and robust representations. With both modules, the aggregated global model achieves desired group fairness despite the data heterogeneity in local clients. Finally, we show that FF-DVP is flexible and can be extended to other parameter-efficient fine-tuning (PEFT) methods, such as adapter-style tuning (Houlsby et al., 2019) or low-rank adaptation (LoRA) (Hu et al., 2021). We summarize the contributions of our paper as follows1:
1. To the best of our knowledge, this work is first to study inherent bias of the pretrained VLMs in FL applications. Compared to centralized debiasing methods, we focus on the unique data-heterogeneity-triggered bias in FL, and proposed a scalable fair VLM-based FL framework.
2. Technically, FF-DVP debiases the pre-trained VLM (e.g., CLIP) through a novel fairnessaware deep visual prompting. Moreover, as a parameter-efficient adaptation method, we show that our method could be easily extended to other PEFT schemes.
3. We evaluate our method on federated face attribute recognition (FAR) for its privacy and ethical implications. On different FAR applications, experimental results suggest that FF-DVP can effectively improve fairness of FL models, outperforming the state-of-the-art baselines by a significant margin.
1We adopt publicly available datasets and will release the code upon acceptance.

heading: 2 Related Work
text: Foundation Models and Federated Learning. Large foundation models (e.g., GPT family (OpenAI, 2023), LlaMA (Touvron et al., 2023), CLIP (Radford et al., 2021)) exhibit robust generalization capabilities across a wide spectrum of tasks. Recently, efforts have been made to integrate foundation models (e.g., CLIP) into FL frameworks for better personalization and generalization (Lu et al., 2023; Yang et al., 2023; Guo et al., 2023a; Chen et al., 2023). For instance, Guo et al. (2023a,b) focus on learning text prompts to personalize CLIP on client data, whereas Li et al. (2023) leverage visual prompts for the same goal. In addition to prompt learning, Lu et al. (2023); Chen et al. (2023) fine-tune CLIP with light-weighted adapters to adapt CLIP to the personalized data on clients. However, existing literature that combines CLIP and FL merely focuses on accuracy, largely overlooking the inherent bias of CLIP and its impact on fair FL applications. In contrast, our work is the first to consider the bias issue of CLIP in FL and leverage biased CLIP to build fair FL frameworks.
Group Fairness in Federated Learning. For identity-sensitive applications, concerns have been raised about the fairness of the FL models (i.e. group fairness): they could perform discriminatively against under-represented demographic groups (Ezzeldin et al., 2023). In terms of group fairness in FL, Cui et al. (2021) strive to satisfy the local personalized fairness on each client. In comparison, a larger amount of studies (including our work) focus on a more general notion of global fairness, for its profound significance in real-world applications Mohri et al. (2019); Du et al. (2021); Ezzeldin et al. (2023); Hong et al. (2021). For in-
stance, agnostic federated learning (Mohri et al., 2019) achieves the good-intent fairness that protects the worst-case performance on any client. A fairness-aware aggregation protocol is introduced in (Ezzeldin et al., 2023) to obtain a fair global model. However, above methods usually suffer from the issue of scalability: in the era of large foundation models, the performance of traditional fair FL methods might degrade significantly when the data is complex and heterogeneous (Zhou et al., 2022). In this work, we present the very first framework that leverages large foundation models (e.g., CLIP) to address the bias issue in FL.

heading: 3 Preliminaries
text: Federated Learning. Assume there are K clients in an FL application. For all datasets, each data point consists of an input feature x ∼ X , a demographic attribute a ∼ A and a label y ∼ Y . A local dataset of client k is denoted as D(k) = {(x(k)1 , a (k) 1 , y (k) 1 ), ...}. For simplicity, if not specified, we use the notations without client index k to represent the data of an arbitrary client.
To find the optimal global model f∗θ in an FL application, McMahan et al. (2017) proposed FedAvg (Appendix A). Specifically, at each round, each local client trains its local model with its own data. Then, clients send the trained local model weights to the central server for aggregation. On the central server, the global model will be updated using a weighted-average of the received weights. However, if the local data distributions are imbalanced, the locally trained models are also biased. This eventually leads to an unfair global model after the aggregation (Ezzeldin et al., 2023).
Fairness Notions. To measure the model fairness and performance, we adopt the commonly used demographic parity Φdemo, equalized odds Φeq (Hardt et al., 2016), accuracy parity ΦA (Makhlouf et al., 2021) and balanced accuracy AB (Brodersen et al., 2010). Due to space limit, we have summarized the formal definition of fairness notions (i.e., Φdemo, Φeq, ΦA and AB) in Appendix A. CLIP. CLIP predicts which images are paired with which text prompts. We use fI to denote the CLIP image encoder, and fT for the CLIP text encoder. For a query image x and a list of candidate prompts for |Y| classes (e.g., a photo of [class c]), CLIP selects the candidate prompt with the highest normalized cosine similarity to x as the predicted class (more details in Appendix A):
ŷ = argmax c
exp ( cos(z, tcandidatec)/τ ) ∑
c′ exp ( cos(z, tcandidatec′ )/τ ) ,
where z = fI(x),
tcandidatec = fT (a photo of [class c]), c ∈ {1, 2, ..., |Y|}. (1)
For instance, in smiling detection (Figure 1), for a query image, CLIP selects either “a smiling person” or "an unsmiling person" as its prediction based on similarity. However, due to the inhere bias of CLIP (Chuang et al., 2023), directly integrating CLIP into FL results in a biased FL model.

heading: 4 Algorithm

heading: 4.1 Fairness-aware Deep Visual Prompting
text: To exploit CLIP for a fair FL model, we propose to firstly debias CLIP on each client. Specifically, we present fairness-aware deep visual prompting (DVP) that suppresses the demographic-related signals in CLIP features and preserves domaingeneralized ones. For a better understanding of our fairness-aware DVP, we expand the image encoder fI into L layers (Figure 3 (left)). As in (Jia et al., 2022), at the input layer, CLIP divides a query image x into J fixed-sized image patches {I1, I2, ...Ij , ..., IJ |Ij ∈ R3×h×w}, where the size of each image patch is h× w. Note that the index of image patches j is different from the index of samples i defined in Section 3. The image patches are then embedded into embeddings using the embedding layer (i.e., layer 0) of fI :
e0,j = fI,Embed(Ij), e0,j ∈ Rd, j ∈ {1, 2, ..., J} (2)
After the embedding layer, the transformer layers of fI then extract visual features from the image embeddings using different transformer layers. In particular, at the l-th transformer layer of fI :
[el,0, el,1, ..., el,J ]
= fI,Transformerl([el−1,0, el−1,1, ..., el−1,J ]), (3)
where l ∈ {1, 2, ..., L} and [·, .., ·] represents the concatenation operation along the dimension of sequence length. For simplicity, we use El to denote [el,1, ..., el,J ]. Thus, Equation 3 is simplified as:
[el,0, El] = fI,Transformerl([el−1,0, El−1]). (4)
Note that in Equation 4, there is usually a [CLS] token prepended to the visual tokens. Thus, at the output layer, the first feature (corresponding to [CLS]) is used as the classification token to compute the cosine similarity: z = eL,0.
To address the bias issue of CLIP, we insert fairness-aware visual prompts V = {p ∈ Rd} at the embedding layer and the transformer layers of fI as in Figure 3 (right). At the embedding layer, the prompted image embeddings are formulated as: [ e0,0, V0, E0 ] =
[ e0,0, p0,1, ..., p0,P ,︸ ︷︷ ︸
prompts
e0,1, ..., e0,J︸ ︷︷ ︸ embeddings
] ,
(5) where P is the length of visual prompts, and V0 is the visual prompt for the embedding layer. Similarly, for the transformer layers, the forward pass of each layer is formulated as:
[ẽ1,0, _, Ẽ1] = fI,Transformerl([e0,0, V0, E0])
[ẽl,0, _, Ẽl] = fI,Transformerl([ẽl−1,0, Vl−1, Ẽl−1]), (6)
where ẽ and Ẽ are prompted representations, and l ∈ {2, ..., L}. Therefore, at the output layer of the prompted fI , the extract visual representation z is also prompted:
z̃ = ẽL,0. (7)
The intuition of debiasing CLIP is to mitigate the demographic-related information within z̃. For instance, as in Figure 1, if the goal is to debias CLIP for smiling detection w.r.t. gender, z̃ should not encompass information that could infer the gender of the query image.
To debias CLIP w.r.t. the demographic attribute a, we propose to construct a set of demographiconly prompts that depict the demographic information of the input images (Figure 4a). As before, CLIP encodes the demographic-only prompts into representations: {ta1 , ta2 , ..., ta|A|}. With encoded demographic-only text prompts and the prompted visual representation z̃, we then propose a fairnessloss to debias the CLIP feature for an input x: lfair(x) = KL(P̂ r(A) ∣∣∣∣U(1, |A|)),
where
P̂ r(A) = Softmax (cos(z̃, ta1)
τ , ...,
cos(z̃, ta|A|)
τ
) ,
U(1, |A|)) = [ 1|A| , ..., 1
|A| ]. (8)
Equation 8 computes the KL-divergence between the normalized cosine similarities P̂ r(A) and a uniform distribution U(1, |A|). Each element in P̂ r(A) measures the relevance between the prompted visual representation z̃ and a demographic-only text prompt (e.g., a photo of a [male].). By minimizing Equation 8, all demographic-only text prompts become equally relevant to z̃. This indicates that CLIP can no longer distinguish which demographic group is more related to z̃ than others, thereby, debiasing CLIP.
In addition to fairness, we also propose to optimize a contrastive loss using the groundtruth prompts to maintain an overall high performance (Figure 4b). Formally, for training dataset
D = {(xi, ai, yi)}, we construct the ground truth prompts for xi that contains textual description of yi. Then, we compute the CLIP contrastive loss over the prompted visual representation and the prompts as in (Radford et al., 2021):
LCLIP = 1
|D|
|D|∑
i=1
− log e z̃i·tgti
∑|D| j=1 e z̃i·tgtj
+ 1
|D|
|D|∑
i=1
− log e z̃i·tgti
∑|D| j=1 e z̃j ·tgti ,
(9)
where z̃i is the prompted visual representation (i.e., Equation 7) and tgti = fT (ground-truth prompt of xi) is the encoded ground-truth prompt. To adapt CLIP into task-specific data and suppress the inhere bias of CLIP, Equation 9 and Equation 8 are optimized jointly, balanced via a non-negative factor λ1:
LCLIP + λ1 ∗ 1
|D|
|D|∑
i=1
lfair(xi) (10)
Finally, we highlight that FF-DVP is adjustable, allowing users to determine which transformer layers to prompt or not (i.e., not necessarily prompting all layers). We experimented this design by intervening only the first layer, first half layers and all layers of the CLIP image encoder in Section 5.3. We found that more intervened layers would generally improve the performance.

heading: 4.2 Client-Specific knowledge and Fairness
text: In FL, it is not always feasible to collect Internetscale image data and text data on each client to finetune CLIP. Such data sparsity leads to training instability and performance degradation. Therefore, we further propose to build a shared lightweighted modality-fused classifier fcls (i.e., a two-
layer fully connected network) to stabilize the training of CLIP on clients and help CLIP learn robust latent representations (Appendix B).
In our FL setting, each client fine-tunes CLIP as well as trains this classifier to the FAR application using the client’s local data. The classifier takes the prompted visual representation and the text representation as input. It then fuses the features from both data modalities:
ŷ = fcls([Π(z̃),Π(tgt)]), (11)
where z̃ is the prompted visual representation of x, and tgt = fT (ground-truth prompt of x) is the encoded ground-truth prompts. [·, ·] represents the concatenation operation. In Equation 11, we introduce a learnable projection matrix Π to reduce the dimensionality of z̃ and tgt. To train fcls, we optimize the cross entropy loss as well as a fairness regularizer Φ̂. over the training set on each client:
1
|D|
|D|∑
i=1
lce(ŷi, yi) + λ2 · 1
|D|
|D|∑
i=1
Φ̂.(xi, yi, ai).
(12) The rationale behind this design is that: despite the debiasing efforts on CLIP features, the ultimate classification performance of the model does not necessarily fulfill the fairness constraints on local clients. Therefore, training fcls on local data contributes to learning client-specific knowledge and fairness constraints. This design improves the fairness of the model in terms of the ultimate prediction performance after the global model aggregation. The fairness regularizers Φ̂. could be implemented flexibly using arbitrary differentiable version of fairness notion defined in Section 3.

heading: 4.3 Overall Framework
text: Training. The fairness-aware DVPs and clientspecific classifiers are jointly trained in a standard FL fashion (i.e., local training and global aggregation). In our implementation, FedAvg is used for aggregation. The pseudocode of the training pipeline is summarized in Algorithm 1 (Appendix C). Finally, we highlight that the weights of CLIP encoders are not updated and exchanged. FF-DVP is a parameter-efficient method.
Inference. During the inference phase, we only use the prompted CLIP image encoder f̃I and the original text encoder fT to perform inference for the query images. fcls is dropped on purpose because fcls is only designed to help CLIP learn
client-specific knowledge and fairness constraints during training. Similar to Equation 1, the cosine similarity between prompted visual representations z̃ and candidate prompts is computed for making predictions.

heading: 5 Experiments

heading: 5.1 Experimental Setup
text: Dataset. We use CelebA (Liu et al., 2015) and FairFace (Karkkainen and Joo, 2021) to study different FAR applications in the context of FL. Due to space limit, we chose smiling and age as our predictive face attributes. As mentioned in (Shen et al., 2017), smiling detection is objective since smiling or not is easy to judge. In comparison, age detection is more challenging: it is formulated as a binary task of classifying ’young’ and ’old’, but both age groups exhibit a broad age range, causing a vague and hard-to-learn boundary. Finally, the age label is the only shared label in both datasets, which help us to test the generality of our method. Without loss of generality, we choose gender as the demographic attribute.
FL setup. During experiments, the training of some baseline methods could not converge under the high data complexity and data heterogeneity of FAR applications. Therefore, for fair comparison, we compare all methods under a setting of 5 clients, where all baseline methods could converge. Moreover, for training convergence and computational efficiency, we downsample 20000 images from both datasets and distribute the samples images to the 5 clients. We explicitly control population shifts for all clients, so that the local training data distributions are imbalanced and non-i.i.d. Finally, to eliminate the potential bias in the test data distribution that could affect the fairness evaluation, we sample a balanced test set of size 5000 to evaluate the FL model. More implementation details (i.e., local data distribution configuration, prompt design, hyperparameters, CLIP version) are summarized in Appendix D.
Baseline algorithms and models. Due to the lack of baselines that explore pre-trained foundation models for fair FL, we select FedAvg (McMahan et al., 2017), AFL (Mohri et al., 2019), FairFed (Ezzeldin et al., 2023), and FADE (Hong et al., 2021) as the baselines for comparison. Moreover, to better demonstrate the necessity of using pretrained foundation models, we adopt a relatively
smaller model, ResNet-18 (pre-trained on ImageNet) to run the selected baseline methods.

heading: 5.2 Evaluation: Fairness and Accuracy
text: Firstly, we compare FF-DVP with the baselines methods and present evaluation results in Table 1. Following (Lu et al., 2023), the experiments are repeated for 3 times. We observe: (1) the proposed FF-DVP effectively reduces the demographic bias of the FL model compared to all baseline fair FL methods. For instance, compared to CLIP (zeroshot), the model bias is reduced by approximately 87% w.r.t. Φeq on smiling detection; (2) FF-DVP achieves better fairness without necessarily sacrificing the accuracy. This is expected because our test dataset is subsampled from the original dataset and is balanced. After deploying FF-DVP, the FL model’s performance on the minority group would be improved, leading to a natural increase in overall accuracy with the balanced test dataset. This indicates that the fairness-accuracy trade-off of FFDVP is less pronounced. (3) FF-DVP achieves better training convergence than the baseline methods. Compared to baselines, FF-DVP achieves performance improvements in terms of both accuracy and fairness, whereas barely can traditional FL methods converge. We attribute the success of FF-DVP to the pre-trained foundation model as well as our novel fairness-aware adaptation strategy.

heading: 5.3 Fairness-aware PEFT
text: Next, we show that FF-DVP can generalize to other parameter-efficient fine-tuning (PEFT) methods, including adapter-style fine-tuning and Low-Rank Adaptation (i.e., LoRA). For the adapter-style fine-
tuning, we combine FF-DVP with the attentionbased adapter proposed in (Lu et al., 2023) (abbreviated as FF-ADP). As for LoRA, we combine FFDVP with LoRA (abbreviated as FF-LoRA), and set the LoRA rank as 8, considering the communication cost of FL. As shown in Table 2, under the same FL setup, our debiasing strategy could still be effective in terms of debiasing FL models for both adapter-style fine-tuning and LoRA. Furthermore, we note that the number of intervened layers can influence the debiasing performance. For instance, FF-ADP only intervenes the output layer of the image encoder, which makes FF-ADP perform worse than FF-DVP and FF-LoRA. In comparison, FF-LoRA intervenes in all layers of the image encoder, and FF-LoRA generally achieves better performance than the others. As such, we highlight that our method is adjustable: FF-DVP allows the users to specify which transformer layers to debias, taking into account application-specific factors such as communication cost or dataset size.

heading: 5.4 Ablation Study
text: We conduct an ablation study to evaluate the contribution of each key module of FF-DVP, namely the fairness-aware DVP and client-specific classifiers. The results are reported in Table 3. As expected, we observe that it is necessary to use contrastive loss to fine-tune the deep visual prompts. For instance, without the contrastive loss (as well as Lfair), the adaptation basically fails. In comparison, fcls indeed contributes to the fair representation learning of the model. For instance, the model fairness w.r.t demographic parity and equalized odds increase without using fcls, indicating the contribution of fcls to fair representation learning.

heading: 5.5 Scalability Study
text: We further study the scalability of FF-DVP w.r.t. the number of clients. We exclude the results of baseline methods in Figure 5, because baseline methods could barely converge. They make predictions randomly and achieve almost perfect but trivial fairness. We keep scaling up the number of clients until the training time exceeds the limit of a week. Moreover, we increase the number of clients but still control the group shifts to simulate non-i.i.d. heterogeneous and imbalanced data distributions (Appendix D). The results on smiling detection are visualized in Figure 5. We observe that, under a larger number of clients, our method could still converge and achieve similar or better fairness than the CLIP zero-shot performance.

heading: 5.6 Robustness Study
text: We finally study the relationship between the performance of FF-DVP and the length of visual prompts P . This hyperparameter is directly related to the communication cost in FL, as the visual prompts are shared and updated during the training process. In Table 4, we observe that there exists an
optimal length of 20 to use the deep visual prompting. With shorter prompts, the expressive power of the model is reduced, indicating both debiasing and adaptation process is under-fitting the data. In comparison, with more tunable tokens, the training becomes slower and unstable. The instability is caused by the scarcity of the textual modality: there is insufficient amount of text data to train such a large model. For instance, in FairFace age detection, the FL model only achieves 78.5% accuracy with even 50 visual tokens. This observation indicates that more visual tokens will not necessarily improve the performance.

heading: 6 Conclusion
text: This work presents a novel fair FL framework using biased vision language models. FF-DVP provides an effective solution to develop fair ML models while protecting data privacy on users’ end. We highlight the significance of this work: despite the promising integration of large foundation models with FL applications, existing literature largely overlooks the inherent bias of these foundation models. In contrast, our work is the first to address the inherent bias of these foundation models and their bias implication on FL applications. Finally, in addition to the performance of FF-DVP, we show that our method could easily extended to other PEFT methods for the adaptation of VLMs.

heading: 7 Limitations
text: One limitation of this work is that our method introduces extra hyperparameters. For different applications, one might need to finetune these hyperparameters, which brings extra computational cost, such as the trade-off factor λ and number of tunnable visual tokens. As for the actually trainable modules, there is only a small two-layer network and light-weight perturbations. Another limitation of this work is that our method only focuses on the bias, whereas CLIP has encoded other ethicsrelated issues (e.g., stereotypical data, racism and hate speech). Such malicious contents could have negative ethical implications on downstream FL applications as well. Therefore, a future research direction is to develop a benign, fair FL framework.

heading: Acknowledgement
text: This research is supported in part by the National Science Foundation under Grant No. IIS-2202481, CHE-2105032, IIS-2130263, CNS-2131622, CNS2140999. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.

heading: A Additional Prelinminaries
text: A.1 Federated Averaging
To find the optimal global model f∗θ in an FL application, McMahan et al. (2017) proposed Federated Averaging (FedAvg). Specifically, at each round, each local client firstly receives a copy of the global model fθ from the central server and trains the model with its own data. All clients will then obtain different local models (fθ(1) , fθ(2) , ...fθ(K)), and send the trained model weights to the central server. Next, on the central server, the global model will be updated using a weighted-average of the received local model weights. Formally, FedAvg operates as follows:
θ∗ = 1∑
k |D(k)|
K∑
k=1
|D(k)| · θ(k)
s.t. θ(k) = argmin θ 1∑ k |D(k)|
|D(k)|∑
i=1
l(fθ(x (k) i , y (k) i )
k ∈ {1, ...,K}.
(13)
A.2 Fairness Notions
Definition 1 (Demographic Parity Φdemo (binary case)). For a classifier fθ, demographic parity Φdemo is defined as:
Φdemo(·) = |Pr(fθ(X) = 1|A = 0)− Pr(fθ(X) = 1|A = 1)|. (14)
Furthermore, we also use fairness-aware accuracy metrics to measure the model’s performance as in Yue et al. (2022), namely sub-group accuracy gap and balanced accuracy.
Definition 2 (Sub-group Accuracy Asub). For a classifier fθ, the sub-group accuracy Asub is the accuracy on a specific demographic group characterized by A and Y .
Asub(·) = Pr(fθ(X) = Y |A = a) (15)
Definition 3 (Accuracy Parity ΦA). For a classifier fθ, the accuracy parity ΦA sums up the absolute error among all demographic groups.
ΦA(·) = ∑
a
∑
y
∑
a′
∑
y′ |Asub(a, y)−Asub(a′, y′)| (16)
Definition 4 (Balanced Accuracy AB). For a given classifier fθ, the balanced accuracy computes the averaged sub-group accuracy for all demographic groups.
AB(·) = ∑ a ∑ y Asub(a, y)
|A| · |Y| . (17)
Since the model parameter is learned based on the training data distribution, local fairness could be highly heterogeneous across different local clients due to the discrepancy across different local data distributions.
A.3 CLIP Inference
CLIP is a large foundation model pre-trained with 400 million image-caption pairs. During pre-training, CLIP is trained to predict which images are paired with which texts. With such scale of pre-training data, CLIP is a powerful zero-shot image classifier and generalizes to different image classification tasks. To perform image classification, CLIP firstly encodes a query image and a set of text descriptions into latent representations. Next, CLIP computes the cosine similarity between the image representations and
text representations. To produce the final prediction, CLIP selects the text with highest cosine similarity among all texts as the final prediction.
Formally, we use fI to denote the CLIP image encoder and fT to denote the CLIP text encoder. For a query image x and |Y| classes, we firstly craft a set of candidate prompts that contain class information (e.g., { a photo of [class 1], a photo of [class 2]...}). Then, CLIP encodes x into z, and encodes the candidate prompts into representations {tcandidate1 , tcandidate2 , ..., tcandidate|Y|}, respectively. After computing cosine similarity between the image representation z and candidate prompt representations, CLIP selects the text with highest cosine similarity as the final prediction:
ŷ = argmax c cos(z, tcandidatec)∑ c′ cos(z, tcandidatec′ ) , where z = fI(x),
tcandidatec = fT (”a photo of [class c].”), c ∈ {1, 2, ..., |Y|}.
(18)

heading: B Modality-fused Classifier
text: The modality-fused classifier takes the prompted visual representation and the text representation as input. It then fuses the features from both data modalities:
ŷ = fcls([Π(z̃),Π(tgt)]), (19)
where z̃ is the prompted visual representation of x, and tgt = fT (ground-truth prompt of x) is the encoded ground-truth prompts. [·, ·] represents the concatenation operation.

heading: C Algorithm in Pseudocode
text: Algorithm 1: FF-DVP 1 Input CLIP image encoder fI , CLIP text encoder fT , classifier fcls and projection matrix Π,
datasets of local clients D1,D2, ...,DK , task prompts T = {T1, T2, ..., TC}, demographic-only prompts TA = {Ta1 , Ta2 , .., Ta|A|};
2 Hyperparameters learning rate η, trade-off factor λ1 and λ2, length of visual prompts P ; 3 Initialize visual prompts V of length P at the central server ; 4 Clients download fI and fT ; 5 for global epochs do 6 for k=1,2,...,K do 7 Receive trainable models: f (k)cls = fcls, V
(k) = V ; 8 for local epochs do 9 compute L1 = LCLIP + λ1 · 1|D| ∑|D| i=1 lfair(xi) ;
10 compute L2 = Lcls + λ2 · 1|D| ∑|D| i=1 Φ̂.(xi); 11 Update V (k) and f (k)cls with gradient descent; 12 end 13 Send V (k) and f (k)cls to the central server ; 14 end 15 Aggregate the received V (k)s and f (k)cls s ; 16 end 17 Output the prompted CLIP image encode f̃I ;
D Implementation Details
D.1 Non-i.i.d. Local Data Distributions We explicitly control population shifts for all clients, so that the local training data distributions are imbalanced and non-i.i.d. Below, we visualize the training data distribution of local clients.
D.2 Hyperparameters and CLIP Configuration For FF-DVP, FF-ADP, FF-LoRA, we select CLIP with configuration of ViT-L/14@336px to run experiments, and the learning rate is initialized as 1e-5. The models are optimized via AdamW. The local training epoch is 2 and the global epoch is also 2. For other baselines, including FedAvg, AFL, FairFed, FADE, we train the models by starting with the default training configuration and recommendations. However, we observed training divergence, and therefore, increased both local epochs (up to 30) and global epochs (up to 30). For all methods with key hyperparameters, we firstly performed grid search with the resolution of 0.1 until find the best performance. Based on that, we further reduce the search resolution to 0.01 until find best performance. Our hardware is NVIDIA A40.
D.3 Prompt Engineering As a necessary input for CLIP, we manually design sets of prompts for different FAR applications. To determine whether the prompts are informative, we firstly compare the zero-shot performance of CLIP on FAR applications with biased or unbiased prompts. A biased prompt contains sensitive demographic information in addition to the class information. On the contrary, an unbiased prompt only contains class information. To enable FF-DVP, we also craft the demographic-only prompts defined in Section 4.1. For better understanding, we provide exemplar prompts for the FAR applications in Table 5 and Table 6.
In our experiments, the biased prompts generally achieve better accuracy because they impose demographic information on the face image, which potentially helps CLIP to recognize the face image. Given the zero-shot performance, we use biased prompts in experiments. Therefore, in our experiments, the biased prompts are used as the textual input for CLIP. Note that even if the biased prompts contain demographic information, FF-DVP can still effectively debias CLIP.

heading: Future Work
text: Therefore, a future research direction is to develop a benign, fair FL framework