Year: 2021

Abstract:
A common approach for personalized federated learning is fine-tuning the global machine learning model to each local client. While this addresses some issues of statistical heterogeneity, we find that such personalization methods are often vulnerable to spurious features, leading to bias and diminished generalization performance. However, debiasing the personalized models under spurious features is difficult. To this end, we propose a strategy to mitigate the effect of spurious features based on our observation that the global model in the federated learning step has a low accuracy disparity due to statistical heterogeneity. Then, we estimate and mitigate the accuracy disparity of personalized models using the global model and adversarial transferability in the personalization step. We theoretically establish the connection between the adversarial transferability and the accuracy disparity between the global and personalized models. Empirical results on MNIST, CelebA, and Coil20 datasets show that our method reduces the accuracy disparity of the personalized model on the bias-conflicting data samples from 15.12% to 2.15%, compared to existing personalization approaches, while preserving the benefit of enhanced average accuracy from fine-tuning.

heading: 1 INTRODUCTION
text: Federated learning (FL) is a leading framework for clients to collaboratively train a shared global machine learning (ML) model without releasing their local private datasets (McMahan et al., 2017; Kairouz et al., 2019). The jointly trained global model could be further fine-tuned on each client’s local dataset to produce personalized models (Fallah et al., 2020; T. Dinh et al., 2020; Li et al., 2021). While existing theoretical and empirical results highlight how personalized models improve accuracy on local data, few works consider what features the personalized models learn from the local dataset. Our motivating hypothesis is that not all local features are beneficial.
For example, consider a gender prediction task using face images, where the ML model learns to predict gender based on hair color because females are more likely to have blond hair (Sagawa et al., 2020). In this case, the hair color is called a spurious feature because it only statistically correlates with the gender on biased-aligned samples but not necessarily on the overall population. Thus, the accuracy of a model that relies on spurious features such as hair color is likely to drop significantly on bias-conflicting samples where the spurious correlation does not hold, e.g., for blond male images (Sagawa et al., 2020). This paper calls the accuracy difference of an ML model on the dataset with spurious features and the dataset without spurious features accuracy disparity. More broadly, the accuracy disparity caused by spurious features leads to issues in both fairness (McNamara et al., 2019; Zhao & Gordon, 2019; Agarwal et al., 2019; Chi et al., 2021), i.e., racial bias (Khani & Liang, 2021) and robustness, i.e., accuracy decrease under distribution shift (Zhao et al., 2019; Koh et al., 2021). Compared to the global model, because of the local fine-tuning, the personalized models are more vulnerable to spurious features and have a larger accuracy disparity.
Empirically, we observe that the typical non-i.i.d. data distributions in FL settings reduce the accuracy disparity of the global model. One potential explanation is that the statistical heterogeneity (Wang
et al., 2021) under spurious features across users is larger than that of non-spurious features. As a result, the aggregation operation in the central server averages out the diverse shifts under the spurious features (Figure 1) so that the global model becomes more robust against spurious features on the benchmark datasets. On the other hand, the local fine-tuning step will result in a personalized model that entangles spurious features and becomes biased. Hence, it remains a key challenge how to debias the personalized models.
Various methods (Wang et al., 2019; Sagawa et al., 2020; Liu et al., 2021) have been developed to disentangle spurious features from ML models, but few can be easily applied to personalized models under the federated learning setting. The main reason lies in their reliance on bias-conflicting samples. The bias-conflicting samples can be rare. For example, in the CelebA dataset, only around 1.7k samples are bias-conflicting out of more than170k samples. If we distribute the CelebA dataset across users, nearly half do not have any bias-conflicting samples. Prior work (Li & Wang, 2019) has tried using a global proxy dataset for training FL models. However, a global proxy dataset may not characterize the local samples well. Furthermore, estimating the accuracy disparity of the personalized model becomes difficult without access to bias-conflicting samples.
To approach this problem, we propose a novel method to reduce the accuracy disparity of personalized models. Our proposed method does not rely on the bias-conflicting samples, e.g., blond male images in the aforementioned task of gender prediction, which are often rare and may not be available for every user. Instead, inspired by prior works (Tramèr et al., 2017; Liang et al., 2021) on the transferability of adversarial examples, we use the global model as a reference and the adversarial transferability between the global model and the personalized models as a proxy to estimate the accuracy disparity of the personalized models. The intuition is that if two ML models use disjoint subsets of features, the adversarial examples that one ML model generates do not transfer to the other ML model. Based on this intuition, we propose the following hypothesis:

heading: If the personalized models entangle spurious features (thus increasing the accuracy
text: disparity), the adversarial examples generated by the global model (that uses non-spurious features) do not transfer to the personalized models.
Empirically, we show that the adversarial transferability between the global and personalized models strongly correlates with the accuracy disparity between the global and personalized models (Section 4), validating our hypothesis. We further theoretically connect the adversarial transferability to the accuracy disparity (Section 5). Based on the empirical observations and theoretical results, we develop a method that enforces adversarial transferability between the global and the personalized models to reduce the accuracy disparity of personalized models (Section 6). Our contributions are summarized as follows:
• We empirically evaluate the accuracy disparity of the global and personalized models in a federated learning setting with spurious features, highlighting a risk of existing personalization methods. • We design a method to estimate the accuracy disparity of the personalized models, based on the low accuracy disparity global model and the adversarial transferability between the global and personalized models.
• We theoretically connect the adversarial transferability and the accuracy disparity of the global and personalized models. • We develop a methods to reduce the accuracy disparity of personalized models by enforcing the adversarial transferability between the global and personalized models.
Empirically, we conduct extensive experiments to validate the effectiveness of the proposed methods in mitigating accuracy disparity under the FL setting. Our experiments on MNIST (Deng, 2012), CelebA (Liu et al., 2015), and Coil20 (Nene et al., 1996) datasets show that the proposed approach reduces the accuracy disparity of personalized models from 15.12% to 2.15%, which is closer to that of the global model (−0.63%). Our method also preserves the benefit of the enhanced average accuracy from fine-tuning, resulting in 3.43% accuracy improvement on the biased test set and 0.85% accuracy improvement on the biased-conflicting test set.

heading: 2 RELATED WORK
text: Personalized Federated Learning Fine-tuning is typical for personalization methods. The metalearning-based method first trains a global model and fine-tunes the global model locally (Fallah et al., 2020). Other methods using multi-task learning (Li et al., 2021) or Moreau envelopes (T. Dinh et al., 2020) have an interpretation as fine-tuning the local model along with training the global model. Fine-tuning is also compatible with clustering-based method (Ghosh et al., 2020).
Debiasing ML Models A few prior works (Li & Vasconcelos, 2019; Sagawa et al., 2020) utilize group labels, which might require human annotation, to debias ML models. For example, the group distributional robust optimization (DRO) method (Sagawa et al., 2020) aims to optimize the worst-case error rate of ML models across different (often manually annotated) groups. Some groups contain bias-conflicting samples while others do not. Residual learning-based methods (He et al., 2019; Nam et al., 2020; Liu et al., 2021) train a biased ML model and up-weight the residual, which mainly contains bias-conflicting samples that the biased ML model mis-predicts. Chi et al. (2021) aims to mitigate the accuracy disparity in regression problems via learning the appropriate representations. However, all these methods rely on the explicit access to the bias-conflicting samples, making them difficult to apply on personalized federated learning, where bias-conflicting samples may not be accessible for every client.

heading: 3 PRELIMINARIES
text: Definitions and Notation A data sample is a vector x = [xr,xs], where xr corresponds to the robust and non-spurious features and xs are the spurious features. ds is the dimension of spurious features. Let y be a label, and define ` : Y × Y −→ R to be a λ-smooth, twice differentiable loss function and L(f,D) = E(x,y)∼D[`(f(x), y)] to be the empirical risk. wg and wp are the weights for the global model fg : X −→ Y and personalized model fp : X −→ Y , respectively. γ is the ratio between the gradient norms of the global and personalized models, ‖∇x`(fg(x),y)‖‖∇x`(fp(x),y)‖ . Let supp(D) be the support of distribution D. We define a global data distribution Dg , a biased local data distribution Db, and assume a bias-conflicting local data distribution Dbc. We define the “pseudo-gradient” as the difference between the updated local model and the global model from the previous round (sometimes we will use the term “gradient” when it is clear from context). 〈· , ·〉 denotes an inner product of two vectors and ·_ · denotes a concatenation of two vectors. θ is the angle between ∇x`(fg(x), y) and ∇x`(fp(x), y). θg is the angle between ∇x`(fg(x), y) and ∇xr`(fg(x), y) _ 0, which measures the entanglement of the global model to spurious features.
Training and Personalization Methods We train the global model using the federated averaging algorithm (McMahan et al., 2017), which learns a model f : X −→ Y that minimizes: L(f,Dg) =∑N i=1 |Dbi |/|Dg| · Ex,y∼Dbi `(f(x), y), where N is the number of clients, Dbi is the biased local dataset for client i, and Dg = ∪Ni=1Dbi is the global dataset. When the global model fg converges, fg will be sent to local clients for further fine-tuning by minimizing Ex,y∼Dbi `(f(x), y).
Adversarial Examples and Transferability We can generate an adversarial example xadv given data sample x with label y by solving:
xadv = arg max ‖x′−x‖≤
`(f(x′), y), (1)
where f is the victim ML model, and is the attack budget. In this example, we consider the L2 attacks, but extensions to general Lp attacks are straightforward. We say an adversarial example to be transferable if it also fools another ML model (e.g., a personalized model) other than the original victim model f (e.g., the global model).

heading: 4 AN EMPIRICAL STUDY WITH SPURIOUS FEATURES
text: To gain some insights into the problem, we first perform an empirical study on the accuracy disparity of the global and personalized models in an FL setting. In this study, the personalization method is fine-tuning. Our results highlight the risk of existing fine-tuning-based personalization methods and the difficulty of mitigating the risk. We also highlight the correlation between the adversarial transferability and the accuracy disparity between the global and personalized models. We provide additional theoretical analysis in Section 5 to support the observed correlation. The spurious features in the empirical study are as follows.
Spurious Features We consider color as the spurious feature for the MNIST, CelebA, and Coil20 datasets. In the MNIST and Coil20 datasets, we manually color the objects according to their labels to create spurious correlations, as Figure 2 shows. The spurious correlations vary across clients for the MNIST and Coil20 data (e.g., the red color correlates with label zero on the first client and with label one on the second client) to create additional statistical heterogeneity. In the CelebA dataset, the hair color attribute correlates with the gender label. We assign disjoint subsets of celebrities to different users, which naturally increases statistical heterogeneity for the spurious correlation.

heading: 4.1 STATISTICAL HETEROGENEITY REDUCES ACCURACY DISPARITY
text: Figure 3 shows the accuracy disparity of ML models on biased and bias-conflicting test sets. Compared to the models trained in the centralized setting, where the spurious correlations are fixed, the accuracy disparity of models trained in the federated setting decreased significantly. These empirical results suggest that the global model in FL is more robust to spurious features if the spurious features are non-i.i.d. across clients.
To explain this observation, consider the relationship between the gradient directions and the learned features. For the spurious features, its correlation with the label may change across clients. As
an example, in some users’ local datasets, the blond hair does not correlate with the gender label because the dataset does not contain any blond female image or the dataset has blond male images, as visualized in Figure 1a. Therefore, the gradient directions for the spurious features are diverse across clients, as shown in Figure 1b. The divergence between the gradient directions, as a consequence, makes learning spurious features difficult. In contrast, the non-spurious features, e.g., shape features, are more consistent across clients, leading to a more consistent gradient direction.

heading: 4.2 PERSONALIZATION MAY EXACERBATE ACCURACY DISPARITY
text: Although the global model in the federated setting has a lower accuracy disparity than in the centralized setting, the advantage could vanish during the personalization step.
The Personalized Model Entangles Spurious Features As can be observed in Figure 4, the accuracy first increases and decreases on the MNIST bias-conflicting test set and slowly decreases on Coil20. These two observations indicate that the personalized model entangles spurious features and exacerbates the accuracy disparity in a few batches. Although in principle, one may resort to early stopping, this is not feasible when the bias-conflicting dataset is unavailable or scarce.

heading: 4.3 ADVERSARIAL TRANSFERABILITY INDICATES ACCURACY DISPARITY
text: Because the bias-conflicting test set is often unavailable, it is infeasible to directly measure the accuracy disparity across personalized models. To this end, in this section, we focus on methods that implicitly measure the accuracy disparity and determine whether the personalized models entangle spurious features. Following our hypothesis in Section 1, we consider using the adversarial transferability between the global and personalized models as a proxy for the accuracy disparity measurement. Figure 5 plots the accuracy disparity and the adversarial transferability during finetuning. As the accuracy disparity of the personalized models increases and drifts away from that of the global model, the adversarial transferability between the global and personalized models decreases. This result empirically validates our hypothesis.

heading: 5 THEORETICAL INSIGHTS
text: This section presents our theoretical result that supports our hypothesis in Section 1 and the experimental results in Section 4.3. Our theoretical result applies the loss, which has similar behavior to the accuracy as is shown in the empirical results in Section 7. Before we proceed, some additional definitions and notations are needed for the presentation, and we provide a table summarizing all the notations used in Appendix A to ease the reading. Then, we connect both the loss disparity and the adversarial transferability to the angle between the gradients of the global and personalized models. In what follows, we shall show an upper bound of the loss disparity of a personalized model, which consists of the adversarial transferability between the global and personalized models and an indicator of the entanglement of the global model to spurious features.

heading: 5.1 MORE DEFINITIONS AND NOTATION
text: We define natural perturbation ∆ to model the distribution shift between the bias-conflicting Dbc and biased Db. ∆ could change a bias-aligned sample to a corresponding bias-conflicting sample. The distribution D∆|x of the natural perturbation ∆ conditions on the data sample x. Formally, for any x ∈ Rd, we have: Prx∼Dbc(x) = ∑ x′∈Rd,∆∈Rd 1{x=x′+∆} · Prx′∼Db(x′) · Pr∆∼D∆|x′(∆).
Running Example For a non-blond male image x, we could draw a natural perturbation ∆ from D∆|x that change the hair color in x to blond. That is saying, x+∆ is a blond male image. Iteratively drawing data samples from the biased dataset and applying the sampled natural perturbations to the data samples result in a dataset with bias-conflicting samples.
Another perturbation to consider is the adversarial perturbation δf, = xadv−x that is generated using f with budget . Plugging the definition of δf, into Eq. (1), we have δf, = arg max‖δ‖≤ `(f(x+ δ), y). Since the budget is small, we could approximate the loss function ` using the first-order gradient: δf, = arg max‖δ‖≤ ∇x`(f(x), y)>δ = · ∇x`(f(x),y) ‖∇x`(f(x),y)‖ (Miyato et al., 2018; Liang et al., 2021). With the adversarial perturbation, we define the adversarial transferability loss:
`trans(fg, fp,x, y) = ( `(fg(x+ δfg, ), y)− `(fg(x), y) ) − ( `(fp(x+ δfg, ), y)− `(fp(x), y) ) ,
which indicates the effectiveness of the adversarial perturbation generated using the global model applied to the personalized models.

heading: 5.2 LOSS DISPARITY AND ADVERSARIAL TRANSFERABILITY
text: With the definitions of natural and adversarial perturbations, this section shows that both the loss disparity and the adversarial transferability connect to an angle θ. Next, we outline the assumption:
Assumption 1. The distribution shift does not exacerbate the entanglement of a model f to spurious features xs, which is measured by ∇xs`(f(x), y):
E(x,y)∼Db,∆∼D∆|x,y [ ∫ 1 α=0 〈∇xs`(f(x+ α ·∆), y) ,1〉dα] ≤ E(x,y)∼Db [〈∇xs`(f(x), y) ,1〉].
Under Assumption 1, the following Lemmas hold.
Lemma 1. Under Assumption 1, let ∆ be the natural perturbation, θ be the angle between ∇x`(fg(x), y) and ∇x`(fp(x), y), θg be the angle between ∇x`(fg(x), y) and ∇xr`(fg(x), y) _ 0, and γ be ‖∇x`(fg(x),y)‖‖∇x`(fp(x),y)‖ , we have:
L(fp,Dbc)− L(fp,Db) = E(x,y)∼Db,∆∼D∆|x [ ∫ 1 α=0 〈∇xs`(fp(x+ α ·∆), y) ,1〉dα]
< E(x,y)∼Db,∆∼D∆|x [ √ ds γ · ‖∇x`(fg(x), y)‖ · (sinθg + sinθ)]
(2)
Lemma 1 connects the loss disparity to θ. The θg , differing from θ, is an indicator of the entanglement of the global model to spurious features and is a constant during the personalization step.
Lemma 2. Let be the attack budget, θ be the angle between ∇x`(fg(x), y) and ∇x`(fp(x), y), γ be ‖∇x`(fg(x),y)‖‖∇x`(fp(x),y)‖ , and the loss function ` : Y × Y −→ R be λ-smooth, twice differentiable, we have
· ‖∇x`(fg(x), y)‖ · (1− 1
γ · cosθ)− λ · 2 ≤ `trans(fg, fp,x, y)
≤ · ‖∇x`(fg(x), y)‖ · (1− 1
γ · cosθ) + λ · 2
(3)
Lemma 2 connects the adversarial transferability loss to θ. In the following analysis, we connect the loss disparity to adversarial transferability via θ.

heading: 5.3 A GENERALIZATION UPPER BOUND
text: We now present an upper bound of the disparity L(fp,Dbc)− L(fp,Db). The main idea is to derive an upper bound of ‖∇x`(fg(x), y)‖ · sinθ in Eq. (2) from Eq. (3). Theorem 3. Let γmin be the minimum of γ, with Lemmas 1-2, we have:
L(fp,Dbc)− L(fp,Db) < √ ds · ( ( sinθg + √ 2
γmin − 1) · E(x,y)∼Db [‖∇x`(fg(x), y)‖]
+ 1
· E(x,y)∼Db [`trans(fg, fp,x, y)] + λ · ) Theorem 3 suggests (1) debiasing the global model fg, whose entanglement to spurious features is measured by θg , and (2) enforcing the adversarial transferability between fg and fp help reducing the loss disparity of personalized models. For the constants γmin and λ, we further explore their impacts in the following section and Appendix D.1, respectively.

heading: 6 METHODS
text: With the empirical and theoretical results in Sections 4.3 and 5, respectively, it is natural to ask if enforcing the adversarial transferability in the personalization step reduces the accuracy disparity. In this section, we first introduce adversarial examples to the personalization step as a regularization term added to the original loss function, aiming to enforce the adversarial transferability. However, the accuracy disparity still increases, albeit much slower, even if the adversarial transferability remains high. One possible reason is that the personalized model increases its gradient norm, which helps preserve the adversarial transferability but does not prevent the personalized model from entangling spurious features. To this end, we add an L2 regularization term to the loss function, aligning the gradient norms of the global and personalized models. Combing these two methods addresses the accuracy disparity. Both methods are relatively light-weight from a computational perspective.

heading: 6.1 ENFORCING ADVERSARIAL TRANSFERABILITY
text: We enforce that global and personalized models make consistent predictions on adversarial examples, such that adversarial examples transfer from one to the other.
Generating Adversarial Examples The projected gradient descent (PGD) attack (Madry et al., 2018) is an effective attack method that uses the neural network’s first-order gradient, and is easy to compute. Additionally, . The attack solves xadv = arg max‖x′−x‖≤ `(f(x
′), y) iteratively. At iteration t+ 1, the adversarial example is: xt+1adv = Proj‖xadv−x‖≤ (x+α · sign(∇xtadv`(fg(x t adv), y))), where Proj is a projection operator.
Enforcing Consistent Predictions Both the global model fg and the personalized model fp take the adversarial example xadv as input and output zg and zp from their last layers, respectively. We enforce the adversarial transferability by adding the following regularization term, which maximizes the cross-entropy between zg and zp. Since the global model fg is fixed as a reference in the personalization step and its low accuracy disparity is desirable, we use zg as the ground-truth:
Radv(zg, zp) = K∑ i=1 [zgi · log(zpi) + (1− zgi) · log(1− zpi)],
where K is the number of classes. The local model has access to the global model, so there is no additional communication overhead for implementing this regularization. The adversarial examples are computed using the global model once for all. The computation only needs a few back-propagations, much less than training the global model.

heading: 6.2 ALIGNING GRADIENT NORMS
text: In Eq. (3), we have seen that the adversarial transferability loss depends not only on the angle θ, which connects the transferability to the disparity but also on γ := ‖∇x`(fg(x),y)‖‖∇x`(fp(x),y)‖ . A small γ indicates that the personalized model increases its gradient norm. Then, the personalized model could entangle the spurious features and increase θ without decreasing the adversarial transferability.
To prevent γ from decreasing, we employ a simple and effective strategy by adding an L2 regularization term RL2 = ‖wg − wp‖2 to the loss function. The motivation behind the L2 term is straightforward: if two models have similar weights, they have similar gradients. Empirical results in Appendix D.2 show the effectiveness of the L2 term in controlling γ. Although prior works (Li et al., 2020; T. Dinh et al., 2020; Li et al., 2021) have explored similar regularization methods, we develop the regularization term from a different perspective.

heading: 7 EXPERIMENTS
text: This section presents our experimental results, demonstrating that our method reduces the accuracy disparity. We also show that the benefit of enhanced average accuracy from fine-tuning is preserved.

heading: 7.1 SETTING
text: Data Partition We distribute the MNIST and Coil20 dataset across 50 clients where each client have 5 different classes. The local dataset on each client is further partitioned to train/validation/test set with a ratio of 72:8:20, following prior work (Li et al., 2021). We make two data partitions for CelebA: CelebA R using a real partition and CelebA S using a synthetic partition, both have 508 clients. In both CelebA partitions, each client represent a disjoint set of celebrity (Li et al., 2021).
Due to the limited space, we further detail the data partition in Appendix C.1, report the hyperparameters in Appendix C.2, and list the neural network architecture in Appendix C.3.

heading: 7.2 THE EFFECTIVENESS OF PROPOSED METHODS
text: We conduct an ablation study on the MNIST dataset. Figures 6b and 6f demonstrate the effectiveness of enforcing adversarial transferability. However, the accuracy and loss disparity still increase during personalization, which is potentially caused by the gradient norm issue (Section 6.2). Aligning the gradient norms by applying the L2 regularization term while enforcing adversarial transferability address the accuracy and loss disparity as Figures 6d and 6h show, respectively. Figure 6c and 6g further show that applying the L2 regularization term alone does not address the accuracy or loss disparity. Compared to naive fine-tuning, which is reported in Figures 6a and 6e, our method mitigates the accuracy and loss disparities by ∼ 50%.

heading: 7.3 ANALYSIS
text: We compare our method to no personalization (Global), naive fine-tuning (FT), Ditto (Li et al., 2021), up-weighting (UW) (Sagawa et al., 2020), and just train twice (JTT) (Liu et al., 2021). The up-weighting method is implemented via sampling bias-aligned and bias-conflicting samples with equal probability (Sagawa et al., 2020). Up-weighting and JTT are not applicable to Coil20 due to the lack of bias-conflicting samples. We use the local finetuning version of the Ditto solver because the local finetuning solver performs fewer local updates than that of the joint optimization solver and therefore entangles spurious features less. Each experiment is repeated 9 times with 3 random seeds for the federated learning step and 3 for the personalization step. We select models using the validation accuracy minus the decrease of adversarial transferability and using the validation accuracy for other baseline and competitor methods.
Tables 1 shows the main result. Our method reduces the accuracy disparity of personalized models from 15.12% to 2.15%, compared to other personalization methods. Our method also preserves the
enhanced average accuracy from fine-tuning, resulting in 3.43% accuracy improvement on the biased test set and 0.85% improvement on the biased-conflicting test set compared to the global model. In contrast, the naive fine-tuning method sacrifices the accuracy on the bias-conflicting test set by up to 14.3% and increase the accuracy disparity by 15.12%. We also find that our methods outperform the supervised up-weighting method and the unsupervised JTT method, which increase the average accuracy disparity to 7.56% and 17.76%, respectively. One possible reason is that the diversity of the up-weighted bias-conflicting samples are small. Therefore, the neural network could memorize them instead of discarding spurious features. Appendix D.3 further shows empirical results that support our analysis.

heading: 8 CONCLUSION
text: In this work, we show the risk of prior federated learning personalization methods with spurious features, which lead to high accuracy disparity between the global and local models. Then, we develop a strategy by enforcing the adversarial transferability between the global and personalized models to reduce the accuracy disparity. Both empirical and theoretical results show that our strategy is effective.

heading: 9 ETHICS STATEMENT
text: Our method mitigates the issue of spurious features, which lead to bias towards minority groups, in personalized federated learning. However, completely disentangling spurious features remains challenging and is an issue for many federated learning methods.

heading: 10 REPRODUCIBILITY STATEMENT
text: Our implementation, including data partition scripts, is based on the FedML library (He et al., 2020), which is open-sourced. The proofs of the theoretical results are in Appendix B. The datasets in our experiments are publicly available. We detail the data partition in Appendix C.1, report the hyper-parameter tuning in Appendix C.2, and list the neural network architecture in Appendix C.3.

heading: B PROOFS
text: B.1 PROOF OF LEMMA 1
Lemma 1. Let ∆ be the natural perturbation, θ be the angle between ∇x`(fg(x), y) and ∇x`(fp(x), y), θg be the angle between ∇x`(fg(x), y) and ∇xr`(fg(x), y) _ 0, and γ be ‖∇x`(fg(x),y)‖ ‖∇x`(fp(x),y)‖ , we have:
L(fp,Dbc)− L(fp,Db) = E(x,y)∼Db,∆∼D∆|x [ ∫ 1 α=0 〈∇xs`(fp(x+ α ·∆), y) ,1〉dα]
< E(x,y)∼Db [ √ ds γ · ‖∇x`(fg(x), y)‖ · (sinθg + sinθ)]
Proof. Rewriting L(fp,Dbc) and introducing ∆:
L(fp,Dbc) = E(x,y)∼Dbc [`(fp(x), y)] = E(x,y)∼Db,∆∼D∆|x [`(fp(x+ ∆), y)] = E(x,y)∼Db,∆∼D∆|x [`(fp(x), y) + `(fp(x+ ∆), y)− `(fp(x), y)] = E(x,y)∼Db,∆∼D∆|x [`(fp(x), y)]
+ E(x,y)∼Db,∆∼D∆|x [`(fp(x+ ∆), y)− `(fp(x), y)] = E(x,y)∼Db [`(fp(x), y)]
+ E(x,y)∼Db,∆∼D∆|x [`(fp(x+ ∆), y)− `(fp(x), y)] = L(fp,Db) + E(x,y)∼Db,∆∼D∆|x [ ∫ 1 α=0 〈∇x`(fp(x+ α ·∆), y) ,1〉dα]
= L(fp,Db) + E(x,y)∼Db,∆∼D∆|x [ ∫ 1 α=0 〈∇xs`(fp(x+ α ·∆), y) ,1〉dα]
Moving L(fp,Db) to the left-hand-side (LHS), we have:
L(fp,Dbc)− L(fp,Db) = E(x,y)∼Db,∆∼D∆|x [ ∫ 1 α=0 〈∇xs`(fp(x+ α ·∆), y) ,1〉dα]. (4)
According to Assumption 1, we further have:
E(x,y)∼Db,∆∼D∆|x [ ∫ 1 α=0 〈∇xs`(fp(x+ α ·∆), y) ,1〉] ≤ Ex∼Db [〈∇xs`(fp(x), y) ,1〉] (5)
Next, we connect 〈∇xs`(fp(x+α ·∆), y) ,1〉 to ‖∇x`(fg(x), y)‖·sinθ. The first step is connecting 〈∇xs`(fp(x+ α ·∆), y) ,1〉 to ‖∇xs`(fp(x), y)‖ using Cauchy-Schwarz inequality:
〈∇xs`(fp(x+ α ·∆), y) ,1〉 ≤ √ 〈∇xs`(fp(x+ α ·∆), y) ,∇xs`(fp(x+ α ·∆), y)〉 · 〈1 ,1〉
= √ ds · ‖∇xs`(fp(x), y)‖
(6)
Then, we connect ‖∇xs`(fp(x), y)‖ to ‖∇x`(fg(x), y)‖. Assuming the global model fg entangles spurious features and the angle between∇x`(fg(x), y) and∇xr`(fg(x), y) _ 0 is θg , we have:
‖∇xs`(fp(x), y)‖ ≤ ‖∇x`(fg(x), y)‖ · sin(θg + θ). (7)
Since it is easy to see that θ ∈ [0, π4 ] and the gradient of sinθ is monotonically decreasing in [0, π 4 ], we have:
sin(θg + θ) = ∫ θg+θ 0 ∇sinθdθ
= ∫ θg+θ 0 cosθdθ
< ∫ θg 0 cosθdθ + ∫ θ 0 cosθdθ
= sinθg + sinθ
(8)
Combining Eq. (7) and Eq. (8), we have:
‖∇xs`(fp(x), y)‖ < ‖∇x`(fp(x), y)‖ · (sinθg + sinθ) (9)
Recalling the defition of γ := ‖∇x`(fg(x),y)‖‖∇x`(fp(x),y)‖ and combining Eq. (4), Eq. (5), Eq. (6), Eq. (9) complete the proof.
B.2 PROOF OF LEMMA 2
Lemma 2. Let be the attack budget, θ be the angle between ∇x`(fg(x), y) and ∇x`(fp(x), y), γ be ‖∇x`(fg(x),y)‖‖∇x`(fp(x),y)‖ , and the loss function ` : Y × Y −→ R be λ-smooth, twice differentiable, we have
· ‖∇x`(fg(x), y)‖ · (1− 1
γ · cosθ)− λ · 2 ≤ `trans(fg, fp,x, y)
≤ · ‖∇x`(fg(x), y)‖ · (1− 1
γ · cosθ) + λ · 2
Proof. Under the definition of the adversarial perturbation, it is easy to see that δf, = · ∇x`(f(x),y)‖∇x`(f(x),y)‖ and δf, increases the loss by:
`(fg(x+ δfg, ), y)− `(fg(x), y) = δfg, ∇x`(fg(x), y) + 1
2 δ>fg, ∇ 2 x̃g`(fg(x̃g), y)δfg,
= · ‖∇x`(fg(x), y)‖+ 1
2 δ>fg, ∇ 2 x̃g`(fg(x̃g), y)δfg,
where x̃g is a linear interpolation between x and x+ δfg, , by the Lagrange’s mean-value theorem. Similarly, for a transferable adversarial example from fg applies to fp, δfg, could increase the loss of fp by:
`(fp(x+ δfg, ), y)− `(fp(x), y) = δfg, ∇x`(fp(x), y) + 1
2 δ>fg, ∇ 2 x̃p`(fp(x̃p), y)δfg,
= · ‖∇x`(fp(x), y)‖ · cosθ + 1
2 δ>fg, ∇ 2 x̃p`(fp(x̃p), y)δfg,
where cosθ = ∇x`(fg(x),y)·∇x`(fp(x),y)‖∇x`(fg(x),y)‖‖∇x`(fp(x),y)‖ . Plugging the approximations above to the adversarial transferability loss, we have:
`trans(fg, fp,x, y) = ( `(fg(x+ δfg, ), y)− `(fg(x), y) ) − ( `(fp(x+ δfg, ), y)− `(fp(x), y) ) = · ‖∇x`(fg(x), y)‖ − · ‖∇x`(fp(x), y)‖ · cosθ
+ 1
2 · δ>fg, ∇ 2 x̃g`(fg(x̃g), y)δfg, −
1 2 · δ>fg, ∇ 2 x̃p`(fp(x̃p), y)δfg,
Under the λ-smooth assumption on the loss function, the spectral norms of the Hessian metrics are bounded. Therefore, we could bound the norm of the deviate between the quadratic terms (Nesterov, 2003, Proof of Theorem 2.1.5) in the adversarial transferability loss:
‖δ>fg, ∇ 2 x̃g`(fg(x̃g), y)δfg, − δ > fg, ∇ 2 x̃p`(fp(x̃p), y)δfg, ‖ ≤ 2λ · δ > fg, δfg, = 2λ · 2 (10)
Since the quadratic terms in Eq. (10) are scalars, we have:
− 2λ · 2 ≤ δ>fg, ∇ 2 x̃g`(fg(x̃g), y)δfg, − δ > fg, ∇ 2 x̃p`(fp(x̃p), y)δfg, ≤ 2λ · 2 (11)
Plugging Eq. (11) and the definition of γ to `trans(fg, fp,x, y) completes the proof.
B.3 PROOF OF THEOREM 3
Theorem 3. Let γmin be the minimum of γ, under Assumptions 1 and Lemmas 1-2, we have:
L(fp,Dbc)− L(fp,Db) < √ ds · ( ( sinθg + √ 2
γmin − 1) · E(x,y)∼Db [‖∇x`(fg(x), y)‖]
+ 1
· E(x,y)∼Db [`trans(fg, fp,x, y)] + λ · ) Proof. According to Lemma 2, we know:
`trans(fg, fp,x, y) ≥ · ‖∇x`(fg(x), y)‖ · (1− 1
γ · cosθ)− λ · 2
where cosθ = ∇x`(fg(x),y)∇x`(fp(x),y)‖∇x`(fg(x),y)‖‖∇x`(fp(x),y)‖ . Then, we derive an upper bound of ‖∇x`(fg(x), y)‖ · sinθ from `trans(fg, fp,x, y). It is easy to see that θ ∈ [0, π4 ]. Therefore, we have:
`trans(fg, fp,x, y)
≥ · ‖∇x`(fg(x), y)‖ · (1− 1
γ · cosθ)− λ · 2
= γ · ‖∇x`(fg(x), y)‖ · (1− cosθ + γ − 1)− λ · 2
= γ · ‖∇x`(fg(x), y)‖ · (1− cosθ) + · (γ − 1) γ · ‖∇x`(fg(x), y)‖ − λ · 2
= γ · ‖∇x`(fg(x), y)‖ · (2 · sin2
θ 2 ) + · (γ − 1) γ · ‖∇x`(fg(x), y)‖ − λ · 2
= γ · ‖∇x`(fg(x), y)‖ · (sinθ + 2 · sin2
θ 2 − sinθ)
+ · (γ − 1)
γ · ‖∇x`(fg(x), y)‖ − λ · 2
= γ · ‖∇x`(fg(x), y)‖ · sinθ + γ · ‖∇x`(fg(x), y)‖ · (2 · sin2
θ 2 − sinθ)
+ · (γ − 1)
γ · ‖∇x`(fg(x), y)‖ − λ · 2
≥ γ · ‖∇x`(fg(x), y)‖ · sinθ + γ · ‖∇x`(fg(x), y)‖ · (1−
√ 2)
+ · (γ − 1)
γ · ‖∇x`(fg(x), y)‖ − λ · 2
= γ · ‖∇x`(fg(x), y)‖ · sinθ +
· (γ − √ 2)
γ · ‖∇x`(fg(x), y)‖ − λ · 2
Moving ‖∇x`(fg(x), y)‖ · sinθ to the left hand side (LHS): ‖∇x`(fg(x), y)‖ · sinθ
≤ γ · `trans(fg, fp,x, y) + ( √ 2− γ) · ‖∇x`(fg(x), y)‖+ γ · λ ·
(12)
According to Lemma 1, we know: L(fp,Dbc)− L(fp,Db) = E(x,y)∼Db,∆∼D∆|x [ ∫ 1 α=0 〈∇xs`(fp(x+ α ·∆), y) ,1〉dα]
< E(x,y)∼Db [ √ ds γ · ‖∇x`(fg(x), y)‖ · (sinθg + sinθ)]
(13)
Combining Eq. (13) and Eq. (12), and taking expectation of x, y over Db:
Ex∼Db [ √ ds γ · ‖∇x`(fg(x), y)‖ · (sinθg + sinθ)]
≤ √ ds γ · ( E(x,y)∼Db [‖∇x`(fg(x), y)‖ · sinθg] + E(x,y)∼Db [ γ · `trans(fg, fp,x, y)]
+ E(x,y)∼Db [( √ 2− γ) · ‖∇x`(fg(x), y)‖] + γ · λ · )
≤ √ ds · ( ( sinθg + √ 2
γmin − 1) · E(x,y)∼Db [‖∇x`(fg(x), y)‖]
+ 1
· E(x,y)∼Db [`trans(fg, fp,x, y)] + λ ·
) (14)
Plugging Eq. (14) back to Eq. (13) completes the proof.

heading: C MORE EXPERIMENTAL SETTING
text: C.1 DATA PARTITION
We distribute the MNIST and Coil20 dataset across 50 clients. Each client has data from 5 different classes. The local dataset on each client is further partitioned to train/validation/test set with a ratio of 72:8:20, following prior work (Li et al., 2021). The test set here is biased. We alternate the spurious features in biased test sets by recoloring the data to create a bias-conflicting test set. For the CelebA dataset, we consider two partitions. In the first partition (CelebA R), each client represents 20 celebrities. One celebrity only appears on one client. The blond male images in the biased test sets are copied to bias-conflicting test sets. We use all clients for training the global model. In the personalization step, we select the clients who have more than 5 blond female training samples and more than 5 blond male test samples. We select these clients because they provide enough samples to create spurious correlations and bias-conflicting test sets. Although the first partition on CelebA is real, the number (161) of blond male images is small. To make the result clearer, we create another synthetic CelebA partition. In the second partition (CelebA S), there are 650 blond male images, which achieve a similar bias-conflicting test set size as prior works Sagawa et al. (2020); Liu et al. (2021). The 650 images are distributed to 3 clients with 2350 other images. The rest of the images are distributed in the same way as the first partition. Tables 3, 4, and 5 provide more details about the 3 clients.
We use Adam optimizer (Kingma & Ba, 2015) throughout our experiments with learning rate 1e-4. Although stochastic gradient descent (SGD) optimizer is more common in vision-related tasks, we find that the Adam optimizer always leads to lower accuracy disparity. We train the global model for 500 rounds. 5 clients are selected per round, and each performs 5 epochs of local updates. We tune the coefficients of the adversarial transferability and L2 regularization terms from {0.01, 0.1, 1.0, 10.0} and select the largest value that does not decrease the validation accuracy during penalization. We start the attack budget at 0.031 and gradually decrease it such that 30%− 50% of the attack succeeds. A large budget will make the attack too strong and push the adversarial examples far across the decision boundary, making the regularization method less effective. We configure to 0.031/0.01/0.015 for MNIST/CelebA/Coil20, respectively. We fine-tune the global model for 5 epochs on MNIST and
10 epochs on CelebA/Coil20, which are sufficient for the personalized models to converge. The clients with the most data samples fine-tune the penalized models for a total of 80/40/30 batches on MNIST/CelebA/Coil20 datasets. Note that we may not select the personalized model with the most fine-tuning batches for reporting. In the just train twice (JTT) method, we up-sample the residuel by a factor of 50. In Ditto, we tune its λ from {0.1, 1.0}.
C.3 NEURAL NETWORK ARCHITECTURE
We use CNN 28x28 for MNIST dataset and CNN 64x64 for CelebA and Coil20 datasets.

heading: D MORE EXPERIMENTAL RESULTS
text: D.1 FIRST-ORDER APPROXIMATION OF ADVERSARIAL TRANSFERABILITY LOSS
To explore the impact of the λ term in Lemma 2 and Theorem 3, we measure the relative difference between `trans(fg, fp,x, y) and · ‖∇x`(fg(x), y)‖ · (1− 1γ · cosθ). In other words, we measured the accuracy of a first-order approximation of `trans(fg, fp,x, y). If the approximation is accuracy is high, it implies that the impact of λ · 2 is low. Specifically, we compute an approximation error:
E(x,y)∼Db [∣∣`trans(fg, fp,x, y)− · ‖∇x`(fg(x), y)‖ · (1− 1γ · cosθ)
`trans(fg, fp,x, y) ∣∣∣]. On MNIST, CelebA and Coil20 datasets, we find that the approximation error is 0.019, 0.058, 0.084, respectively. These results suggests that using · ‖∇x`(fg(x), y)‖ · (1− 1γ · cosθ) to approximate `trans(fg, fp,x, y) results in a decent accuracy and the impact of λ · 2 is low. The possible reason is that the attack budget is usually small (e.g., 0.031), such that the gradient of a function changes little in a small neighborhood defined by .
D.2 EFFECTIVENESS OF L2 REGULARIZATION TERM
Figure 7 shows the distribution of γ before and after applying the L2 regularization term on the CelebA dataset. Here, the γ is computed once per data sample. We keep the global model fixed and fine-tune the personalized model for 1 epoch.
D.3 DIVERSITY OF BIAS-CONFLICTING SAMPLES IMPACTS DEBIASING
To explore the impact of the diversity of bias-conflicting samples on debiasing, we vary the diversity of bias-conflicting samples and adjust the up-weighting factors accordingly. Specifically, we sample a factor of 0.02, 0.025, 0.033, 0.05, and 0.1 biased data samples from the MNIST dataset and re-color them to become bias-conflicting. The factor in the sampling step is called sampling factor. Then, we up-weight the bias-conflicting samples by a factor of 50, 40, 30, 20, 10, respectively, keeping the total number of bias-conflicting samples consistent. Here, the bias-conflicting samples have less diversity if generated by a small number of biased data samples with a large up-weighting factor. Experimental results in Figure 8 show that, as the diversity reduces, the accuracy disparity of personalized model on the biased dataset and bias-conflicting dataset increases, supporting our analysis. Therefore, our method is applicable in the scarcity of bias-conflicting samples while the up-weighting method fails.

Review summaries:
The paper considers the problem that personalization methods in federated learning may cause the personalized models to overfit on spurious features, thereby increasing the accuracy disparity compared to the global model. To mitigate this accuracy disparity, the paper investigates adversarial transferability, which is shown to correlate with disparity. Thus, the paper proposes a federated personalization approach based on adversarial transferability and catastrophic forgetting that reduces accuracy disparity to the level of the global model while maintaining the higher accuracy of prior personalization methods. The paper evaluates the approach on three real-world datasets.

This work explores the possibility of personalisation methods entangling spurious features that can undermine their generalization in case of federated learning. It proposes to use a combination of a consistency term for adversarial transferability and an L2 regularisation term to help reduce this disparity.  The approach is evaluated on artificial settings with spurious features.  

The authors have proposed a new FL training strategy to reduce the performance discrepancy between the central model and the client models. The PGD generated adversarial examples are fed into both central model and client models and their outputs are used to minimize the entropy loss. The computation is further simplified using Taylor expansion.