Towards Understanding Adversarial Transferability in Federated Learning
Authors: Yijiang Li
Ying Gao
Haohan Wang
Venue: Accepted by TMLR ()

Abstract: We investigate a specific security risk in FL: a group of malicious clients has impacted the model during training by disguising their identities and acting as benign clients but later switching to an adversarial role. They use their data, which is part of the training set, to train a substitute model and conduct transferable adversarial attacks against the federated model. This type of attack is subtle and hard to detect because these clients initially appear to be benign.

The key question we address is: How robust is the FL system to such covert attacks, especially compared to traditional centralized learning systems? We empirically show that the proposed attack imposes a high-security risk to current FL systems. By using only 3\% of the client's data, we achieve the highest attack rate of over 80\%. To further offer a full understanding of the challenges the FL system faces in transferable attacks, we provide a comprehensive analysis of the transfer robustness of FL across a spectrum of configurations. Surprisingly, FL systems show a higher level of robustness than their centralized counterparts, especially when both systems are equally good at handling regular, non-malicious data.

We attribute this increased robustness to two main factors:
1) Decentralized Data Training: Each client trains the model on its own data, reducing the overall impact of any single malicious client.
2) Model Update Averaging: The updates from each client are averaged together, further diluting any malicious alterations.
Both practical experiments and theoretical analyses support our conclusions. This research not only sheds light on the resilience of FL systems against hidden attacks but also raises important considerations for their future application and development。

Published in Transactions on Machine Learning Research (11/2024)
Towards Understanding Adversarial Transferability in Feder- ated Learning
Yijiang Li University of Colifornia San Diego
vijiangli edu
Ying Gao
srogpingtscuu..duuccr
South China University of Technology
Haohan Wang
haohanu@illinois.coom edu
University of Alinois Urbana- Chanppign
Reviewed on OpenReview: https: opwnreview net/ forum? id= hafnY2PP7h
Abstract
We investigate specific security risk in FL: a group of malicious clients has impacted the model during training by disguising their identities and acting benign clients but later switching to adversarial role. They use their data, which was part of the training set, to train a substitute model and conduct transferable adversarial attacks against the federated model. This lype of attack is subtle and hard to detect because these clients initially appear to be benign.
The key question we address is: How robust is the FL system to such covert attacks, especially compared to traditional centralized learning systems? We empirically show that the proposed attack imposes a high security risk to current FI. systems. By using only 3% of the client data, we achieve the highest attack rate of over 80%. To further offer a full understanding of the challenges the FL system faces in transferable attacks, we provide a comprehensive analysis over the transfer robustness of FL across spectrum of configurations. Surprisingly, FI. systems show higher level of robustness than their centralized counterparts, especially when both systems are equally good at handling regular, non-malicious data.
We attribute this increased robustness to two main factors: 1) Decentralized Data Training: Each client trains the model on its own data, reducing the overall impact of any single malicious client. 2) Model Update Averaging: The updates from each client are averaged together, further diluting any malicious alterations. Both practical experiments and thcoreti- cal analysis support our conclusions. This research not only sheds light on the resilience of FL systems against hidden attacks but also raises important considerations for their future application and development.
1 Introduction
Although Federated Learning (FL) provides a promising solution to collaboratively train models without exchanging data, especially in privacy-concernee areas Li et al. (2022). it is still susceptible to attacks such data poisoning Huang al. (2011). model poisoning Bhagoji et al. [2019] Bagdasaryan al. (2020); Huang ct al. (2023). freo-riders attack Lin et al. (2019). and reconstruction attacks Geiping et al. (2020); Zhu et al. (2019) It is also vulnerable to adversarial atlacks during inference Biggio el al. (2013): Szegedy el al. 2013)), including adversarial examples designed to deceive the model Zizzo et al. (2020). Research on robust FL methods against adversarial examples has primarily focused on white-box setting where attackers have full model access Zhou al. (2020); Reisizadeh et al. (2020a): Hong et al. (2021); Qiao al. (2024). However, real-world FT. applications, like Gboard Hard et al. (2018) usually restrict such access.

---

Published in Transactions on Machine Learning Research (11/2024)
We observe distinct FL security challenge: malicious clients may pose benign contributors, only revealing their adversarial intent. post-training. This setting raises new security challenge because these clients have access to a subset of the training data, potentially leading to a better surrogate model for transfcrable attacks. Current FL applications lack effective mechanisms to climinate such hostile participants
Hard al. (2018), even if selection mechanisms exist, such as Krum Fang et al. (2020); Li et al. (2020a); Bagdasaryan et al. (2020), as attackers do not exhibit hostile behavior during training. After obtaining data,
an altacker could train a surrogate model for transfer-based black-box attacks.
In this paper, we pioneer an exploration of this practical perspective of FL robustness. Stemming from the above scenarios, we propose a simple yet practical assumption: the attacker possesses a limited amount of the users' data but no knowledge about the target model or the full training set. To assess current FL system robustness and guide future research in this regard, we investigate the adversarial transferability under a spectrum of practical FL settings. Adversarial transferability refers to the ability of adversarial examples generated from the source model to successfully attack the target model, which measures the amount of threat white-box attack poses to the model and system.
We first evaluate the transferability of adversarial examples generated from different source models to attack federated-trainet model. Then comprehensive evaluation of practical configurations is conducted to assess the feasibility of our attack. We further investigate two properties ies of FL: decentralized training and the averaging operation and their correlation with federated robustness. To provide comprehensive evaluation in a practical aspect, we consider the attack timing, the architecture, and different aggregation methods in our experiments. We have the following findings:
We find that, while there are indeed socurity challenges from the novel attack sctting, the federated model is more robust under white-box attack compared with its centralized-Lrainee counterpart when their accuracy on clean images is comparable.
We investigate the transferability of adversarial examples generated from models trained by various numbers of users' data. We observe that without any elaborated techniques such as dataset synthesis Papernot et al. (2017) or attention Wu et al. (2020b), a regularly trained source model with only limited users' data can perform the transfer attack. With ResNet50 on CIFAR10, we achieve a transfer rate of 70% and 80% with only 5% and 7% of users with augmentations and further improve this number to 81% and 85% with AuloAttack Croce & Hein (2020).
We investigate two intrinsic properties of the FL: the property of distributed training and the averaging operation and discover that both heterogeneity and dispersion degree of the decentralized data as well as the averaging operations can significantly decrease the transfer rate of transfer- based black-bor box attack. To further understand the phenomenon, We provide theoretical analysis to further explain the observations.
2 Background
2.1
Adversarial Robustness
The adversarial robustness of model is usually defined as the model's ability to predict consistently in the presence of small changes in the input. Intuitively, if the changes to the image are tiny that they are imperceptible to humans, these perturbations will not alter the prediction. Formally, given well trained classifier f and image-label pairs (z, v) on which the model correctly classifies f(z) y, f is defined to be e-rbbust with respect to distance metric d() if
E(
min 2'ddlr af((''0)) v) E(any a((=))))
(1)
which is usually optimized through maximizing:
E B(aa))
min a(f(x';0),)) a'idiz' S
(2)
where a denotes the function evaluating prediction accuracy. In the case of classification, ((f 0), y) yields if the prediction f(x; 0) equals ground-truth label y, otherwise. The distance metric d(:) usually
2

---

Published in Transactions on Machine Learning Research (11/2024)
approximated by Lo, Lg or Lo to measure the visual dissimilarity of original image and the perturbed image x'. Despite the change to the input is small, the community have found a class of methods that can easily manipulate model's predictions by introducing visually imperceptible perturbations in images [Zzeedy ct al. (013): Goodfellow et al. (2014); Doosovi-Dezzoooii et al. (2016). From the optimization standpoint, it is achieved by maximizing the loss of the model on the input Madry el al. (2017):
+ max [1f(x++;00)00) 6) s.t. d(x f( <E
(3)
8
where I(:-)) denotes the loss function (e.g., cross-entropy loss) for training the model I parameterized by 0. While these attack methods are powerful, they usually require some degrees of knowledge about the target model f (e.g., the gradient). Arguably, for many real-world scttings, such knowlodge is not available, and we can only expect less availability of such knowledge on FL applications trained and deployed by service providers. On the other hand, the hostile attacker having access to some but limited amount of users' data is much more realistic scenario. Thus, we propose the following assumption for practical attack in FL: given the data of n malicious users Dm = -h=1 D, where D, {(rk: ) 1,.... m,)(9 contains m, data points, we aim to acquire a transferable perturbation 8 by maximizing the same objective as in Equation g but with surrogate model f' parameterized by 8' trained by Dm:
5 = arg max 1(f"(x + 5:6').)) v) s.t. d(x + ), <E
(4)
5
We hope to lest whether this ò can be used to deceive the larget model ʃ as well.
2.2 The Security and Robustness of Federated Learning
Poisoning and Backdoor Attack. Poisoning atlacks Biggio et al. (2012): Fang et al. (0020) aim to disrupt the global model by injecting malicious data or manipulating local model parameters on compromised
devices. In contrast, backdoor attacks Bagdasaryan et al. (000);; Sun et al. 2019 ; Wang et al. (2023)
infuse a malicious task into the existing model without impacting its primary task accuracy Zhang et al.
[2023:: Huang et al. (0244), including model replacement, Chen et al. (0077), label-flipping Fung et al. (018))
lixed triggerrgger backdoor Dai & Li 2023 Liu et al. (0024) and trigger-opdiiiiztionn [Huang] (2020); Li et al. (2023a) Nguyen et al.
2024). Defenses against these attacks often involve anomaly detection methods, such
as Byzantine-tolerant aggregation Shejwalkarr & Houmansadr (2021) (e.g., Krum, MultiKrum Blanchard
ct al. (2017), Bulyan Guerraoui et al. (2018), Trimmed-mean and Median YYn et al. (2018)), focusing on the geometric distance between hostile and benign gradients. More advanced defenses take detection, aggregation, detection, and differential privacy into consideration Huang et al. (2023): Nguyen et al. (2022). The robustness
and attack performance of backdoor attacks is significantly influenced by FL data heterogeneity Zawad et al,
2021
Transfer Attack. Transfer-based adversarial allacks employ the full training set of the larget model to train a surrogate model Zhou et al. (2018), a challenging condition to meet in practice, especially in FL where data privacy is paramount. Another line of inquiry delves into the mechanisms of black-box attacks, which exploit the high transferability of adversarial examples even between different model architectures (Szegedy et al. (2013); Goodfellow et al. (2014). This transferability is partially attributed to the similarity between source and target models Goodfellow et al. (2014); Liu et al. (2016); Li et al. (2023b), as adversarial perturbations align closely with a model's weight vectors, and different models learn similar decision boundaries for the same task. Tramer et al. (2017) found that adversarial examples span a large, contiguous subspace, facilitating transfcrability. Mcanwhile, Tlyas et al.] (2019) posits that adversarial perturbations are non-robust features captured by the model, and Waased al. (2023) utilizes this theory to explain differing mistakes in transfer attacks. Additionally, Demontis et al. (2019) : Zhang et al. (2024) reveals that similar gradients in source and target models and lower variance in loss landscapes increase transfer attack probability. Despite transfer attacks being more realistic and practical, not much attention has been focused on this aspect of the safety and robust of FL. We take the initiative to investigate this area and underscore our setting's 's distinctiveness and importance compared to others.
Key Difference 1: Different from query- based or transfer- based black-box attack, we assume the malicious clients possess the data themselves, impacting the target model during training and attack during inference.
3

---

Published in Transactions on Machine Learning Research (11/2024)
We also present comparison of our attack setting and the query-based attack in Section 4.3 attack setting doesn't ' contradict the query-based attack. In fact, we can perform with both if the FL system allows a certain number of queries, which we leave to future works to explore.
Note that our
Poisoning attack or backdoor attack manipulates the parameters update during target model training which can be defended by anomaly detection. Moreover, in practice, despite clients preserving the training data locally, the training procedure and communication with the server are highly encapsulated and encrypted with secret keys, which is even more unrealistic and laborsome to manipulate. Our attack setting circumvents this risk since hostile action is performed during the training but successfully boost the attack possibility during inference time.
Key Difference 2:
Significance: Besides the potential data leakage by malicious participants, we also emphasize that despite, ideally, each participant having access to the global model, in real-world applications (e.g. Gboard Hard et al. (2018)) ; the infrastructure provider will impose additional protection such as encryption or encapsulation over the local training. For instance, Google's Gboard provides next-word prediction with FL, which requires users to install an app to participate. For an adversary, it's impractical to obtain the global model without breaking or hacking the app or hijacking and decrypting the communication, despite all the things happening 'locally". We believe this is much more difficult and laborsome than our setting which significantly boosts the transferability by simply acting as a benign.
3 Investigation Setup and Research Goals
We aim to investigate the possibility of transfer attack with limited data and validate whether it is possible and practical for the attacker to lay benign during the training process and leverage the obtained data to perform the adversarial attack.
GOAL 1:
We aim to explore how different degrees of decentralization, the heterogeneity of data and the aggregation, i.e. average affect the transferability of the adversarial examples against. the FL model in a practical configuration.
GOAL 2:
3.1 Experiment Setup
Thrcat Model. Following Zizzo et al. 2020). we use PGD (aady et al. 2017 with 10 itcrations, no random restart, and an epsilon of 8 255 over Lex norm on CIFARIO. For experiments on ImageNet200, we use PGD (Madry et al. 2017) of the same setup but with an epsilon of 2 255.
Scttings. We first build up the basic FL setting. We split the datatset into 100 partitions of equal size in an iid fashion. We adopt two models for the experiments: CNN from (McMahan et al., 2017) since it is commonly used in the FL literature and the widely used ResNet50 (He et al. 2016) which represents more realistic evaluation. We conduct training in three paradigms: the centralized model, the federated model and the source model with a limited number of clients' data. For the federated model, we use SGD without momentum, weight decay of 1e-3, learning rate 0.1, and local batch size 50 following (Acar et al. 2021 b. We follow the cross-device setting and use a subset of clients in each round of training. We use a 10% as default where not specified. We train locally 5 epochs on ResNet50 and epoch on CNN. For centralized and source model training, we leverage SGD with momentum of 0.9, weight decay of 1e-3, learning rate of 0.01 and batch size of 64. For adversarial training, we use the same setting as centralized and leverage PGD to perform the adversarial training. We refer to (Zizzo et al. 2020 for the details of adversarial training. All experiments are conducted with one RTX 2080Ti GPU.
Metrics. We report Accuracy (Acc) and adversarial accuracy (Adv.Acc) for the performance and the robustness of white-box attack, and for adversarial transferability, we report transfer accuracy (T.Acc) and transfer success rate (T.Rate) detailed in 3.2
4

---

Published in Transactions on Machine Learning Research (11/2024)
3.2 Adversarial Transferability in Federated Learning
To define the transferability of adversarial examples, we first introduce the definition of the source model, target model and adversarial example. The source model is the surrogate model used to generate adversarial examples while the target model is the target aimed to attack. Given the validation set x = {( (s)]: source model f', target model f and adversarial perturbation function adv(..)) (e.g., PGD), we first define the following sets:
81 (xxl vi), f')) + uc
s2-
s3 (ailf(x;)= Vit:
s4 {zilf(ado(<z; f')) vi}
Adversarial examples are defined as those samples that are originally correctly classified by model f' but are misclassified when the adversarial perturbation added, i.c., sl s2. Adversarial transferability against the target model refers to the ability of adversarial examples generated from the source model attack the target model (become an adversarial example of the target model). We define transfer rate (T.Rate) and transfer || where
accuracy (T..cc) to measure the adversarial transferability: T.Rate = [|s1^>2n<3]]
[ T.Acc = 1 - |[=||
||- denotes the cardinality of set. We are the first one to propose and use the Transfer Rate metric to measure the transferability of adversarial examples which measures the transferability of the surrgatte model by measuring the portion of transferable examples. This serves as a complimentary to accuracy as plain accuracy fails to accurately measure robustness.
4 Experiments
4.1 Robustness with Comparable Accuracy
In order to provide a preliminary understanding about the robustness of the FL model, we train the centralized model for 200 epochs and the federated model for 400 rounds resulting in decent accuracy of over 90% (see the regular column of Tab. D. For the CNN model, we train 200 epochs for the centralized model and 600 rounds for the federated model to achieve accuracy of over 75%.
Table 1: Centralized and federated model under white-box attack. We can see that, with compa- rable clean accuracy, the FL model shows greater robustness against white-box attacks compared with the centralized model. This observation is consistent across different datasets and model ar- chitectures.
We can observe that the federated model's clean and ad- versarial accuracy is lower than its centralized counterpart, aligned as the result in (Zizzo el al.j 2020). However, we conjecture that such an increase in adversarial accuracy not attributed to the intrinsic robustness of the centralized model but largely due to its high clean accuracy. To validate this hypothesis and facilitate a fair comparison between the two paradigms, we early-stop both models when their clean accuracy reaches 90% (75% for CNN) and report the results in the same-ace column of Tab. [ We early stop at 80% for adversarial training (72% for CNN). We can see that when both models reach comparable clean accuracy, the FL model shows greater robustness against white-bor box attacks compared with the centralized model.
Paradigm Architecture
same-acc Acc
regular Ace
Adv.A
Adv.A
CIFAR10
R50
95.24
0.40
R50 (adv)
81.23
23.27
89.46
46.09
centralized
CNN
75.06
1.24
82.41
0.35
CNN (adv)
73.15
20.89
76.78
28.92
R50
90.29
0.05
92.31
0.02
R50 (adv)
$0.05
36.44
81.05
39.11
federated
CNN
75.09
3.68
76.83
3.98
CNN (adv)
72.85
25.5
72.87
24.35
ImagoNNt220
R50
55.05
3.68
65.79
8.41
centralized
R50 (adv)
50.04
31.20
55.59
32.84
R50
55.03
13.42
60.59
15.68
federated
R50 (adv)
50.18
38.31
54.92
41.13
To further validate our hypothesis, we perform the experiment on a much larger and more realistic dataset, í.e. ImageNet200. We early stopped both models at 55% accuracy. Results can be seen the bottom two rows of Tab. [ We can see that FL models demonstrate superior robustness against white-box attacks compared with the centralized model on both same-ace and regular settings.
5

---

Published in Transactions on Machine Learning Research (11/2024)
4.2 Robustness Against Transfer Attack
We turn to the black-box attack which is more practical and realistic in real-world applications. We explore the examples generated by two different training paradigms and their transferability to different models. Since the similarity of decision boundary and clean accuracy influences and reflects the transferability between models (Goodfellow
Table 2: T.Rate and transfer accuracy of PGD attack between pairs of models us- ing various training paradigms. The row and column denote the source and largel model respectively. For each cell, the left is the transfer accuracy and the right is the T. Rate.
al.) 2014 Liu et al., 2016;; Demontis al.) 2019), we early stop both federated and centralized models. For CIFAR10, we early stop both models at. 90% of accuracy (75% for CNN). For ImageNet200,, we follow Section 4.1 and early stop at 55%. We follow this training setting for the rest of this paper.
federated
contralized
CIFAR10
Tab. 2 shows that the adversarial examples generated by the fed- erated model are highly transferable to both the federated and
federated 0.15 / 99.83
2.01 / 99.67
R50
centralized 24.28 71.94
7.41 / 91.48
centralized model while adversarial examples generated by the cen- tralized model exhibit less transferability. The T.Rate of federated- to-centralized attack is even larger than centralizaa--o-ccnttaallzzz allack. Secondly, T.Rate of adversarial examples between models trained under same paradigms is larger than models trained under different paradigms, which can be attributed to the difference of the two training paradigms, e.g., the discrepancy in the decision
federated 19.31 / 76.32
21.59 / 71.84
CNN
centralized 30.57
56.59
22.62 / 68.19
ImageNet200
federated
2.29 / 95.60
6.52
/ 86.74
R50
centralized
22.72 54.00
8.27
83.13
boundary Gooddellow et al., 2014 Liu et al., 2016) or different sub-space Tramer et al.
2017
4.3 Transfer Attack with limited data
In this section, we comprehensively evaluate the practicality and plausibilitty of our proposed attack setting, e.e. transfer attack with limited data. We present the overview of our attack setup in Algorithm [ To simulate this scenario, we fix the generated partition used in the federated training and randomly select a specified number of users as malicious clients whose data is available for performing the attack. To perform the transfer atlack, we train a surrogate model in a centralized manner with the collected data. Training details are specified in Section 3.1
Algorithm 1: Our Attack as Benign Setting
Input: set of N clients {C1, C2,..., CN) where M clients (Ci, Co. Ca....,,Ca Cs3 are corrupted by attacker, datasets X, for each client C,, sampling rate K, total communication round T, local iteration number E and surrogate model training iteration number Tg.
Output: Adversarial perturbation ō on example
1 Normal Federated Training with corrupted clients acting benign */ initialize FL model fo; for each round t == 1,2..... T do
Server samples K devices St and distributes the global model fe for each client C, in S, do in parallel
fi+1 + Client (ieentupdate(:....
end
f Aggregate(f4+11 ff41)
end
y Train surrogale model *
Adversary collects data from corrupted clients (C1, "aa......,,,, initialize surrogate model f6; for each iter t = 2, ......... Ts do
ft Clientt ClientUpdate((f_ U M Xi)
end
+ AdvPerturb(['..o
One of the key differences between the proposod sctting and the conventional transfer-basod attack is the amount of available data to train the substitute (source) model, which key factor for a successful attack since one would reason that more data will lead to higher success rate. This is measured by the number of

---

Published in Transactions on Machine Learning Research (11/2024)
clients used to train the source model. To provide overview of the transferability of adversarial examples generated by the source model trained with different numbers of clients, we plot their relation in Fig. 1 We have the following observations:
Observation 1. T.Rate increases as the mumber of users increases, which is consistently observed in both centralized and federated models.
Observation 2. With only 20% of clients the source model achieves an T.Rate of 90% and 50% with ResNet50 and CNN respectively. We notice that with ResNet50, the T.Rate of 20% clients is even larger than a transfer attack with full training data (71.94% T.Rate). With CNN, the source model trained with 20% clients can achieve 50% T. Rate which only lags behind the transfer attack by 6% (56.59%).
From observation 2, we can see that the proposed attack can achieve comparable or even better T. or lower T.Acc. Consequently, we can conclude that the proposed attack setting with limited data is likely to cause significant security brcaches in the current and future FL applications. To further explain observation 1 and an intriguing phenomenon that the T. T.Rale of ResNet50 model rises to the peak and then decreases, provide the following hypothesis: When the mumber of clients used to train the source model is small, the clean accuracy of the source model is also low, leading to large discrepancy in the decision boundary. Increasing the number of users used in the source model minimizes such discrepancy until the amount. of data is sufficient. to train a source model with similar accuracy. At this point, the difference between the federated and centralized (Caldarola et al. 2022) becomes the dominant factor affecting the transferability since the source model is trained in the centralized paradigm.
[s]ReeNes00 on CIFARIO
(bjCNN on CIFAR10
(e (hhnsstig trander rate
(d)trander rate in different training stege
100
cestrdloed
traader aet
Selvaae
trandder
200e
(eBReSSetaeonIIAARRO with and 30
trander rate
100
inander
Inndre
28
number of 100e
manbbbr 1040
number of uses"
1M
number of user
number of user
Figure 1: Attack with data from limited number of users. (a) We show the transfer rate of our attack with ResNet50 on the CIFAR10 dataset. We additionally experiment with attackers of different architectures. (b) experiment with CNN on CIFAR10 dataset. (c) we boost the performance of our atlack with standard augmentation and pretraining techniques. (d) we perform our attack on different training stages. (e) we provide experiments on easier scenarios with 10 and 30 users, which further demonstrate the threat our attack poses.
Training surrogate model in a federated manner. To validate the conjecture above, we train the surrogate model in federated manner. Specifically, since we know each sample's client, we partition the collected data from malicious clients as the federated model and train the source model in a federated manner. Results can be seen in (c) of Fig. [ with a federated source model, the T. RaRa can be slightly boosted at the beginning (limited number of clients) but continue to increase as the number of users increases and finally contributes to a significantly high T.Rate of 99%. This demonstrates our conjecture and also shows that if the hostile party trains the surrogate model in a foderated manner, the T.Rate can be further incrcasod. We further evaluate whether the partition information is crucial to the higher transferability in Appendix A
Boosting transfer rate. Following the conjectures above, we propose several effective approaches to enhance the transfcrability of the surrogate model trained by the malicious data only. We first boost the T.Rate of the source model trained with limited data with data augmentation and model pretraining. Without loss of generality, we leverage AutoAugment (Cubuk et al. 2018) and ImageNet pretrain. We believe other forms of augmentation and pretrained weights will exhibit similar effects. From (c) of Fig. [ we can see, with these techniques we can successfully increase the T. Rate of 1% and 2% of clients from around 3% to 36% and 48% respectively. With 7% or 10% of clients' data, the proposed attack setting achieves a high T. Rate of more
7

---

Published in Transactions on Machine Learning Research (11/2024)
than 80% (10% higher than the transfer-based attack). With simple training techniques, malicious clients can attack with more than 40% success rate with one or more clients and 80% with 7 to 10 clients.
Advanced Transfer Attack. To further show the threat posed by our attack, we leverage more advanced attacks to replace the default PGD attack as shown in (c) of Fig. [ both AutoAttack (Croce & Hein 2020
and skip attack (Wu et al. 2020a) achieve a transfer rate of over 80% with only 3 and 10 users respectively. Noticeably, AutoAttack achieves the highest transfer rate of 85% with 10% of the users.
Comparison with query-based attack As elaborated in Sec. 2 our transfer attack with limited data is similar but different from the query-based black-box attack, as we assume the malicions clients possess a portion of the original training data. Query-based black-box attack, on the other hand, aims to attack the larget model with limited queries to the larget model. Through these queries, the source model manages to optimize the decision boundary towards the target model. However, most APIs and FL-based systems roquire charges access the service or equip anomaly detection that detects multiple or malicious querics.
We provide comparison of our proposed attack setting and the query-based black-box attack. To facilitate follow the same experiment setting to train the query-based model. We plot the comparison in (a) of Fig. 2 Despite query-based outperforming the default of our proposed attack by a slight margin at the cost of expensive queries, our attack combined with federated training outperforms the query-based attack consistently at all configurations. It is also pertinent to note that facilitating the queries for data from only one client necessitates the execution of 500 requests to the target model (in a 100-client partition). Further, the query cost demonstrates proportional escalation as the number of required queries increases, which may be regarded impracticable in tangible operational contexts. Moreover, that our attack does not counteract the query-based attack. In fact, we can perform both if the FL system allows a certain nunber of queries. We believe that, combined with a limited nunber of queries, our attack strategy will be augmented to attain a higher transfer rate since the queries from the target model can help the surrogate model learn a decision boundary that more congruently aligns with that of the target model.
fair comparison,
underscore
10
100
100-
20-
80-
transfer rate
transfer rate
trarsfer
40
40
20
quary-lused
20-
20-
SOUROe moel braind on CIFARIOD
ours (ruadar)
ear
0-
aus (feebentte))
(fedorlee +AudloAuggonn))
dia bmputiilative nnm moddl partiduutins tww. tideeeo Das tote
0-
0
20
3D
100
10
20
30
30
60)
80
100
number of USETS
number uSETS
Taumber of users
Figure 2: (a) Comparison with query-based black-box attack. (b) Attack with the surrogate model trained with the same distribution (no participation in the FL training) (c) Attack with the surrogate model trained with similar knowledge, Le. CIFAR100.
Practical evaluation. To practically evaluate our attack setting, we consider two cases: 1. the malicious clients can only participate in some periods during the FL training and 2. the attacker has no knowledge of the model deployed (i.c. unknown architecture). We simulate these situations by attacking in different. training stages and with different architectures, as shown in (d) of Fig, I
Is Data Distribution Similar Knowledge Sufficient? We emphasize the significance of the process that malicious clients stay benign during training and affect the training with their own data. We demonstrate in this section that this obtained training data is crucial to a successful attack.
We first show that using a similar distribution is not as good as the actual training data to train a surrogate model, as shown in (b) of Fig. 2 We note that the inherent hetcrogeneity of FL training will further impose challenges. The distinct distribution of each client's data makes it nearly impossible to approximate an overall training distribution due to the scale and variability of the data.
8

---

Published in Transactions on Machine Learning Research (11/2024)
Training the surrogate model with dataset of similar knowledge or characteristics is also ineffective. We simulate this by using the CIFAR100 dataset to train surrogate model and perform the attack against the FL model trained on the CIFARIO dataset, as shown in (c) of Fig. 2 We also want to emphasize that our approach to obtaining the training data by acting benignly during training is practically impossible to defend against as there no way to distinguish a benign client and malicious client. if they stay benign.
More Configurations. We also conduct experiments with 10 and 30 users as shown in (e) of Fig. ! Whiich is relatively simpler setting. Specifically, uses, attack achieves the highest 90% transfer rate with only 2 and 5 malicious clients respectively. When there are only 1 malicious client in the 10-user setting and 3 malicious clients in the 30-user setting, we can achieve over 60% transfer rate and over 70% transfer rate. We further perform experiments on CIFAR100, SVHN and a much larger and more realistic dataset ImageNet200, as demonstrated in (a) of Fig. where similar trends are demonstrated. These results emphasize the threat of our attack setting.
emphasize that when the federated model is trained on 10 and 30
5 Two intrinsic properties contributing to transfer robustness
To fully undersland how adversarial examples transfer between centralized and federated models, study two intrinsic properties of FL and its relation with transfer robustness. To protect the privacy of clients and leverage the massive data from user-end, FL utilizes distributed data to train a global model through local updates and aggregation at the server (McMahan el al. 2017). As a consequence, the heterogeneity of the distributed data and the aggregation operation the core component of an FL method. In this section, we study how these two properties affect the transfer robustness of the FL model.
(a) ResNet50 on CIFARI0
(c) transfer rate v.s. dirichlei alpha different training stage
40
2.44
18]
200
mumber soers
J00
400
6.2
0.5
alpho
imbolance sgm
0.5
0.0
Irainiing
(b) CNN on CIFAR10
(d) transffer rale v.s. dirichlet alpha
Crasster
10
0.2
USErS
dirichler alpha
Insbolance sgm
nlpha
Figure 3: T.Rate vs. data of different heterogeneity and dispersion degree. (a): top 3 are results of ResNet50; (b): bottom 3 are results of CNN; Left: T.Rate as a function the number of users in federated training; Middle: T.Rate as function of dirichlet alpha; Right:T Rate as function of unbalanced sgm.
5.1 Decentralized training and data Heterogeneity
This section aims to explore the relationship between the degree of heterogeneity and transfer robustness.
Control the degree of dispersion and heterogeneity. To explore the impact of distributed data on adversarial transferability,
control decentralization and heterogencity through four indexes. By varying the number of clients in the partition, we alter the degree of dispersion of distributed data. We provide two approaches to control the hetcrogeneity of the distributed data. 1. Change the number of maximum classes per client (McMahan et al., 2017). 2. Change the alpha value of Dirichlet distributions (a) (Wang el. al. 2020: Yurochkin et al. 2019) (smaller a means a more non-iit -id partition) and the lor-normal variance (sgm) of the Log- Normall distribution (larger variance denotes more unbalanced partitions) used in unbalanced partition ZZeng al. 2021). We leverage the FedLab framework to generate the different partitions (Zeng al. 2021 ). For simplicity, leverage the centralized trained model as the source model for the rest of the experiments.
9

---

Published in Transactions on Machine Learning Research (11/2024)
Degree of decentralization reduces transferability. We first explore the relation between decentralization and transferability. To control the degree of decentralization, we generate partitions with different numbers of clients and train target federated models on these partitions. Then we perform the transfer attack using the centralized model as the source model. As seen in Fig. t (left of (a) and (b)), we can observe that despite some fluctuations, T.Rale drops with an increasing number of users, which demonstrates that more decentralized data leads to lower transferability. We provide statistical testing for the correlation coefficient in Appendix [ to further validate the above observation. With the Spearman correlation coefficient, we report significant negative correlation on both ResNet50 and CNN between the degree of decentralization and R.Rete under significance level of 0.1 with p-value=0.03 and 0.01 respectively for ResNet50 and CNN. We also visualize linear regression to fit the negative correlation between the degree of decentralization and T.Rale the in Appendix H
Data heterogeneity affects transfer attack. As discussed in Soc. 5.1 we provide two approaches for controlling the helerogeneity of the data. First, we alter the alpha values of the Dirichlet distributions to generate heterogencous data of different degroes. As per the middle plot of (a) and (b) in Fig. 3 we
GO Mo
OUONN CHARD
Metlen
can see that T.Rate increases as a increases.
maddmum cfclaase Loeer
We also alter the maximum number of classes per client to generate heterogeneous data of different de- grees. Results are reported in Fig. We can observe from Fig. 4 a clear increase trend in both the 10- user partition and the 100-nser partition setting with ResNet50 and CNN. As the degree of heterogeneity decrcases, the transfcrability increases. Experiments in both settings can illustrate our findings. This proves that our observations hold wider circum- stances and scenarios.
Figure 4: Transfer rate v.s. maximum number of classes per client; (a): Results of ResNet50 on CIFARIO dataset; (b): Results of CNN CIFAR10 dataset; It is notewor- thy to observe that the transfer rate in the 10-user setting persistently surpasses that in the 100-user setting. This fur- ther substantiates the proposition that more decentralized training leads to lower adversarial transferability for the federated model.
We further explore whether unbalanced data (i.e. different clients possess different numbers of samples) leads to decrease in transferability in Fig. 3 (right of (a) and (b)). We can observe that a larger variance leads to a lower transfer rate, meaning that unbalanced data also contribute to higher robustness.
To validate the above observation, we provide statistical testing for the correlation coefficient Appendix C With the Spearman correlation coefficient, we report significant negative correlation on ResNet50 between log-normal variance and T.Rate under significance level of 0.1 (p-value-0.05). We report significant correlation on all results with level of 0.05 except the CNN experiments with different Dirichlet and unbalanced sgm. Visualization of linear fitting is in Appendix H To provide a more practical evaluation, we follow the sctting in practical evaluation of Sec. 4.3 and evaluate the above findings in different training stages and with different architectures. As per (c) and (d) of Fig. 3 we can observe a similar correlation and trends between Dirichlet alpha and transfer rate. This further illustrates that our findings hold in various settings and configurations.
5.2 Averaging Leads to Robustness
We explore the other core property, averaging operation, of FL and its correlation with transfer robustness. To change the degree of averaging in FL, we alter the number of clients selected to average at each round. To comprehensively illustrate this finding, we use two source models to attack. The first one is centralized model trained with all the data. The Second surrogale model is trained with 30% of all the clients' data, simulating the attack in Sec. [.3] We plot the relation of T.Rate and #averaged users in (a) to (d) of Fig. 6 Both CNN and ResNet50 exhibit a docreasing trend as more users participate in the averaging operation. This demonstrates that the averaging operation contributes the robustness of the FL model and more clients to average per round leads to higher transfer robustness. We provide isailltiaal testing to validate
10

---

Published in Transactions on Machine Learning Research (11/2024)
the correlation in Appendix C With the Spearman correlation coefficient, report significant negative correlation in all four experiments (all p-values are less than .001).
To provide more practical evaluation, we follow the setting in practical evaluation of Sec. 4.3 and evaluate the above findings different training stages and with different architectures. As shown in (e), (f) of Fig. 6
we can observe a similar trend as mentioned above. This demonstrates that this phenomena hold more practical settings and wider configurations. We further illustrate that this observation generalizes to different
aggregation methods in Appendix B
ResNet18 SVHN
Jearce tralad
Sederated
tramneeran
trandder
ResNet50 CIFAR100
dirichlet
11507 anvernge
seree triled
trander
rrvestre
18
10
380
number
ResNet50 ImageNet200
12
transteraa
Trandter cale
transfer rale
Sndersted
controllzed
dliens
193
number users
dirichlet
usems sverraee
attack
(b) decentralization
(c) heterogeneity
averaging
Figure 5: Additional experiments on SVHN, CIFAR100 and ImageNet.200. (a) Our attack with ResNet18 on SVHN (first row), ResNet50 CIFAR100 (second row) and ResNet50 on ImageNet200 (third row). (b) How decentralization relates to the transfer robustness FL model on SVHN (first row), CIFAR100 (second row) and ImageNet200 (third row) datasets. (c) How heterogeneity relates the trausfer robustuess of FL model SVHN (first row), CIFAR100 (second row) and ImageNet200 (third row) datasets. (d) How averaging operation relates to the transfer robustness of
FL. model SVHN dataset (first row), CIFAR100 (second row) and ImageNet200 (third row) datasets.
5.3 Discussion
We summarize the above investigations as the below take-home messages and provide implications for understanding adversarial transferability FL and secure FL applications:
The heterogencous data and large degree of decentralization both result in lower transferability of adversarial examples from the surrogate model. + The attacker benefit from closing the discrepancy between the surrogate model and the target model (e.g., train the surrogate model in federated manner).
With more clients to average at each round, the federated model becomes increasingly robust to black-box attacks. + Defenders can benefit from increasing the number of clients selected at each round to average. In addition, we also identify a different, simpler, but practical attack evaluation for FL, which can serve as the standard robustness evaluation for future FL applications.
11

---

Published in Transactions on Machine Learning Research (11/2024)
ResNee50((eettraliiee))
(c) (NN(centralize))
(e) transfer rate v.s. Buser average
70
68
transfer rate
64
rate
10
76-
62-
30
60
50
58
70
68
56
100
64
20
50
10
30
100
50
20%
40%
80%
100%
number of user average per round
number of user average per round
training stage
(b) (esSee50(30users)
(d) CNN(30users)
(f) transfer rate Puser average
96
185
RoNNa50-RosNee5
80-
MANN650-VGG11
transfer rate
48
transfer rate
75
VGG16-Bes NetSo
86-
84-
82
44
42
40
38
72
70
100
30
100
70
number of user to average per round number of user to average per round
number of user to average per round
Figure 6: Transfer rate v.s. different number of clients selected to average in each round; (a) ResNet50 results with source model trained in centralized manner with full data; (b) ResNet50 results with source model trained with 30 users; (c) CNN results with source model trained in centralized manner with full data; (d) CNN results with source model trained trained with 30 users;
6 Supporting Theoretical Evidence
Notation. We use (X. Y) to denote a dataset, where X E RRXP and Y E RR. We use (xr, y) to denote
sample following Section 2 We consider the problem of federated learning optimization model: ming ((())
((f(;; (x; v), N is the total number devices and ng is the number E (0)), where (0) + ,
mg
of samples possessed by device k. Pk is the weight of device with the constraint that Pt. 1. Following Li et al. (2020b). we assume the algorithm performs total of TT' stochastic gradient descent (SGD) iterations and each round of local training possesses E iterations. E is thus the total communccation times. At each communication, maximal number of K devices will participate in the process. We quantify the degree of non-iid using I' ===4 1, Emm Pulka where I, and 4. is the minimum value of ((0) and ( (0) respectively. We use the relative increase of adversarial loss Demontis et al. (2018) to mcasure the transferability for casier derivation, which defined as the loss of the target model an adversarial example, simplified through a linear approximation of the loss function: T [(f(x + 8; y) = (z; 0). y) + 8 (a,((((; 0), y), where å is some perturbation generated by the surrogate model, which corresponds the maximization of an inner product over e-sized ball under the above linear approximation:
E arg max ((f' (x + 8;0)), [
max 8" =(V
where Ill, and ||]l are dual norm.
Without loss of generality, we take p 2 and gives optimal 8 cx) (.) 0'), y)12 from the surrogate model. Substituting it back to the Equation t we have the loss increment, bounded by the loss of the white-box attack:
V V=l(f(<; V selVVAA(((0))
Al
(V We define the relative increase loss the black-box case compared the white box as R(x, v), which we show has lower bound.
".
R(z.y,0,0)=
(5)
12

---

Published in Transactions on Machine Learning Research (11/2024)
Then we provide low bound for R. Theorem 6.1. With Assumptions 2-8, we have:
2p(7 110000
2,, 4(B C)K B22 -1112
(6)
where L, pl ok. are defined in the assumptions, k - 7 max 8kc, Et B B= +
and C AE 0, is the parameter after one step update of SGD. O. is the optimal parameter for the centralized model.
We refer to Appendix D for proof.
Corollary 6.2. We derive the lower bound of the expectation of R(x, 0z 0") by setling E = and K = 1, where 0" represents the centralized source model.
2p(7 + T 1
E[R(2, y, 0.,00,00 0,),
4 of + 67T 4Ga) + p² 0- 021
Remark 6.3. The difference between the lower bound of FL (Lemma D.1 and the centralized model (Corollary 6.2 lies in the denominator. With FL model,
B B+C = Σ
4 pi + 677 8(E-+8
E² K
1=1
while contralized gives a smaller
B B+C= Σ PRo + 6L7 + 4G
leading to larger lower bound. Thus, the transferability of adversarial examples generated by the surrogate centralized model to attack the federated model is less than that when attacking centralized model.
Remark
6.3 supports the empirical findings in Section 4.2
Remark 6.4. With Thcorem 6.1 we see the degree of non-iid I lies in the denominator of the lower bound, meaning that the larger the degree of non-iid among devices, the less the trausferability of examples generated by centralized surrogate model.
Remark 6.4 aligns with our empirical findings in Section 5.1 which will provide more insights to future research on federated adversarial robustness.
Theorem 6.5. Let T denote the train set (X,)) sampled from some data distribution and x' denole the adversarial ezample following some adversarial distribution. Let f(. T) be the model trained with dataset T and f(.) [T)(( the expectation of the model trained on dataset T. We denote the an average of n models as fn En TT(: Then we have:
ES ET Ill fa(x') [[3] E.t a [lll [(x')12) + 1 E,,, [11) [( T
Remark 6.6. In Theorem 6.5 E. Illy - [(x)) and Fel.T(1l[('')) - f(a, T) only depends on f and are fixed with respect n. Thus, n becomes larger, the expected error decreases.
Remark 6.6 supports the observation in Scction Scction 5.2
7 Conclusion
We explore the potential for malicious clients to masquerade benign entities in Federated Learning, then exploit this position to launch transferable attacks. We provide thorough investigation of the proposed
13

---

Published in Transactions on Machine Learning Research (11/2024)
attack with limited data setting. Our evaluation shows that limited data yield comparable transfer rate to full-dataset attack.
To fully understand how adversarial examples transfer between centralized and federated models, we further study two intrinsic properties of FL and its relation with transfer robustness. We discover that decentralized training, heterogeneous data, and averaging operations enhance transfer robustness and reduce the transfer- ability of adversarial examples. We provide evidence from both the perspective of empirical experiments and theoretical analysis.
Our findings have implications for understanding the robustness of federated learning systems and poses practical question for federated learning applications.
References
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint ar arXiv:2111.04263, 2021.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Arlificial Intelligence and Statislics, pp. 2938 2948. PMLR, 2020.
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through an adversarial lens. In International Conference on Machine Learning, pp. 634-643. PMLR, 2019.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. arXiv
preprinl ar 1206 6389, 2012.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning test time. In Joint European conference machine learning and knowledge discovery in databases, pp. 387-402. Springer, 2013.
Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient. descent. Advances in Neural Information Processing Systems, 30, 2017.
Debora Caldarola, Barbara Caputo, and Marco Ciccone. Improving generalization in federated learning by secking flat minima. arXiv preprint ar Xix:22200. 11834, 2022.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep Icarning systems using data poisoning. Xi preprind fll:1712.05522 2017.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206 2216. PMLR, 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv: 095011 2018.
Yanbo Dai and Songze Li. Chameleon: Adapting to peer images for planting durable backdoors in foderatod learning. In International Conference on Machine Learning, pp. 6712 6725. PMLR, 2023.
Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, B. Biggio, Alina Oprea, C. Nita-Rotaru, and F. Roli. Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks. Useniz Security Symposium, 2018
Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli. Why do adversarial attacks transfer? explaining transfcrability of evasion and poisoning attacks. Tn 28th USENIX securily symposium (USENIX security 19), pp. 321 338, 2019.
14

---

Published in Transactions on Machine Learning Research (11/2024)
Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. Local model poisoning attacks to {Byzantine- Robust} federated learning. In 29th USENIX Security Symposium. (USENIX Securily 20), pp. 1605-12 1622, 2020.
Clement Fung, Chris JM Yoon, and Ivan Beschastnikh. Mitigating sybils in federated learning poisoning. arXiv preprinl or 1rx: 04866, 2018.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients- easy is it to break privacy in foderated learning? Advances in Neural Information Processing Systems, 33: 16937-16947, 2020.
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural computation, 4(1):1-55 58, 1992.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. Xrv preprinl 14 2014.
Rachid Guerraoui, Sébastien Rouault, et al. The hidden vulnerability of distributed learning in byzantium. In International Conference on Machine Learning, pp. 3521-3530. PMLR, 2018.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arxiv:181 1. .03604, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on compuler vision and pallern recognilion, pp. 770 778, 2016.
Junyuan Hong, Haotao Wang, Zhangyang Wang, and Jiayu Zhou, Federated robustness propagation: Sharing adversarial robustness in federated learning. arXiv preprint arXi::2106.1011 2021.
Robert Hönig, Yiren Zhao, and Robert Mullins. Dadaquant: Doubly-adaptive quantization for communication- efficient federated learning. In International Conference Machine Learning, pp. 8852 8866. PMLR, 2022.
Anbu Huang, Dynamic backdoor attacks against federated learning. arXiv preprint orXic:2011.07429, 2020.
Ling Huang, Anthony D Joseph, Blaine Nelson, Benjamin IP Rubinstein, and J Doug Tygar. Adversarial machine learning. In Proceedings of the 4th A ACM workshop on Security and arlificial intelligence, pp- 43 58, 2011.
Siquan Huang, Yijiang Li, Chong Chen, Leyu Shi, and Ying Gao. Multi-metrics adaptively identifies backdoors in federated learning. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 4652-4662 2023.
Wenke Huang, Mang Ye, Zekun Shi, Guancheng Wan, He Li, Bo Du, and Qiang Yang. Federated learning for generalization, robustness, fairness: A survey and benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Andrew llyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are fcatures. Advances in neural information processing systems, 32, 2019.
Henger Li, Chen Wu, Sencun Zhu, and Zizhan Zheng. Learning to backdoor federated learning. aixiv preprint urXiv:2303.03322 2023a.
Suyi Li, Yong Cheng, Wei Wang, Yang Liu, and Tianjian Chen. Learning to detect malicious clients for robust federated learning. aixio preprint 2xXi:22002.00211 2020a.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In 8th International Conference on Learning Represenlations, ICLR 2020, Addis Ababo, Ethiopia, April 26-30, 2020. OpenReview.net, 2020b. URL httpps///oenrrvviee.ne//ooooo
HJ*MANVEDS!
Yijiang Li, Wentian Cai, Ying Gao, Chengming Li, and Xiping Hu. More than encoder: Introducing transformer decoder to upsample. In 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 1597-1602. IEFE, 2022.
Yijiang Li, Xinjiang Wang, Lihe Yang, Litong Feng, Wayne Zhang, and Ying Gao. Diverse cotraining makes strong semi-supervised segmentor. arXiv preprint arXiv:2308. 092001 2023b.
Jierui Lin, Min Du, and Jian Liu. Free-riders in federated learning: Attacks and defenses. arXiv preprint arXiv:1911.1255 2019.
Tao Liu, Yuhang Zhang, Zhu Feng, Zhiqin Yang, Chen Xu, Dapeng Man, and Wu Yang. Beyond traditional threats: A persistent backdoor attack on federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 21359 21367, 2024.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transfcrable adversarial examples and black-box allacks. arXiv preprint urXiv:1611.02770 2016.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint or arXie 1706 06080,, 2017.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communicttion- efficient. learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp- 1273 1282. PMLR, 2017.
Seyse-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference computer vision and pollern recognition, pp. 2574-2582, 2016.
Thien Due Nguyen, Phillip Rieger, Roberta De Viti, Huili Chen, Björn B Brandenburg, Hossein Yalame, Helen Möllering, Hossein Fereidooni, Samuel Marchal, Markus Miettinen, et al. (FLAME}: Taming backdoors in federated learning. In 31st USENIX Security Symposium (USENIX Security 22), pp.
2022.
Thuy Dung Nguyen, Tuan A Nguyen, Anh Tran, Khoa D Doan, and Kok-Seng Wong. Iba: Towards irreversible backdoor attacks in federated learning. Advances in Neural Information Processing Systems, 36, 2024.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp. 506-519, 2017.
Yu Qiao, Apurba Adhikary, Chaoning Zhang, and Choong Seon Hong. Towards robust federated learning via logits calibration on non-iid data. arXiv preprint arXiv:2403 20t.:22000000033, 02803, 2024.
Amirhossein Reisizadeh, Farzan Farnia, Ramtin Pedarsani, and Ali Jadbabaic. Robust federated learning: The case of affine distribution shifts. Advances in Neural Information Processing Systems, 33:21554 21566, 2020a.
Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fed- paq: A communictiion-efficaann federated lcarning method with periodic averaging and quantization. In International conference on artificial intelligence and slatistics, pp. 2021 2031. PMLR, 2020b.
Virat Shejwalkar and Amir Houmansadr. Manipulating the byzantine: Optimizing model poisoning attacks and defenses for federated learning. In NDSS, 2021.
Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. Can you really backdoor federated learning? Xix preprint 191.0:9911.077663, 2019.
16

---

Published in Transactions on Machine Learning Research (11/2024)
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint
2013.
Florian Tramer, Nicolas Papernot, Ian Goodfellow, Dan Bonch, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint arXiu: 1rXii:1004.000 704.03453, 2017.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint Xrx222002 06440, 2020.
Xinjiang Wang, Xingyi Yang, Shilong Zhang, Yijiang Li, Litong Feng, Shijie Fang, Chengqi Lyu, Kai Chen, and Wayne Zhang. Consistent-teacher: Towards reducing inconsistent psoudo-targets in semi-supervised object detection. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 3240 3249, 2023.
Futa Waseda, Sosuke Nishikawa, Trung-Nghia Le, Huy H Nguyen, and Isao Echizen. Closer look at the transferability of adversarial examples: How they fool different models differently. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1360-1368, 2023.
Dongxian Wu, Yisen Wang, Shu- Xia, James Bailey, and Xingjun Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. arXiv preprint urXiv:2002.55990 2020a.
Weibin Wu, Yuxin Su, Xixian Chen, Shenglin Zhao, Irwin King, Michael R. Lyu, and Yu-Wing Tai. Boosting the transferability of adversarial samples via attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 1161-1170, 2020b.
Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning, pp. 5650-5659. PMLR, 2018.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International Conference Muchine Learning, pp. 7252-7261. PMLR, 2019.
Syed Zawad, Ahsan Ali, Pin-Yu Yu Chen, Ali Anwar, Yi Zhou, Nathalie Baracaldo, Yuan Tian, and Feng Yan. Curse redemption? how data heterogeneity affects the robustness of federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.
2021.
Dun Zeng, Siqi Liang, Xiangjing Hu, and Zenglin Xu. Fedlab: A flexible federated learning framework. arXiv preprint arXiv:2107.11621, 2021.
Yechao Zhang, Shengshan Hu, Leo Yu Zhang, Junyu Shi, Minghui Li, Xiaogeng Liu, and Hai Jin. Why does little robustness help? a further step towards understanding adversarial transferability. In Proceedings of the 45th IEEE Symposium on Security and Privacy (SEP'24), volume 2, 2024.
Zaixi Zhang, Qi Liu, Zhicai Wang, Zepu Lu, and Qingyong Hu. Backdoor defense via deconfounded representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12228-1 112338, 2023.
Wen Zhou, Xin Hon, Yongiun Chen, Mengyun Tang, Xiangqi Huang, Xiang Gan, and Yong Yang. Transferable adversarial perturbations. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 452 467, 2018
Yao Zhou, Jun Wu, Haixun Wang, and Jingrui He. Adversarial robustness through bias variance decomposition: A new perspective for federated learning. arXie preprint urXiv:2009.0900 2020.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in neural information processing systems, 32, 2019.
Giulio Zizzo, Ambrish Rawat, Mathieu Sinn, and Beat Buesser. Fat: Federated adversarial training. arXiv preprint trXiv:2012.01791 2020.
17

---

Published in Transactions on Machine Learning Research (11/2024)
A Attack with Same or Different Partition
We show in Section 4.3 that training surrogate models in federated manner can lead to even higher transferability of adversarial examples. However, in the experiment in Section 4.3 we partition the collected data from malicious clients as the federated model and train the source model in a federated manner. In this section, we further evaluate whether this partition information is crucial to the higher transferability. That is, whether different partitions used by the source model affect the transferability of adversarial examples against the target model. To simulate this setting, we first randomly generate two different partitions with
90
80
75
80
65
00
transfer Tale
70
60
60
25%-75% (different partition)
55
25%-75% (same partition)
50
45
50
Range within 1.5IQR
40
40
Median Line
35
Mean
30
A8
30
Outtiers
25
20
20
15.
10
10
5 7
10 20
30 50
100
20 30 50 70 90 100 number of users
number of users
Figure 7: Difference between same partition and different partition; Left: transfer accuracy v.s. number of users' data leveraged in source model training; Right: transfer rate v.s. number of users' data leveraged source model training
distinct random seeds and then perform the source model training and the target model training the two different partitions. Then transfer attack is performed with the source model against the target model following the above configuration. We repeat the experiment 4 times with different random seeds and report the averaged results in Figure 7 We observe no significant. difference between the same partition and the different partition settings. To further validate this observation, we perform a Hypothesis Test on the obtained results with the Paired Sample T-Test and achieve p-value of .393 meaning that there no significant difference. This further demonstrates the possibility of attacking federated learning system through our proposed attacks and illustrates higher sccurity risk.
B Averaging Leads to Adversarial Robustness with Different Aggregation Methods
We show in Section 5.2 that the averaging operation contributes to the robustness of the FL model and more clients to average per round leads to higher transfer robustness. We further validate that this observation generalizes to different aggregation methods, í.e. Krum Blanchard et al. (017)). Geometric Mean Yin et al. (2018) and Trimmed Mean Yin et al. (2018). As per Figure we can see that with all three aggregation methods, there are decreasing trends as the number of averaged users per round increases.
Figure 8: Transfer rate v.8. #clients selected to average in each round with difference aggregation methods.
18

---

Published in Transactions on Machine Learning Research (11/2024)
C Statistical Hypothesis Testing on Spearman correlation coefficient
In Section 5.1 we elucidate the correlation between the degree of decentralization and the heterogeneity of training data with the transferability of adversarial examples. The findings denote that, with more decentralized, heterogeneous data, the federated model is more robust to transfer attack. Furthermore, Section 5.2 portrays disccrnible inverse relationship between the number of clients and the average transfer success rate, as depicted through box plots, which illustrate the averaging operation leads to better robustines against adversarial examples.
To statistically validate these corrclations, perform two-tailed Hypothesis Testing Spcarman correlation coefficient. To conduct Hypothesis Testing for Spearman correlation coelficient specified correlation, first calculate the Spearman correlation coefficient on the two sets of points (e.g., T.Rate and Dirichlet a):
cov(X,Y
(X))((Y)
where cou(:,) denotes the covariance and (-) represents the standard deviation. To perform the Hypothesis Test, we first have the Null Hypothesis Hy and Alternate Hypothesis Ha:
Ho p0
Ha #
We choose the significance level to be 0.1, which means we reject Ho is p-value is smaller than 0.1. We report the p-value and Spearman correlation coefficient in Table 3
We can see, as reported in Section 5. and 5.2 all experiments except the CNN experiments on dirichlet a and unbalance sgm can deomenstrates significant correlation under significance level of 0.1. More to the point, report numerous correlations with p-value less than .001 (significant under level of .001). This Hypothesis Testing validated the findings of our investigation.
Table 3: Spearman correlation coefficient, and p-value
Architecture
p- p-value (two-tailed)
Spearman cocfficient
dirichlet a
.006
.59
unbalance sgm number of user in partition maximum number of classes (10 users)
.05
-44
.003
-63
ResNet50
.001
0.80
maximum number of classes (100 users)
< .001
.76
number of user to average (30 users)
.001
-71
number user to average (contralized)
< .001
-79
dirichlet a
.76
.074
unbalance sgm
.84
.049
number of user in partition
.001
-88
CNN maximum number of classes (10 users) maximum number of classes (100 users)
.009
.45
.005
.67
number of user to average (30 users) number of user to average (centralized)
.001
-77
< .001
.83
Proof of Theorem 6.1
Following assumptions from Li et al. (020b) and additional assumptions for adversarial transferability, we provide a low bound for R.
Assumption 1 (f 0)/rx 8T ((xx where p(x,8 0) is an arbitrary function in the forms of p(æ, 8) RIxp x Rxx1_ R.
19

---

Published in Transactions on Machine Learning Research (11/2024)
There are many functions following standards (e.g., f2-norm regularized linear regression, logistic regression and softmax classifier). For the rest of our discussion, we define VI([(z; 0). (f;x;))
Lemma D.1. With Assumplion 1, we have
(xx,y,0,0) |00 0 2
With Lemma D.1 0 can be directly measured by the cosine similarity of the parameters 0 and 0', Furthermore, need to offer discussion regarding multiple aspects of the model, such as the loss, the parameters, and the data, we follow the previous convention Li et al. (000bb) to focus on a narrower scope the model family:
Assumption 2 Model f is in the form of f(0) [xfx(=)))
Notice that this is not significant deviation from previous studies Li et al. (200bb) that focus f(0) A0 for detailed investigation of the parameter behaviors.
Assumption 3
The covariance of the samples we study positive semidefinite, ie., x &
Assumption 4:
The loss function I L-smooth: for all v and w, ((v) - (V) + 1/2 11 w u1.
Assumption 5:
The loss function / is p-strongly convex: for all and w, [(v) 2 ((w) + (v w) V +
f 11-wl 1
Assumption 6: Let ti be sampled from the k-th device's local data uniformly at random in iteration The variance of stochastic gradients in each device is bounded by oi: Vl 05) v((00) o
Assumption The expected squared norm of stochastic gradients is uniformly bounded, i.e., E| 0/) G2
G2 for all k 1,..,,,,. NN and 1,".,,T --1.
Assumption 8: We assume the federated algorithm is FedAvg. Assume S, contains a subset of K indices raudomly selected with replacement according to the sampling probabilities PI,'.- ,PN- The aggregation step of FedAvg performs 0, # Ekes, 0r. 0 denotes the parameters of device l iteration
We Theorem 2 from Li ct al. (020b) as our lemma to prove Theorem 6.1
Lemma D.2. With assumption 4 to 8 and L, μ1, a% and G be defined therein. Let T denote the minimum loss obtained by optimal estimation from the centralized model and 1(0') denotes the loss of the federated model. Then
2(B + C)) E E(((0)) - L' 7
MY E - 9+1l)
+
(8)
4
2
where L, M4 ok, G is defined in the assumptions, L n² and C = 4 G² 0, is the parameter after one step update of SGD. 02 is the optimal porameter for centralized model.
y= max (8x, E}} B B E + +
K
Now, we prove Theorem 6.1
Proof. We first write out
((6') L = 0TT XØ' xx_x" x0. (0' x - 02) > 110'121100111
(9)
(10)
(11)
when X is p.s.d.
On the other hand, we write 1(0') L* + where 0. Due to the construction of our model, write
0' = xx (x0,)++
(12)
20

---

Published in Transactions on Machine Learning Research (11/2024)
by solving linear system. Thus, can get
00 0 (x ((X.)2
(13)
(14)
0-
(15)
Thus, by connecting the above terms, we will have
8T
0T
R(x, y. 0., 0,,0'
2
(16)
110112/10.111 1(0)) - L
Finally, by substituting Equation to the denominator, we have
(++ 1)
E E (.0,,,0'))). 2
(17)
2( + C) μ
+ 197 01 0.112)
2
2/177 - T 1)0 0.
(18)
4(B + C)K + 222xBB11 0.11
E Proof of Theorem 6.5
Proof. We adopt the Bias Variance decomposition Geman et al. (1992) to provide theoretical justification for the observation in Section 5.2 Let 'T denote the train set T Y) sampled from some data distribution and x' denote the adversarial example following some adversarial distribution. First, we give the Bias Variance decompasition follows:
Bylly [xx:; R.y Ill f(x')l 1
(19)
Blas? Ey Bew.TIIV (1l s(x'; T
Variance
where f(:T) denotes model f trained on dataset T. f(:)= Er[((; is the expected the model over the data distribution of T. By using an average of multiple model f(:T), denoted by fn fa(; Ta) and with (.) 1 En ()] f(.), we can show the following:
Ey B, fn (x))))) Es Mlly fa (x)))
(20)
Blas
+ Ey [ll (2')))
Variance
1 -E,r - f
E.r.ylly flll--((x')11 f(2')111)
n
Inequality holds due to that Var(# + 2 (1 Ti) = d Var ( (((TT)) and since each Ss trained independently, y, zi Var(En fi(; T.)) nz E Var ((;;
II
F
Attacking Communication-efficieet FL Methods
We also explore whether our attack performs well on commuiictiion-ffiiiant federated methods such as DAdaQuant Hönig et al. (2022) and FedPAQ Reisizadeh al. (2020b). Specifically, we compare the standard Avg, Krum, Trimmed Mean and FedPAQ Reisizadeh al. 2020b
21

---

Published in Transactions on Machine Learning Research (11/2024)
100
transfer rate
60
FedAvg
40
Trimmed Mean
Krum
20
FedPAQ
20
40
60
80
100
#mmllciious dients
Figure 9: Attacking communication--ffiiee. FL.
G Comparison between transfer attack and attack using traces of training.
In this section, we explore using the traces of training (chockpoints during the training of the foderated model) to attack and compare with the transfer attack presented in the paper. We must emphasize that the former is different threat model with the capability of keeping traces of the model at different stages of training. That is to say, the attack is purely white-box one.
We present the attack result using checkpoints of different stages of the training as shown in Tab 4 We early stopped the model when the accuracy reaches 90% which is around 75 epochs. Then we use the checkpoints from 20, 40 and 60 epochs to perform the attack.
white-box
20e
40e
60e
transfer rate
91.48
89.98
91.45
94.66
1% transfer 10% transfer
30% transfer
100% transfer
transfer rate
2.49
57.69
83.91
68.94
Table 4: Comparison between using checkpoint. (traces of training) allack and transfer attack.
H linear regression to visualize the correlation
To better demonstrates the correlation between various factors and adversarial transferability, we perform linear regression with hypothesis testing on the experiment results. We plot scatter graph and linear regression line on each of the correlation and corresponding experiment result as shown the following figures:
22

---

Published in Transactions on Machine Learning Research (11/2024)
Lemutier
rate
tcensfor
100
Licour fil of tomalse F250
transt rate
trunster zate
20-
20-
0.2
0.5
1.0
number of usera
dirichlet wlphe
(a) ResNet50: transfer rate v.s, number of total
(b) ResNet50: transfer rate dirichlet alpha.
in the partition.
cransfer
tesusCer
transtor rate
10
transfer rate
60
50
80
28-
10
10
20
30
40
50
80
80
90
100
110
number clients to averaae
(c) ResNet50: transfer rate v.s. unbalance sgm.
(d) ResNet50: transfer rate v.s. number of users to average per round (source model trained in central- ized manner with full training dataset).
Figure 10: Linear regression visualization
rate [iinear fit of transfer rate
90
80
70
tranefer rate
60
50
40
20-
19
-10
0
10
20
30
40
50
60
70
90
100
110
nunber of clients to average
Figure 11: Res5et50: transfer rate v.8. number of users to average per round (source model trained in centralized manner with 30 client's data).
23

---

Published in Transactions on Machine Learning Research (11/2024)
110
transfer rate
rate
100
trensfor rate
70
meximum number classes
Figure 12: ResNet50: transfer rate v.s. maximum number of classes per user (10 users).
100
rate Linear fit of trannfer rate
70
transfer zate
50
20
0
naximun rumber of classes
Figure 13: ResNet50: transfer rate v.8. the maximum number of classes per user (100 users).
transfer rate
Linesr fit of truewfar rale
90
30
70
transfer rate
60
50
2718
number of
Figure 14: CNN: transfer rale v.8. the number of total users in the part ition.
24

---

Published in Transactions on Machine Learning Research (11/2024)
transter rate
Linear fit of transfer rate
50
transfer rate
60
30
10-
0.2
0.4
0.6
0.8
dirichlet alpha
Figure 15: CNN: transfer rate v.s. dirichlet alpha.
Lineor filt transfer
Urtine Cer 1tle
50
10
imbalance sgn
Figure 16: CNN: transfer rate v.8. unbalance sgm.
Lrattsfer fer rale
90
Linear Til of transfer rate
80
transfar rake
60
60
20
40
30
100
110
nunber
clients
average
Figure 17: CNN: transfer rate v.s. number of users to average per round (source model trained in centralized manner with full training dataset).
25

---

Published in Transactions on Machine Learning Research (11/2024)
100
tranifer
inner fit of transfer rate
80
transfor vato
60-
40
10
number clienta average
Figure 18: CNN: transfer rate v.8. number of users to average per round (source model trained in centralized manner with 30 client's data).
trunsfar rale
100
Lineur Flt trmtefer zate
dirichlet alphe
Figure 19: CNN: transfer rate v.s. maximum number of classes per user (10 users).
Crunsfer
50
Linosr fit traesfo
40
traneeer
40
20
200
400
600
800
100G
Figure 20: CNN: transfer rate v.s. the maximum number of classes per user (100 users).
26


Reviews:
[{"review_id": "cEX9HoVfiJ", "reviewer": "Reviewer_g9J4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This work studies a unique FL security risk and conducts extensive experiments to judge the degree of robustness inherent to such systems against adversarial transfer attacks.\n\nWeaknesses:\n\n1. I am concerned about the motivation and broader applicability for this work as mentioned on page 2: \"Stemming from the above scenarios, we propose a simple yet practical assumption: the attacker possesses a limited amount of the users’ data but no knowledge about the target model or the full training set\". If the attacker participated as a benign client in the FL training and then turned malicious, it would end up having access to the target model as communicated by the server. In that scenario, I am wondering if model inversion attacks seem more plausible. Gboard example makes sense in a setting where models are protected by a layer of encryption as mentioned, but my concerns remain about the broad applicability of the observations presented in this paper.\n\n2. Another assumption I have concerns with is the ability of several malicious clients to jointly train a model in a federated manner to increase the transfer rate of the attack. How do these malicious clients access the server after the original FL training to launch an attack while the server cannot detect their intentions? In my understanding, in FL the server holds control over what kind of training should be done and when, hence I am not convinced that a set of malicious clients can decide to use PGD without the server's awareness. Please provide some clarifications in case I misunderstood the setting described on page 6 under \"Training substitute model in a federated manner\".\n\n3. Some experimental results and claims are unclear (see Requested Changes).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper analyzes the impact of black-box transfer attacks in a federated learning scenario when a set of benign clients turn malicious after the original FL training. The results demonstrate that as compared to a centralized setting,FL systems are more robust to such adversarial attacks. The authors attribute this to heterogeneous data distributions over the participating clients and the averaging operation common in FL.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Since the malicious clients could participate in FL training by pretending to be benign, would the transfer attack accuracy increase if the malicious clients kept traces of the model at different stages of training and utilized that to launch an attack? Please clarify and provide a discussion on this scenario.\n\n2. In Figure 1(d), it is unclear why the attack success decreases over training, please provide a discussion on the same.\n\n3. In Figure 3(b) on the trends with Dirichlet coefficient alpha, the trend that the transferability increases with decreasing degree of heterogeneity does not hold.\n\n4. The robustness of the averaging operation is interesting, but it is equally important to evaluate the impact of sampling the benign clients that turn malicious. In other words, can you provide some experiments to discuss the impact of different rates of benign client sampling during FL training upon the transfer rate of the attack?\n\n5. Communication efficiency is another crucial aspect of federated learning, with several communication compression techniques proposed to reduce the communication overhead [1]. Can you comment upon the attack transfer rate assuming the FL training was done in a communication-efficient manner?\n\n[1] R. Honig et al., \"DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning\", ICML'22.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Skh3BvjhUW", "reviewer": "Reviewer_i4Nu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. Figures looks good\n2. Paper well written\n\nCons:\n1. Not very related to machine learning: most of the contributions are empirical observations and analysis, the contents are not very related to machine learning\n2. Insufficient experiments: experiments mostly on small scale datasets (CIFAR10/100, SVHN). The Federate learning has it's most potential usage in large scale datasets. Results (especially empirical observations) on small scale dataset is not convincing enough.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper analyze the \"transfer-based black-box attacks\", where a subset of training data is assumed to be used to train an adversarial attack model. The main contribution includes:\n1. Empirically observed that federated model is more robust under white-box attack than centralised models\n2. Empirically observed that models trained with the subset of train set can be used for adversarial attack\n3. Empirically investigate the methods that is able to increase/decrease model's robustness to transfer-based black-box attacks\n4. Provides some theoretical analyze on the transfer-based black-box attacks on federated model/centralised model", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "The contribution looks good empirically, but I think this paper may not of interest for machine learning community.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ol7B5RjSjm", "reviewer": "Reviewer_jQmf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The threat model appears to be a natural one.\n- The related work section is fairly rich and I appreciate the flagging of Key Differences.\n- I appreciate the variety of experiments\n\nWeaknesses [see also Requested Changes]\n- Discussion and Conclusion sections are quite limited\n- Clarity of exposition can be improved in multiple places in the submission.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work considers inference/validation-time attacks where adversarial perturbations are constructed from a fraction of the training data. The authors operate in a black-box setting: they do not assume the adversary has knowledge of the target's training procedure. Conversely, the authors assume the adversary is passive during training; it is only during afterward that they act by perturbing a validation set. Success of the attack is measured by a transferability rate (T.Rate) metric and success of the target model is measured by transfer accuracy (T.Acc) metric. The authors' main interest is in the success or failure of these attacks against federated learning.\n\nOne baseline finding is that white-box attacks against centralized models are more successful than those against federated models with the same accuracy. A second baseline finding is that adversarial examples generated by federated training are highly transferable in the unlimited data setting; as shown in Fig 1b and 1c, it holds true in the authors' limited-data setting\n\nMore generally, the authors find (in Section 4.3) that only a small fraction of corrupted clients are enough to attack federated models. As an example, they show T.acc drops to (near) 0 with ~30/100 users on CIFAR10 (Figure 1a)\n\nOn the other hand, introducing more \"federated-ness\" in the training seems to mitigate transferability. Specifically, the authors give experimental evidence that T.Rate is reduced by dividing the dataset across more partitions and by making the distributions of data more heterogeneous.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Post-rebuttal comment**: the authors responded to my questions and requests to my satisfaction\n\n---\n\n*Clarifications*\n\nIs the following a correct description of the interactions between the participants?\n1. Adversary corrupts some fraction of clients/devices\n2. All clients/devices receive data and honestly participate in the training process of M\n3. Meanwhile, adversary feeds the data of the corrupted clients/devices into its own (ML training) algorithm to obtain some perturbation $\\delta$\n4. Adversary receives a set of examples and corrupts them using $\\delta$\n\nIf it is a correct description, consider including a summary like it to make it easier for the reader to understand all the important features of the setting at a glance. As it is, this information is scattered throughout the paper.\n\nPlease clarify the threat model in Section 4.2. In particular, how does it differ from the steps above?\n\nPlease clarify whether the data of the corrupted clients is chosen by the adversary or if they come from the same distribution as the others. Similarly, how is the validation set created?\n\n---\n\n*Questions*\n\nIs the boldface conclusion in Section 4.3 actually unique to FL target models? That is, if we were to repeat the experiments with centralized training, would we get reduced transfer rate?\n\nThe experiments were on CIFAR10. Do you conjecture that similar behavior would occur on different datasets? Why or why not? And what about different types of classifiers (e.g. different architectures)?\n\n---\n\n*Presentation*: Consider re-distributing the information conveyed by the figures. For example Figure 1c has information relevant to \"Training substitute model in a federated manner\" \"Boosting Transfer Rate\" and \"Advanced transfer attack\" which is a bit much for one subfigure.\n\nDo \"source model\" \"substitute model\" and \"surrogate model\" all refer to the same thing?\n\n---\n*Smaller editing errors*:\n- In equation (4) where does the data of the malicious users play a role? It seems x is some fixed value.\n- typo: equation (4) should read argmax instead of argmin\n- typo: Threat Model paragraph begins with the word \"follwing\"\n- typo: Figure 1 (b)'s caption says \"ResNet50\" when the title says \"CNN\"", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The recommendation is based on the reviewers' comments, the action editor's evaluation, and the authors’ response. 

This paper studies a practical transfer attack setting in federated learning and discovers improved robustness in such a scenario. 
All reviewers find the studied setting novel and the results provide new insights. The authors’ rebuttal has successfully addressed the major concerns of reviewers. Therefore, I recommend acceptance of this submission. I also expect the authors to include the new results and suggested changes during the rebuttal phase to the final version.
Decision: Accept (poster)