FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneity
Authors: Kai Yi
Nidham Gazagnadou
Peter Richtárik
Lingjuan Lyu
Venue: ICLR 2024 poster (2024)
Keywords: Network Pruning
Federated Learning
Model Heterogeneity
Primary area: optimization
TL;DR: We introduce FedP3, a flexible and privacy-aware federated network pruning framework, addressing model heterogeneity and allowing client-specific personalization.

Abstract: The interest in federated learning has surged in recent research due to its unique ability to train a global model using privacy-secured information held locally on each client. This paper pays particular attention to the issue of client-side model heterogeneity, a pervasive challenge in the practical implementation of FL that escalates its complexity.  Assuming a scenario where each client possesses varied memory storage, processing capabilities and network bandwidth - a phenomenon referred to as system heterogeneity - there is a pressing need to customize a unique model for each client. In response to this, we present an effective and adaptable federated framework FedP3, representing Federated Personalized and Privacy-friendly network Pruning, tailored for model heterogeneity scenarios. Our proposed methodology can incorporate and adapt well-established techniques to its specific instances. We offer a theoretical interpretation of FedP3 and its locally differential-private variant, DP-FedP3, and theoretically validate their efficiencies.

Published as a conference paper at ICLR 2024
FEDP3: FEDERATED PERSONALIZED AND PRIVACY- FRIENDLY NETWORK PRUNING UNDER MODEL HET- EROGENEITY
Kai Yi'; Nidham Gazagnadou", Peter Richtárik', Lingjuan Lyu² KAUST, Sony Al
kai. www.kyikkkusttcemm edu. nidham. gawagnadoudssnyy..ooooo com peter wwwterrrttaaaikkkuueteeee edu. sa, lingjuan wwe.ooay....ss....oo.m..m com
ABSTRACT
The interest in federated learning has surged in recent research due to its unique ability to train a global model using privacy-secure information held locally on each client. This paper pays particular attention the issue of client-side model heterogeneity, a pervasive challenge in the practical implementation of FL that escalates its complexity. Assuming a scenario where each client possesses varied memory storage, processing capabilities and network bandwidth phenomenon referred to as system heterogeneity there is pressing need to customize unique model for each client. In response to this, we present an effective and adaptable federated framework FedP3, representing Federated Personalized and Privacy- friendly network Pruning, tailored for model heterogeneity scenarios. Our pro- posed methodology can incorporate and adapt well-established techniques its specific instances. We offer theoretical interpretation of FedP3 and its locally differential -private variant, DP-FedP3, and theoretically validate their efficiencies.
INTRODUCTION
Federated learning (FL) (McMahan et al. 2017 Konečný et al., 2016) has emerged as a signif- icant machine learning paradigm wherein multiple clients perform computations their private data locally and subsequently communicate their findings to a remote server. Standard FL. can be articulated as an optimization problem, specifically the Empirical Risk Minimization (ERM) given by
min f(W)
Σ M(W),
WERd
where W represents the shared global network parameters, fi denotes the local objective for client and is the total number of clients.
Distinguishing it from conventional distributed learning, FL. predominantly addresses heterogeneity stemming from both data and model aspects. Data heterogeneity characterizes the fact that the local data distribution across clients can vary widely. Such variation is rooted in real-world scenarios where clients or users exhibit marked differences in their data, reflective of the variety of sensors or software Jiang et al. (2020), of users' unique preferences, etc. [i et al.] (2020a). Recent works Zhao jet al. (2018) 8) showed how detrimental the non-iidness the local data could be on the training of a FL model. This phenomenon known as client-drift, is intensively studied to develop methods
limiting its impact on the performance (Karimireddy et al. 2020 Gao et al. 20226 Mendieta al., 2022).
Furthermore, given disparities among clients in device resources, e.g., energy consumption, compu- tational capacities, memory storage network bandwidths, model heterogeneity becomes pivotal consideration. To avoid restricting the global model's architecture to the largest that is compatible with all clients, recent methods aim at reducing its size differently for each client to extract the ut- most of their capacities. This can be referred to as constraint-based local model personalization (Gao)
Work done during internship Sony AT.

---

Published as a conference paper at ICLR 2024
et al. 2022a). In such context, clients often train a pruned version of the global model (Jiang al. 2022b Diao et al. 2021) before transmitting it to the server for aggregation (Li et al. (021). A con- temporary and influential offshoot of this is Independent Subnetwork Training (IST) (Yuan et al. 2022]. It hinges on the concept that each client trains a subset of the main server-side model, sub- sequently forwarding the pruned model to the server. Such approach significantly trims local computational burdens in FL (Dun et al. (023).
Our research, while aligning with the IST premise, brings light some key distinctions. A signif- icant observation from our study is the potential privacy implications of continuously sending the complete model back the server. Presently, even pruned networks tend to preserve the overarching structure of the global model. In this paper, we present an innovative approach to privacy-friendde pruning. Our method involves transmitting only select segments of the global model back to the server. This technique effectively conceals the true structure of the global model, thus achieving a delicate balance between utility and confidentiality. As highlighted in Zeiler & Fergus] (2014), different layers within networks demonstrate varied capacities for representation and semantic in- terpretation. The challenge of securely transferring knowledge from client to server, particularly amidst notable model heterogeneity, is area that has not been thoroughly explored. It's pertinent to acknowledge that the concept of gradient pruning as a means of preserving privacy was initially popularized by the foundational work of Zhu al. (209)). Following this, studies such Huang et al. 2020) have further investigated the efficacy of DNN pruning in maintaining privacy.
Besides, large language models (LLMs) have garnered significant attention and have been applied to a plethora of real-world scenarios (Brown et al. 2020; Chowdhery et al., [2022] Touvron et al. 2023) recently. However, the parameter count of modern LLMs often reaches the billion scale, making challenging to utilize user or client information and communicate within a FL framework. We aim to explore the feasibility of training more compact local model and transmitting only a subset of the global network parameters to the server, while still achieving commendable performance.
From a formulation standpoint, our goal is to optimize the following objective, thereby crafting a global model under conditions of model heterogeneity:
min Wi....W,, W,, ERdd
f(W) = (f((W), fwW.))
where W; denotes the model downloaded from client to the server, which can differ as we allow global pruning or other sparsification strategies. The global model W is function of {W1, W W,J. ft the local objective for client and n the total number of clients. Func- tion h is the aggregation mapping from the clients the server. In conventional FL, it's assumed that function h is the average and all W1 = :... Wa = W, which means the full global model is downloaded from the server to every client. When maintaining global model W. this gives us f(x) = E (iW), which aligns with the standard empirical risk minimization (ERM).
In this paper, we introduce an efficient and adaptable federated network pruning framework tai- lored to address model heterogeneity. The main contributions of our framework, denoted as FedP3 (Federated Personalized and Privacy-friendly network Pruning) algorithm, are:
Versatile Framework: Our framework allows personalization based on each client's unique constraints (computational, memory, and communication).
Dual-Pruning Method: Incorporates both global (server to client) and local (client- specific) pruning strategies for enhanced efficiency.
each client by limiting the data shared with the server to only select layers loca-local training.
Frivacy-Friendl) Approach: Ensures privacy-friendde
Managing Heterogeneity: Effectively tackles data and model diversity, supporting non- iid data distributions and various client-model architectures.
Theoretical Interpretation: Provides a comprehensive analysis of global pruning and personalized model aggregation. Discusses convergence theories, communication costs, and the advantages over existing methodologies.
Local Differential-Privacy Algorithm: Introduces LDP-FedP3, a novel local differential privacy algorithm. Outlines privacy guarantees, utility, and communication efficiency.
2

---

Published as a conference paper at ICLR 2024
Algorithm FedP3
1: Input: Client has data X, for i E [n], the number of local updates K, the number of com- munication rounds T, initial model weights W; = [W?, ...... W?' on the server for t
2: Server specifies the server pruning mechanism P., the client pruning mechanism Qi, and the set of layers to train I, C [T.] for each client i E [n}
3: for t 0, I...., T do
4: Server samples a subset of participating clients C. C [n]]
5: Server sends the layer weights W for E Lg to client i C, for training
6: Server sends the pruned weights P. @ W. for I & L; to client E C.
7: for each client E C2 parallel do
8:
Initialize W
1,0 W/ for all E [L.] and W P. o Wt for all I & [4.]
9:
for 1: = 0, 1.... K do
10:
Compute Wi,++1 Low.aoouaaaaee//ocoo 1,k; X,, L, k), where W == {W t,k. W i.k...., W(A
11: end for
Send Uier Wi, the server
end for
14: Server aggregates W = Aggregation(Uhetn Uten w(,k)
15:
end for
Output: WT
2
APPROACH
We focus on the training of neural networks within the FL paradigm. Consider a global model
W = Wwo,W' ....,,',, Won)).
where 0 represents the weights of the input layer, Wout the weights of the final output layer, and L the number of hidden layers. Each W for all l = {0, 1....., L), denotes the model parameters for layer 1. We distribute the complete dataset X across n clients following specific distribution, which can be non-iid. Each client then conducts local training on local data denoted by X;-
Algorithmic overview. In Algorithm [ introduce the details of proposed general frame- work called Federated Personalized and Privacy-friendly network Pruning (FedP3). For every client i e [n]] we assign predefined pruning mechanisms P/ and Qi determined by the client's computa- tional capacity and network bandwidth (see Line 2). Here, P. denotes the maximum capacity of pruned global model W sent client i, signifying server-client global pruning. On the other hand, Qi stands for the local pruning mechanism, enhancing both the speed of local computation and the robustness (allowing more dynamics) of local network training.
In Line 4 we opt for partial client participation by selecting a subset of clients C, from the total pool [m). Unlike the independent subnetwork training approach, Lines 5 employ personalized server-client pruning strategy. This aligns with the concept of collaborative training. Under this approach, we envision each client learning a subset of layers, sticking to smaller neural network architectures of the global model. Due to the efficient and sevaacyfrriedly communication, such method is not only practical but also paves a promising path for future research in FL.-type training and large language models.
The server chooses a layer subset L, for client and dispatches the pruned weights, conditioned by P;, for the remaining layers. Local training spans K steps (Lines 12). detailed Algorithm To uphold a privacy-friendee i are transmitted to the server (Line 12). The server concludes by aggregating the weights received from every client to forge the updated model W4+1, as described in Algorithm an intuitive pipeline in Figure T
framework, only weights Uler., W'' necessary for training of each client
We also provide
Local update. Our proposed framework. FedP3, incorporates dynamic network pruning. In ad- dition to personalized task assignments for each client i, our local update mechanism supports diverse pruning strategies. Although efficient pruning strategies in FL remain active research
3

---

Published as a conference paper at ICLR 2024
(W? WT.Q. W
(WF Q7 Wille (We W. Q. @ W Ln
Server model W
Client model W1,1
Client model Wa,
Client model Wi,
W t
W
W
W
W
W
W
W Lo
W
W2,1
W
W
LT|P1
LT/P:
LT P.
Client model W
Client model W
Client model W
Server model We+1
W
Wous E
Wout
W
W
W
W
W
t,3
W
w/
W
W
W
Aggregation
Figure 1: Pipeline illustration of our proposed framework FedP3.
Algorithm LocalUpdate
1: Input: W.,,, X,, I4; Qi,
2: Generate the step-wise local pruning ratio dı,k conditioned on P: and Q;
3: Local training WIEL,WW, (UIgL, ② P. PROR Wi) using local data X;
4: Output: Wc,++1
area (Horváth et al. 2021: Alam et al. 2022 Liao et al. 2023), we aim to determine if our frame- work can accommodate various strategies and yield significant insights. In this context, we examine different local update rules as described in Algorithm [ We evaluate three distinct strategies: fixed without pruning, uniform pruning, and uniform ordered dropout.
Assuming our current focus is on W t,ke where & L,, after procuring the pruned model conditioned on P3 from the server, we denote the sparse model we obtain by P; @ W Here:
Fixed without pruning implies that we conduct multiple steps of the local update without additional local pruning, resulting in P. @ W W.
Uniform pruming dictates that for every local iteration k, we randomly generate the proba- bility 9. and train the model 9a P. @ Wt W.K-
Uniform ordered dropout is inspired by Horváth et al. 2021 In essence, if P: @ W xd2 (extendable 4D convolutional weights; however, we reference 2D fully con- nected layer weights here), we retain only the subset P. @ W q1,d1. d2 for train- ing purposes. [: [d represents we select the first qi,k x dj elements from the total dj elements.
Regardless of the chosen method, the locally deployed model is given by (UIEL, Wi.e)
(Uigt, Pr PrO P.O...k) as highlighted in Algorithm Line 3
4

---

Published as a conference paper at ICLR 2024
Algorithm 3 Aggregation
Input: Usein] Uter, W
2: Simple Averaging:
3: Wi+1 < Avg (W*,,,) for all nodes with E T4p
4: Weighted Averaging:
5: Construct the aggregation weighting a, for each client
6: WT < Avg (a, x,,,)) for all nodes with I L,
7: Attention Averaging:
8: Construct an attention mapping layer annoted by function h
9: Wi+1 < h (Wi, t,K,s) for all nodes with / E L;
10:
Output: Wi+1
Layer-wise aggregation. Our Algorithm| distinctiiil deviates from existing methods in Line as each client forwards only portion of information to the server, thus prompting investigation into optimal aggregation techniques. Algorithm we evaluate three aggregation methodologies:
Simple averaging computes the mean of all client contributions that include a specific layer
1. This option presented in Line
Weighted averaging adopts a weighting scheme based on the number of layers client i is designated to train. Specifically, the weight for aggregating W W.,,s from client is given by |Tal// Σj=1 TTyl, analogous to importance sampling. This option is presented in Line
Attention-baae based averaging introduces an adaptive mechanism where an attention layer is learned specifically for layer-wise aggregation. This option is presented in Line
3 THEORETICAL ANALYSIS
Our work refines independent subnetwork training (IST) by adding personalization and layer-level sampling, areas yet to be fully explored (see Appendix A.2 for related work). Drawing the sketch- based analysis from Shulgin Richtárik (2023), we aim to thoroughly analyze FedP3, enhancing the sketch-type design concept in both scope and depth.
Consider a global model denoted as w E Rd. In Shulgin Richtárik (2023). a sketch C E Rdxd represents submodel computations by weights permutations. We extend this idea to more general case encompassing both global pruning, denoted as P E Rdxd, and personalized model aggrega- tions, denoted S E dkd Now first present the formal definitions.
Definition 1 (Global Pruning Sketch P). Let a random subset S of [d] a proper sampling such that the probability cj Prob(j S) > 0 for all E [d]. Then the biased diagonal sketch with S is P Diag(p!. p2. P.), where pl. if S otherwise 0.
Unlike Shulgin & Richtárik 2023) we assume client-speeific sampling with potential weight over- lap. For simplicity, we consider all layers pruned from the server to the client, a more challenging case than the partial pruming in FedP3 (Algorithm T]. The convergence analysis of this global prun-
ing sketch is in Appendix C.4
Definition (Personalized Model Aggregation Sketch S). Assume d n, = sn, where an integer. Let = (T1," .T)) be a random permutation of the set [d]. The number of parameters per layer ny, assume s can be divided by ni Then, for all x E IRd and each [n], we define as : Eei 1-=s(i-1)41 Cn, '.y
is
Sketch S is based on the permutation compressor technique from Szlendak et al. (2021) Extending this idea to scenarios where d is not divisible by follows similar approach as outlined in Szlendak ct al. 2021) To facilitate analysis, we apply a uniform parameter count nı across layers, preserving layer heterogeneity. For layers with fewer parameters than d,, zero-padding ensures operational consistency. This uniform distribution assumption maintains our findings' generality and simplifies the discussion. Our method assumes divides d,, streamlining layer selection over individual ele- ments. The variable v denotes the number of layers chosen per client, shaping a more analytically conducive framework for FedP3, detailed in Algorithm 4 in the Appendix.
5

---

Published as a conference paper at ICLR 2024
Theorem (Personalized Model Aggregation). Let Assumption 7 holds. Iterations K, choose step- size y Σ (1/mae, LLasek Denote Do == f(w") -f Then for any K Z 1, the iterates w
of FedP3 in Algorithm 4 satisfy
E [V](w>>])) 2(1 + LL 2
min OSRSK-
(3)
We have achieved total communication cost of O (4/2), marking significant improvement over unpruned methods. This enhancement is particularly crucial in FL for scalable deployments, es- pecially with a large number of clients. Our approach demonstrates a reduction in communication costs by a factor of O (*/e). In the deterministic setting of unpruned methods, we compute the exact gradient, in contrast bounding the gradient as in Lemma [I] Remarkably, by applying the smoothness- based bound condition (Lemma 1 to both FedP3 and the unpruned method, we achieve a communication cost reduction by a factor of ((/)) for free. This indicates that identifying tighter upper gradient bound could potentially lead to even more substantial theoretical improve-
ments communication efficiency. A detailed analysis is available in Appendix C.2
We have
also presented an analysis of the locally inteerential-privaatt variant of FedP3, termed LDP DPFFedP3, in
Theorem
Theorem (LDP-FedP3). Under Assumptions anddd 2
with the use of Algorithm 5 consider the
number of samples per client to be and the number of steps to be K. Let the local sampling probability be q = b/m. For constants d' and e, and for any < c²2 and E (0, 1). LDP-FedP3
achieves (e,6)-LDP with o² eKC² 1/c)
m
Set K = max
min { Cvedd ed (og(1/)) ed log(1/0/ } and 7 =ymmmm
LAo
m 22
(oocloog1101 }
CmeVL }.
have:
20 Lcd log(///(///)) =0
C CTTllog(1/0)
K Σ E
[[)((())
me
mc
Consequently, the total communication cost is:
CLDP-FedP3 = me//LLo m²c²
C-llog(1/0) log(1/8)
We establish the privacy guarantee and communication cost of LDP-FedP3. Our analysis aligns with the communication complexity in Li et al. (002)) while providing more precise convergence bound. Further details and comparisons with existing work are discussed in Appendix C.3 C.3
EXPERIMENTS
4.1 DATASETS AND SPLITTING TECHNIQUES
We utilize benchmark datasets CIFAR 10/100 Krizhevsky et al. (2009). subset of EMNIST la- beled EMNIST-L. Cohen et al. (2017), and FashionMNIST Xiao et al. 7), maintaining standard train/test splits in McMahan et al. (0177) and Li et al. (222b). While CIFAR 100 has 100 labels, the others have 10, with a consistent data split of 70% for training and 30% for testing. Details on
these splits are in Table in the Appendix. For non-iid splits in these datasets, employ class-wise and Dirichlet non-iid strategies, detailed in Appendix B.2
4.2 OPTIMAL LAYER OVERLAPPING AMONG CLIENTS
Datasets and Models Specifications. In this section, our objective to develop a communication- efficient architecture that also preserves accuracy. We conducted extensive experiments recog- nized datasets like CIFAR 10/100 and FashionMNIST. using a neural network with two convolutional layers (denoted Conv) and four fully-con fully-comnected layers (FC). For EMNIST-L, our model includes four FC layers including the output layer. This approach simplifies the identification of optimal layer overlaps among clients. We provide the details of network architectures in Appendix B.3

---

Published as a conference paper at ICLR 2024
95
90
90-
Top-1 Accuracy
80
Top-1 Accuracy
85
FedAvg-S1
X
FedAug-S1
70-
X
FedAvg-S2
FedAvg-52
LowerB-S1
75
OPU3-S1
LowerB-S2
OPU3-52
OPU2-51
70-
X
FedCR-S1
60
OPU2-S2
X
FedCR-S2
0PU3-S1
65
FedCR-OPU3-S1
FedCR OPU3 S2
50
60
CIFARIO
CIBAR100
EMNIST-L
FashionMNIST
CIFAR10
CIRAR100
EMNIST-L
FashionMNIST
Figure 2: Comparative Analysis of Layer Overlap Strategies: The left figure presents a comparative study of different overlapping layer configurations across four major datasets. On the right, we ex- tend this comparison to include the state-of-thee art personalized FL method, FedCR. In this context, S1 refers to a class-wise non-iid distribution, while 52 indicates a Dirichlet non-iid distribution.
Layer Overlapping Analysis. Figure 2 presents a comparison of different layer overlapping strategies. For Optional Pruning Uniformly with selection of layers (OPU2) represents the se- lection of two uniformly chosen layers from the entire network for training, while OPU3 involves such layers. LowerB denotes the scenario where only one layer's parameters are trained per client, serving as a potential lower bound benchmark. All clients participate training the final FC layer (denoted as FFC). "S1" and "S2" signify class-wise and Dirichlet data distributions, respectively. For example, PedAvg-S1 shows the performance of FedAvg under class-wise non-iid setting. Given that few layers are randomly assigned for each client to train, we assess the communica- tion cost on average. In CIFAR10/10 and FashionMNIST training, by design, we obtain a 20% communication reduction for OPU3, 40% for OPU2, and 60% for LowerB. Remarkably, OPU3 shows comparable performance to FedAvg, with only 80% of the parameters communicated. Com- putational results in the Appendix B.3 (Figure () elucidate the outcomes of randomly sampling single layer (LowerB). Particularly in CIFAR10, clients training on FC2+FFC layers face commu- nication costs more than 10,815 times higher than those training Conv1+FFC layers, indicating significant model heterogeneity.
Beyond validating FedAvg, FedCR ZZang et al. (2023) (details in Appendix [.4). as shown on the right of Figure 2 Our method (FedCR-OPU3), despite 20% lower communication costs, achieves promising performance with only a 2.56% drop on S1 and 3.20% drop on S2 across four datasets. Additionally, Figure 2 highlights the performance differences between the two non-iid data distribution strategies, $1 and s2. The average performance gap across LowerB, OPU2, and OPU3 is 3.55%. This minimal reduc- tion performance across datasets underscores the robustness and stability of our FedP3 pruning strategy in diverse data distributions within FL.
compare with the state-of-the-art personalized FL method
Larger Network Verifications. Our assessment extends beyond shallow networks to the more complex ResNet18 model He et al. (016),, tested with CIFARIO and CIFAR100 datasets. Figure 3 iillustrates the ResNet18 8 architecture, composed of four blocks, each containing four layers with skip connections. plus input and output layer, totaling 18 layers. A key focus of our study is to evaluate the efficiency of training this heterogeneous model using only a partial set of its layers.
We performed layer ablations blocks and 3 (B2 and B3), as shown in Figure []] The notation -B2-B3 (full) indicates complete random pruming of B2 or B3, with the remaining structure sent to the server. -B2 (part:) refers to pruning the first or last two layers B2, We default the global pruning ratio from server to client at 0.9, implying that the locally deployed model is approximately 10% smaller than the global model. Results Figure [ demonstrate that dropping random layers from ResNet18 does not significantly impact performance, sometimes even enhancing it. Com- pared with Full, -B2 (part) and -B3 (part) achieved 6.25% reduction in communication costs with only a 1.03% average decrease in performance. Compared to the standard FedAvg without pruning, this is a 16.63% reduction, showcasing the efficiency our FedP3 method. Remarkably, -B3 (part) even surpassed the Ful model in performance. Additionally, -B2-B3 (full)

---

Published as a conference paper at ICLR 2024
Output Layer
Method
CIFAR10
CIFAR100
Block
Full
73.25
63.33
3x3 conv. 128
-B2-B3 (full)
65.68
58,26
Block
3x3 conv, 128
-B2 (part)
72.09
61.11
Block
-B3 (part)
73.47
62.39
3x3 conv, 128
Block
3x3 conv, 128, /2
Performance of ResNet18 under class- wise non-iid conditions. The global pruning ra- tio from server to client is maintained 0.9 for all baseline comparisons by default.
Table 1:
Input Layer
Figure 3: ResNet18 architecture.
90
CIFARLO 51
CIFAR10-52
p=1.0
CIFAR100-S1
80
CIFAIC100-52
Combinee Certereeeeerrtrree Pertomaaii
p=05
Combined Peeformance
peos
70
ENNISTU-S1
TashionMN/ST.S1
FashionMNI5S52
40
1.00
Pruning
Weighting
Figure 4: Comparative Analysis of Server to Client Global Pruning Strategies: The left portion displays Top-1 accuracy across four major datasets and two distinct non-IlD distributions, varying with different global pruning rates. On the right, we quantitatively assess the trade-off between model size and accuracy.
sulted in 12.5% average reduction in communication costs (21.25% less compared to unpruned FedAvg), with just a 6.32% performance drop on CIFAR10 and CIFAR100. These results demon- strate the potential of FedP3 for effective learning in LLMs.
4.3 KEY ABLATION STUDIES
Our framework, detailed in Algorithm critically depends on the choice of pruning strategies. The FedP3 algorithm integrates both server-to-client global pruning and client-specific local pruning. Global pruning aims minimize the size of the model deployed locally, while local pruning focuses on efficient training and enhanced robustness.
4.3.1 EXPLORING SERVER TO CLIENT GLOBAL PRUNING STRATEGIES
We investigate various global pruning ratios and their impacts, as shown in the left part of Figure global pruming rate of 0.9 implies the local model has 10% fewer parameters than the global model. When comparing unpruned (rate 1.0) scenarios, we note an average performance drop of 5.32% when reducing the rate 0.9, 12.86% to 0.7, and a significant 27.76% 0.5 across four major datasets and two data distributions. The performance decline is more pronounced at 0.5 pruning ratio, indicating substantial compromises in performance for halving the model parameters.
In the right part of Figure 4 evaluate the trade-off between model size and accuracy. Assuming total global model parameters as N and accuracy as Acc, the global pruning ratio as r, we weigh the local model parameters against accuracy using a factor represents Acc + a/r. A higher a indicates a focus on reducing parameter numbers for large global models, accepting some performance loss. This becomes increasingly advantageous with higher values, suggesting promising area for future exploration, especially with larger-seale models.
N/Ace >> 0, where the x-axis
4.3.2 EXPLORING CLIENT-WISE WISE LOCAL PRUNING STRATEGIES
Next, interested in exploring the influence of different local pruning strategies. Building upon our initial analysis, we investigate scenarios where our framework permits varying levels of local network pruning ratios. Noteworthy implementations in this domain resemble FjORD (Horváth
8

---

Published as a conference paper at ICLR 2024
Table 2: Comparison of different network local pruning strategies. Global pruning ratio p 0.9.
Strategies
CIFAR10
CIFARI00
EMNIST-L
FashionMNIST
Fixed
67.65 61.17
65.41 57.38
88.75 / 86.33
81.75 84.27
Uniform (p 0.9)
65.51 / 60.10
64.33 / 58.20
85.14 / 84.29
78.81 / 77.24
Ordered Dropout (p 0.9)
61.73 58.82
61.11/53.25
82.54 / 80.18
75.45 73.27
Uniform (p- 0.7)
60.78 / 56.41
60.35 / 54.88
77.39 75.82
72.66 70.37
Ordered Dropout (p 0.7)
58.90 53.38
59.72 50.03
72.19 / 70.30
70.21 / 67.58
al. 2021), FedRolex (Alam al., 2022). and Flado (Liao al. 2023 Given that the only par- tially open-source code available is from FjORD, we employ their layer-wise approach to network sparsity. The subsequent comparisons and their outcomes
presented in Table2 The details of different pruning strategies, including F xed, Uni form and Ordered Dropout are presented in the above Approach section. "Fixed", "Uniform", "Ordered Dropout" represents Fixed without praning, Uniform prwning, and Uniform order dropout in the Approach section, respectively. From the results in Table. [ we can see the difference between Uniform and Ordered Dropout strategies will be smaller with small global pruning ratio from 0.9 to 0.7. Besides, in our experi- ments, Ordered Dropout no better than the simple Uni forf strategy for local pruning.
4.3.3 EXPLORING ADAPTIVE MODEL. AGGREGATION STRATEGIES
In this section, explore range of weighting strategies, including both simple and advanced averaging methods, primarily focusing on the CIFAR datasets. We assign clients with 3 layers (OPU1-2-3) or 2 - 3 layers (OPU2-3) randomly. In Algorithm 3 we implement two aggregation approaches: simple and weighted aggregation.
Let I denote the set of clients involved in training the 1-th layer, where 1 E L. The server's received weights for layer from client i are represented as Wi,,,. The general form model aggregation
is thus defined as:
Top-1 Accuracy
64
5123-S1
X
L' W 2+1 Σ a.W. LKK,
5123-52
62
W123-52
523-51
60
523-52
or initialized as 1/14%, this constitutes si simple mean aver- aging. Considering N, as the total number of layers for client i and as the total number of clients, if a; No/E N,, this method termed weighter averaging. The underlying idea is that clients with more comprehensive network information should have greater weight in parameter contribution. A more flexible ap- proach is attent i on averaging, where is learnable, encom- passing a simple and we giteed averaging as specific cases.
W23-51
58
W23-52
QIAR11
CIFAR10D
Figure Comparison of various model aggregation strategies. 0.9.
Future research may delve into a broader range of aggregation strategies. Our findings, shown in Figure 3 include S123 3-33-3133 S1 for the OPU1 2--1--33 method with simple aggregation in class-wise non- iid distributions, and W23-3 -2 for OPU2-3 with weighted aggregation in Dirichlet non-iid. The data illustrates that we weighted averaging relatively improves over simple le averaging by 1.01 % on CIFAR 10 and 1.05% on CIFAR100. Furthermore, OPU-2-3 consistently surpasses OPÚ1-2-3 by 1.89%, empirically validating our hypotheses.
CONCLUSION
In this paper, we introduce FedP3, a nuanced solution designed to tackle both data and model het- erogeneities while prioritizing privacy. We have precisely defined the concepts of personalization, privacy, and pruning as central to our analysis. The efficacy of each component is rigorously vali- dated through comprehensive proofs and extensive experimental evaluations.

---

Published as a conference paper at ICLR 2024
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Samiul Alam, Luyang Liu, Ming Yan, and Mi Zhang, Fedrolex: Model- boodel-hetrrogeneuus federated learning with rolling sub-model extraction. Advances Neural Information Processing Sys- tems, 2022.
Miguel E Andrés, Nicolás E Bordenabe, Konstantinos Chatzikokolakis, and Catuscia Palamidessi. deeo-ndistinguishability Differential privacy for location-bast systems. In Proceedings of the 2013 ACM SIGSAC conference Computer & communications security, pp- 901-914, 2013.
Yossi Arjevani, Ohad Shamir, and Nathan Srebro. A tight convergence analysis for stochastic gra- dient descent with delayed updates. In Algorithmic Learning Theory, pp. 111-132. PMLR, 2020.
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. 2014 IEEE 55th annual symposium on foundations of computer science, pp. 464-473. IEEE, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 2020.
Konstantinos Chatzikokolakis, Miguel E Andrés, Nicolás Emilio Bordenabe, and Catuscia Palamidessi. Broadening the scope of differential privacy using metrics. In Privacy Enhanc- ing Technologies: 13th International Symposium, PETS 2013, Bloomington, IN, USA, July 10-12, 2013. Proceedings 13, pp. 82-102. Springer, 2013.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0231 2022.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. 2017 international joint conference on neural networks (ICCN), pp.
2921-2926. IEEE, 2017.
Leonardo Cunha, Gauthier Gidel, Fabian Pedregosa, Damien Scieur, and Courtney Paquette. Only tails matter: Average-case universality and robustness the convex regime. In International Conference on Machine Learning, pp. 4474 4491. PMLR, 2022.
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. In International Conference on Learning Represen- tations. 2021.
Jiahao Ding, Guannan Liang, Jinbo Bi, and Miao Pan. Differentially private and communication efficient collaborative learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7219-7227, 2021.
Chen Dun, Cameron Wolfe, Christopher M Jermaine, and Anastasios Kyrillidis. Resist: Layer- wise decomposition of resnets for distributed training. In Uncertainty in Artificial Intelligence, pp. 310-620. PMLR, 2022.
Chen Dun, Mirian Hipolito, Chris Jermaine, Dimitrios Dimitriadis, and Anastasios Kyrillidis. Effi cient and weight federated learning via asynchronous distributed dropout. In International Conference on Artificial Intelligence and Statistics, pp. 6630-6660. PMLR, 2023.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp. 265-284, Springer, 2006.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):211-407 2014.
10

---

Published as a conference paper at ICLR 2024
Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: optimal rates in linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 439-449, 2020.
Dashan Gao, Xin Yao, and Qiang Yang. A survey heterogeneous federated learning. arXiv preprint arXiv:2210.04500. 2022a.
Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu. Fedde: Feder- ated learning with non-iid data via local drift decoupling and correction. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10112-10121, June 2022b.
Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochas- tic programming. SIAM Journal on Optimization, 23(4):2341 (0l:)2341-22200 1-268, 2013.
Baptiste Goujaud, Damien Scieur, Aymeric Dieuleveut, Adrien B Taylor, and Fabian Pedregosa. Super-acccleration with cyclical step-sizes. International Conference on Artificial Intelligence and Statistics, pp. 3028-30200 30-60 PMLR, 2022.
Chaoyang He, Erum Mushtaq, Jie Ding, and Salman Avestimehr. Fednas: Federated deep learning via neural architecture search. 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Samuel Horváth, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 34: 14:12876-12889, 2021.
Hong Huang, Lan Zhang, Chaoyue Sun, Ruogu Fang, Xiaoyong Yuan, and Dapeng Wu. Fedtiny: Pruned federated learning towards specialized tiny models. arXiv preprint arXiv:2212.01977, 2022.
Yangsibo Huang, Yushan Su, Sachin Ravi, Zhao Song, Sanjeev Arora, and Kai Li. Privacy- preserving learning via deep net pruning. arXiv preprint arXiv:2003.01876 2020.
Roger Iyengar, Joseph P Near, Dawn Song, Om Thakkar, Abhradeep Thakurta, and Lun Wang. To- wards practical differentially private convex optimization. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 299-316. IEEE, 2019.
Martin Jaggi, Virginia Smith, Martin Takáe, Jonathan Terhorst, Sanjay Krishman, Thomas Hofmann, and Michael Jordan. Communication--ffiiee distributed dual coordinate ascent. Advances in neural information processing systems, 27, 2014.
Chu Jiang, Burak Kantarci, Sema Oktug, and Tolga Soyata. Federated learning in smart city sensing: Challenges and opportunities. Sensors, 20(21):6230, 2020.
Yuang Jiang, Shigiang Wang, Victor Valls, Bong Jun Ko, Wei-Han Lee, Kin K Leung, and Leandros Tassiulas. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems, 2022a.
Yuang Jiang, Shiqiang Wang, Victor Valls, Bong Jun Ko, Wei-Han Lee, Kin Leung, and Leandros Tassiulas. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems, 2022b.
Peter Kairouz, Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 14(1-2):1-210, 2021.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference machine learning, pp. 5132-5143. PMLR, 2020.

---

Published as a conference paper at ICLR 2024
Ahmed Khaled and Peter Richtárik. Better theory for sgd in the nonconvex world. arXiv preprint arXiv:2002.03329, 2020.
Jakub Konečný, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint urXiv:1610.02555 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint DrXiv:1910.03355 2019.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10713-10722, 2021.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 3703),50-60, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and sys- tems, 2:429 450, 2020b.
Thize Li and Jian Li. Simple and optimal stochastic gradient methods for nonsmooth nonconvex optimization. The Journal of Machine Learning Research, 23(1):1081110995) 2022.
Zhize Li, Haoyu Zhao, Boyue Li, and Yuejie Chi. Soteriaft: A unified framework for private fed- erated learning with communication compression. Advances in Neural Information Processing Systems, 35:4285-4300 2022.
Dongping Liao, Xitong Gao, Yiren Zhao, and Cheng-Zhong Xu. Adaptive channel sparsity for federated learning under system heterogeneity. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 20432-20441 2023.
Fangshuo Liao and Anastasios Kyrillidis. On the convergence of shallow neural network training with randomly masked neurons. Transactions on Machine Learning Research, 2022.
Tao Lin, Linging Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning, Advances in Neural Information Processing Systems. 33:2351-2388 2020.
Andrew Lowy, Ali Ghafelebashi, and Meisam Razaviyayn. Private non-convex federated learning without a trusted server. In International Conference on Artificial Intelligence and Statistics, pp- 5749-5786. PMLR, 2023.
Chenxin Ma, Virginia Smith, Martin Jaggi, Michael Jordan, Peter Richtárik, and Martin Takác. Adding averaging in distributed primal-dual optimization. In International Conference Machine Learning, pp. 1973-1982, PMLR, 2015.
Grigory Malinovsky, Kai Yi, and Peter Richtárik. Variance reduced Proxskip: Algorithm, theory and application federated learning. NeurlPS, 2022.
Brendan McMahan, Eider Moore, Daniel Ramage. Seth Hampson, and Blaise Aguera y Arcas. Communication-efficti learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Matias Mendieta, Taojiannan Yang. Pu Wang, Minwoo Lee, Zhengming Ding, and Chen Chen. Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEF/CV Conference on Computer Vision and Patlern Recognition (CVPR), pp. 8397-8406. June 2022.
Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, and Peter Richtárik. ProxSkip: Yes! Local gradient steps provably lead to communication acceleration! Finally! In 39th International Conference Machine Learning (ICMI. 2022), 2022.
12

---

Published as a conference paper at ICLR 2024
Yurii Nesterov. Introductory lectures convex optimization: A basic course, volume 87, Springer Science & Business Media, 2003.
Mher Safaryan, Filip Hanzely, and Peter Richtárik. Smoothness matrices beat smoothness constants:
Better communication compression techniques for distributed optimization. Advances in Neural Information Processing Systems, 34:25688-25702 2021.
Egor Shulgin and Peter Richtárik. Towards a better theoretical understanding of independent sub- network training. arXiv preprint arXiv:2306. 16484, 2023.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. Advances in neural information processing systems, 30, 2017.
Rafal Szlendak, Alexander Tyurin, and Peter Richtárik. Permutation compressors for provably faster distributed nonconvex optimization. arXiv preprint arXiv:21 2021.
Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fed- proto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8432-8440, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini 139,111,,2.. 2023.
Bokun Wang, Mher Safaryan, and Peter Richtárik. Theoretically better and numerically faster dis- tributed optimization with smoothness-awasn aware quantization techniques. Advances in Neural Infor- mation Processing Systems, 35:9841-998 -9852, 2022.
Di Wang, Minwei Ye, and Jinhui Xu. Differentially private empirical risk minimization revisited: Faster and more general. Advances in Neural Information Processing Systems, 30, 2017.
Cameron R Wolfe. Jingkang Yang, Fangshuo Liao, Arindam Chowdhury, Chen Dun, Artun Bayer, Santiago Segarra, and Anastasios Kyrillidis. Gist: Distributed training for large-scale graph con- volutional networks. Journal of Applied and Computational Topology, pp. 1-53, 2023.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 100.::7708.77777 2017.
Mang Ye, Xiuwen Fang, Bo Du, Pong C Yuen, and Dacheng Tao. Heterogeneous federated learning: State-of-the-art and research challenges. ACM Computing Surveys, 56(3): 2023.
Kai Yi, Laurent Condat, and Peter Richtárik. Explicit personalization and local training: Double communication acceleration in federated learning. arXiv preprint arXiv:2305. 2023.
Binhang Yuan, Cameron R Wolfe, Chen Dun, Yuxin Tang, Anastasios Kyrillidis, and Chris Jer- maine. Distributed learning of fully connected neural networks using independent subnet training. Proceedings of the VLDB Endowment, 1588):158--1590, 2022.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part / 13, pp. 818-833. Springer, 2014.
Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger B Grasse. Which algorithmic choices matter at which batch sizes? insights from noisy quadratic model. Advances in neural information processing systems, 32, 2019.
Hao Zhang, Chenglin Li, Wenrui Dai, Junni Zou, and Hongkai Xiong. Feder: Personalized feder- ated learning based on across-client common representation with conditional mutual information regularization. 2023.
Xin Zhang, Minghong Fang, Jia Liu, and Zhengyuan Zhu. Private and commumication-efiicii edge learning: a sparse differential gaussian-masking distributed sgd approach. In Proceedings of the Twenty-Fire First International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing, pp. 261-270 2020.
13

---

Published as a conference paper at ICLR 2024
Yang Zhao, Jun Zhao, Mengmeng Yang, Teng Wang, Ning Wang, Lingjuan Lyu, Dusit Niyato, and Kwok- Yan Lam. Local differential privacy-based federated learning for internet of things. IEEE Internet of Things Journal, 8(1 8(11):8836- 8853, 2020.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:/80..0000 2018.
Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan, and Mikhail Belkin. Quadratic models for understanding neural network dynamics. arXiv preprint arXiv:2205. 2022.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances neural infor-
mation processing systems, 32, 2019.
14

---

Published as a conference paper at ICLR 2024
CONTENTS
Introduction
Approach
3
Theoretical Analysis
Experiments
Datasets and Splitting Techniques
4.2
Optimal Layer Overlapping Among Clients
4.3
Key Ablation Studies
4.3.1
Exploring Server to Client Global Pruning Strategies
4.3.2
Exploring Client-Wise Local Pruning Strategies
4.3.3
Exploring Adaptive Model Aggregation Strategies
Conclusion
Extended Related Work
16
Federated Network Pruning
16
A.2 Subnetwork Training
16
Model Heterogeneity
17
Experimental Details
17
I Statistics of Datasets
17
B,2 Data Distributions.
17
Network Architectures
18
B.4 Training Details
18
Quantitative Analysis Reduced Parameters
18
Extended Theoretical Analysis
20
Analysis of the General FedP3 Theoretical Framework
20
C.2 Model Aggregation Analysis
20
Differential Privent FedP3 Analysis
22
C.4 Global Pruning Analysis
24
D Missing Proofs
25
D. I Proof of Theorem
25
D.2 Proof of Theorem
27
D.3 Proof of Theorem
30
15

---

Published as a conference paper at ICLR 2024
A EXTENDED RELATED WORK
A.1 FEDERATED NETWORK PRUNING
We introduce two distinct types of network pruning within our study: 1) global pruning, which extends from server to client, and 2) local pruning, where each client's network is pruned based on its own specific data. In our setting, we assume federated pruning is the scenario with both possible global and local pruning. Federated network pruning, a closely related field, pursues the objective of identifying the optimal or near-optimal pruned neural network at each communication from the server to the clients, as documented in works of Jiang et al. 2022a) and Huang et al. (2022). for example.
During the initial phase of global pruning, (Jiang et al. 2022a) isolates a single potent and reliable client to initiate model pruning- The subsequent stage of local pruning incorporates all clients, advancing the adaptive pruning process. This process involves not only parameter removal but also the reintroduction of parameters, complemented by the standard FedAvg (McMahan et al.) 2017). However, the need for substantial local memory to record the updated relevance measures of parameters in the full-scale model poses a challenge. As a solution to this problem. Huang et al. (2022) proposes an adaptive batch normalization and progressive pruning modules that utilize sparse local computation. Yet, these methods overlook explicit considerations for constraints related to client-side computational resources and communication bandwidth.
Our primary attention gravitates towards designing distinct local pruning methods, such as (Horváth) et al. (022). (Alam et al. 2022), and (Liao et al. (202)). Instead of learning the optimal or subop- timal pruned local network, each client attempts to identify the optimal adaptive sparsity method. The work Horvááh al. 2021 has been groundbreaking, as they introduced Ordered Dropout to navigate this issue, achieving commendable results. It's noteworthy that our overarching framework is compatible with these methods, facilitating straightforward integration of diverse local pruning methods. There are other noticeable methods, such as (Diao et al., [2021), which focuses on re- ducing the size of each layer in neural networks. In contrast, our approach contemplates a more comprehensive layer-wise selection and emphasizes neuron-oriented sparsity.
As of our current knowledge, existing literature directly aligns with our approach, despite its practicality and generality. Even the standard literature regarding federated network pruning appears be rather constrained.
A.2 SUBNETWORK TRAINING
Our research aligns with the rising interest in Independent Subnetwork Training (IST), a technique that partitions neural network into smaller components. Each component is trained in distributed parallel manner, and the results are subsequently aggregated to update the weights of the entire model. The decoupling in IST enables each subnetwork to operate autonomously, using fewer pa- rameters than the complete model. This not only diminishes the computational cost on individual nodes but also expedites synchronization.
This approach was introduced by Yuan et al. (2022) for networks with fully connected layers and was later extended to ResNets Dunn et al. (2022) and Graph architectures Wolfe et al. (2023). Empirical evaluations have consistently posited IST as an attractive strategy, proficiently melding data and model parallelism to train expansive models even with restricted computational resources.
Further theoretical insights into IST for overparameterized single hidden layer neural networks with ReLU activations were presented by Liao & Kyrillidis (2022). Concurrently, (Shulgin & Richtárik 2023 revisited IST, exploring it through the lens of sketch-type compression.
While acknowledging the adaptation IST to FL using asynchronous distributed dropout tech- niques Dun et al. (2023), clients should not relay the entirety their subnetworks to the central server-both both to curb exces- sive networking costs and to safeguard privacy. Moreover, model envisions each client akin to an assembly line component: each specializes in a fraction of the complete neural network, guided by its intrinsic resources and computational prowess.
approach diverges significantly from prior works. We advocate that
16

---

Published as a conference paper at ICLR 2024
compared our study with pivotal existing research, focusing on federated network pruning and subnetwork training. Responding to reviewer feedback, we have broadened the scope of our related work section to include more extensive comparison with other significant studies.
In Section A. and A.2
A.3 MODEL HETEROGENEITY
Model heterogeneity denotes the variation in local models trained across diverse clients, as high- lighted in previous research (Kairouz et al. 2021 Ye et al. 2023). ). A seminal work by |mmth et al.) (2177) extended the well-known COCOA method (Jaggi et al. 2014 Ma et al. 2015). incorporat- ing system heterogeneity by randomly selecting the number of local iterations or mini-batch sizes. However, this approach did not account for variations in client-specific model architectures or sizes. Knowledge distillation has emerged as prominent strategy for addressing model heterogeneity in Federated Learning (FL). Wang (2019) demonstrated training local models with distinct archi- tectures through knowledge distillation, but their method assumes access to large public dataset for each client, a premise not typically found current FL scenarios. Additionally, their approach, which shares model outputs, contrasts with method of sharing pruned local models. Building this concept. Lin et al. (2020) proposed local parameter fusion based on model prototypes, fusing outputs of clients with similar architectures and employing ensemble training on additional unla- beled datasets. Tan et al. (2022) introduced an approach where clients transmit the mean values of embedding vectors for specific classes, enabling the server to aggregate and redistribute global pro- totypes to minimize the local-global prototype distance. He et al. (2021) developed FedNAS, where clients collaboratively train global model by searching for optimal architectures, but this requires transmitting both full network weights and additional architecture parameters. Our method diverges from these approaches by transmitting only weights from a subset of neural network layers from client to server.
B EXPERIMENTAL DETAILS
B. STATISTICS OF DATASETS
We provide the statistics of our adopted datasets in Table. 3
Dataset
# data
# train per client
# test per client
EMNIST-L (Cohen et al. 2017)
48K+8K
392
168
FashionMNIST (Xiao et al. 2017
60K+10K
490
210
CIFARIO (Krizhevsky et al. 2009)
50K+10K
420
180
CIFARIO0 100 (Krizhevsky et al. 2009)
50K+10K
420
180
Table 3: Dataset statistics, with data uniformly divided among 100 clients by default.
B.2 DATA DISTRIBUTIONS
We emulated non-iid data distribution among clients using both class-wise and Dirichlet non-iid scenarios.
Class-wise: we designate fixed classes directly to every client, ensuring uniform data vol- ume per class. As specifics, EMNIST-L, FashionMNiST, and CIFARIO assign classes per client, while CIFAR100 allocates 15 classes for each client.
Dirichlet: following an approach similar to FedCR (Zhang et al. 2023) ) we use Dirichlet distribution over dataset labels to create a heterogeneous dataset. Each client is assigned vector (based on the Dirichlet distribution) that corresponds to class preferences, dictating how labels-and consequently images-are selected without repetition. This method contin- ues until every data point is allocated to client. The Dirichlet factor indicates the level of data non-iidness. With Dirichlet parameter of 0.5, about 80% of the samples for each client on EMNIST-L. FashionMNIST, and CIFAR10 are concentrated four classes. For CIFAR 100, the parameter is set to 0.3.
17

---

Published as a conference paper at ICLR 2024
B.3 NETWORK ARCHITECTURES
Our primary experiments utilize four widely recognized datasets, with detailed descriptions provided in the Experiments section. For the CIFAR10/100 and FashionMNIST experiments, opt for CNNs comprising two convolutional layers and four fully-connected layers as our standard network architecture. In contrast, for the EMNIST-L experiments, we employ a four-layer MLP architecture. The specifics of these architectures are outlined in Table 4 Additionally, the default ResNet18 network architecture is selected for our layer-overlapping experiments.
Layer Type
Size
# of Params.
Cony + ReLu
x 5 64
4,864/ / 1,664
Layer Type
Size
# of Params.
Max Pool
2 x
FC + ReLu
784 x 1024
802,816
Cony ReLu
x 5 64 2 x 2
102, 464
FC + ReLu
1024 x 1024
1,048,576
Max Pool
0
FC + ReLu
1024 x 1024
1,048,576
FC ReLu
1600 X 1024
1,638,400
FC
1024 x 10
10,240
FC ReLu
1024 X 1024
1,048,576
FC ReLu
1024 10/ 100
10,240 / 102.400
Table 4: The left figure depicts the neural network architecture employed for the CIFAR10/100 and FashionMNIST experiments. Conversely, the right figure illustrates the default ML.P (Multi-l.ayer Perceptron) architecture used specifically for the EMNIST-L experiments.
B.4 TRAINING DETAILS
Our experiments were conducted NVIDIA A100 V100 GPUs, depending on their availability in our cluster. The framework was implemented in PyTorch 1.4.0 and torchvision 0.5.0 within Python 3.8 environment. Our initial code, based FedCR Zhang et al. (202)), include hyper-parameter fine-tuning. significant modification was the use of an MLP network with four FC layers for EMNIST-L performance evaluation. We standardized the experiments to 500 epochs with a local training batch size of 48. The number of local updates was set at 10 to assess final performance. For the learning rate, we conducted grid search, exploring range from 10-5 to 0.1, with a fivefold increase at each step. In adapting FedCR, we used their default settings and fine-tuned the parameter across values 0.0001, 0.0005, 0.001, 0.005, 0.01 for all datasets.
refined to
B.5 QUANTITATIVE ANALYSIS OF REDUCED PARAMETERS
We provide quantitative analysis of parameter reduction across four datasets, shown in Fig- ure 6) The x-axis represents different global pruning ratios, and the y-axis indicates the number of parameters. For simplicity, layer, each client trains only one additional layer, akin to the LowerB method used our earlier experiments. For instance, the label FC refers condition where only FC2 and the final layer fully trained, with other layers being pruned during server-to-client transfer and dropped in server communication.
consider scenario where, aside from the final fully-connected
With a constant global pruning ratio, the left part of the figure shows the total number of parame- ters in the locally deployed model post server-to-client pruning, while the right part illustrates the communication cost for each scenario. The numbers atop each bar indicate the relative differences between the largest and smallest elements under various conditions. Across all datasets, we note that higher global pruning ratios result in progressively smaller deployed models. For example, at 0.5 global pruning ratio, the model size for clients training the Convl layer is 57.93% smaller than those training FC2. Moreover, there is a significant disparity in communication costs among clients. The ratios of communication costs are 10815 for CIFARIO, 1522.91 for CIFAR 100, 13749.46 for NNShionMNIST, and 30.23 for EMNIST-L.
18

---

Published as a conference paper at ICLR 2024
FC2
FC2
FC1
6.24
FC1
Conv2
Conv2
2.5
57.93
36
2.0
of parameters
parameters
10815:25
10815
10815
10815:25
1622.91
122..91
622.91
622.91
1.5
0.5
0.5
0.0
0.0
1.0
0.9
0.7
0.5
1.0
0.9
0.7
0.5
global pruning rati
global pruning ratio
(a) CIFARIO
(b) CIFARI00
1c6
1c6
FC2
0.00
FC2
6.49
2.5
24.99
Convl
of parameters
of parameters
1874946
740
13749
1.5
1.5
30.23
50.23
30.23
80.23
1.0
1.0
0.5
0.9 global pruning ratio
0.5
global pruning ratio
(c) FashionMNIST
(d) EMNIST-L
The number of parameters across multiple layers, varying according to different global pruning ratios, spans across four distinct datasets. For each global pruning ratio, the left side of the bar graph shows the total number of parameters in the model after server-to-client pruning when deployed locally. Conversely, the right side details the communication cost associated with each scenario. Atop each bar, we indicate the relative ratio between the layers with the largest and smallest number of parameters, i.e., value (largest smmallsst)) smmallest. / For (d), since the size of parameters of FC2 and FC3 are the same, we omit plotting FC3 to avoid overlapping.
Figure 6:
19

---

Published as a conference paper at ICLR 2024
Algorithm 4 FedP3 theoretical framework
1: Parameters: learning rate > O, number of iterations K, sequence of global pruning sketches (P....., P n kg aggregation sketches (S!..... S kgk initial model w R
2: for k ===00 0. .....................
do 3: Conduct global pruning W/ for E [n] and broadcast all computing nodes
4: for = ...... n in parallel do
5:
Compute local (stochastic) gradient w.r.t. personalized model: P (PP
6: Take (maybe multiple) gradient descent step u w (P
7: Send of = S u to the server
8: end for
9: Aggregate received subset of layers: x+1 Ei=1 o
10: end for
C EXTENDED THEORETICAL ANALYSIS
C.I ANALYSIS OF THE GENERAL FEDP3 THEORETICAL FRAMEWORK
Line demonstrates the global pruning process, employing biased sketch over randomized sketches P; for each client i [n). as defined in Definition I The procedure from Lines 4 8 details the local training methods, though we exclude further local pruning for brevity. Notably, our framework could potentially integrate various local pruning techniques, an aspect that merits future exploration.
We introduce the theoretical foundation of FedP3, detailed in Algorithm 4
Our approach uniquely compresses both the weights w* and their gradients fP( For sake of clarity, we assume Line that each client calculates the pruned full gradient PPV concept that could be expanded to encompass stochastic gradient computations.
In alignment with Line 6, our subsequent theoretical analysis presumes that each client performs single-step gradient descent. This assumption stems from observations that local steps have not demonstrated theoretical efficiency gains in heterogencous environments until very recent studies, such as Mishchenko et al. (2022) and its extensions like Malinovsky et al. (0022;; Yi et al. (2023). which required extra control variables not always viable in settings with limited resources.
Diverging from the method in Shulgin & Richtárik (202)). model involves explicitly sending selected subset of layers of from each client to the server. The aggregation of these layer subsets meticulously described in Line 9.
Our expanded theoretical analysis is structured as follows: Section C.2 focuses on analyzing the convergence rate of our innovative model aggregation method. In Section CK FedP3, a novel diffcrential-privaln variant of FedP3, and discuss its communication complexity in local differential privacy setting. Section C.4 then delves into the analysis of global pruning, as detailed in Algorithm
we introduce LDP-
C.2 MODEL AGGREGATION ANALYSIS
In this section, our objective is to examine the potential advantages of model aggregation and present the convergence analysis of our proposed FedP3. Our subsequent analysis adheres to the standard nonconvex optimization framework, with the goal of identifying an e-stationary point where:
[f(w)) Se
(4)
Here, [[|] represents the expectation over the inherent randomness in E Rd, Moving forward, our analysis will focus primarily on the convergence rate of our innovative model aggregation strategy. To begin, we establish the smoothness assumption for each local client's 's model.
Assumption (Smoothness). There exists some L; 2 0, such that for all i E [n), the function si is L.-smooth, i.e. [[V]((>) [((ii))l T ||x -vll
Vx, E Rd.
20

---

Published as a conference paper at ICLR 2024
This smoothness assumption is very standard for the convergence analysis (Nesterov 2003 Ghadimi & Lan 2013 Mishchenko et al. 2022; Malinovsky et al., 2022 Li & Li 2022] Yi et al. 2023). 3). The
smoothness of function f is L 2 L;, denote Lmax maXçen
We demonstrate the convergence of our proposed FedP3, with a detailed proof presented in Sec- tion D.I Here, restate Theorem [ elarity:
Theorem (Personalized Model Aggregation). Let Assumption TaaioomThoolo Tholds. Iterations K, choose step- size 7 S {1/mmass V/Vi.lmmax Denote Do f(w²) pint. Then for any K 2 1, the iterates we of FedP3 in Algorithm 4 satisfy
2(1 + LLmmx )))
min
E
Ao.
(3)
Next, we interpret the results. Utilizing the inequality 1 + S exp(w) and assuming ? s
we derive the following:
V VLLmark
(1 + L S exp( 2 S exp(1) 3.
Incorporating this into the equation from Theorem ! we ascertain:
6
min
E
S >
Au-
OSkSK-
To ensure the right-ham side of the above equation is less than e, the condition becomes:
6
6
S
c
K
K
7E
Given 7 S V.J.maa K"
it follows that K
36(An) 12
O (+)
LLmaae
Considering the communication cost per iteration is n x v cost is:
d d, the total communication
CreaP3 = O
We compare this performance with an algorithm lacking our specific model aggregation design, namely Distributed Gradient Descent (DGD). When DGD satisfies Assumption 4 with A C 0, B = as per Theorem the total iteration complexity to achieve an e-stationary point O (t) Given that the communication cost per iteration nd, the total communication cost for DGD is:
nd
CDGD = 0
We observe that the communication cost of FedP3 more efficient than DGD by factor (n/e)). This is particularly advantageous in practical Federated Learning (FL) scenarios, where large number clients are distributed, highlighting the suitability of our method for such environments. This efficiency also opens avenues for further exploration in large language models.
Although we have demonstrated provable advantages in communication costs for large client num- bers, anticipate that method's 's performance exceeds current theoretical predictions. This expectation is based on the comparison of FedP3 and DGD under Lemma For DGD, with param- eters A L,, B C = 0. the iteration complexity aligns with O(-)), leading to communication cost of:
COGD = O nid
This indicates a significant reduction in communication costs by a factor of n without additional requirements. It implies that if we could establish a tighter bound on [[)]( beyond the scope of Lemma our theoretical results could be further enhanced.
21

---

Published as a conference paper at ICLR 2024
Algorithm 5 Differential-Prive FedP3 (LDP-FedP3)
1: Parameters: learning rate > 0, number of iterations K, sequence of aggregation sketches
........ S Sk) kg perturbation variance o², minibatch size
2: for k == 0. 1, do
3: Server broadcasts w to all clients
4: for each client = l..... in parallel do
5: Sample random minibatch I& with size from Icoal dataset Dj
6:
Compute local stochastic gradient o = Σjez, (i.(("))
7:
Take (maybe multiple) gradient descent step u w got
8: Gaussian perturbation achieve LDP: प. u. Ct where S ~ N/(00
9: Send of = S ut to the server
10: end for
11: Server aggregates received subset of layers: w+1
n E "
12: end for
C.3 DIFFERENTIAL-RRVVTTE FEDP3 ANALYSIS
The integration of gradient pruning as a privacy preservation method was first brought to prominence by Zhu al. 2019). Further studies, such HHang al. (2020), have delved into the effectiveness of DNN pruning in protecting privacy.
In our setting, we ensure that our training process focuses on extracting partial features without rely- ing on all layers to memorize local training data. This is achieved by transmitting only a select subset of layers from the client the server in each iteration. By transmitting fewer layers layers-effectively implementing greater pruning from clients to the server-we enhance the framework.
of our
This section aims to provide a theoretical exploration of the "privacy-friendyy aspect of our work. Specifically, we introduce differential privatte version of our method, LDP-FedP3, and discuss its privacy guarantees, utility, and communication cost, supported by substantial evidence and rigorous proof.
Local differential privacy is crucial in our context. We aim not only to train machine learning models with reduced communication bits but also to preserve each client's local privacy, an essential element in FL applications. Following the principles of local differential privacy (LDP) as outlined in works like Andrés et al. (2013); Chatzikokolakis define two datasets D and D' as neighbors if they differ by just entry. We provide the following definition for LDP:
al. 2013); Zhao et al. (2020); Li et al. (2022), we
Definition 3. A randomized algorithm A D F, where D is the dataset domain and F the domain of possible outcomes, is (e, -lloocay differentially private for client if, for all neighboring datasets D,, D2 DD client and for all events S within the range of A, it holds that:
PrA(D,) E ES S PrA(()) S
This LDP definition (Definition 3) closely resembles the original concept of (c, 5)-DP (Dwork et al., 2014 2006), but the FL context, emphasizes each elient's responsibility safeguard its privacy. This is done by locally encoding and processing sensitive data, followed by transmitting the encoded information to the server, without any coordination or information sharing among clients.
Similar to our previous analysis of FedP3, we base our discussion here the smoothness assump- tion outlined in Assumption T For simplicity, and because our primary focus in this section is on privacy concerns, assume uniform smoothness across all clients, i.c., L, L.
Our analysis also relies on the bounded gradient assumption, which is a common consideration in differential privacy analyses:
Assumption 2 (Bounded gradient). There exists some constant C > 0, such that for all clients i [n) and for any Rdd the gradient norm satisfies [[V[(()]] < C.
22

---

Published as a conference paper at ICLR 2024
Table 5: Comparison of communication complexity in LDP Algorithms for nonconvex problems across distributed settings with n nodes.
Algorithm
Privacy
Communication Complexity
(1+n/ (m+m/(m==))m²> dlog(1/6)
Q-DPSGD-1 DDing et al.
2021
(c,8)-LDP
LDP SVRG/SPIDER (Lwwy et al.
2023)
(c,8)-LDP
16 3/2 meva
v /lo(1188
SDM-DSGD Zhang et al.
2020
(c,8)-LDP
7/2
(1+w))/2 log(116)
(1+w) log(1/8)
CDP-SGD et 2022
(e,8)-LDP
3/2 mevd
2
(1+w)33/2
/lo((1/6) (1+w) (1g()
LDP-FedP3 (Ours)
(c,
meva m
Alg(115)
log(1/8)
This bounded gradient assumption aligns with standard practices in differential privacy analysis, as evidenced in works such as (Bassily et al.) 2014 Wang et al. 2017 yynggrr et al., 2019 Feldman
ct al. 2020 Li et al. 2022).
We introduce a locally differentially private version of FedP3, termed LDP-FedP3, with detailed al- gorithmic steps provided Algorithm [5] This variant differs from FedP3 Algorithm 4 primarily by incorporating the Gaussian mechanism, as Abali et al. (2016), to ensure local differential pri- vacy (as implemented
Line 8 of Algorithm (5). Another distinction the allowance for minibatch sampling per client in LDP-FedP3. Given that primary focus in this section is privacy, set aside the global pruning aspect for now, considering it orthogonal to our current analysis and not central our privacy considerations. In Theorem 2 encapsulate the following theorem:
with the use of Algorithm 5 consider the number of samples per client be m and the number of steps be K. Let the Tocal sampling probability be q = b/m. For constants e' and c, and for any < dq K and d 1). LDP-FedP3 achieves (c, 5)D with o² eKC 1oc(1/c)
Theorem 2 (LDP-FedP3). Under Assumptions 7 and 2
m²c
LAo [od(1/1) ad log(1/0) and 7 -mmninn {t.
m 22
Doocd log(1/)
max
we have:
OmevL
2C Led
(VLolog(1//)
Σ E
[[VS(w))))
me
mc
Consequently, the total communication cost is:
mE dLA. (oolo((1/)) log(1/8)
m²
CIDPP-RodP3 =
In Section D.2 we provide the proof for our analysis. This section primarily focuses analyzing and comparing our results with existing literature. Our proof pertains to local differentially--rivace Stochastic Gradient Descent (SGD). We note that et al. (2022) offered a proof for CDP-SGD using a specific set of compressors. However, our chosen compressor does not fall into that category, as discussed more comprehensively Szlendak al. (002)) Considering the Rand-t compressor with t = d/ //n, it's established that:
[[R w||?] s w w where
d
t
Setting the same K and and applying Theorem from Li et al. (2022), obtain:
k
5C BCVLouk Lod = / (VVLooog(1/)) (
E
K
me
23

---

Published as a conference paper at ICLR 2024
which aligns with our theoretical analysis. Interestingly, observe that our bound is tighter by factor of 2/5, indicating a more efficient performance in our approach.
We also compare our proposed LDP-FedP3 with other existing algorithms in Algorithm | An intriguing finding is that our method's efficiency does not linearly increase with higher number of clients, denoted
Notably, communication complexity remains independent of n. This implies that in practical scenarios with a large n, our communication costs will not escalate. We then focus on methods with a similar structure, namely, SDM-DSGD and CDP-SGD. For these, the communication cost comprises two components. Considering a specific case, Rand-t, where / is deliberately set to d/n, we derive w 4/t-1 = n 1. This results in a communication complexity on par with CDP-SGD, but significantly more efficient than SDM-DSGD, Moreover, it's important to note that the compressor in LDP-FedP3 differs from that in CDP-SGD. Our analysis introduces new perspectives and achieves comparable communication complexity to other well-established results.
C.4 GLOOBA PRUNING ANALYSIS
Our methodology relates to independent subnetwork training (IST) but introduces distinctive fea- tures such as personalization and explicit layer-level sampling for aggregation. IST, although con- ceptually simple, remains underexplored with only limited studies like Liao & Kyrillidis) (2022), which provides theoretical insights for overparameterized single hidden layer neural networks with ReLU activations, and (hulliin & Richtárik (2023), which revisits IST from the perspective of sketch-type compression. In this section, we delve into the nuances of global pruning as applied in Algorithm 4
For our analysis here, centered on global pruning, we simplify by assuming that all personalized model aggregation sketches S, are identical matrices, that is, S, I. This simplification, how- ever, does not trivialize the analysis as the pruning of both gradients and weights complicates the convergence analysis. Additionally, we adhere to the design of the global pruning sketch as per Definition 1 which results in a biased estimation, i.e., E[P,w] f w. Unbiased estimators, such as Rand-t that operates over coordinates, are more commonly studied and offer several advantages in theoretical analysis.
For Rand-t, consider a random subset S of [d] representing a proper sampling with probability 9 = Prob(j S) > for every [d]. R2 == Diag(r!,
rr)). where r 1/e, if j E S and () otherwise. In contrast to our case, the value on each selected coordinate in Rand-t. is scaled by the probability ps, equivalent Sl/d.. However, the implications of using biased estimator like ours are not as well understood.
Our theoretical focus is Federated Learning (FL. in the context of empirical risk minimization, formulated in (T) within quadratic problem frameworks. This setting involves symmetric matrices L, as defined in the following equation:
f(w) = 1 f((w), where f((w) = 2 L,W w b.
While Equation 5 siimlliiess the loss function, the quadratic problem paradigm is extensively used in neural network analysis (Zhang et al. 2019 Zhu et al. 2022]] Shulgin & Richtárik (0023). Its in- herent complexity provides valuable insights into complex optimization algorithms (Arjevani al. 2022), thereby serving as a robust model for both theoretical
2020 Cunha et al. 2002 Goujaud et al.
examination and practical applications. In this framework, (xx is L-smooth, and V = L bb, where L = Σ L;, and 5 = Σ4 b,
At this juncture, we introduce fundamental assumption commonly applied in the theoretical anal- ysis of coordinate descent-type methods.
Assumption 3 (Matrix Smoothness). Consider a differentiable function f Rd - R. We say that is L-smooth if there exists positive semi-defiiite matrix L E Rdxd satisfying the following condition for all x, h E Rdd
f(x + (VF(z),
f())
(Lh, h).
(6)
24

---

Published as a conference paper at ICLR 2024
The classical L-smoothness condition, where L == I, is particular case of Equation equation 6 The concept of matrix smoothness has been pivotal in the development of gradient sparsification methods, particularly in scenarios optimizing under communication constraints, shown in [Sa-] faryan et al. (202T); Wang et al. (2022). We then present our main theory under the interpolation regime for quadratic problem with = 0, detailed in Theorem 3
We first provide the theoretical analysis of biased global pruning as implemented in Algorithm 5 To the best of our knowledge, biased gradient estimators have rarely been explored in theoretical analysis. However, our approach of intrinsic submodel training or global pruning is inherently biased. [Shulgin Richtárik (2023) proposed using the Perm-K K (Szlendak al., 2021) the global pruning sketch. Unlike their approach, which assumes a pruning connection among clients, our method considers the biased Rand-K compressor over coordinates.
Theorem 3 (Global pruning). In the interpolation regime for a quadratic problem 5 with L > 0 and by = 0, let L 1 n + P. L.P. Assume that W = ! LB +P* B
L
and
there exists a constant 0 > such that E[B" & LB"] 3 OW. Also, assume f(P"=*) S (1 + (2)5 fiul for some h > O. Fixing the number of iterations K and choosing the step size
min
1 1 the iterates satisfy:
AK 8
4 K
E
| [V]
WL
<
where Do = f(w") finf
By employing the definition of y. we demonstrate that the iteration complexity is O(1/e²). Com- pared with the analysis [Shulgiin&& Richtárik (2023), we allow personalization and do not constrain the global pruning per client to be dependent on other clients. Global pruning is essentially biased estimator over the global model weights, concept not widely understood. Our theorem provides insightful perspectives on the convergence of global pruning.
Our theory could also extend to the general case by applying the rescaling trick from Section 3.2 in Shulgin & Richtárik (2023). This conversion of the biased estimator to unbiased one leads to general convergence theory. However, this is impractical for realistic global pruning analysis, as involves pruning the global model without altering each weight's scale. Given that IST and biased gradient estimators are relatively new in theoretical analysis, we hope our analysis could provide some insights.
D MISSING PROOFS
D.1 PROOF OF THEOREM 1
Building on the smoothness assumption of L; outlined in Assumption [ the following lemma is established:
Lemma 1. Given that a function fo satisfies Assumption for each i e [n], then any w E Rd
holds that
[v)(())] 2L.(f((w) fiul).
Proof. Consider w' - (ww) By applying the L--smoothness condition of ʃ per Assumption
we obtain
L [lw'' S fi(w) + (VS( w/ - w [V/[21))1
Taking into account that piul < f((w), it follows that
ginl < f(w')
1
1 [[
fi(w)
+
L
21
1
f((w) 2L,
25

---

Published as a conference paper at ICLR 2024
Rearranging the terms yields the claimed result.
Since in this section, we are primarily interested in exploring the convergence of our novel model aggregation design, we set P for all E [n] and k [K]. Our analysis focuses on exploring the characteristics of S, which leads to the following theorem.
By the definition of model aggregation sketches in Definition 2 next iterate can be represented as
we have 1 n Ex=1 S; -. Thus, the
w
S (w - (V[i("
==1
n
s wi
Σ S (ww*)
(8)
71
w n
Bounding g crucial part of our analysis. align with existing works on non-convex opti- mization, numerous critical assumptions are considered. Extended reading this can be found in Khaled Richtárik (2020). Here, we choose the weakest assumption among all those listed in Khaled & Richtárik (200).
Assumption (ABC Assumption). For the second moment of the stochastic gradient. it holds that [I][())]?? s 2A(](w) - finf) + [ + +C,
for certain constants A, B, C and all RRd.
Note that in order accommodate heterogeneous settings, we assume a localized version of As-
sumption 4 Specifically, each of = ((V])((") is bounded for some constants A., Bi, Ci 2 0 and all w E R.d.
Lemma 2. The ge defined in Eqn. satisfies Assumption with A Lmaax B= 0.
Proof. The proof is as follows:
E&
[1l
E2
-
S
n
| fo (w
11
2La(Jiw**) )
(10)
< 2 2 (s(w*) pint),
where Equation follows from Lemma
We also recognize certain characteristics of the unbiasedness and upper bound of model aggregation sketches, as elaborated in Theorem 4
Theorem (Unbiasedness and Upper Bound Model Aggregation Sketches). For any vector w e Rdd the model aggregation sketch St. for each i [m], is unbiased. meaning E[S,w] Moreover; for any set of vectors yı y2, y/ E RRd, the following inequality is satisfied:
E
Σ S.y
<
M
1=1
26

---

Published as a conference paper at ICLR 2024
Proof. Consider vector RR, where x; denotes the i-th element of We first establish the unbiasedness the model aggregation sketch (Definition I::
S n
E[va,er,]= n
d
ng
d
I
I.
(11)
gmglt-1 1)+1
(i-
1)+
im
Next, examine the second moment:
21
d =n² |1=11/ = nn 1|/2/112
E [| = n²
Σ
Σ
|
j= +1+1
For all vectors yt, .... Un Rd, the following inequality holds:
27
Styk
1
Σ
E EI| II+
Σ
EKS
Σ
E S.y
1
Σ
11
71
n
n²
72
(12)
Integrating Equation
with Equation
we also deduce:
Σ
n
E
yb
U4
(13)
n
1
71
72
i=1
We now proceed to prove the main theorem of model aggregation, as presented in Theorem theorem is restated below for convenience:
This
Theorem (Personalized Model Aggregation). Let Assumption Tholds. Iterations K, choose step- size y < (1/Lmar, K. Denote Ao f(w") fint Then for any K 2 1, the iterates w of FedP3 in Algorithm satisfy
2(1 + 2 K
min osksK-1
E
Do-
TK
Our proof draws inspiration from the analysis in Theorem of Khaled & Richtárik (020) and is reformulated as follows:
Theorem (Theorem 2 in Khaled & Richtárik (0200). Under the assumptions that Assumption
and 4 are satisfied, let us choose a step size 7 such that & S T. Define A = f(w f Then, holds that
E [[]V < LCy + 2(1 L2 DSASK-
min
K A.
Careful control of the step size is crucial to prevent potential blow-up of the term and to ensure convergence c-stationary point. Our theory be seen special case with A Immax, B =
0, C' = 0, as established Lemma Thus, we conclude our proof.
D.2 PROOF OF THEBRRM 2
To establish the convergence of the proposed method, begin by presenting crucial lemma which describes the mean and variance the stochastic gradient. Consider the stochastic gradient
of = 1/5 Ejez, (w*) as outlined in Line of Algorithm
27

---

Published as a conference paper at ICLR 2024
Lemma 3 (Lemmaa 9 in Li al. 2022 Given Assumption for any client i, the stochastic gradient estimator g; is an unbiased estimator, that is,
1
E
V (wo V.fi(w)),
JEIs
where En denotes the expectation conditioned on all history up to round k. Letting q b the following inequality holds:
m.
2
(1 (1-9)C2
E&
- -VI((()
b
b
JEIn
Considering the definition of S we observe that 1 El= 8 next iteration w+1 of the global model is given by:
According Algorithm the
w: +1
St (wo* go + +c+) = w
St (ng. 5.
Employing the smoothness Assumption] and taking expectations, we derive:
L _ Ex |
Ex f(w En (VS(w") G*)
(14)
Given that c: 02), have Ea as follows:
Consequently, we can analyze E, (Vf(w*)) G*))
E, (V
Es
V Vf(w
s
(::0)
==1
Es
v (
1
Ex
v)(w
(15)
To bound the last term E& ||
in Equation 14
proceed as follows:
28

---

Published as a conference paper at ICLR 2024
Ex
||a||2 Ex
(1g 5i
1=1
12
E5 |M, 1|?
Ex |1 - -/,11²
Ex || # 1|2 ddo do²
Ex || -Vf((*)) + V)(("*)]| +
1=1
V fi (10)0 >|² +T
Ex 11 Vs((w>)]1 do²
1=1
2 0)
02
+ do
(16)
b
Incorporating Equations 16 and 15 into Equation
14
we obtain the following inequality for the
expected function value at the next iteration:
L [u f(w²)=(())) -2 [V((">)] + (PCO++ (1(1-)) + do?):
(17)
Before proceeding further, it is pertinent to consider the privacy guarantees of FedP 3, which based on the analysis of SoteriaFL as presented in Theorem this theorem follows:
al. (202)). We reformulate
Theorem (Theorem 2 in Li et al. 22022).. Assume each client possesses m data points. Under Assumption 3 in Li et al. (2022) and given two bounding constants C'A and CB for the decomposed gradient estimator, there exist constants e and c'. For any d and 5 e (0, SoteriaFL satisfies (e,8)-Local Differential Privacy (LDP) if we choose
(C2 C K log(1/5) m²c2
In the absence of gradient shift consideration within SoteriaFL, the complexity of the gradient esti- mator can be reduced. We simplify the analysis by substituting the two bounds CA and Cp with single constant C. Following similar setting, we derive the privacy guarantee for LDP-FedP3 as:
cC2 K log(1/8)
(18)
m²22
which establishes that LDP-FedP3 (e. LDP compliant under the above condition.
Substituting from Equation 18 and telescoping over iterations k = I,..., K, we can demonstrate the following convergence bound:
k
< f(w") - 8r
7(1 - q)0² (ogglog((//5) c2
[V
+
b + ym262
2
Ao
L
71b+1- -2
edC K log(1/0)
+
b
mm262
A
L
cdC K @dC2Klog(1/5)
+
722
+
K
ym?6
29

---

Published as a conference paper at ICLR 2024
To harmonize analysis with existing works, such CDP-SGD proposed Li al. 2022) which compresses the gradient and performs aggregation on the server over the gradients instead of directly on the weights, reframe Algorithm [ accordiingly. The primary modification involves defining M = 79." 7S,, where ç scaled by factor of 7. This leads to the following convergence result:
K K
Ao yk
yLC2
edK log(1/8)
(19)
2
m²²²
Optimal choices for K and y that align with this convergence result be defined
< mevAu
m m²c
K
(20)
CVLcdlog(1//0
cd log (1/0)
Adhering to the relationship established in Equation equation straint 7 S 1. define:
20 and considering the stepsize con-
m
K max
C log( (1/5) ed log((1/))
1
v Dood lon(loo//// (1/8)
7 min
T
CmevL L
Substituting these into Equation 19
we obtain:
A
LC
cdK log(1/8)
E
|
S
1
K
K
2
m²c²
A
K
m²c²
Au
K
m²e2
20 /odlog((18)
me
C Ldlo
me
Neglecting the constant the total communication cost for LDP-FedP3 is computed as:
CLDP-ROPPS nーk K dK
n
mey dL
m²
max
Co/log(1/8)
log(1/0)
me mevaLAo
m²=²
cloog(1/0) log(1/8)
D.3 PROOF OF THEOREM 3
We consider the scenario where P acts as a biased random sparsifier, and S = I. In this case, the update rule is given by:
w
6+1
p t
7P
V x
71
30

---

Published as a conference paper at ICLR 2024
Let E Rd and let S represent the selected number of coordinates from d. Then, P; is defined as:
P; = Diag(C.: Cg , where c² = {
if E S, if $ S.
Given that P, I, it follows that + P; I. In the context where P; is biased sketch, we introduce Assumption
Assumption 5. For any learning rate Y > 0, there exists a constant > such that, for any P E Rdxd w E RR, we have:
f(Pw) s ()w) plat).
Assumption 5 assumes the pruning sketch bounded. Given that the function value should remain finite, this assumption is reasonable and applicable.
In this section, for simplicity, we focus on the interpolation case where fi(z) f/v' L,w. The extension to scenarios with b; * 0 is left for future work. By leveraging the L-smoothness of function f and the diagonal nature of Ps, we derive the following:
f(wbtt)==
=
(P w
17
1
=
pk
p p w
11
n
1=1
(21)
p
B
2
f(P" w (P w²
B
w
22
2
af(w*) - - (Lp p* w B w
B w
72
af(w*) ((u* T pBL'' w +
(w*)
B
LB
2
Considering the conditional expectation and its linearity, along with the transformation properties of symmetric matrices, we obtain:
w
T L
Tww
(I+I') w. 2
1
By defining W we derive the following:
/EE
+p* B" I
and setting the stepsize to be less than equal to
31

---

Published as a conference paper at ICLR 2024
El {( 1 of(w*) - ((x T p LB
wh + ( LB
w
2
= =af(w*) -(w)) W w + 22 (w E B
LB
w
=af(w*) (VS(w*)) WL Vs(w*) + (VS(w
E B LB" L VS(w
マ 2
S saf(w") (V WL VS(w")
+ (VS(w>)) 0 OWL VS(w*)
2
fy
af(w*) 7 / f(w WI
= af(w") - 71 - -1/2) Vf(
I-' WWWW
S saf(w ? [ 'WL
(22)
Our subsequent analysis relies on the following useful lemma:
Lemma 4. Consider two sequences {x*)k20 and (Y&)k20 of nonnegative real numbers satisfying, for each is 2 0, the recursion
X S axxx Yv c
where >> 1 and are constants. Let K 2 1 be fixed. For each k 0, 1...... K 1, define the probabilities
K- Pk
K
where SK
Σ
a
(k+1))
SK
k=
Define random variable Y such that Y = Y2 with probability Pk. Then
a X
Xx
ak
E[
t
Xo + c.
SK
SK
Proof. We start by multiplying the inequality Yv S aXx Xkt1 by (k+1) for each k yielding
K-(K+1)y S a K K xx, - a (k+11 ( + a k-(+11)
Summing these inequalities for ho 0,,, ..., K 1, observe that many terms cancel out in telescopic fashion, leading to
Σ Kk (K-(+++) S a Xx Xx + Σ
k-(++) xx Xx
-X + Skc.
k=
k=0
Dividing both sides of this inequality by Sk, we get
K-1
K Xo
Xx
Σ P&Y& S
+ C,
SK
0=
where the left-hand side represents E[[)]
Building upon Lemma and employing the inequality + x S e", which valid for all x 2 0,
along with the fact that Sx K, we can further refine the bound:
k (1 (1+ (a - 1))
(aa-1)K 1)K
(23)
Sk
K
K
To mitigate the exponential growth observed in Eqn Setting the step size as
23
we choose
1 + 72h for some h 0.
log 2
7 5
hK
32

---

Published as a conference paper at ICLR 2024
ensures that >2h Σ log 2, leading to
(a-1))
E
S
SK
K
K
K
Incorporating Lemma 4 into Eqn 22 and assuming step size establish the following result:
S
log2 hK
for
some h > 0, we
400
V
(24)
33


Reviews:
[{"review_id": "ykL2P9lgVL", "reviewer": "Reviewer_2HRe", "summary": "This paper highlights the growing interest in federated learning (FL) for its privacy-preserving capabilities. It particularly addresses the challenge of client-side model heterogeneity in FL, driven by variations in client resources. Termed \"system heterogeneity,\" this scenario necessitates customizing a unique model for each client. The paper introduces FedP3, a Federated Personalized and Privacy-friendly Network Pruning Framework, designed to address model heterogeneity effectively. FedP3 can adapt established techniques to specific instances, offering a practical solution.", "questions": "Comments:\n\n-\tPrivacy analysis and convergence analysis need to be included. Indeed, separating model parameters from the network parameter architecture is intuitively beneficial to FL privacy protection, but whether existing gradient reconstruction attacks or other privacy attacks will challenge this personalization technology has not been fully explored. Therefore, it would be better if the authors could analyze the privacy performance empirically or theoretically.\n\n-\tFurthermore, the convergence analysis of the proposed method has not given any explanation. For model heterogeneous scenarios, reviewers expect to see rigorous analysis and discussion of the convergence and stability of this method.\n\n-\tThis paper claims that the proposed personalized pruning technique can well alleviate the system heterogeneity and model heterogeneity problems, but the reviewer has not seen any discussion and numerical results on the system heterogeneity scenario.\n\n-\tLimited in-depth comparison with state-of-the-art solutions. In fact, personalized pruning technique is not the first time to be applied to FL to solve efficiency and heterogeneity problems. The following literature needs to be included in the baseline solutions and explain the differences and connections between this paper and them.\n\n[1] Zhou X, Jia Q, Xie R. NestFL: efficient federated learning through progressive model pruning in heterogeneous edge computing[C]//Proceedings of the 28th Annual International Conference on Mobile Computing And Networking. 2022: 817-819.\n\n[2] Li A, Sun J, Li P, et al. Hermes: an efficient federated learning framework for heterogeneous mobile clients[C]//Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. 2021: 420-437.\n\n[3] Pase F, Isik B, Gunduz D, et al. Efficient federated random subnetwork training[C]//Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022). 2022.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "==*== Strengths\n+ This work offers an effective and adaptable FL framework FedP3 tailored for model heterogeneity scenarios.\n+ The proposed personalized network pruning technique is applicable to diverse scenarios.", "weaknesses": "==*== Weaknesses\n- The outcomes of the experiment need to be made more convincing. \n- Limited in-depth comparison with state-of-the-art solutions.\n- Privacy analysis and convergence analysis need to be included.\n- Empirical verification of the claimed contributions is necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1HL1LmHBLH", "reviewer": "Reviewer_mPei", "summary": "The appeal of Federated learning lies in its privacy-aware model training using data held locally on clients. However, the issue of varied client capacities, termed system heterogeneity, complicates its implementation. This paper proposes the FedP3 framework, emphasizing federated, personalized, and privacy-friendly network pruning, to cater to such diverse client scenarios.", "questions": "1. How much the few-layer communication can save communication costs and bandwidth?\n2. How the other FL aggregation strategies work in combination with FedP3?\n3. How to choose and tune the hyperparameter?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tThe proposed system comprehensively considers a holistic management of heterogeneity, i.e., it effectively manages both data and model disparities. It supports data distribution among clients, class-wise or Dirichlet non-iid. Moreover, it accommodates variance in the models between the server-client and among individual clients.\n2.\tThe proposed methods are inspiring for real-world scenarios, including dual pruning that supports both global pruning (from server to client) and local pruning by individual clients; few-layer communication from the clients to the server after local training. \n3.\tExperiments validate that the proposed FedP3 is both effective and adaptable. It paves the way for personalized and privacy-conscious pruning in a heterogeneous federated setting.", "weaknesses": "1.\tFew-layer communication can also significantly reduce communication costs and save bandwidth, a detailed numerical analysis would help.\n2.\tThe results on various FL aggregation strategies shall be considered for completeness.\n3.\tHyperparameter tuning part is not so crystal. More explanation is needed on how to choose and tune the hyperparameter (maybe via grid-search?) to deliver the best possible results for each model.\n4.\tBroader Literature Review is expected. While this work focuses primarily on the discussion and analysis of the most relevant and typical works, this approach might overlook other pertinent past research that holds tangential relevance to their study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XK073o9trN", "reviewer": "Reviewer_CVkW", "summary": "The paper focus on the problem of federated learning with model heterogeneity among clients and designs an algorithm that (1) allows each client only to perform training on a small subnetwork and (2) incorporates model pruning between server and clients to meet the different memory and communication constraints of individual clients. For subnetwork selection, the paper allows each client to train a randomly selected subset of layers of the global model. For model pruning, the paper explores two approaches: uniform pruning and uniform-ordered dropout. Finally, for aggregation of clients' updates, the paper explores simple averaging, weighted averaging (with the weight proportional to the number of layers each client trains), and attention averaging.", "questions": "1. Could the authors comment on whether the algorithm would perform well if the subnetworks trained by individual clients are significantly smaller than the global model? Table 2 shows that the algorithm performs poorly when each client only trains one layer.\n2. Could the authors comment on an alternative approach of global model pruning + standard FL on the pruned model?\n3. Several parts of the algorithm could be clarified more.\n   - Figure 1 shows that each client still needs to store a large proportion of the unpruned global model. Is that correct?\n   - In Algorithm 1, the pruned weights $P_i(W_t^l)$ are not used anywhere later.\n   - In eq (2), the objective function $h$ does not appear to be defined for the problem considered in this paper.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem of designing FL algorithms under model heterogeneity is well-motivated from practice. \n- The paper introduces the new ingredient of subnetwork pruning between the server and the clients and investigates the effect of pruning strategies and pruning ratio on the performance of FL.\n- The authors performed necessary ablation studies for the proposed algorithm, such as different subnetwork selections, data heterogeneity levels, sizes of total networks, and aggregation methods.", "weaknesses": "- There is a discrepancy between the experiment setting and the motivation problem setting in the introduction. \n  - The experiment setting does not reflect the heterogeneity of memory and communication constraints between clients, even though this is a major motivation mentioned in the introduction. Specifically, in all experiments, different clients train subnetworks roughly the same size. \n  - The subnetwork is not significantly smaller than the global model. For the ResNet-18 experiments, the subnetwork of each client appears to be at least around half the scale of the global model. This differs from the interesting scenario mentioned in the introduction, where each subnetwork is significantly smaller than the global model.\n- Lack of baselines for interpreting the significance of the proposed algorithm. One baseline that is not mentioned is an approach that performs model pruning on the global model first and then performs standard FL on the pruned smaller model. This is for understanding the necessity of personalized model pruning among clients.\n- Algorithm descriptions sometimes need more clarity; see question 3 for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ko44TTxBow", "reviewer": "Reviewer_Kwpv", "summary": "This paper focuses on addressing the model heterogeneity problem in federated learning. Concretely, it proposes an adaptable federated framework that leverages the personalized pruning technique in a privacy-preserving way. Experiments show that FedP3 can deal with data and model heterogeneity and adapt to various pruning strategies.", "questions": "1. Please refer to weakness.\n\n2. How to conduct federated training of LLMs based on the proposed method?\n\n3: In the algorithm, predefined pruning mechanisms are assigned for each client. Is this a common operation in FL under model heterogeneity issues? Are there any previous works conducting similar operations?\n\n4: There are multiple pruning and averaging strategies that can be adopted in the proposed FedP3. For a specific case, how to select the most appropriate strategy? \n\n5. In practice, what if we set several types of models to tackle various heterogeneous devices? Can you set a baseline algorithm to test it?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. An interesting network pruning-based pipeline is proposed. In this pipeline, the size of training parameters can be personalized for each client.\n\n2. Various local pruning and global aggregation strategies are developed, which present high flexibility of the proposed framework. \n\n3. Limitations of this paper including theoretical analysis and LLMs aspects are well discussed.", "weaknesses": "1. The relation between privacy-friendly property and network pruning is implicit. It is better to provide a detailed illustration of the reason why existing pruning-based FL methods have privacy concerns. This can help readers to better understand the motivation of the privacy-preserving part. \n\n2. The authors have emphasized the possibility of utilizing the proposed method in LLMs-based scenarios but there is a lack of related experiments supporting that point. Only shallow neural networks and ResNet are considered in the experimental part.\n\n3. To show the improved communication efficiency as mentioned in the 5th paragraph of the Introduction, it is better to provide some quantitive results on the number of communicated parameters. \n\n4. The presentation of the paper should be further enhanced and some parts should be reorganized. For example, the training goal and variable notation should be given in the Preliminary or Methodology section rather than the Introduction section. \n\n5. Lack of empirical comparison with existing FL methods that also address model heterogeneity problem. \n\n6. To solve the model heterogeneity problem, some existing work needs to be discussed, e.g. Knowledge Distillation-based FL [1],  Prototype-based FL [2], and NAS-based FL.\n\n[1] Ensemble Distillation for Robust Model Fusion in Federated Learning, NeurIPS 2020\n\n[2] FedProto: Federated Prototype Learning across Heterogeneous Clients, AAAI 2022\n\n[3] FedNAS: Federated Deep Learning via Neural Architecture Search, arXiv 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z5oW3eCGnG", "reviewer": "Reviewer_8d9P", "summary": "The authors present a \"privacy-preserving\" pruning mechanism that is tailored for heterogeneous clients/devices. In this approach, only a part of the model is transmitted to the server, saving resources and enhancing privacy.", "questions": "Please see my comments above.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- The paper is well written, presented and easy to understand. \n- Tackling device heterogeneity on federated learning is an important problem\n- Model pruning and submode research is an important step towards addressing device heterogeneity and our ability to train larger models with federated learning", "weaknesses": "The main weaknesses of this paper are:\n- The authors claim that this method is \"privacy-preserving\" and designed to maximise privacy overall. At the same time, there is no evaluation at all wrt to privacy (either analytical or through some empirical attacks). Furthermore, the privacy aspect is not discussed, practically assuming that fewer layers sent -> more privacy. While there might be some correlation there, I would expect these claims to be backed up with some thorough evaluation/analysis. \n\n- Similarly, the authors claim that resources are saved (energy, memory, cpu)  and larger models can be trained. But there is no evaluation wrt to any such savings. There are no numerical results wrt to any energy savings, the memory consumption savings, understanding how wide can the heterogeneity can we have wrt to device capabilities. Finally, It would be great to have a thorough study on the convergence speed. \n\n- Finally, given that there are a number of sub-model FL training methods proposed (the authors cite a few), it would be great if the evaluation could be expanded to compare with the state of the art.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pdzBR0T87N", "reviewer": "Reviewer_45Hg", "summary": "Federated learning has gained attention for its capability to train global models while maintaining local data privacy. This paper delves into the challenge of client-side model heterogeneity, exacerbated by differences in clients' memory, processing, and network capabilities (system heterogeneity). The proposed FedP3 framework can well address these challenges.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Adaptable Design: FedP3 caters to model diversity by allowing personalization based on each client's specific capacities, including computational, memory, and communication constraints.\n- Novel Dual-Pruning Strategy: FedP3 integrates a dual-pruning approach. This encompasses both global pruning (server to client) and local pruning executed by individual clients.\n- Strong Privacy Commitment: FedP3 prioritizes user privacy. The design ensures that full client data is kept confidential as only the selected layers are transmitted from the client to the server post-local training.", "weaknesses": "- Absence of Theoretical Insights: This work seems lacks a strong theoretical foundation or interpretation.\n- More Ablations: The current model does not explore all potential ablations, for instance, different data aggregation techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)