Why Go Full? Elevating Federated Learning Through Partial Network Updates
Authors: Haolin Wang
Xuefeng Liu
Jianwei Niu
Wenkai Guo
Shaojie Tang
Venue: NeurIPS 2024 poster (2024)
Keywords: Federated Learning
Partial Network Updates
Convergence Efficiency
Computational and Communicational Overhead Reduction
Primary area: other
TL;DR: We observe the layer mismatch problem in federated learning and propose a partial network update method to address it, improving convergence speed, accuracy, and reducing computational and communication overhead.

Abstract: Federated learning is a distributed machine learning paradigm designed to protect user data privacy, which has been successfully implemented across various scenarios. In traditional federated learning, the entire parameter set of local models is updated and averaged in each training round. Although this full network update method maximizes knowledge acquisition and sharing for each model layer, it prevents the layers of the global model from cooperating effectively to complete the tasks of each client, a challenge we refer to as layer mismatch. This mismatch problem recurs after every parameter averaging, consequently slowing down model convergence and degrading overall performance. To address the layer mismatch issue, we introduce the FedPart method, which restricts model updates to either a single layer or a few layers during each communication round. Furthermore, to maintain the efficiency of knowledge acquisition and sharing, we develop several strategies to select trainable layers in each round, including sequential updating and multi-round cycle training. Through both theoretical analysis and experiments, our findings demonstrate that the FedPart method significantly surpasses conventional full network update strategies in terms of convergence speed and accuracy, while also reducing communication and computational overheads.

Why Go Full? Elevating Federated Learning Through Partial Network Updates
Haolin Wang Xuefeng Liu Jianwei Niu Wenkai Guo St, Shaojie Tang
State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering. Beihang University, Beijing, China Center for Al Business Innovation, Department of Management Science and Systems, University at Buffalo, Buffalo, New York, USA.
V Zhongguancun Laboratory, Beijing, China
(wanghaolin, liu_xuefeng, niujianwei, kyeguo)-bbuaa, edu, cn shaojietSbuffalo edu
Abstract
Federated learning is distributed machine learning paradigm designed to protect user data privacy, which has been successfully implemented across various scenar- ios. In traditional federated learning, the entire parameter set of local models is updated and averaged each training round. Although this full network update method maximizes knowledge acquisition and sharing for each model layer, it prevents the layers the global model from cooperating effectively to complete the tasks of each client, challenge we refer to as layer mismatch. This mismatch problem recurs after every parameter averaging, consequently slowing down model convergence and degrading overall performance. To address the layer mismatch issue, we introduce the FedPart method, which restricts model updates to either single layer few layers during each communication round. Furthermore, maintain the efficiency of knowledge acquisition and sharing, we develop several strategies select trainable layers in each round, including sequential updating and multi-round cycle training. Our theoretical analysis and experimental results show that the FedPart method significantly outperforms traditional full-network update strategies, achieving faster convergence, greater accuracy, and reduced communication and computational overhead.
Introduction
Federated learning is machine learning framework that protects data privacy, which has attracted widespread attention from researchers recent years [McMahan et al. 2017 Kairouz et al. 2019 Li et al. 2019]. In traditional federated learning, after receiving the global model sent by the server, each client uses their local data to update the entire model parameters set for several iterations; then, the server averages the updated models obtain new global model and broadcasts all clients, starting the next training round.
While this approach has proven effective in many applications (Hard et al. 2018 Rieke et al. 2020), its convergence speed and ultimate performance are often lower than those of centralized schemes [McMahan et al. 2017 Zou et al. 22023). even when data across clients identically distributed di.. This suggests that while full network updates and sharing enrich each model layer with more knowledge, they may also introduce factors that negatively impact final performance. To further investigate the underlying reason, we conduct an experiment to visualize
independently and
Equal contribution. Corresponding Author, The source code is available at: https://githhit
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the update step sizes during each iteration. Typically, in centralized learning, the update step sizes of the model show a downward trend, indicating that the model is gradually converging. However, in federated learning, as shown in Fig- [a| the update step sizes significantly increase after each parameter averaging. This suggests that after averaging, the gradients calculated by subsequent layers become notably large, indicating inadequate cooperation among layers within the global model, a phenomenon we term as layer mismatch. The cause of this issue is illustrated in Figure 22.. The middle section of the figure depicts the local models of each client, which have undergone sufficient local training. Within these local models, the layers cooperate effectively, demonstrating match. However, upon aggregating the parameters of each layer, the averaged layers may struggle maintain this cooperation, resulting in mismatch. This layer mismatch can result in two key issues: first, it may prevent the global model in federated learning from converging to the optimal point of the global loss function, thereby compromising performance. Second, the persistent mismatch disrupts the federated learning process at the server level, significantly reducing training efficiency.
2.0
Full Network Up sate
Ful Network Jpdace Partia Network Update
Upd33t 2005 50305
18
22
40
(a) Update step sizes of traditional federated learning with full network updates.
(b) Update step sizes comparison between full network updates and partial network updates.
Figure I: Update step sizes for each iteration. The experiment uses the ResNet-8 model with 20,000 CIFAR-100 images distributed in an i.i.d. manner across 40 clients.
Trainable Parametors
Broadcasting
Parameters
Global Model (t-1)
Client Model 1
Client Model N
Global Model
loper #
layer
Match
Masch
Match
Milmmathh
Tryer
lapal
Averaging
(a) FedAvg: Full network updates lead to layer mismatch.
Broadcasting
Global (I-1)
Client
Client
Globall Modell
axa
Match
Maxch
Match
Malch
layor
Toyer
Averaging
(b) FedPart: Partial network updates help to reduce layer mismatch.
Figure 2: Mechanism for layer mismatch in FedAvg and FedPart.
To address the aforementioned problems, we propose FedPart, which employs partial network updates. Our main motivation is illustrated in Fig.2h In this toy example, we assume that only the i-th layer of the network is trainable in a given round. During local training on each client, this trainable layer can naturally align with the fixed parameters of other layers, which serve as anchors that constrain its update direction. This makes the averaged layers better align with other layers. To validate this approach, we conduct experiments and visualize the results in Figure Tb The curves demonstrate that partial network updates significantly reduce the increase in update step sizes after averaging, confirming their role in alleviating layer mismatch.

---

However, as a trade-off, training and transmitting only portion of the parameters at time might limit the efficiency of knowledge learning and sharing. Through thorough analysis, we identify a solution to this challenge, which involves carefully selecting trainable parameters. This solution is based on two key principles. The first principle sequential updating. We train the network layers sequentially, from shallow to deep, one layer at a time. This design is based on the observation that the shallower layers of neural network typically converge to their final parameters faster than the deeper ones (Raghu et al. 2017]. To align with this natural order, we adopt similar sequential strategy for layer selection. The second principle is the multi-round cycle training strategy. Our method emphasizes the importance of repeating the process of training from shallow layers to deep layers multiple times. During the original full network updates, shallow layers often learn low-level features, while deep layers learn high-level semantic features [Zeiler and Fergus 2014 Erhan et al. 2009). To preserve this property, inspired by the idea of Block Coordinate Descent (BCD) [Poczos and Tibshirani,, we propose the multi-round eycling training method to retain this characteristic to the greatest extent.
In addition, FedPart offers improved computation and communication efficiency, making it highly suitable for edge computing scenarios [Wang et al. 2019a b. Abreha et al.) 2022]. This is because FedPart only needs to train part of the neural network each training round, thereby significantly reducing the computational overhead of each client in each iteration. At the same time, since clients only need to upload and download the parts of the model that need updating, the amount of parameters to be transmitted is also greatly reduced.
To validate the effectiveness of FedPart, we explore its performance from both theoretical and experimental perspectives. Theoretically, we demonstrate that FedPart has a superior convergence rate under non non-convex settings compared to FedAvg. Experimentally, we perform extensive evaluations on various datasets and model architectures. The results indicate that the FedPart method significantly improves convergence speed and final performance (e.g., improvement of 24.8% on Tiny-ImaeeNet with ResNet-15 18), while also reducing both communication overhead (by 85%) and computational overhead (by 27%) simultaneously. Furthermore, our ablation experiments demonstrate how each of the proposed strategies contributes to enhancing the overall performance of FedPart. We also conduct comprehensive visualization experiments to illustrate the underlying rationale of FedPart. In summary, the contributions of this paper are as follows:
We identify the issue of layer mismatch in federated learning, which arises from updating and aggregating all parameters in each training round. This phenomenon potentially impact the model's ' convergence speed and overall performance.
To mitigate the effects of layer mismatch, we introduce FedPart, which implements partial network updates. Additionally, develop corresponding strategies for selecting trainable parameters.
We theoretically analyze the convergence rate of FedPart in a non-convex setting, demonstrating its advantages over full network updates.
We perform extensive experiments, showing that FedPart achieves significant improvements across multiple evaluation metrics compared to the full network updates scheme. Additionally, ablation and visualization experiments enhance our understanding of the rationale behind FedPart.
2 Related Work
Current research on partial parameter training or aggregation in federated learning has led to various applications, broadly categorized into three types:
Train all parameters, aggregate partial parameters. Also known as personalized federated learning, this approach involves each client training all parameters but only aggregating some of them [Tan et al. 2022]. For example, FedPer (Arivazhagan al.] 2019 and FedBN [Li al. 2021b personalizes classification and monichhhommaizztiinn layers respectively. FedRoD [Chen and Chao 2021 applies both global and local classifier heads. Other works may only upload low-rank space of parameter matrices Waag et al. 2023 Wu et al. [024a]. Although these methods achieve impressive results in hata-heterogeneous scenarios, they usually exhibit performance degradation when datasets across clients distributed in an i..d. manner. Moreover, they do not effectively
3

---

reduce computational overhead, and any reduction in communication overhead is minimal, as the personalized components are typically small.
Train partial parameters, aggregate all parameters. This category refers to each client training a different part of the model, with global update to the entire model during aggregation. For example, PVT Yang et al. 2022] and FedPT (Sidahmed et al. [021] strategically assigns specific model layers each client, Federated Dropout [Caldas et al. 2018 and FedPMT [Wu et al. 2023] randomly assign a neurons in different layers to clients, FeteroFL Diao et al. 2020 and FjORD Horvath et al.)
2021 deterministically decide a trainable subnetwork based on client computational power, FedRolex
Alam et al. 2022] further introduces sliding window method, and CoCoFL (Pfeiffer et al. 2022] introduces a quantization technique for overhead reduction. The primary objective of these methods to reduce client overhead by dynamically leveraging varying computational capacities among clients. However, compared to full network updates, these methods often result in performance degradation and slower convergence speed.
This approach starts with a small model and gradually increases its size until the entire network is trained [Rusu et al., (0166. This training paradigm has gain attention in the field of federated learning as its efficieccy in reducing resource consumption (e.g., ProgFed (Wang et al. 2022] and ProFL [Wu et al. 2024b)). However, because these methods eventually train a full model, they are able to solve the layer mismatch problem. Moreover, while these methods aim to reduce resource consumption, they often lead to performance losses compared to full network training. To the best of our knowledge, our FedPart is the first approach to simultancously enhance both convergence accuracy and efficiency.
Progressive Training.
3 Method
Generally speaking. FedPart based on partial network updates, which trains and aggregates only a few layers of the global network model for each training round. At the beginning of each training round that requires partial network update, the server first determines which layers need to be trained and sends this information to all clients. Subsequently, each client trains the corresponding layers, transmitting them to the server for aggregation, and the server broadcasts the averaged results to each client for next training round. We elaborate two key components of FedPart in the following subsections: partial network updates and the strategic selection of trainable layers.
3.1 Partial Network Updates
The partial network updates involve training and aggregating only few layers of the global network model each later communication rounds. Specifically, we partition the layers of global model into trainable ones and frozen ones. For each training iteration t and client the optimization objective is: mig us DE [[rr v))), where wi and ot respectively denotes parameters trainable and non con-trainable layers, wt & {ıūf, w represents the total parameter set, Dj represents the local data distribution of client and L(-) refers to the loss function. To optimize this objective, we adopt the following gradient descent formula:
w/ 7 S
anf L(x(w))) x Dj
Here, is the learning rate, S is a binary mask that selectively enables updates only for trainable parameters and O denotes element-wise product. After performing several local training iterations, the parameters of these selected layers are sent to the server and globally averaged at iteration t= T: WT NV Σ wif where N' represents the number of clients. For the sake of simplicity in formulation, the above equation aggregates and calculates the gradient for all parameters. However, only update and transmit the trainable components, significantly
in practical implementation, reducing both computational and communication costs.
3.2 Selecting Trainable Layers
Although training only subset of parameters can largely mitigate the layer-mismatch issue, it may limit the efficiency of knowledge learning and sharing. Therefore, we propose to carefully select trainable layers to address this limitation. As illustrated in Fig following the initial full network

---

updates. we train parameters layer by layer from the shallowest to the deepest. Subsequently, we cycle back the shallowest layer and periodically repeat this process. Generally, this strategy is driven by two key principles:
Sequential updating. This principle refers training model layers in sequence, from shallow to deep layers one at time. Our motivation is based on the fact that the convergence of neural networks follows natural intrinsic order, with shallower layers typically converging earlier than deeper ones |Rahu et al. 2017]. By partially updating network in accordance with this inherent training order, we can largely replicate the convergence process of full network updates while preserving training efficiency simultaneously.
Multi-round cycle training. This principle refers repeating the process of updating the neural network layers from shallow to deep multiple times. To illustrate our motivation, consider a fully trained neural network: shallow layers primarily focus on low-level semantic features (such the edges in images), while deeper layers focus on higher-level semantic features (such as the main objects in images). However, during partial network updates, because the deeper layers are initially non-trainable, shallow layers are forced to learn complex high-level semantic features, which disrupts the original information hierarchy in the neural network. Through multi-round cycle training, return to the shallow layers after training the deep layers. This strategy can reduce the burden on the shallow layers and helps approximate the final results of full network updates.
Model
Untrainable Layer
Trainable Layer
outout
Full Network Updates
Sequential Training (1 cycle)
Input
rounds
Multi-round Cycle Training
Figure 3: Strategy for selecting trainable layers.
3.3 Convergence Analysis for Fed
To analyze the convergence of FedPart, we first introduce some definitions and notations. Let each client use a uniform loss function C(x(w) with parameters for the data æ to calculate the loss function value, fiw Eg-D,, [C(x] w)] which is the expected loss function of client i. In this setting, the overall optimization goal of federated learning can be written as the sum of the expected loss functions of each client, that is: f(w) N Eimsii Si(w). Additionally, for notational simplicity, we denote the parameters of the i-th client at time i as w., the computed stochastic gradient vector as G. and the average of all client models at time as w
To represent partial network updates, we add binary matrix St as a mask for each update process of each client. To keep consistency with the methods section, we assume that for each mask S 37 of the elements are 1, and the rest are set to O. Before proving the convergence of our FedPart under non-convex conditions, we first introduce three assumptions:
1
Assumption 1: The expected loss function of any client is L-smooth, namely:
(w) (. < L|[ u/I, Vi, w.
(2)
Assumption 2: The variance and second-order moments of the gradients are bounded, that is:
Eg- (V - V (w)
E,- 1t 3 G V.,
5

---

Assumption 3: The variance of the gradients is approximately equal under all permissible masks:
Ex-D, |||S o (VL(xlw) -VC(ele) D, ||| & (VC(=)w) Vf((w))] Vi, Dj, Sr.
(3)
The first two assumptions are common in the literature, ensuring certain necessary characteristics of the loss function. The third assumption ensures that the specific choice of the mask matrix does not have too much impact on the final update variance, provided that the mask meets the required criteria. further discussion about the third assumption can be found in Appendix G
Based on these assumptions, we can analyze the convergence of FedPart. In terms of the approximate convergence rate, we align with related literature (Alistarh et al., 2017 Lian et al. 2017 Ghadimi and Lan 2013), using the average magnitude of the expected gradient over iterations, and finally obtain the following theorem:
Theorem 1: Under assumptions 1-3, with the total number clients as N, and all parameters divided into M groups of trainable parameters, the convergence rate of FedPart satisfies:
Ells VSW ||] O( VMNTT
where @ denotes element-wise product. A detailed proof of this theorem is provided in Appendix B O(>NT)
The results show that compared to the convergence rate of full network updates O VNT Yu et al.
2019), FedPart demonstrates significantly better convergence performance. This advantage becomes more pronounced as the choice amount of partial parameters per instance is reduced, which aligns with our original intention to reduce layer mismatch. However, it should be noted that convergence analysis can only indicate the difficulty of converging to stationary point, and cannot measure the model's 's performance after convergence. Therefore, it is not advisable to arbitrarily reduce the number of parameters trained in each iteration.
3.4 Analysis for Communication and Computational Cost
Communication Cost. Suppose FedPart divides all layers into M groups, and only one group is trained during each partial training session, with one communication round for each group. is easy to verify that the averaged communication costs of FedPart during the partial network update phase is of the original costs.
reduced to 37
Computational Cost. Assuming that computational costs are uniform across all layers, our method can reduce the overall computational expense during the partial network update phase by The primary reason for this reduction is that in FedPart, there is no need to compute gradients for the layers preceding the trainable parameters. To analyze this quantitatively, suppose the total overhead for both forward and backward propagation in complete model satisfies Diot D for + Dink. The ratio computational costs over single training cycle can then be written as
CompENU M Doo - + EM DoDum
M Djor ΣM (M+1)-Deae
CompENU M D Dor + M * Dbak
M D for + M * Dbak
Moreover, it is widely accepted in the literature that the computational cost of backward propagation approximately twice that of forward propagation [Rasley et al., 2020 Hobbhahn and Sevilla 2021 Therefore, the above equation can be rewritten as:
CompPNU Re COmpENU
M*Dfor M * Dfor for + (M+1) Djor 2
x
(5)
M D 2M Dior
Experiments
In the experimental setup, we primarily choose clients, with local epochs to 8. We test the global model on a balanced set. Unless specifically stated otherwise, the training datasets across all clients are independently and identically distributed (i.i.d... We utilize the Adam optimizer [Kingma and Ba 2014) with learning rate of 0.001, which is determined to be the optimal learning rate. In line

---

with prior references [i et al. 20216 Chen et al. 2022]. we refrain from uploading local statistical information during model aggregation. Each experiment is conducted three times with different random seeds to ensure robustness. The experimental results for additional scenarios, including learning-rate tuning and client sampling, are presented in Appendix
When choosing experimental metrics, we employ three distinct measures to capture various aspects of the benefits. These metrics include: Best Acc., which represents the ultimate accuracy achieved in classification tasks; Comm., indicating the total upstream transmission volume required by each client for given training round (in GB); and Comp., which illustrates the total floating-poit computation required by each client (in TFLOPs). All experiments are conducted on server equipped with 8xA100 GPUs, and we provide the complete source code in our supplementary material.
4.1 Main Properties
Comparison with full network updates. We apply the FedPart method to three classic federated learning algorithms: FedAvg McMahan et al., 2017]. FedProx (Sahu et al. 2018 and FedMOON et al., 2021a), and compare the results with their full network updates (FNU) counterparts. We utilize ResNet-8 [He et al. (0166] (detailed in Appendix (A) and update only one layer in each two consecutive training rounds (denoted as 2 R/L). Additionally, we insert five rounds of full network training between each cycle in FedPart. We conduct experiments on the CIFAR-10 (Krizhevsky
et al. 2010], CIFAR-100 Krizhevsky et al. 2009]. and Tiny TinylmageNet [Le and Yang 2015 datasets.
Table I: Performance of FL algorithms with full network and partial network updates.
FedAvg
FedProx
FedMoon
Comm. FNU FedPart
Comp. FNU FedPart
Data
FNU
FedPart
FNU
FedPart
FNU
FedPart
1
(61.0(1111) 57.7 (+0.5)
54.4 (+2.1) 57.5 (+0.6)
58.9 (+0.5) 57.8 (#0.4)
4,83
1.35
4,38
3.21
CTF
2
58.6 (+1.6) 60.2 (+0.4)
60.2 (+1.5) 59,9 (+0.5)
61.1 (=0.1) 59.4 (=0.2)
9.65
2.70
8.76
6.43
AR-
3
59.6 (+1.7) 61.7 (+0.3)
62.3 (#0.7) (+0.1)
62.3 (#0.4) 59.8 (=0.1)
14.5
4.05
13.2
9.64
60.7 (#1.3) 62.8 (+0.2)
62.8 (+1.1) 62.3 (+0.1)
62.3 (#0.4) 60.5 (#0.6)
19.3
5.40
17.5
12.9
30.9 (+0.4) 31.0 (+0.5)
30.6 (+0.3) 30.9 (+0.5)
31.0 (=0.5) 30.9 (=0.4)
4.92
1.38
4.39
3.22
CIF
2 32.9 (+0.3) 34.8 (+0.5) 33.6 (+0.5) 34.7 (#0.4) 33.2 (=0,9) 35.1 (#0.4)
9.65
2.75
8.78
6.44
AR-
3 34.3 (+0.2) 36,1 (+0.5) 34.5 (+0.5) 36.7 (#0.4) 34.6 (#1.1) 36.5 (=0.6)
14.8
4.13
13.2
9.66
100
4 35.6 (+0.3) 37.0 (+0.6)
35.8 (+0.2) 37.1 (#0.4)
35.0 (+1.0) 37.2 (#0.6)
19.7
5.51
17.6
12.9
5 35.6 (+0.3) 37.2 (#0.7)
36.2 (+0.5) 37,5 (=0.2)
35.4 (+0.8) 37.6 (+0.5)
24.6
6.88
21.9
16.1
1
15.6 (20.6) 17.1 (10.2)
15.8 (10.4) 16.8 (20.2)
17.5 (10.6) 17.3 (20.3)
5.02
1.40
17.5
12.9
Tiny-
2
17.0 (+0.8) 20.3 (40.1)
17.2 (#1.0) 20.1 (.0.2)
17.5 (#0.6) 20.5 (20.0)
10.0
2.81
35.1
25.7
Imag
3
17.6 (+0.4) 20.8 (+0.2)
18.0 (+0.5) 20.7 (=0.1)
18.4 (=0.8) 21.1 (=0.1)
15.1
4.21
52.6
38.6
eNet
4
17.7 (+0.4) 21.1 (+0.1)
18.2 (+0.7) 21.2 (#0.1)
18.4 (+0.8) 21.5 (=0.1)
20.1
5.62
70.1
51.4
5
17.7 (#0.4) 21.4 (+0.2)
18.4 (+0.8) 21.5 (+0.2)
18.4 (=0.8) 21.7 (=0.1)
25.1
7.02
87.7
64.3
The results in Table show that our FedPart method demonstrates rapid convergence and consistently outperforms traditional FNU methods across all training cycles C, ultimately achieving significantly higher accuracy (e.g., improving FedAvg -may- by 21 21).. At the same time, its commu- nication and computational costs are only 28% and 73% of those required by FNU. Furthermore, we observe that in some scenarios, the performance improvements of other federated learning algorithms even surpass those observed with FedAvg. This highlights that the layer mismatch problem identified in this paper is novel and cannot be addressed by any existing methods. However, our results on CIFAR- 10 are less impressive. This suggests that in simpler datasets, the primary issue might be the client drift problem explored in previous studies, whereas the layer mismatch problem becomes more prominent in complex datasets.
FedPart with deeper models. To evaluate the effectiveness of FedPart with deeper networks, conduct experiments on ResNet-18 8 (detailed in Appendix (A). This presents a more challenging scenario, as the proportion of trainable parameters significantly decreases in each round. Our experimental setup also follows the R/L pattern, with five additional full network updates inserted between cycles. The results, displayed in Table 2 show that in deeper networks, FedPart not only maintains its advantages in convergence speed and accuracy but also provides even greater reductions in communication and computational costs (by 85% and 27% compared to full network updates).
7

---

Table 2: Performance of FedPart for ResNet-18.
FedAvg-FNU Comm.
PerrattFedPPat Comm.
Data
C
Best Acc.
Comp.
Best Acc.
Comp.
1
59.4 (+1.5)
82.1
11.2
53.5 (+0.5)
12.2
8.19
C-FAR- 10
2
61.4 (+0.1)
164
22.3
57.5 (+0.6)
24.5
16.4
61.7 (+0.2) 3
246
33.5
59.2 (#0.4)
36.7
24.6
30.4 (+0.4)
82.5
11.2
27.8 (+0.5)
12.3
8.20
CIFAR- 100
2
31.9 (40.6)
165
22.4
31.6 (+0.4)
24.6
16.4
32.0 (20.5) 3
247
33.5
33.4 (#0.4)
36.8
24.6
Tiny-
1
13.7 (+0.2)
82.8
44.7
12.0 (+0.2)
12.3
32.8
2
13.7 (+0.2)
166
89.4
15.1 (+0.3)
24.7
65.5
ImageNet
13.7 (+0.2) 3
248
134
17.1 (+0.2)
37.0
98.3
PedPart for language modality. We also extend the FedPart method to the
Table 3: Performance of FedPart on NLP datasets.
field of natural language processing and evaluate it on AGnews and So- gouNews [Zhang et al. 2015] datasets. We choose the transformer architec- ture VVaswani al., 2017 for exper- iments. As shown in Table 3 the re- sults indicate that FedPart performs well on language tasks, not only main- taining comparable performance as FNU, but also reducing communication and computational overhead by 66% and 25%, respec- tively. This demonstrates method's scalability.
Data
C
FedAvg-FNN Best Acc.
FedAyg-FedPPan
Comm. Comp.
Best Acc.
Comm Comp.
AG
1 91.4 (+0.3)
22.3
5.58
91.1 (#0.2)
7.43
4.16
3
92.0 (+0.2)
66.9
16.7
91.5 (#0.2)
22.3
12.5
News
92.1 (+0.3)
106
27.9
92.0 (=0.3)
37.2
20.8
Sogou
94.2 (+0.2)
51.8
5.58
93.8 (+0.2)
4.16
3 94.3 (#0.2)
155
16.7
94.3 (=0.2)
51.8
12.5
News
94.4 (20.2)
259
27.9
94.4 (20.2)
86.3
20.8
FedPart under data heterogeneity. We also evaluate the performance of FedPart under scenarios involving data heterogeneity. The results in Table show that our FedPart consistently improve final performance (e.g., an improvement of 3.4% on Inn-ImmaeNNet in the presence of data heterogeneity. However, the extent of performance improvement is relatively smaller. This suggests that client drift (Karimireddy et al.] 2020) may have a more pronounced negative impact on our method. We conduct experiments with extreme data heterogeneity (a 0.1) in Appendix F3
Table 4: Performance of FedPart under data heterogeneity (Dirichlet, a 1).
Table 5: Performance of FedPart with different training rounds per layer.
Dataset
FedAvg-FNU
FedPart
Dataset
R/L r=15
r=25 r=35
1=45
r=55
r=65
57.7 (+ 0.7)
57.8 (# 0.4)
58.06 59.35 60.06 60.56
61.12 61.2
CIFAR-
59.2 (+ 0.7)
59.2 (# 0.4)
CIFAR- 10
56.85
58.80
58.80 60.46
60.46 61.25
60.4 (# 1.1)
60.7 (# 0.4)
56.17 58.76 59.60 59.60 59.60 59.60
60.4 (# 1.1)
61.4 (# 0.4)
10
48.22
54.65 57.40 57.40 59.03 59.03
33.1 0.4)
34.4 0.1)
28.10 29.86 31.25 32.17 32.60 33.09
CIFAR- 100
34.3 0.6)
35 0.2)
CIFAR-
2
24.47 30.07 30.07 32.53 32.53 33.59
(.)))
36.8 (2
100
4
23.56 26.26 28.19 32.01 32.01 32.01
(.5)
10
22.83 23.51 26.04 26.43 29.21 30.94
16.9
19.8 0.4))
14.37
16.33 18.02 19.27 19.88 20.18
Tiny-
0.1)
20.3 (# 0.1)
Tiny-
11.32
16.00 16.00 19.25 19.25 20.69
ImageNet
(+
20.4 (# 0.1)
ImageNet
4
9.09
11.44
15.21 17.89
17.89 17.89
17.4 0.1)
20.8 (+ 0.3)
10
11.33
12.03
12.03 12.03
12.03 16.16
4.2 Ablation Study
Training rounds per layer. In our FedPart, the training rounds per layer (denoted R/L) is an important hyperparameter. A larger R/L value means more thorough training in each cycle, but also results decrease in the number of cycles within the same number training rounds. We explore the performance of FedPart under different R/L. From the results in Table 5 when R/L,1,, the outcome shows limited final performance due to insufficient training for each layer. However, further increasing the R/L value not only fails to improve the final performance but also

---

reduces the convergence speed. In extreme cases, when R/L=10, only one cycle is conducted overall, which significantly affects both convergence speed and final accuracy. This indicates that generally, increasing the number of cycles is more effective than extending their duration. This aligns with the motivation behind our proposal of multi-cycling training.
Rounds of initial warm-up updates. To explore the im- pact of the duration of the initial full network updates phase (i.e. warm-up stage), we conduct experiments with this stage set to lengths 0, 5, and 60. In Table the term state refers to the period before or after partial network updates, which follow the warm-up up phase. The experimen- tal results clearly show that initial full network updates is crucial to the final model's accuracy. Notably, extending the full network update phase yields diminishing returns. However, even when the model is trained with FNU until no further accuracy improvement is observed (60 init.), utilizing FedPart still enhances the model's accuracy. This confirms FedPart's 's capability to improve the convergence of the final global model and reduce layer mismatch.
Table 6: Impact of the warm-up up rounds.
Dataset
State
[ init.
5 init. 60 init.
CIFAR-
bef.
0
41.56
58.92
10
aft.
58.48
61.25
66.18
CIFAR-
bef.
0
20.38
34.16
100
aft.
29.53
33.59
36.65
Tiny- ImageNet
bef. 0
9.11
16.25
afi.
16.81
20.69
19.99
Table 7: Impact of training sequences.
Dataset
|C|
Seq.
Rev.
Ran.
1 58.80 58.53 59.62
CIFAR- 10
Different orders for selecting trainable layers. We ex- periment with three different orders for selecting trainable parameters: sequential, reverse, and random. Sequential is the default configuration of FedPart, selecting layers from shallow to deep. In contrast, the reverse sequence selects layers from deep to shallow, while the random sequence selects layers randomly in each round. The results of the experiments are depicted the effectiveness of the three methods ranks as follows:
2
60.46
59.76 59.97
61.25
60.19
60.23
30.07
27.84
29.58
CIFAR-
32.53
29.41
30.92
100
3
33.59
31.79
31.44
1 16.00
13.15
15.91
Tiny- ImageNet
2 19.25
15.62
17.71
Table 7 demonstrating that
3 20.69
18.33
18.99
sequential > reverse > random. This aligns with the intrinsic convergence order of neural networks and meets experimental expectations.
4.3 Visualization Results
In this section, we conduct experiments to demonstrate why our proposed parameter selection strategy can enhance final performance, and what the impact of layer-wise information exchange has on privacy leakage. Our experiments are based on ResNet-8 and the CIFAR- dataset. We analyze the models obtained from four different methods: 1) FedAvg-100, which represents training with full network for 100 rounds; 2) FedPart(No Init. 1C), which represents using FedPart for one cycle without initial full network updates; 3) FedPart(1CC, which involves initial full network updates followed by one cycle of FedPart training: 4) FedPart(5C), which involves initial full network updates followed by five cycles of FedPart training. The visualization results are follows.
Activation maximization visualization. Activation maximization (Erhan et al., 2009 involves finding an input that maximizes a specific activation value within a neural network, reflecting the feature patterns the neuron focuses on. We use this method to explore the visual patterns captured by different models and measure their similarity using SSIM (Structural Similarity Index Measure) (Hore and Ziou| 2010). The results in T multiple cycles, the features captured by the FedPart model significantly differ from those of the FedAvg model. However, this discrepancy decreases after applying our layer-selcction strategy, suggesting that the model better recognizes the hierarchical nature of different semantic information, thus enhancing its performance. Additional visual results are provided in Appendix C
that, without initial full network updates and
Convolutional kernel visualization. We also analyze how different models extract semantic informa- tion by visualizing the convolutional kernels. We find that in the full network updates represented by FedAvg- 100, the shallow convolutional kernels primarily function as edge/comerr detectors. However, direct training of partial networks disrupts this property. Further, by initially employing full network updates and adding multiple training cycles, we gradually restore this characteristic. This effectively explains the impact of the parameter selection strategy on the final model formation. For specific visualization results of the convolutional kernels, please refer to Appendix D

---

4.4 Impact on Privacy Protection
We next demonstrate that FedPart offers enhanced privacy protection, as transmits less information in each communication round. Formally, we can abstract the model training process (for both full and partial parameter training) as a mapping: (Aw1, Awg, ..., Aw,) f(z). where the left hand side denotes the updates to each model parameter, and is the training data. From privacy attack perspective, the goal to find the best & such that the updates to w are as close as possible to the actual updates in each dimension. This resembles solving system of equations, where are the unknowns, and each dimension of w update represents an equation. With partial network training, the unknowns remain unchanged compared with full parameter training, but the number of equations decreases (i.e., less information for the attacker to leverage). Therefore, we believe that partial network training generally leaks less information.
To verify this experimentally, we conduct several rounds of federated learning using both full network and partial network updates. We employ DLG (Deep Leakage from Gradients) [Zhu et al.] (019) to attempt the recovery of original images and use PSNR (Peak Signal-to-Noiss Ratio) [Hore and Ziou 2010] to measure the extent of privacy leakage. DLG is a classic privacy leakage scheme, which aims finding an input that produces gradients most similar the gradients calculated from a given sample. In this way, DLG can approximately recover the input sample. Let the original model input be v, then the specific formula for recovering the input is as:
min [V V(((w))
We use PSNR to measure the quality of the reconstructed image. Given that x denotes the original image and & denotes the reconstructed image, the PSNR is calculated as follows:
PSNR -10 1og (MSB(x, 2))
where MSE(2, i)) denotes the mean square error between m x matrices a and ê, given by:
m
MSE(x, 2)
(8)
m
A larger PSNR value means better quality of the reconstructed image, which further implies higher risk privacy leakage.
The results in Table 9 show that, for different trainable layers, our method consistently exhibits better privacy protection both average and worst-case scenarios compared to full network updates. Attacking examples are provided in Appendix E
Table 8: SSIM activation maximization images between FedAvg and FedPart.
Table 9: Average and Max PSNRs of recon- structed images for FedAvg and FedPart models.
#1 (Conv) #10 (FC)
Model
Param. Avg. PSNR Max PSNR
FedPart(No Init. 1C)
0.680
0.896
FedAvg-100
All
17.07
25.57
FedPart(1C)
0.863
0.955
FedPart(SC) #1 (conv)
12.53
15.02
FcdPart(5C)
0.865
0.980
#10 (fc)
13.84
16.88
Conclusion and Limitation
We observe that the model averaged in federated learning is not directly applicable to the specific tasks of each client, situation we refer as layer mismatch. To address this issue, we propose the FedPart method, which introduces a strategy for selecting and training partial networks. We validate the effectiveness FedPart both theoretically and experimentally. In future work, we plan to evaluate our method on a wider range of model architectures and apply it to larger-scale datasets to further investigate its effectiveness and scalability.
ti represents the i-th layer of the model, with detailed partitioning method is presented in Appendix
10

---

Acknowledgement
This work was supported by the National Natural Science Foundation of China under Grants 62372028 and 62372027.
References
H. G. Abreha, M. Hayajneh, and M. A. Serhani. Federated learning in edge computing: systematic survey. Sensors, 22(2):450, 2022.
S. Alam, I.. Liu, M. Yan, and M. Zhang. Fedrolex: Model-heteroggeneuu federated learning with rolling sub-model extraction. Advances in neural information processing systems, 35:29677-299 2022.
D. Alistarh, D. Grubic, Li, R. Tomioka, and M. Vojuovic. Qsgd: Communication sgd via gradient quantization and encoding. Advances in neural information processing systems, 30, 2017.
M. G. Arivazhagan, V. Aggarwal, A. Singh, and S. Choudhary. Federated learning with personal- ization layers. arXiv preprint arXiv:191..00 (et.:::912.0081.. 2019.
S. Caldas, J. Konečny, H. B. McMahan, and A. Talwalkar. Expanding the reach of federated learning by reducing client resource requirements. arXiv preprint 11Xiv:1812.07210 2018.
D. Chen, D. Gao, W. Kuang, Y. Li, and B. Ding. pfl-bench: A comprehensive benchmark for personalized federated learning. Advances in Neural Information Processing Systems, 35:9344- 9360, 2022.
H.-Y. Chen and W.-L. Chao. On bridging generic and personalized federated learning for image classification. arXiv preprint arXiv:2107.00778, 2021.
E. Diao, Ding, and Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01264, 2020. 2020.
D. Erhan, Y. Bengio, A. Courville, and P. Vincent. Visualizing higher-layer features of deep network. University of Montreal, 1341(3):1, 2009.
S. Ghadimi and G. Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic program- ming. SIAM journal on optimization, 23(4):2341-236 2013.
A. Hard, K. Rao, R. Mathews, F. Beaufays, S. Augenstein, H. Eichner, C. Kiddon, and D. Ramage. Federated learning for mobile keyboard prediction. ArXiv, abs/181 11181100004 2018.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
M. Hobbhahn and J. Sevilla. What's 's the borkkward-forwaade flop ratio for neural networks?, 2021. URL wtwpaa//r/rooooo//ooddrr---dddee
https://epochai.cc org/ http://////pcca...ooooo/o/////// http://aawwwaadaaaae forward FLOP ratio Accessed: 2024-04-22.
A. Hore and D. Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pultern recognition, pages 2366-2369. IEEE, 2010.
S. Horvath, S. Laskaridis, M. Almeida, I. Leontiadis, S. Venieris, and N. Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 34: 12876-12888, 2021.
Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. Bhagoji, K. Bonawitz, Z. B. Charles, G. Cormode, R. Cummings, R. G. L. D'Oliveira, S. Y. E. Rouayheb, D. Evans, J. Gardner, Z. Garrett, Gascón, Ghazi, P. Gibbons, M. Gruteser, Harchaoui, C. He, L. He, Z. Huo, B. Hutchinson, J. Hsu. M. Jaggi, T. Javidi, G. Joshi, M. Khodak, J. Konecný, A. Korolova, Koushanfar, O. Koyejo, T. Lepoint,
Liu, P. Mittal, M. Mohri, R. Nock, A. Özgür, R. Pagh, M. Raykova, H. Qi, D. Ramage, R. Raskar, D. X. Song, W. Song. S. U. Stich, Z. Sun, A. T. Suresh, F. Tramèr, Vepakomma, J. Wang, L. Xiong, Z. Xu, Q. Yang, F. X. Yu, H. Yu, and S. Zhao. Advances and open problems in federated learning. Found. Trends Mach. Learn., 14:1-210, 2019.

---

S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5132-5143. PMIR, 13-18 Jul 2020. URI. ItR.attss://proceedigs
mlr. presss/v119/ariimrrett
html
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
A. Krizhevsky, V. Nair, and G. Hinton. Cifar-10 (canadian institute for advanced research), 2010.
Y. Le and X. Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Q Li, B. He, and D. X. Song. Model-contrastive federated learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10708-10717, 2021a.
T. Li, A. K. Sahu, A. Talwalkar, and V. Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37:50-60, 2019.
X. Li, M. Jiang, X. Zhang, M. Kamp, and Q. Dou. Fedbn: Federated learning non-iid features via local batch normalization. arXiv preprint arXiv:2102.07625, 2021b.
X. Lian, C. Zhang, H. Zhang, C.-J. Hsieh, W. Zhang. and J. Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems, 30, 2017.
B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Areas. Communication-efficin learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
K. Pfeiffer, M. Rapp. R. Khalili, and J. Henkel. Cocofl: Communication-ang and computation-aware federated learning via partial nn freezing and quantization. arXiv preprint arXiv:2203.054663 2022.
B. Poczos and R. Tibshirani. Coordinate descent. https: stat. cmu www.cme...u..e.....m....... edu/"ryantibbs/ http://pppppp/ieeettree///--oooo
pdf [Online; accessed March 31, 2024).
M. Raghu, Gilmer, J. Yosinski, and Sohl-DDckstein. Sveca: analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Singular vector canonical correlation
J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020. URL https://app
semanticscholar Caceaarr.gg///oaaaoe Trgg/CorpusID::: :C:rppss1::21111111999 221191193
N. Rieke, J. Hancox, W. Li, F. Milletari, H. R. Roth, S. Albarqouni, S. Bakas, M. Galtier, B. A. Landman, H. Maier-Hein, S. Ourselin, M. Sheller, R. M. Summers, Trask, D. Xu, M. Baust, and M. J. Cardoso. The future of digital health with federated learning. NPJ Digital Medicine, 3, 2020.
A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and Hadsell. Progressive neural networks. arXiv preprint arXiv: 04671, 2016.
A. K. Sahu, T. Li, M. Sanjabi, M. Zaheer, A. Talwalkar, and V. Smith. Federated optimization in heterogeneous networks. arXiv: Learning, 2018.
H. Sidahmed, Z. Xu, A. Garg, Y. Cao, and M. Chen. Efficient and private federated learning with partially trainable networks. arXiv preprint / 2021.
A. Z. Tan, H. Yu, Cui, and Q. Yang. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2022.
12

---

A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017.
H. Wang, X. Liu, J. Niu, and S. Tang. Svdfed: Enabling communication-effcccnnt federated learning via singllar-vatue-dccooossttnn In IEEE INFOCOM 2023-IEEE Conference Computer Communications, pages 1-10. IEEE, 2023.
H.-P. Wang, S. Stich, Y. He, and M. Fritz. Progfed: effective, communication, and computation efficient federated learning by progressive training. In International Conference Machine Learning, pages 23034-23054, PMLR, 2022.
S. Wang. T. Tuor, T. Salonidis, K. K. Leung, C. Makaya, T. He, and K. Chan. Adaptive federated learning in resource constrained edge computing systems. IEEE journal on selected areas in communications, 37(6):1205-122) 2019a.
X. Wang, Y. Han, C. Wang. Q. Zhao, X. Chen, and M. Chen. In-edge ai: Intelligentizing mobile edge computing, caching and communication by federated learning. leee Network, 53(5):156-) 2019b.
H. Wu, P. Wang. and A. C. Narayan. Modee--ettrogencouus federated learning with partial model training. In 2023 IEEE/CIC International Conference on Communications in China (ICCC), pages 1-6. IEEE, 2023.
X. Wu, X. Liu, Niu, H. Wang, S. Tang, G. Zhu, and H. Su. Decoupling general and personal- ized knowledge in federated learning via additive and low-rank decomposition. arXiv preprint arXiv:2406. 19931, 2024a.
Y. Wu, Li, C. Tian, and Xu. Breaking the memory wall for heterogeneous federated learning with progressive training. arXiv preprint arXiv:2404. 13349, 2024b.
T.-. Yang, D. Guliani, F. Beaufays, and G. Motta. Partial variable training for efficient on-device federated learning. ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4348 4352. IEEE, 2022.
H. Yu, S. Yang, and S. Zhu. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. Proceedings the AAAI conference on artificial intelligence, volume 33, pages 5693-5700, 2019.
M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 818-833 Springer, 2014.
X. Zhang, J. Zhao, and LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.
L. Zhu, Z. Liu, and S. Han. Deep leakage from gradients. In Neural Information Processing Systems, 2019.
L. Zou, Z. Huang, X. Yu, J. Zheng, A. Liu, and M. Lei. Automatic detection of congestive heart failure based multiscale residual unet++: From centralized learning federated learning. IEEE Transactions on Instrumentation and Measurement, 72:1-13, 2023.
13

---

A Implementation Details
In Section 4 we primarily adopt ResNet and language transformer for experiments, whose architec- tures illustrated in Fig. and Fig. respectively.
We also demonstrate the detailed partitioning method in our FedPart. Taking ResNet-8 (on the left in Fig. D as an example, we divide the trainable parameters of the model into 10 layers, corresponding the numbers #1-#10. Among these, the trainable parameters of #1 include only the weights of the convolutional layers but also the weights and biases of the accompanying BN layers after the convolutional layers. The other models follow the same representation method layer partitioning. During the sequential training phase of the FedPart method, we select one single layer to train in the order their numbering #i.
Input
input
81
7+7 64, 12 BN 4
7x7 84, BN
pool,
#2
3-3 cony, 84 BN
#2
3+3 conv, 64 BN
#3
3x3 64 BN
#3
3+3 64 BN
#4
3+3 conv. 64 BN
corv,
128,
#5
3H3 conv, 128 BN
#7
3H3 cony, 128 BN
1x1 cony. 128, BN
#6
1x1 cony, 128,
#9
3+3 conw, 128 BN
conv,
#7
conv, 256,
comw, 256,
#8
343 conw, 256
BN
#12
3H3 258 BN
1x1 cony, 256, BN
#9
1x1 conv, 256, BN
#13
W14
3+3 corv, 256 BN
comv,
#16
3x3 conv, 512,
#17
3x3 512
1x1 conv, 512, BN
#18
#19
3x3 cony, 512 BN
avg pool
avg pool
#10
fc. CLASS NUM
H21
CLASS NUM
Figure 4: Model architecture and layer partitioning about our ResNet-8 ResNet- 18 model.
14

---

LN Linear+ 2010
LN Linear Drogout 200
LN Linear Dropout 200
NN Linear- Drapout 200
Embeeddmn 200
Self-At Self-Attentinn h=2
Belf-Allention hm
Linear Dropout 200
Selt-Altenton h=2
Tc CLASS NUM NUM
Input
Drapout
#3
#5
#6
#7
Figure 5: Model architecture and layer partitioning for language transformer.
B Proof for Convergence Rate of FedPart
Before beginning the proof, we need to analyze the upper bound of the gradient variance after parameter selection. According to Assumption 3, we know that for any mask matrices S1, Sg. it holds that:
E E D. ||| (VL(zlw) Vf((w)IIl
E EB-D, || 3 (VL(=)w) - [[W)]]] I k, vi, w. E Dj. S1,
Constructivelly we set series of mask matrices S,, ... SM that have no overlapping 'I' elements the same positions and their sum exactly forms all-one matrix. Clearly, each of these mask matrices meets our requirements. Therefore, we can derive:
BE_.D,IIS, O (VL(x)w) vf((w))
122 EE-D, o (VL(z/w) v (())))) (10)
Summing over j from 1 to M, the left-hand side of the inequality is exactly the variance of the gradient without any mask matrices. Therefore:
E B. @ (VL(x))) ((((w)) DE [IVVL())))) [(())
M
Eg |||S @ (VL(xlw) [.(()))
For following proof, we mainly refer to Yu et al. of the left side of the inequality is o², so we finally obtain general upper bound for the gradient with masks:
2019).
According Assumption 2, the upper limit
o2k2
EE O (VL(z)w) (f(
Vi, w, x E D
(11)
M
With the groundwork laid, we are now ready to begin the formal proof process. First, based on Assumption 1, as the loss function L-smooth, we have:
Ef (w") s [[f + Bl(s) & Vf(w"-1), wt - 11---w--11 w' ++ Blll" wI
(12)
Next, we analyze the third term on the right-hand side of the inequality above to derive the following inequality:
B|||w* w² 112 7
o
N
=
1
Σ
S o (G) Vf(w 1)1
Σ S Vf
|
N
72 N
Σ
Ei|ls @ (G:
Vf((w! 1) + =
Σ st о Vf
N²
N
222222
+
St o V
MN
N
1=1
15

---

The last inequality comes from the derived Eq. 11 side of Eq. 12
Next, we analyze the second term on the right-hand
N
5
1
St
N
N
5 --71B
S O
N
ssllaa
V
s @ \)1
1
|5
0
S +
V Vi ||
N
1=1
< -Blss V
Σ
N
N f((
Further expanding the right-hand side of the above incquality, we obtain:
1 N Blll
V fu(
V/( Σ Vi(
N
E1|
( 1) v)(
No
1 N
E b w 119
In the above derivation, we have used the assumption of L-smoothness and Jensen's inequality. Next, we will continue to estimate the upper limit of this term. Assuming that the last parameter aggregation occurred at time t = to. and the next aggregation will take place at = to + E, then:
Elllo'
w||| = El1
Σ 1
Σ
ST o
O
-7
Σ
Seo NoGp G
r=to+1
-11
S GOG
Σ
1=1
r=
< 221
Σ
S
+11 +
Σ
+=
+=o++1
N
22(1
2E
S
O
TOGII+ +
Σ
||ST o G
N
+=
1=1
r=o+1
N Σ
22(1 - 22(1-
2
|S o G +
Σ
||ST о G
7= t=1
r=o++1
< - 20 < GC2
16

---

Substituting all the above inequalities into the right side of Eq. using a learning rate S 7 S I. satisfies:
12
we can finally obtain that when
B[f(w') S E[f(w' 7 -2
N
E
S
V
2 B vf(w ||
+
2
G
L
2
2NM
B|f(a"
2 B .(
G
L
0202k2
2N
After rearranging the above inequalities, we obtain
2 -
L
S o V 1|2
2o2k?
(13)
NM
Finally, summing the inequalities from t = 1, ; ,,, and multiplying both sides by 7. we obtain:
T E O
VS( D1 S 77 (
20 10
L
(14)
NM
Selecting learning rate 7
NM
we obtain: + B[|ls} @
VNNT (f(w))-
21
LVT
(*) + 4NME
Furthermore, by choosing E S
T1/4
we can derive the following
T
VNT
[MNj334
corollary: E
2L VNMT (f(W)
40
o2k
O
VMNT
VNMT
VNMT
This proves the convergence rate of FedPart.
C Visualizations for Activation Maximization
For better visualizing the semantic information recognized by each layer in the different models, in Fig. present representative results from the first and last layers of models under four scenarios: FedAvg-100, FedPart(No Init. IC), FedPart(1C), and FedPart(5C).
From the visualization results, it can be observed that FedAvg- 100, due to being full network update, captures low-level semantic features (such as clear boundaries) in shallow layers, while deeper layers capture complex semantic information. However, the results of FedPart(No Init. IC) exhibit noticeable differences in color and structural features compared to the full network update. This confirms our belief that partial network updates are detrimental to establishing hicrarchical information extraction approach, resulting in the model converging to possible local minima. Additionally, we observe that by including the initial phase of full network updates and multiple rounds of sequential training, the similarity of semantic information obtained by the model gradually approaches that of FedAvg. Therefore, the results sufficiently demonstrate that although only train one layer of the network each time, by employing an appropriate layer selection scheme, we ultimately achieve results comparable those of full network updates.
D Visualizations for Convolutional Kernel
To visually depict the characteristics the convolutional kernels in the first convolutional layer of different models, conduct kernel visualization. The four models select come from the following scenarios: FedAvg-100, 100, FedPart(No Init. 1C), FedPart(11C, and FedPart(SC).
In Fig.Z7 we present a comparison of results for planes in the first convolutional layer. It can be seen that the kernels in the first convolutional layer of the FedAvg- 100 model are mostly edge and corner detectors. In contrast, the results of FedPart(No Init. 1C) and FedPart(1C) 1C) appear more random and irregular. However, after training to convergence, the results of FedPart(5C) are noticeably more similar to those of FedAvg-100, and start to exhibit characteristics of simple feature extractors. This indicates that through partial network updates, the layers of the model gradually coordinate with each other, yielding cooperative effect.
17

---

FedAvg-100
FedPart(No Init. 1C) FedPart(1C)
FedPart(5SC)
Channel-10
Channel-39
#1(conv)
Channel-55
Channel-63
Channel-14
Channel-18
#10(fc)
Channel-57
Channel-62
Figure 6: Activation maximization images of different channels within different layers.
FodAvg-100
FadPart/No Init. 1C)
FedPart(1C)
FedPart(SC)
Figure 7: Convolutional kernel visualization results planes in the first convolutional layer. Fach plane include three color channels of image.
18

---

E Robustness to Privacy Attack
In this section, we will examine the privacy leakage using the DLG method full network updates and partial network updates, We perform DLG attacks in four settings: transmiting all parameters in FedAvg-10 model; and transmiting only the parameters of layers #1, #9, and # separately in FedPart(5CC model. In Fig. [8] we select some representative reconstructed images. The leftmost column represents the original images, while the four columns on the right show the reconstructed images obtained through DLG attacks under different settings.
It can be observed that in the FedAvg- 100 scenario, the reconstructed images have the highest quality, exhibiting significant similarity to the original images. However, when adopting partial network updates, the reconstruction quality poor. Apart from minor color correlations, the reconstructed images exhibit significant differences in structural features compared to the originals. This validates our claim that under the FedPart method, transmitting only subset of parameters effectively preserve data privacy.
FedAvg-100: all
FedPart(5C): #1(conv)
FedPart(5SC: #9(conv)
FedPart(SC): #10(fc)
Original Image
Image=1
Image-2
Image=3
Image-4
Image-5
Figure 8: The reconstructed images from DLG attacks on full network of FedAvg- 100 and different partial network of FedPart(5C..
F
Additional Experiments
F1
Learning Rate Tuning
In this section, we explore the appropriate learning rate for our experimental configurations. We conduct experiments the -IFAR-100 dataset using ResNet-8 for both FNU and PNU methods. The experimental results for Adam optimizer with different learning rates are shown in Table I0
From the results, it can be seen that both FNU and PNU methods perform best with leamning rate of 0.001. our experimental configurations, we ultimately select the Adam optimizer with learning rate of 0.001.
19

---

Table 10: Performances (Best Acc.) for different learning rate in full network and partial network updates.
FedAvg-FNU
FedPart
Dataset
Cycle
1r=0.000
1r=00001
1r=0001
1r=0.0001
Ir=0.001
Ir=0.01
24.82
30.68
28.15
16.56
31.36
23.41
30.08
32.53
30.13
21.88
34.70
30.10
CIFAR- 100
31.91
34.02
30.93
25.42
36.34
31.94
32,45
35.91
30.93
27.85
37.12
32.54
32.95
35.91
31.65
29.98
37.70
33.31
F22 Evaluation of Client Sampling
In this section, we conduct experiments with 150 clients, randomly sampling 20% the clients for training and aggregation in each communication round. The experimental results shown in Table IT Our method achieves final performance improvements of +2.1%. +1.6%, and +3.4% on CIFAR-IO, CIFAR-I00, and Tiny-ImageNet, respectively, indicating that FedPart performs better than FedAvg in this scenario.
Table 11: Performance of FedPart with client sampling.
Dataset
C
FedAvg-PNU
FedPart
2
60.82
63.22
CIFAR- 10
3
61.50
63.22
4
64.34
66.08
65.00 5
67.08
2
34,82
37.13
CIFAR- 100
39,36
37.13
4
39.64
41.00
40.55 5
42.12
2
19,63
23.06
Tiny- ImageNet
3
19.63
23.06
4
22.01
26.03
23.33 5
26.75
F.3 Analysis under Extreme Data Heterogeneity
In this section, we conduct experiments with an a 0.1 setting data heterogeneity is more severe. The experimental results are shown in Table 12
It can be seen that, in this extreme non-IID scenario (a 0.1), the model accuracy of our method is roughly on par with that of the full parameter method. However, this does not imply that FedPart offers performance advantages the benefits primarily arise from reduced communication and computation costs. The results indicate that FedPart can achieve similar accuracy to FedAvg while significantly reducing communication and computation costs (these metrics consistent with those observed in the IID scenario). As shown in Table when training on Tiny-ImageNet, FedPart reduces communication overhead by 72% and computation overhead by 27%. Therefore, believe that even in such an extreme scenario of data heterogeneity, our method still holds practical value.
Table 12: Performance of FL algorithms with full network and partial network updates under extreme data heterogeneity (Dirichlet, a 0.1)
FedAvg
FedProx FNU
Data
C
FNU
FedPart
FedPart
CIFAR-
33.79
44.02
39.64
43.85
44.08
44.41
46.88
45.42
20

---

G JJustification of Assumption 3
In Assumption 3 in Section
3.3
we assume that for any mask matrices S1, S2, holds that:
E || o (VZ(x)w)
v((w))II
S k, vi, w. 2 D1,
(15)
E D. ||| [V(x=))
f())))]
Regarding the value of k on the right side of the equation above, recall Eq. convergence rate of FedPart satisfies:
14 in Appendix B the
E E V/ S (0(09) (1) 4
L NM
Therefore, theoretically, the smaller the value of the smaller the value on the right side of this inequality, leading to improved convergence of FedPart. Hence, it is important to carefully examine the value range of & in practice.
We begin with analysing the lower bound of k. Since S₁ and S₂ are arbitrary, it possible that S = S2, indicating lower bound of for the value k. As for approximating the upper bound the k. we conduct Monte Carlo simulations on real-world nueral networks.
We test the & values in three neural networks at different training stages. For each neural network, conduct Monte Carlo simulations to collect 10,000 samples to accurately approximate the value of k. The experimental results shown in Table [3] We can that is close to 1 under different settings, which proves that the effect of applying different masks to the variability of gradient is similar, thus strongly supporting Assumption
Table 13: Monte Carlo simulation experiments for the value of k.
ResNet-8
ResNet-18
0% Training (Random initialized)
1.09
1.08
50% Training (Intermediate)
1.13
1.18
100% % Training (Fully trained)
1.13
1.17
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification; The following content of this paper centered on the partial network updating method introduced in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
21

---

Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed our limitations in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The full set of assumptions is provided in Section 3.3 and a complete proof is
available in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
22

---

Justification:
We release our complete code in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open tions to faithfully reproduce the main experimental results, as described in supplemental material?
the data and code, with sufficient instruc-
Answer: [Yes]
Justification: This full code is released in the supplementary materials, and the datasets are also public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce)
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips cc/ wwwblic/guides//odeeuuuuuue
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
23

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes)
Justification: We state important details in our main paper, while the full information can be viewed in our released code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The main results our paper is repeated for different random seeds and we have shown the error bar for each metric.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
24

---

Justification:
Descriptions about the resources required is stated in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper may have positive societal impacts due to its better ability for privacy protection, which is discussed in Section 4
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
25

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Owners of the assets used in this paper is properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. www com/datasets: has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code for this paper is well documented and is provided in the supplimentary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
26

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "IdOb5P63TW", "reviewer": "Reviewer_ZWNU", "summary": "The paper proposes a new and novel method to partially train networks to achieve better training efficiency but also, in some cases, better performance.", "questions": "Based on my comments above, I would like to ask the following questions,\n\n - How would the method perform on IID and non-IID data?\n - Would it be possible to add some experiments regarding the previously mentioned question?", "limitations": "The authors have sufficiently addressed limitations of this work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Training efficiency is an extremely important and timely topic. Given that FL aims to have massive networks to train upon any efficiency gains are exacerbated due to the network size. This work, in my opinion, has the following strengths,\n\n - The idea is simple however it is executed well and the intuition behind it is sound.\n - The paper can be fully reproduced as the code and datasets are provided.\n - The experiment section is thorough and has sufficient experiment to validate the authors' claims.", "weaknesses": "As the authors claimed, while the experiments are sufficient - they are rather limited in terms of dataset sizes given the target applications. Further, synthetic data experiments are missing - I would have personally appreciated to see how FedPart performs in the setting of both IID and non-IID data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the reviewers' comments. Here are our responses.\n\n**Q1: How would the method perform on IID and non-IID data?**\n\nThank you for the reviewer's comment. We have conducted additional experiments to enrich our analysis of non-IID data scenarios. We added experiments with an alpha=0.1 setting as data heterogeneity is more severe. The results are as follows. It can be seen that partial network training still exhibits much faster convergence compared to full network training while maintaining the final accuracy. This further shows the robustness of our proposed FedPart under data heterogeneity.\n\n| Dataset  |  C   | FedAvg (FNU) | FedAvg (FedPart) | FedProx (FNU) | FedProx (FedPart) |\n| :------: | :--: | :----------: | :--------------: | :-----------: | :---------------: |\n| CIFAR-10 |  1   |    33.79     |      44.02       |     39.64     |       43.85       |\n|          |  2   |    44.08     |      44.41       |     46.88     |       45.42       |\n\n**W1: ... limited in terms of dataset sizes ...**\n\nWe acknowledge your concern regarding the dataset sizes used in our experiments. As we mentioned in the paper, we did not use particularly large datasets to test our method. However, the datasets we used are comparable in size to those used in related works in the field [1, 2], and we believe that these datasets are sufficient to demonstrate the efficacy of FedPart. We will aim to incorporate more extensive datasets in future iterations of our research.\n\n**W2: ... synthetic data experiments are missing ...**\n\nWe appreciate your suggestion to include synthetic data experiments. This is indeed a worthwhile addition that can provide more insights into the performance of FedPart in various settings. However, due to time constraints, we cannot immediately incorporate synthetic data experiments. We will seriously consider adding synthetic data experiments in both IID and non-IID settings in future versions of our work.\n\n[1] McMahan, Brendan, et al. \"Communication-efficient learning of deep networks from decentralized data.\" Artificial intelligence and statistics. PMLR, 2017.\n[2] Li, Qinbin, Bingsheng He, and Dawn Song. \"Model-contrastive federated learning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dQ1dhx1Ucm", "reviewer": "Reviewer_9NYP", "summary": "This paper discovered the layer mismatch challenge in federated learning due to the full network update. To mitigate this challenge, the authors proposed the FedPart method. Specifically, the FedPart method would ask the clients do full network update in the beginning communication rounds, and then it would ask each client to do only train one layer in each round from the shallowest to the deepest. The experiment results indicate that the proposed method is better than the full network FL baselines.", "questions": "Please see the weakness above.", "limitations": "Please see the weakness above.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper proposed a novel observation of layer mismatch due to the full model training in federated learning.\n\n2. The single-layer sequential training is communicationally and computationally efficient to the edge devices.", "weaknesses": "1. I suggest the author provide more evidence and discussion of the proposed layer mismatch challenge in the paper. In the current version, there is no experiment to validate the proposed challenge and how this challenge happens during the training.\n\n2. In the experiment part, the paper mainly focused on the iid setup. Even in the ablation study (line 273), the paper adopts alpha equals 1 as the non-iid setup. In most of the FL papers, the experiment would adopt an alpha equal to at least 0.1 to simulate the extreme non-iid cases. As a result, the experiment in the paper could not provide useful support to the authors' argument. Also, the client number is very limited (only 40 clients), and the experiment does not contain any client sampling in the discussion.\n\n3. Even though the paper proposed many related studies, the experiment part contains no other layer-wise FL training baselines. As a result, it is really hard to judge whether the proposed method could stand for the SOTA performance.\n\n4. I noticed that some recent papers also proposed similar layer-wise training methods [1,2]. I suggest the author to include them as baselines to compare with.\n\n1. Zhang, Tuo et al. “TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Training.” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2023): 5064-5073.\n2. Lee, Sunwoo et al. “Embracing Federated Learning: Enabling Weak Client Participation via Partial Model Training.” IEEE Transactions on Mobile Computing (2024): n. pag.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the reviewers' comments. Below, we address the concerns raised in the \"Weakness\" section.\n\n**W1: I suggest the author provide more evidence and discussion of the proposed layer mismatch challenge in the paper. In the current version, there is no experiment to validate the proposed challenge and how this challenge happens during the training.**\n\nWe conducted experiments to verify the issue of layer mismatch, as shown in Fig. 1. Here we detail the logic of these experiments. In Fig. 1a, we observed that the step size significantly increases after each model averaging, which is detrimental to the final model convergence. We believe this phenomenon is caused by layer mismatch, and thus we designed a method targeting reducing layer mismatch (i.e., layer-wise training). The results show that after using our method, the step size variation is significantly reduced (as shown in Fig. 1b), and the final performance is improved (as shown in Table 1). Therefore, we believe that layer mismatch does exist, affects model convergence, and consequently impacts federated learning performance, which is the core challenge addressed.\n\nReviewers might have questions about the relationship between step size and layer mismatch, so we provide a more detailed explanation here: Formally, step size resembles the *norm of the gradient* in each model update. In a normal neural network training process, the step size gradually decreases towards zero during training, leading the model to converge to a fixed point. However, in federated learning, the step size significantly increases after each parameter averaging (Fig. 1a), indicating that the global model parameters do not collaborate well to complete local tasks (i.e., there is a parameter mismatch). Additionally, because the gradient calculation follows the backpropagation algorithm, the gradient of a parameter in one layer is only directly related to the parameters in the subsequent layers, with no direct correlation among parameters within the same layer. Therefore, we believe this mismatch mainly exists between layers (i.e., layer mismatch).\n\n**W2: In the experiment part, the paper mainly focused on the iid setup. Even in the ablation study (line 273), the paper adopts alpha equals 1 as the non-iid setup. In most of the FL papers, the experiment would adopt an alpha equal to at least 0.1 to simulate the extreme non-iid cases. As a result, the experiment in the paper could not provide useful support to the authors' argument. Also, the client number is very limited (only 40 clients), and the experiment does not contain any client sampling in the discussion.**\n\nThank you for the comments. We conducted additional experiments with 150 clients, randomly sampling 20% of the clients for training and aggregation in each communication round. The results are shown below, indicating that FedPart still performs better than FedAvg in this scenario.\n\n|    Dataset    | Cycle | FedAvg |  FedPart  |\n| :-----------: | :---: | :----: | :-------: |\n|   CIFAR-10    |   1   | 55.18  |   58.95   |\n|               |   2   | 60.82  |   63.22   |\n|               |   3   | 61.50  |   63.22   |\n|               |   4   | 64.34  |   66.08   |\n|               |   5   | 65.00  | **67.08** |\n|   CIFAR-100   |   1   | 30.82  |   31.96   |\n|               |   2   | 34.82  |   37.13   |\n|               |   3   | 39.36  |   37.13   |\n|               |   4   | 39.64  |   41.00   |\n|               |   5   | 40.55  | **42.12** |\n| Tiny-ImageNet |   1   | 17.34  |   17.83   |\n|               |   2   | 19.63  |   23.06   |\n|               |   3   | 19.63  |   23.06   |\n|               |   4   | 22.01  |   26.03   |\n|               |   5   | 23.33  | **26.75** |\n\nWe also added experiments with an alpha=0.1 setting as suggested by the reviewer. The results are as follows. It can be seen that partial network training still exhibits much faster convergence compared to full network training and achieves comparable accuracy to full parameter training. Here, it can be seen that in extreme data heterogeneity, we were unable to achieve accuracy improvements because the main issue in this setting is client drift rather than layer mismatch. Since our method is not specifically designed to solve data heterogeneity, in future work, we may introduce other solutions for data heterogeneity to improve performance further.\n\n| Dataset  |  C   | FedAvg (FNU) | FedAvg (FedPart) | FedProx (FNU) | FedProx (FedPart) |\n| :------: | :--: | :----------: | :--------------: | :-----------: | :---------------: |\n| CIFAR-10 |  1   |    33.79     |      44.02       |     39.64     |       43.85       |\n|          |  2   |    44.08     |      44.41       |     46.88     |       45.42       |\n\n**W3 & W4 & W5 & W6: Even though the paper proposed many related studies, the experiment part contains no other layer-wise FL training baselines. As a result, it is really hard to judge whether the proposed method could stand for the SOTA performance. I noticed that some recent papers also proposed similar layer-wise training methods [1,2]. I suggest the author to include them as baselines to compare with.**\n\nWe appreciate the reviewers' comments. However, these works cannot be used as benchmarks for comparison, as they are designed for a significantly different scenario from ours. Their purpose in using layer-wise training methods is to enable effective training of networks with different architectures in each client. However, if their methods are applied in our scenario (i.e., the same model is used on all clients, and the computational capacity is consistent), these methods will degrade to full parameter training (i.e., FedAvg, which is compared in our paper). Therefore, we believe comparing with these methods is unnecessary.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pxT23SV27F", "reviewer": "Reviewer_SwoS", "summary": "In this paper, the authors suggest a new approach for the network update step in federated learning. Considering that traditional federated averaging updates and aggregates all parameters at once, leading to a divergence between the global model and the local solution to a client’s specific task, they simply propose to only update parts of the network. They show that the layer mismatch between parameters learned locally and their corresponding global version is largely alleviated through this approach.\n\n[Edit after rebuttal: raised score to accept]", "questions": "Major:\n\n- In Figure 1, the update step size after aggregation seems extremely sharp. I was wondering if the momentum terms (or higher order statistics) of the optimizer were properly reset once the client received the global set of parameters? I briefly checked the submitted source code and it seems the reset of Adam or other optimizers is missing, maybe I overlooked it.\n- In Section 3.3 could you please comment on the assumption and put into perspective how realistic they are? A brief discussion on the assumptions would be great – taking them as they are just because they are in the literature feels a bit rough.\n-  Regarding the communication costs, how do clients know the mask $S^t$? It seems that this is missing from the analysis.\n- In Table 6, the ablation on warm-up rounds, it seems that the larger (60 round) warm-up performs significantly better than what was reported before (in Table 2). How does table 2 look when run with longer warm-up? What is the reason not to do that?\n- In the section on the warm-up ablation you mention that FedPart still enhances the model’s accuracy even after so many warm-up rounds (second to last sentence). Where can I see this in the presented numbers?\n- According to the paper checklist you provide repetitions to assess experimental statistical significance. This is not given for Table 6-9, which is in my opinion fine for these ablations, but the checklist argument should hence be No. You can add in the description that you assessed the significance for the main experiments with an indication which one these are.\n      \nMinor:\n      \n- In Figure 2, it is a bit unclear to me why the layers have these different shapes, could you please clarify?\n- The activation maximization approach is extremely old and yields arguably useless results. There is more than a decade of advancements in this field, with MACO [1], yielding great visual representations that would be much more useful in accessing the meaning.\n\n[1] https://arxiv.org/abs/2306.06805", "limitations": "There is almost no discussion of current limitations of the method at hand. The manuscript would benefit from a slightly more critical discussion in the future.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "This paper is well written and the main idea, which seems well motivated and interesting, is communicated clearly. The experiments are extensive and compare across multiple methods, metrics, and ablations.", "weaknesses": "There are two main weaknesses, (1) the suggested layer-wise update means that there is only one (linear) weight matrix that is changed between the global and local models through learning on the local data, which seemingly makes the models much *more* vulnerable to data reconstruction compared to an update of the full model. After all, we only need to reconstruct data from a change in a linear map, rather than a change of a deep, non-linear model. While the authors show that for some specific attacks that were tailored for deep model architectures the approach does fine, a simple attack based on the change in the single layer might be much more effective. At least a discussion on this matter would be important. (2) The motivation and initial results seem convincing, yet the experimental setup is counter-intuitive: you “insert five rounds of full network training between each cycle in […] FedPart”, which means you do more *full* training than partial updates. This seems strange, the motivated layer mismatch should happen consistently in every round. Maybe this is a misunderstanding, a clarification is certainly needed.\n\nThere are several other issues raised in the Question below, however, I do believe the paper has value and I will (significantly) raise my score if the concerns are answered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the reviewers' comments. Here are our responses.\n\n**Q1: In Figure 1, the update step size ... seems extremely sharp ... if the momentum terms ... were properly reset?**\n\nWe did reset the optimizer's state before the start of each local round. The relevant code is at line 131 in the file `fling/component/client/base_client.py`.\n\n**Q2: In Section 3.3 ... please comment on the assumption and put into perspective how realistic they are.**\n\nThank you for the suggestion. Here are more detailed comments about the assumptions.\n\nAssumptions 1 means that the gradient of the loss function does not change too abruptly as we move in the parameter space. This is reasonable for most models, where the loss changes gradually as the parameters are adjusted. Assumption 2 suggests that the variability in the gradient estimates is controlled. This is generally true in datasets without much extreme values, where individual data points do not drastically affect the overall gradient.\n\nAssumption 3 means that the effect of applying different masks to the variability of gradient should be similar. As this assumption is first proposed in this paper, we next give some evidence. We conducted Monte Carlo simulations (10,000 samples) to approximate the value of $k$ in Assumption 3. As is shown the the table below, $k$ is close to 1 under different settings, supporting Assumption 3.\n\n|           | 0% Training (Random initialized) | 50% Training (Intermediate) | 100% Training (Fully trained) |\n| ------ | ------- | ------ | -------- |\n| ResNet-8  | 1.09                             | 1.13                        | 1.13                          |\n| ResNet-18 | 1.08                             | 1.18                        | 1.17                          |\n\n**Q3: Regarding the communication costs, how do clients know the mask ?**\n\nActually, the server does not need to transfer the mask $S^t$ to each client. Since the trainable parameters are determined at the *layer* level, the server only need to transfer the *indexes* of trainable layers, resulting in almost no communication overhead. Such a notation is just for easier formulation.\n\n**Q4: ... the larger (60 round) warm-up performs significantly better ... How does table 2 look when run with longer warm-up? What is the reason not to do that?**\n\n*Extending the warm-up period leads to higher final model accuracy but also increases computational and communication overhead*. For instance, to achieve extreme performance, one might need to train with FedAvg until full convergence before adding several rounds of partial network training. The reason why we did not use such a setting in Table 2 is that, we *want to show a balanced improvement for both model accuracy and system overhead*.\n\n**Q5: In the section on the warm-up ablation ... FedPart still enhances the model’s accuracy after so many warm-up rounds. Where can I see this in the presented numbers?**\n\nThe relevant figures can be found in Table 6. In the table, *bef.* refers to the accuracy before performing partial network updates (at the end of the warm-up stage), and *aft.* refers to the accuracy after partial parameter training. For example, the accuracy after full parameter training at round 60 in CIFAR-10 is 58.92, while after one cycle of partial parameter training, this number become 66.18.\n\n**Q6: ...repetitions to assess experimental statistical significance is not given for Table 6-9 ...**\n\nWe appreciate the reviewer’s suggestions. We will amend the checklist arguments accordingly.\n\n**Q7: In Figure 2, it is a bit unclear to me why the layers have these different shapes**\n\nSorry for the confusion. In Figure 2, we used different shapes to represent different network parameters. Misalignment of shapes between layers indicates a mismatch, while aligned shapes indicate a match between layers.\n\n**Q8: The activation maximization approach is old ..., with MACO, yielding great visual representations that would be much more useful in accessing the meaning.**\n\nWe appreciate the suggestion. Due to time constraints, we cannot provide an improved visualization immediately but will adopt modern methods in future versions.\n\nWe next address the concerns raised in the \"Weakness\" section.\n\n**W1: ... (the proposed method) seemingly makes the models much *more* vulnerable to data reconstruction compared to an update of the full model ... a discussion on this matter would be important.**\n\nThanks for the comment. Compared to full-parameter training, we believe our proposed method will not increase the risk of privacy leakage. We can abstract  the model training process (for both full and partial parameter training) as a mapping: $ (\\Delta w_1, \\Delta w_2, ..., \\Delta w_n) = f(x) $, where the left hand side denotes the updates to each model parameter, and $ x $ is the training data. From a privacy attack perspective, the goal is to find the best $ x $ such that the updates to $ w $ are *as close as possible* to the actual updates in each dimension. This resembles solving a system of equations, where $ x $ are the unknowns, and each dimension of $ w $ update represents an equation. With partial network training, the unknowns $ x $ remain unchanged compared with full parameter training, but the number of equations decreases (i.e., less information for the attack to follow). Therefore, we believe partial network training leakages less information in general.\n\n**W2: ... the experimental setup is counter-intuitive: you “insert five rounds of full network training between each cycle in FedPart”, which means you do more *full* training than partial updates ...**\n\nThere may be a misunderstanding. A \"round\" refers to one communication between the server and clients, while a \"cycle\" refers to the entire process of training a partial network from shallow to deep layers, including many rounds. For instance, in ResNet-18 with $R/L=2$, one cycle includes 36 rounds, indicating far more partial-parameter training than full-parameter training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HbfZO9vyWp", "reviewer": "Reviewer_6JAi", "summary": "The authors observe 'layer mismatch' phenomenon in federated learning, and propose FedPart that uses partial network updates to address the former issue. They show that FedPart outperforms previous methods and also reduced communication and computation overhead.", "questions": "1. What exactly is the update stepsize in Fig1? Is it the learning rate? If not, what is it? I might have missed the exact definition of it so I would appreciate if the authors can clarify it here.\n\n2. The comparison in Fig1b really does not look like a big difference to me, for some iterations the partial update actually has a slightly bigger swings? And also I don't understand why Fig1a does not match Fig1b in terms of the blue curve, shouldn't they both be full network update and should be the same?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed idea is neat.\n\nThe demonstrated performances in experimental section are (surprisingly) good.", "weaknesses": "1. The 'layer mismatch' term does not convince me. The authors try to use Figure 1(a) to show that the 'update stepsize' (to be clarified later) increases after each averaging and thus it indicates layer mismatch, I don't fully understand the logic here, in particular why does this draw the conclusion that it is caused by 'inadequate cooperation among layers'? I do see there is a comparison in Fig1(b), but on one hand it does not differ that much (except for the 50 to 60 iterations region), on the other hand the logic that this is due to 'layer mismatch' is still not perfectly sound even if Fig1(b) shows greater difference. Are there other evidences or maybe related observations in previous literature to support this claim?\n\n2. The proposed strategy to select trainable layers does not provide much insight. Before getting to that section, I was expecting some unique or more informative way to choose what layer to update. It seems like the proposed strategy is the first thing anyone would try, thus I would like to see some argument/support for the effectiveness of choosing this strategy.\n\n3. The experiment setup, if I understand correctly, involves 5 rounds of full network training between each cycle, which I assume is a lot given that partial update is about 8 or 18 rounds per cycle (for Resnet-8 and Resnet-18 respectively)? This is where I concern about the impact of these full network trainings. Since the amount of full network trainings is significant in FedPart, one cannot confidently tell if it is really the partial network trainings that are truly beneficial. \n\n4. Regarding communication cost, I might argue that most communication cost are overhead in each communication, which I believe sharing 1/8 of the number of parameters BUT requiring more communication rounds is not necessarily saving communication cost. And my intuition is that only updating one layer at a time increases the total number of communication rounds?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the reviewers' comments. Next, we will address each of the issues raised one by one.\n\n**W1 & Q1: What exactly is the update stepsize in Fig1?  ... The authors try to use Figure 1(a) to show that the 'update stepsize' increases after each averaging and thus it indicates layer mismatch, ... why does this draw the conclusion that it is caused by 'inadequate cooperation among layers'?** \n\nWe address W1 and Q1 together as they are closely related. In this paper, the step size refers to the magnitude of model parameter changes between iterations. Since the learning rate is fixed in this experiment, this value directly reflects the norm of the gradient. Therefore, during a normal neural network training process, the step size gradually decreases towards zero, leading the model to converge to a fixed point. However, we found that in federated learning, the step size significantly increases after each parameter averaging. This increase in step size suggests that while the global model parameters might be generally robust and contain more knowledge, they cannot cooperate well to complete local tasks immediately (i.e., there is a parameter mismatch). Additionally, because the gradient calculation follows the backpropagation algorithm, the gradient of a parameter in one layer is only directly related to the parameters in the subsequent layers, with no direct correlation among parameters within the same layer. Therefore, we believe this mismatch mainly exists between layers and we refer to this phenomenon as layer mismatch.\n\n**W2: The proposed strategy to select trainable layers does not provide much insight ...**\n\nWe appreciate the reviewer's comments. In fact, although our method seems simple and intuitive, it is grounded in significant reasoning. The Sequential Training principle addresses the observation that neural networks converge shallow layers before deeper ones, a trend that is even more pronounced in federated learning [1].  From this perspective, in current federated learning, clients need to train deep networks without the shallow layers being fully converged, resulting in wasted computational power. Our sequential training strategy can effectively address this issue, but can lead to significant degradation in neural network performance. Therefore, we proposed another principle, Multi-round Cycle Training, which is inspired by the classic optimization method BCD (Block Coordinate Descent). By repeating cycles of training multiple times, it effectively alleviates the performance degradation caused by layer-wise training.\n\nIn addition, before arriving at this final solution, we tried many different approaches, such as various parameter selection strategies (both randomly, reversely) and optimizer designs, some of which are presented in the ablation studies. Furthermore, while the final method's design is relatively simple, it still involves many details, such as determining the timing for full parameter updates and balancing the length of each cycle with the total number of cycles.\n\n**W4: Regarding communication cost, ... (does) only updating one layer at a time increases the total number of communication rounds?**\n\nWe will first address the reviewer's fourth weakness, as we believe the third point will be better addressed after clarifying this one. The reviewer misunderstood our experimental results and our method does NOT increase the required number of communication rounds. In all experimental tables, we ensure the number of communication rounds is identical when comparing full parameter and partial parameter training. For example, in Table 1, the row with C=1 shows the performance of each method at the round when partial network training completes the first cycle, ensuring a fair comparison across all algorithms.\n\n**W3: The experiment setup ... involves 5 rounds of full network training between each cycle, which I assume is a lot ... Since the amount of full network trainings is significant in FedPart, one cannot confidently tell if it is really the partial network trainings that are truly beneficial.**\n\nThe reviewer's understanding of the experimental setup is generally correct, but we do not agree with the skepticism. Based on the response to weakness 4 above, our method does not increase the communication rounds. Therefore, from an equivalent perspective, in the process where all rounds involve full parameter training, we modified some rounds to partial parameter training and observed significant performance improvements. Logically, this clearly indicates that partial network updates contributed to the improvement.\n\n**Q2: The comparison in Fig1b really does not look like a big difference to me ... And also I don't understand why Fig1a does not match Fig1b in terms of the blue curve**\n\nRegarding the experimental results in Fig. 1b, the reviewer felt the effect was not significant. We respect the different opinions raised by the reviewer. However, we believe that, at least it is fair to say that FedPart can significantly reduce the degree of layer mismatch in extreme cases and generally maintain the original effect in typical cases, which still supports our claim.\nConcerning the reviewer's comment on the difference in the full parameter training curve, we believe there is some misunderstanding. Indeed, the blue curves in Fig 1a and Fig. 1b are NOT calculated in the same way. As the step size is the total magnitude of model updates, if not all dimensions are updated, this magnitude will naturally decrease, but this is an unfair comparison. To make the comparison as fair as possible, the full parameter training curve in Fig. 1b actually shows the \"update magnitude of the same parameters as those updated in partial parameter training,\" ensuring a fair comparison. We apologize for the confusion and will clarify it in a later version.\n\n[1] Unlocking the Potential of Federated Learning for Deeper Models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new approach for federated learning, where only parts of the networks are updated through communication and averaging between clients. The proposed method reduces the layer mismatch between locally learned models and the global model, which is a nice observation as pointed out by reviewers. Also, the experiment results are extensive  and strong, indicating that the proposed method is better than the full network FL baselines under multiple metrics.
Decision: Accept (poster)