Federated Learning with Manifold Regularization and Normalized Update Reaggregation
Authors: Xuming An
Li Shen
Han Hu
Yong Luo
Venue: NeurIPS 2023 poster (2023)
Keywords: federated learning; manifold regularization; update reaggregation

Abstract: Federated Learning (FL) is an emerging collaborative machine learning framework where multiple clients train the global model without sharing their own datasets. 
In FL, the model inconsistency caused by the local data heterogeneity across clients results in the near-orthogonality of client updates, which leads to the global update norm reduction and slows down the convergence.  Most previous works focus on eliminating the difference of parameters (or gradients) between the local and global models, which may fail to reflect the model inconsistency due to the complex structure of the machine learning model and the Euclidean space's limitation in meaningful geometric representations.
In this paper, we propose FedMRUR by adopting the manifold model fusion scheme and a new global optimizer to alleviate the negative impacts.
Concretely, FedMRUR adopts a hyperbolic graph manifold regularizer enforcing the representations of the data in the local and global models are close to each other in a low-dimensional subspace. 
Because the machine learning model has the graph structure, the distance in hyperbolic space can reflect the model bias better than the Euclidean distance.
In this way, FedMRUR exploits the manifold structures of the representations to significantly reduce the model inconsistency.
FedMRUR also aggregates the client updates norms as the global update norm, which can appropriately enlarge each client's contribution to the global update, thereby mitigating the norm reduction introduced by the near-orthogonality of client updates.
Furthermore, we theoretically prove that our algorithm can achieve a linear speedup property $\mathcal{O}(\frac{1}{\sqrt{SKT}})$ for non-convex setting under partial client participation, where $S$ is the participated clients number, $K$ is the local interval and $T$ is the total number of communication rounds.
Experiments demonstrate that FedMRUR can achieve a new state-of-the-art (SOTA) accuracy with less communication.

Federated Learning with Manifold Regularization and Normalized Update Reaggregation
Xuming An'
Li Shen Han Hu
Yong Luo
School of Information and Electronics, Beijing Institute of Technology, China
2 JD Explore Academy, China 3 School of Computer Science, Wuhan University, China
fanxuming, @hit. cnu mathshenl manessheni@@mmiil..m com, luoyongêshu edu.
Abstract
Federated Learning (FL) is emerging collaborative machine learning framework where multiple clients train the global model without sharing their own datasets. FL, the model inconsistency caused by the local data heterogeneity across clients results in the near orthogggagaly of client updates, which leads to the global update norm reduction and slows down the convergence. Most previous works focus eliminating the difference of parameters (or gradients) between the local and global models, which may fail to reflect the model inconsistency due the complex structure of the machine learning model and the Euclidean space's 's limitation in meaningful geometric representations. In this paper, we propose FedMRUR by adopting the manifold model fusion scheme and a new global optimizer to alleviate the negative impacts. Concretely, FedMRUR adopts a hyperbolic graph manifold regularizer enforcing the representations of the data in the local and global models are close to each other machine learning model has the graph structure, the distance in hyperbolic space can reflect the model bias better than the Euclidean distance. In this way, FedMRUR exploits the manifold structures of the representations to significantly reduce the model inconsistency. FedMRUR also aggregates the client updates norms as the global update norm, which can appropriately enlarge each client's contribution to the global update, thereby mitigating the norm reduction introduced by the near-orthogonality of client updates. Furthermore, theoretically prove that our algorithm can achieve linear speedup property O VSKT for non-convex setting under partial client participation, where S is the participated clients number, K is the local interval and T' the total number of communication rounds. Experiments demonstrate that FedMRUR can achieve new state-of sate-of-the-art (SOTA) accuracy with less communication.
wiw-dmmnnsioal subspace. Because the
Introduction
FL is a collaborative distributed framework where multiple clients jointly train the model with their private datasets [27, 28]. To protect privacy, each client is unable to access the other dataset [2]. centralized server receives the parameters or gradients from the clients and updates the global model[46]. Due to the limited communication resource, only part of the clients is involved in the collaborative learning process and train the local model multiple intervals with their own datasets within one communication round [23]. Due to the data heterogeneity. clients' partial participation and multiple local training yield severe model inconsistency. which leads to the divergences between the directions of the local updates from the clients and thus reduces the magnitude of global updates [17]. Therefore, the model inconsistency the major source of performance degradation FL [40, 15].
"Corresponding authors: Li Shen and Han Hu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

So far, numerous works have focused on the issues of model inconsistency to improve the performance of FL. Many of them [20, 16, 42, 1] utilize the parameter (or gradient) difference between the local and global model to assist the local training. By incorporating the global model information into local training, the bias between the local and global objectives can be diminished at some level. However, the parameter (or gradient) difference may fail to characterize the model bias due to the complex structure of modern machine learning model and the Euclidean space has limitations in providing powerful and meaningful geometric representations [10]. Meanwhile, incorporating the difference introduces extra high computation and communication costs because of the high- dimensional model parameter, which is common in the modern machine learning area[25]. Some other works [22, 39, 44, 24] exploit the permutation invariance property of the neurons in the neural networks to align and aggregate the model parameters for handling the model inconsistency issues, but the extra computation required for neuron alignment may slow down the speed of FL.. In addition, Charles et al. [3] demonstrate that after multiple rounds, the similarities between the client updates approach zero and the local update direction are almost orthogonal to each other in FL.. If the server aggregates the local updates, each client's contribution is little, which reduces the global update step. Therefore, we need to reduce the model inconsistency and compensate for the global norm reduction introduced by the near-orthogonality of client updates.
In order to alleviate the model inconsistency and compensate for the global reduction, we pro- pose practical and novel algorithm, dubbed as FedMRUR (Federated learning with Manifold Regularization and Normalized Update Reaggregation). FedMRUR adopts two techniques to achieve SOTA performance. i) Firstly, FedMRUR adopts the hyperbolic graph fusion technique to reduce the model inconsistency between the client and server within local training. The intuition is that adding the manifold regularizer to the loss function constrain the divergence between the local and global models. Unlike the Euclidean space, the hyperbolic space is a manifold structure with the constant negative curvature, which has the ability to produce minimal distortion embedding[8] with the low storage constraints(30) for graph data. And the neural network, the most prevail machine learning model, has graph structure[34, 24], we map the representations to the hyperbolic space and compute their distance to indicate the model bias precisely. Considering the numerical stability[6], we select the Lorentz model to describe the hyperbolic space and the squared Lorentzian distance [19] to indicate the representations' proximity. By adopting the hyperbolic graph fusion technique, FedM- RUR can constrain model inconsistency efficiently. ii) Secondly, FedMRUR aggregates the client's local updates in novel normalized way to alleviate the global norm reduction. In the normalized aggregation scheme, the server aggregates the local update norms as the global norm and normalizes the sum of the local updates as the global direction. Compared with directly aggregating local updates, the new aggregation scheme enables each customer's contribution to be raised from its projection on the global direction to its own size. As result, the size of the global update becomes larger and compensates for the norm reduction introduced by model inconsistency, which improves the convergence and generalization performance of the FL framework.
Theoretically, we prove that the proposed FedMRUR can achieve the convergence rate of O VSKT on the non-convex and L-smooth objective functions with heterogeneous datasets. Extensive experi- ments CIFAR- 10/100 and Tinylmagenet show that our proposed FedMRUR algorithm achieves faster convergence speed and higher test accuracy in training deep neural networks for FL than several baselines including FedAvg, FedProx, SCAFFOLD, FedCM, FedExp, and MoFedSAM. We also study the impact on the performance of adopting the manifold regularization scheme and normalized follows:
aggregation scheme. In summary, the main contributions
* We propose a novel and practical FL algorithm, FedMRUR, which adopts the hyperbolic graph fusion technique to effectively reduce the model inconsistency introduced by data heterogeneity, and a normalized aggregation scheme to compensate the global norm reduc- tion due to the near-orthooonality of client updates, which achieves fast convergence and generalizes better.
We provide the upper bound of the convergence rate under the smooth and non-convex cases and prove that FedMRUR has a linear speedup property O VSKT
We conduct extensive numerical studies on the 1IFAR-10/100 and TinyImagenet dataset to verify the performance of FedMRUR, which outperforms several classical baselines on different data heterogeneity.
2

---

2 Related Work
McMahan et al. [27] propose the FL framework and the well-known algorithm, FedAvg, which has been proved to achieve linear speedup property [43]. Within the FL framework, clients train local models and the server aggregates them to update the global model. Due to the heterogeneity among the local dataset, there are two issues deteriorating the performance: the model biases across the local solutions at the clients [20] and the similarity between the client updates (which is also known as the hear-orthogonality of client updates) [3], which needs new aggregation scheme to solve. In this work, we focus on alleviating these two challenges to improve the convergence of the FL algorithms.
Model consistency. So far, numerous methods focus on dealing with the issue of model inconsistency in the FL. framework. Li et al. [20] propose the FedProx algorithm utilizing the parameter difference between the local and global model as a prox-correction term to constrain the model bias during local training. Similar to [20], during local training. the dynamic regularizer in FedDyn [1] also utilizes the parameter difference to force the local solutions approaching the global solution. FedSMOO [36] utilizes dynamic regularizer to make sure that the local optima approach the global objective. Karimireddy al. [16] and Haddadpour et al. [9] mitigate the model inconsistency by tracking the gradient difference between the local and global side. Xu et al. [42] and Qu et al. [31] utilize client-level momentum term incorporating global gradients to enhance the local training process. Sun et al. [37] estimates the global aggregation offset in the previous round and corrects the local drift through a momentum- term to mitigate local over-fitting. Liu et al. [26] incorporate the weighted global gradient estimations as the inertial correction terms guiding the local training to enhance the model consistency. Charles et al. [4] demonstrate that the local learning rate decay scheme can achieve a balance between the model inconsistency and the convergence rate. Tan et al. [38] show that the local learning rate decay scheme is unable to reduce the model inconsistency when clients communicate with the server in an asynchronous way. Most methods alleviate the model inconsistency across the clients by making use of the parameter (or gradient) difference between the local and global model.
Aggregation scheme. There numerous aggregation schemes applied on the server side for improving performance. Some works utilize classical optimization methods, such as SGD with momentum [45], and adaptive SGD [5], to design the new global optimizer for FL. For instance, FedAvgM [13, 35] and STEM [17] update the global model by combining the aggregated local updates and a momentum term. Reddi et al. [32] propose a federated optimization framework, where the server performs the adaptive SGD algorithms to update the global model. FedNova [41] normalizes the local updates and then aggregates them eliminate the data and device heterogeneity. In addition, the permutation invariance property of the neurons in the neural networks is also applied for improving robustness data heterogeneity. FedFTG [48] applies the data-free knowledge distillation method to fine-tune the global model in the server. FedMA [39] adopts the Bayesian optimization method to align and average the neurons in layer-wise manner for a better global solution. Li et al. [22] propose Position-Aware Neurons (PANs) coupling neurons with their positions to align the neurons more precisely. Liu al. [24] adopt the graph matching technique to perform model aggregation, which requires large number of extra computing resources in the server. Many deep model fusion methods [21] are also applied in the research field of FL, such as model ensemble [47] and CVAE [12]. The aforementioned algorithms utilize the local parameters gradients directly without considering the near-orthgonnllty of client updates, which may deteriorate the convergence performance of the FL framework.
The proposed method FedMRUR adopts the hyperbolic graph fusion technique to reduce the model inconsistency and normalized update aggregation scheme to mitigate the norm reduction of the global update. Compared with the previous works, we utilize the squared Lorentzian distance of the features in the local and global model as the regularization term. This term can more precisely measure the model bias in the oimeeinnsionnl subspace. For the update aggregation the server, FedMRUR averages the local updates norm the global update norm, which achieves to alleviate the norm reduction introduced by the cear-orthogonality of the client updates.
3 Methodology
In this section, we first formally describe the problem step for Fl. and then introduce the FedMRUR and the two novel hyperbolic graph fusion and normalized aggregation techniques in FedMRUR.
3

---

server
| avap (114p1))
Ag global update-
Update
normalize
Σ
Wgi global parameter
5
Aggregation
① Broadcast
Upload
client p-1
client
Wg
4p local update
Fg
train wp locally with:
data
Euclidian Lorentzian
F (w(wp) fp(w)) + R ((wp.W))
Fp
Ffstance(Fg..F)
Construct the manifold regularizer: w(w
Wp
The workflow of FedMRUR. Once the global parameter is received, the client initializes the local model up and starts hyperbolic graph fusion. In the hyperbolic graph fusion, the client first takes the local and global model to get their representations and maps them into the hyperbolic space. Then, client use their distances in the hyperbolic space as a regularizer to constrain model divergence. Next, the client performs local training and uploads the updates to the server. The server adopts the normalized scheme to aggregate the local updates and performs the global model update.
Figure I:
3.1 Prohlem setup
We consider collaboratively solving the stochastic non-convex optimization problem with P clients:
min f(w) = P
1
Σ [n(w,, with fp(w)= == EE [M(1, =)], PEP
(1)
where w is the machine learning model parameter and = is a data sample following the specified distribution Dp in client y; meanwhile ((w, 2) represents the model loss function evaluated by data z with parameter w. fp (w) and f(w) indicate the local and global loss function, respectively. The loss function l(w, 2), fw(w,, and fww are non-convex due the complex machine learning model. The heterogeneity of distribution Dr causes the model inconsistency across the clients, which may degrade the performance of the FL framework.
Notations. We define some notations to describe the proposed method conveniently. || denotes the spectral norm for a real symmetric matrix or L2 norm for a vector. (.. denotes the inner product of two vectors. For any nature a, b, a b and V denote min {a, b) and max {a, b). respectively. The notation O(-), O(-), and ((-) are utilized to hide only absolute constants that don't it depend any problem parameter.
3.2 FedMRUR Algorithm
In this part, we describe our proposed FedMRUR algorithm (see Figure and Algorithm 1) to mitigate the negative impacts of model inconsistency and improve performance. We add manifold regularization term on the objective function to alleviate the model inconsistency. To climinate the near-orthogonality of client updates, we design new method to aggregate the local updates from the clients. Within one communication round, the server first broadcast the global model to the participating clients. During local training, the client takes the sampled data into the local and received global model and gets the representations. Then the client maps the representations into the hyperbolic space and computes their distance, which is used to measure local and global models. Next, the client adopts the distance as a manifold regular to constrain the model bias, achieving model fusion the hyperbolic graph. After local training, the client uploads its local update to the server. The server aggregates the local update norms as the global update step
divergence between

---

Algorithm FedMRUR
1: Input: initial parameter w": momentum coefficient or perturbation radius pi local interval communication rounds T; set of selected clients S global and local learning rate fg YR-
2: Output: Global parameter w², VI E
3: Initialization: Initialize A and w the global parameter at the server.
4: For f -.,,
do: 5: The server broadcasts parameter w/ and global update A to the selected clients S..
6: For client p E St in parallel do:
7: client p initialize the local parameter wp wt.
8:
For O, K- do:
ui k = w/ + VF, (w,b"")))
9:
1VF III
u. k. aVVF ( (1 a)
w. w.t.ht1 with 1.k m 1,++1
11:
12:
End for.
13:
A w w
14:
End for
Enso, ||
15:
Aggregate A++1
15. pes, AIll Eies, 4p. Update global parameter w+11 =ww 71 At1
17:
End for.
and normalizes the sum of the local updates the global update direction. Utilizing the normalized aggregation scheme, the server can update the model with larger step and improve the convergence.
Hyperbolic Graph Fusion. In FL, the Euclidean distances between parameters[11, 1, 20] (or gradients[16, 16, 42]) between the client and the server is utilized correct the local training for alleviating the model inconsistency. However, the Euclidean distance between the model parameters can't correctly reflect the variation in functionality due to the complex structure of the modern machine learning model. The model inconsistency across the clients is still large, which impairs the performance of the FL framework. Since the most prevail machine learning model, neural network has graph structure and the hyperbolic space exhibits minimal distortion in describing data with graph structure, the client maps the representations the local and global model into the hyperbolic shallow space[29] and uses the squared Lorentzian distance[19] between the representations to measure the model inconsistency.
To eliminate the model inconsistency effectively, we adopt the hyperbolic graph fusion technique, adding the distance of representations the hyperbolic space as a regularization term to the loss function. Then the original problem (1) can be reformulated as:
min F(wo) = (if((w)) + R(up, wg)l, s.t. Wg = p p
1
Σ up
(2)
F
where R(wp, wg) is the hyperbolic graph fusion regularization term, defined as:
R(w, wg) exp (114p- L,122/o) 14, L, = 223 2( Lg)e.
(3)
In (2) and (3), Lp and Ly the mapped Lorentzian vectors corresponding representations from local model wp and global model Wy. y and a are parameters to tune the impact of the model divergence training process. 3 is the parameter of the Lorentz model and (I, y) denotes the Lorentzian scalar product defined as:
Zp and Zg. the
c
900 +
V4
(4)
where and y are d dimensional mapped Lorentzian vectors. The new problem (2) can be divided into each client and client p uses its local optimizer to solve the following sub-problem:
minF,(wp) fo(w,) + 7 R(w w³)-
(5)
5

---

The regularization term R(wp: W'y) has two benefits for local training: (1) It mitigates the local over-fitting by constraining the local representation to be closer to the global representation in the hyperbolic space (Lorentzian model); (2) It adopts representation distances in a low-dimensional hyperbolic space to measure model deviation, which can be more precisely and save computation.
1
4
0
1 cos
0
|
A
A
a cos 0
|
0 1'
4
(a) Vanilla Aggregation.
(b) Normalized Aggregation.
Figure 2: A toy schematic to compare the naive aggregation and normalized aggregation the local updates, where the number of clients is 2 and the local intervals are set as 1. The solid line indicates the client's local update Ar, 9: is the angle between the local update and the global update, and the dotted line represents the clients' contribution on the global update. The red lines the aggregated global update A. The main difference is ||A||, , the norm of the global update. When adopting the naive aggregation method, the global norm ||||| = > || cos O.. When adopting the normalized aggregation method, the global norm |||| = Σ, ||].]]. We can see that the norm of global update in the direct aggregation is less than the norm in the normalized aggregation, duc the fact that cos A < 1.
Normalized Aggregation of Local Updates. According to [3], after a number of communication rounds, the cosine similarities of the local updates across clients are almost zero, which comes from the model(gradient) inconsistency between the server and client sides. In the vanilla aggregation way, the server takes the mean of local updates from participated clients the global gradient. As shown in Figure 2(a), client i makes || || cos f; contribution on the aggregation result. When the data heterogeneous significant, the cosine similarities across the local updates are small. Correspondingly, the cosine similarities cos 0g between the clients and the server narrows down, and the global update norm shrinks which slows down the convergence. To alleviate the negative impact of near-orthogonal local updates, we propose a new normalized aggregation method to compute the global update. The direction of the global update can be acquired by normalizing the sum of the local updates and the result is the same as the one obtained by the vanilla way. For the norm, the server computes by taking the average of norms of the received local updates. As shown in Figure 2(b), with the proposed normalized aggregation method, the client i's contribution on the global update increases from || cos O, to ||A/|]. Accordingly, the norm of the global update || accelerates the convergence.
grows and
Our proposed FedMRUR is characterized by Figure and the detailed training process is summarized in Algorithm 1. Concretely, firstly, the server broadcasts the global parameter and updates it to the selected clients S.. At the initialization stage of local training, client utilizes the manifold regularization scheme to construet its own local loss function F; with the received global parameter w Then, client i adopts the Sharpness Aware Minimization (SAM) [7] optimizer to compute the gradient T. with data sampled randomly. The local updater U; consists of the stochastic gradient ag; and the momentum term (1 a'' the received global update from the last round. Client i applies to perform multiple SGD and uploads the accumulated local update A to the server. The server takes two steps to construct the global update: 1) aggregating and normalizing the accumulated local updates from the participated clients S, as the direction of the global update; 2) averaging the norms of accumulated local updates as the norm of the global update. Finally, the server utilizes the constructed global update to perform one step SGD and get new global parameter.
Remark 1. FedMRUR is the top of MoFedSAM [31] due to its excellent performance and our method can also integrated with other federated learning methods, including FedExp, FedCM, SCAFFOLD, FedDYN, etc., to improve the performance.

---

4 Convergence Analysis
In this section, provide the theoretical analysis of proposed FedMRUR for general non-convex FL. setting. Due to space limitations, the detailed proofs are placed in Appendix. Before introducing the convergence results, we first state some commonly used assumptions as follows.
Assumption 1. fp(x) is L-smooth and R(I, zo) is r-smooth with fixed vo for all client p. i.e.,
[V - VI,(D)I S a|aa- | bll: [[V xo) - V R(b, æo)il S r l|l-b
Assumption 2. The stochastic gradient of with the randomly sampled data on the local client is an unbiased estimator of VVF. (x with bounded variance, i.e.,
VF,(ot,, E $l0g
Assumption 3. The dissimilarity of the dataset among the local clients is bounded by the local and global gradients, i.e.,
E ( - [F(x))] S
(6)
Assumption guarantees the gradient Lipschitz continuity for the objective function and regularizer term. Assumption guarantees the stochastic gradient is bounded by zero mean and constant variance. Assumption 3 gives the heterogeneity bound for the non-iid dataset across clients. All the above assumptions are widely used in many classical studies [1, 43, 33, 42, 14, 16), and our convergence analysis depends them study the properties of the proposed method.
Proof sketch. To explore the essential insights of the proposed FedMRUR, we first bound the client drift over all clients within the t-th communication round. Next, we characterize the global parameter moving within a communication round, which similar to the one in centralized machine learning algorithms with momentum acceleration technology. Then, the upper bound for the global update D is provided. Lastly, we use V Fxx))ll the global gradient norm as the metric of the convergence analysis of FedMRUR. The next theorem characterizes the convergence rate for FedMRUR.
Theorem 1. Let all the assumptions hold and with partial client participation. If 1 S V30aKI. 90n(L+r) 30r( L+r
S 7g 20L(S-1) satisfying 4 KN
2(1-aL) 70(1 K2 m)
28
then for all K and T ' we have:
F
F
VF(ww))]
+ 22
Σ T
Cang Σ dl
1=1
where
1
C (10 + r) () + 350 K(L
(31 6(L +((+r))(+))) 28a 2 K - m)p
42K= a + +r)
a(L+r)Ml
(L
2KS
K2SN
+270N + 1000 5400 (1+ + (L+)
(L + +4NK20 3N -
and de = [Σ, AE convergence rate of the FedMRUR under partial client participation can be bounded as:
E.149
2 1. Specifically, we set 7g = e
YSK /T
and mp
O (STK(L+))) )
the
T
Σ
+0
K ST
+
+
(8)
SKT
VKT
F=-F= Remark 2. Compared with the inequality CongT + T of Theorem D.7 in MoFedSAM paper[31] }, the second constant term in (7) is same and the first term is less than the first term in MoFedSAM paper, which validates FedMRUR achieves faster convergence than MoFedSAM.
Remark 3. From (8), can find that when T is large enough, the dominant term O /SKT) in the bound achieves linear speedup property with respect to the number of clients. It means that to achieve e- pprecision, there are O( ) communication rounds required at least for non-convex and smoothh objective functions.
7

---

Test accuracy CIFAR-100 in the non-iid 0.3, p 0.6. 10 and 20) settings.
100
(b) Test accuracy TinylmageNet in the non-iid -id (μ 0.3, / 0.6, 40 and n = 80) settings.
Figure 3: Test accuracy w.r.t. communication rounds of our proposed method and other approaches. Fach method performs in 1600 communication rounds. To compare them fairly, the basic optimizers are trained with the same hyperparameters.
5 Experiments
In this section, we validate the effectiveness of the proposed FedMRUR algorithm using the exper- imental results CIFAR-10/100 [18] and TinyImageNet [18]. We demonstrate that FedMRUR outperforms the vanilla FL baselines under heterogeneous settings. We also present that both manifold regularization and the proposed normalized update aggregation can improve the performance of SGD in FL. The experiments of CIFAR-10 10 are placed in the Appendix.
5.1 Experimental Setup
Datasets. We compare the performance FL algorithms on CIFAR-10/100 and TinyImageNet datasets with 100 clients. The CIFAR-10 dataset consists of 50K training images and 10K testing images, All the images are with 32 x 32 resolution belonging to 10 categories. In the CIFAR- 100 dataset, there are 100 categories of images with the same format CIFAR-10. TinylmageNet includes 200 categories of 100K training images and 10K testing images, whose resolutions are 64 64. For non-iid dataset partitioning and Dirichlet-ju (abbreviated as Dir(()) sampling as [13], where the coefficient n is the number of data categories each client and z measures the heterogeneity. In the experiments, select the Dirichlet coefficient (a from (0.3. 0.6) for all datasets and set the number of categories coefficient n from {3,0 6) CIFAR-10, (10, 20) CIFAR-100 and (40, 80} TinylmageNet.
clients,
Pathological-n (abbreviated as Path(n))
Implementation Details. For all algorithms on all datasets, following [1, 42], the local and global learning rates set as 0.1 and 1.0, the learning rate decay is set as 0.998 per communication round and the weight decay is set as 5 10 44 ResNet-18 18 together with group normalization is adopted as the backbone to train the model. The clients' settings for different tasks summarized in Table 1. Other optimizer hyperparameters are as follow: P = 0.5 for SAM, a = 0.1 for client momentum, y 0.005. 10000.0 and 3 for manifold regularization.
Table
The experiments settings for different tasks.
Task
num of clients
participated ratio
batch size
local epoch
CIFAR
200
0.05
50
3
Tiny
500
0.02
20
2
Baselines. To compare the performances fairly, the random seeds are fixed. We compare the proposed FedMRUR with several competitive benchmarks: FedAvg [43], the most widely used baseline,

---

Table 2: Test accuracy (%) on CIFAR- 100& TinyImagenet datasets in both Dir(u)) and Path(z)) distributions.
CIFAR-100
Tuylmaggene
Algorithm
Dir(a))
Path(n)
Din(p))
Puth(n)
μ= 0.6
==
n
j== 0.6
== 0.3
n= 80
n
FedAvg
39.87
39.50
38.47
36.67
30.78
30.61
31.62
31.18
Fedlixp
44.51
44.26
43.58
41.00
33.49
33.65
33.39
FedProx
39.89
39.86
38.82
37.15
30.93
31.05
32.09
31.77
SCAFFOLD
47.51
46.47
46.23
42.45
37.14
36.22
37.48
35.32
FedCM
51.01
50.93
50.58
50.03
41.37
40.21
40.93
40.46
MoFedSAM
52.96
51.87
42.52
FedMRUR
55.81
55.49
55.21
53.69
45.54
45.41
45.42
45.71
Table Convergence speed comparison on CIFAR100K Tine datasets. "Ace," represents the target test accuracy on the dataset. "oo" mcans that the algorithm is unable to achieve the target accuracy on the dataset.
Datasets
CIFAR-100
TinyImageNe
Dir(p)
Path(n)
Dir(pe)
Path(n)
Algorithms
Acc.
0.6
0.3
20
10
0.6
0.3
80
40
FedAvg
494
00
972
1078
1002
1176
FedExp
715
782
795
1076
1255
1362
1327
1439
FedProx
480
488
638
1043
1030
1163
1615
SCAFFOLDD
38%
30%
FedCM
120
126
157
255
342
401
366
474
MoFedSAM
154
146
211
300
436
447
415
460
157
179
223
473
517
470
570
FedAvg
20
DC
DC
(0
00
DC
00
FedExp
985
1144
1132
1382
00
FedProx
SCAFFOLD
406
449
558
1289
1444
1206
2064
FedCM
173
193
260
527
599
735
674
879
MoFedSAM
197
192
260
392
598
624
583
685
192
230
707
653
788
FedAvg
0C
0C
00
00
00
0C
00
00
FedExp
FedProx
SCAFFOLD
45%
616
784
00
0c
0C
00
00
FedCM
276
353
470
842
1451
2173
1587
2186
MoledSAM
243
278
400
575
950
1106
959
1162
263
484
948
1050
953
firstly applies local multiple training and partial participation for FL framework: SCAFFOLD [16] utilizes the SVRG method to mitigate the client drift issue; FedProx [20] uses a proximal operator to tackle data heterogeneity; FedCM [42] incorporates the client-momentum term in local training to maintain the model consistency among clients; Based on FedCM, MoFedSAM [31] improves the generalization performance with local SAM [7] optimizer; FedExp [14] determines the server step size adaptively based on the local updates to achieve faster convergence.
5.2 Evaluation Results
Figure 3 and Table 2 demonstrate the performance of ResNet-18 trained using multiple algorithms on CIFAR-I00 and TinylmageNet datasets under four heterogencous settings. We plot the test accuracy of the algorithms for simple image classification task in the figure. We can observe that: proposed FedMRUR performs well with good stability and effectively alleviates the negative impact of the model inconsistency. Specifically, on the CIFAR 100 dataset, FedMRUR achieves 55.49% on the Dirichlet-0.3 setups, which 5.07% higher than the second-best test performance. FedMRUR effectively reduces the model inconsistency and the enlarged global update improves the speed of convergence.
Table 3 depicts the convergence speed of multiple algorithms. From [13], larger indicates less data heterogeneity across clients. We can observe that: 1) our proposed FedMRUR achieves the fastest convergence speed at most of the time, especially when the data heterogeneity is large. This

---

validates that FedMRUR can speed iteration; 2) when the statistical heterogeneity is large, the proposed FedMRUR accelerates the convergence more effectively.
5.3 Ablation Study
Impact of partial participation. Figures 4(a) and 4(b) depict the optimization performance of the proposed FedMRUR with different client participation rates on CIFAR-I00, where the dataset 0.3 and the client participation ratios are
splitting method is Dirichlet sampling with coefficient chosen from 0.02 0.2. From this figure, we can observe that the client participation rate (PR) has positive impact on the convergence speed, but the impact on test accuracy is little. Therefore, our method can work well under low PR settings especially when the communication resource is limited.
(b)
(c)
(d)
Figure 4: (a). Training loss w.r.t different client participation rations; (b). Test accuracy w.r.t different
client participation ratios. (c). Test accuracy with different 7. (d). Train loss with different 7. The
performance of FedMRUR with different parameters on the CIFAR-100 dataset.
Hyperparameters Sensitivity. In Figures 4(c) and 4(d), we compare the performance of the proposed FedMRUR with different ppper-paraeeters on the CIFAR- 100 dataset. From the results, we can see that our algorithm achieves similar test accuracy and training loss under different Y within a certain range (? [0.005, 0.5)) and this indicates the proposed FedMRUR is insensitive to the hyperparameter y. The hyperparameter y represents the method the punishment on the model bias.
Impact of Each Component. Table 4 demon-
strates the impact of each component of FedM- RUR on the test accuracy for CIFAR- 100 dataset on the Dirichlet-0.3 setups. For convenience, we abbreviate normalized aggregation as "normal- ized(N)" and hyperbolic graph fusion as "hy- perbolic(H)), respectively. From the results, can find that both the normalized local update aggregation scheme and the hyperbolic graph fusion can improve performance. This table validates
Table 4:
Test accuracy on CIFAR-106 datasets about without each ingredients of FedMRUR.
Algorithm
normalized(N))
hyperholic(1))
Acc.
MoFedSAM
$2.81
FedMRUR-N
54.27
FedMRUR-H
53.57
FeaMRUR
55.49
that our algorithm design and theoretical analysis are correct and effective.
Conclusion
In this work, propose novel and practical federated method, dubbed FedMRUR which applies the hyperbolic graph fusion technique to alleviate the model inconsistency in the local training stage and utilizes normalized updates aggregation scheme to compensate for the global norm reduction due to the near-orthogonality of the local updates. We provide the theoretical analysis to guarantee its convergence and prove that FedMRUR achieves a linear-speedup property of O <SKT)) We also conduct extensive experiments to validate the significant improvement and efficiency of our proposed FedMRUR, which is consistent with the properties of our analysis. This work inspires the FL framework design to focus on exploiting the manifold structure the learning models.
Limitations& Brouade Broader Impacts. Our work focuses on the theory of federated optimization and proposes a novel FL algorithm. During the local training, the representations of the global model must be stored locally, which may bring extra pressure on the client. This will help us in inspiration for new algorithms. Since FL has wide applications in machine learning, Internet of Things, and UAV networks, our work may be useful in these areas.
10

---

Acknowledgements. This work is supported by National Key Research and Development Program of China under SQ2021YFC333 Grant 61971457. Thanks for the support from CENI-HEFEI and Laboratory for Future Networks in University of Science and Technology of China.
and National Natural Science Foundation of China under
References
[1] Durmus Alp Emre Acar. Yue Zhao, Ramon Matas Navarro, Maithew Mattina, Paul N. Whatmough, and Venkatesh Saligrama. Federated learning based dynamic regularization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, OpenReview.net, 2021.
[2] Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný. Stefano Mazzocchi, Brendan McMahan, et al. Towards federated learning at scale: System design. Proceedings of machine learning and systems, 1:374-388, 2019.
[3] Zachary Charles, Zachary Garret, Zhouyuan Huo, Sergei Shmulyian. and Virginia Smith. On large-cohort training for federated learning. In 34th Advances in Neural Information Processing Systems, NeurIPS 2021, virtual, December 6-14, I, pages 20461-20475, 2021.
[4] Zachary Charles and Jakub Konečný. Convergence and accuracy trade-off's in federated learning and meta-learning. The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event, pages 2575 2583..253.1111 PMLR, 2021.
[5] Ashok Cutkosky and Róbert in Neural Information Processing Systems, 31, 2018.
Distributed stochastic optimization via adaptive sgd. Advances
[6] Shanshan Feng, Lisi Chen, Kaiqi Zhao, Wei Wei, Xuemeng Song, Shuo Shang, Panos Kalnis, and Ling Shao. Role: Rotated lorentzian graph embedding model for asymmetric proximity, IEEE Transactions on Knowledge and Data Engineering, 2022.
[7] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness aware minimization for efficiently improving generalization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
[8] Mikhael Gromov. Hyperbolic groups. In Essays in group theory, pages 75-263. Springer, 1987.
[9] Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi. Federated lcarning with compression: Unified analysis and sharp guarantees. In The 24th International Conference on Artificial Intelligence and Statisticx, AISTATS 2021, April 13-15, 2021, Virtual Event, Proceedings of Machine Learning Research, pages 2350-2358. PMLR, 2021.
[10] William L. Hamilton, Rex Ying, and Jure Leskovec. Representation learning graphs: Methods and applications. arXiv preprint arXiv:1709.0 170.005584 2017.
[11] Filip Hanzely and Peter Richtárik. Federated learning mixture of global and local models. arXiv preprint arXiv:2002.05516 2020.
[12] Clare Elizabeth Heinbaugh, Emilio Luz-Ricca, and Huajie Shao. Data-free one-shot federated learn- ing under very high statistical heterogeneity. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[13] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribu- tion for federated visual classification. arXiv preprint 1.Xiv:1909006333 2019.
[14] Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Fedexp: extrapolation. 11-th International Conference on Learning Represenaatinns, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
Speeding up federated averaging via
[15] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems federated lcarning. Foundations and Trends@ in Machine Learning, (4(1-)):1 1-00, 2021.
[16] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. SCAFFOLD: stochastic controlled averaging for federated learning. In Proc. 37th Int. Conf. Mach. Learn., pages 5132-5143 2020.

---

[17] Prashant Khanduri, Pranay Sharma, Haibo Yang, Mingyi Hong, Jia Liu, Ketan Rajawat, and Pramod K. Varshney. STEM: A stochastic two-sided momentum algorithm achieving near-optimal sample and communication complexities for federated learning. Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 6050-600 1500-601, 2021.
[18] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report. 2009.
[19] Mare Teva Law, Renjie Liao, Jake Snell, and Richard S. Zemel. Lorentzian distance learning for hyperbolic representations. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9- 15 June 2019, Long Beach, California, USA, Proceedings of Machine Learning Research, pages 3672-36000 -6811 2019.
[20] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization heterogeneous networks. In Proc. Mach. Learn. Sys., 2020.
[21] Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. 2023.
[22] Xin-Chun Li, Yi-Chu Xu, Shaoming Song, Bingshuai Li, Yinchuan Li, Yunfeng Shao, and De- Dc-Chuan Zhan, Federated learning with position-awe aware neurons. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, Jane 18-24, 2022, pages 10072-10081. IEEE, 2022.
[23] Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ying-Chang Liang, Qiang Yang, Dusit Niyato, and Chunyan Miao. Federated learning in mobile edge networks: A comprchensive survey. IEEE Communications Surveys & Tutorials, 22(3):2031-206
2020.
[24] Chang Liu, Chenfei Lou, Runzhong Wang, Alan Yuhan Xi. Li Shen, and Junchi Yan. Deep neural network fusion via graph matching with applications to model ensemble and federated learning. In International Conference on Machine Learning, ICMI. 2022, 17-23 July 2022, Baltimore, Maryland, USA, pages 13857-13869. PMLR, 2022.
[25] Weibo Liu, Zidong Wang. Xiaohui Liu, Nianyin Zeng. Yurong Liu, and Fuad E Alsaadi. A survey of deep neural network architectures and their applications. Neurocomputing, 234:11 134:11-26, 26, 2017.
[26] Yixing Liu, Yan Sun, Zhengtao Ding. Li Shen, Bo Liu, and Dacheng Tao. Enhance local consistency in federated learning: A multi-step inertial momentum approach. 2023.
[27] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-effcceent learning of deep networks from decentralized data. In Proceedings of the 20th International Conference Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL USA, Proceedings of Machine Learning Research, pages 1273-1282. PMLR, 2017.
[28] Dinh C Nguyen, Ming Ding, Pubudu N Pathirana, Aruna Seneviratne. Jun Li, and H Vincent Poor. Federated learning for internet things: A comprehensive survey. IEEE Communications Surveys Tutorials, 132))11622-1658, 2021.
[29] Maximilian Nickel and Douwe Kiela. Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, ICML 2018, Stockholmsmässsnn Stockholm, Sweden, July 10-15, 2018, volume 80, pages 3776-3785. PMLR, 2018
[30] Wei Peng, Tuomas Varanka, Ahdelrahman Mostafa, Henglin Shi, and Guoying Zhao. Hyperbolic deep neural networks: A survey. IEEF Trans. Pattern Anal. Mach. Intell., 44(12):10023-10044 2022.
[31] Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, pages 18250-18280, 2022.
[32] Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.
[33] Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net 2021.
12

---

[34] Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020.
[35] Tao Sun, Dongsheng Li, and Bao Wang. Decentralized federated averaging. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2022.
[36] Yan Sun, Li Shen, Shixiang Chen, Liang Ding, and Dacheng Tao. Dynamic regularized sharpness aware minimization federated learning: Approaching global consistency and smooth landscape. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, PMLR, 2023.
volume 202, pages
[37] Yan Sun, Li Shen, Hao Sun, Liang Ding, and Dacheng Tao. Efficient federated learning via local adaptive amended optimizer with lincar speedup. IEEE Transactions on Pottern Analysis and Machine Intelligence, in press:1-12. 12, 2023.
[38] Yue Tan. Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the 36th AAAI Conference on Artificial Intelligence, pages 8432-8440, 2022.
[39] Hongyi Wang, Mikhail Yurochkin, Yuckai Sun, Dimitris S. Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. 8th International Conference on Learning Representations. ICLR 2020, Addis Ababo, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[40] Jany Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated optimization. arXiv preprint (eXiv:2107.06911 2021.
[41] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the objective inconsis- tency problem in heterogencous federated optimization. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[42] Jing Xu, Sen Wang, Liwei Wang, and Andrew Chi-Chih Yao. Fedem: Federated learning with client-level momentum, arXiv preprint arXiv:2 106. 10874, 2021.
[43] Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participation in non-iid federated learning. In 9th Int. Conf. Learn. Reprexentations, 2021.
[44] Fuxun Yu, Weishan Zhang, Zhuwei Qin, Zirui Xu, Di Wang, Chenchen Liu, Zhi Tian, and Xiang Chen. Fed2: Feature-aligned federated learning. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 2074, 2021.
[45] Hao Yu, Rong Jin, and Sen Yang. On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization. In International Conference on Machine Learning, pages 7184-7193. PMLR, 2019.
[46] Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. A survey on federated learning. Knowledge- Bassed Systems, 216: 106775, 2021.
[47] Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. DENSE: data-free one-shot federated learning. In NeurlPS, 2022.
[48] Lin Zhang, Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan, Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In JEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10164-10173. IEEE, 2022.
13


Reviews:
[{"review_id": "GhrBk2aTPM", "reviewer": "Reviewer_mqsa", "summary": "This paper proposes a federated learning framework called FedMRUR to deal with the model inconsistency caused by the local data heterogeneity across clients and insufficient geometric representation ability. To do this, it adopts a hyperbolic graph manifold regularizer to ensure that the representations obtained by the local and global models are close in a low-dimensional subspace. Then it aggregates the client updates norms as the global update norm to mitigate the norm reduction introduced by the near-orthogonality of client updates. A linear speedup property is theoretically proved for the proposed algorithm.", "questions": "Please see the above part.", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The motivation to deal with the model inconsistency and insufficient representation ability is clear.\n+ It is interesting to adopt the hyperbolic graph fusion scheme in federated learning.\n+ The theoretical guarantee is provided for the convergence and especially the proposed method exhibits a linear speedup.\n+ The experimental results seem promising.", "weaknesses": "I'm not very familiar with federated learning. Here are some major concerns:\n- There are many ways to exploit the manifold structure. What's the advantage of adopting the graph fusion scheme, especially in the hyperbolic space? If the hyperbolic space is a good choice, then why not let the whole learning process work in the hyperbolic space?\n- Figure 2 is less informative since the readers cannot understand how the normalized aggregation works and what's the difference between these two operations in terms of their workflow.\n- It seems that the widely used CIFAR-10 benchmark is not involved for evaluation.\n- The experiments to validate the linear speedup property are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments which helps us improve the paper. We have prepared our responses to each of your questions.\n- Weakness:  \n    **- W1: There are many ways to exploit the manifold structure. What's the advantage of adopting the graph fusion scheme, especially in the hyperbolic space? If the hyperbolic space is a good choice, then why not let the whole learning process work in the hyperbolic space?** \n    - A1：For the answer, please refer to Q1 in the Author Rebuttal. \n\n    **- W2: Figure 2 is less informative since the readers cannot understand how the normalized aggregation works and what's the difference between these two operations in terms of their workflow.**\n    - A2: Sorry for the unclear presentation of Figure 2. Figure 2 is a toy schematic to compare the naive aggregation and normalized aggregation of the local updates. The main difference is the norm $\\lVert \\triangle \\rVert$ of the global update, which is the aggregation of the local updates $\\triangle_p$ from the participated clients. When adopting the naive aggregation method, the server directly takes the average of the local updates as the global update. Each client's contribution on the global update is $\\lVert \\triangle_p \\rVert \\cos \\theta_p$, where $\\theta_p$ is the angle between the local update $\\triangle_p$ and the global update $\\triangle$. When the data heterogeneous is significant, the divergence among the directions of local updates is large. Correspondingly, the divergence between the directions of each local update and the global update is large, and the global update norm shrinks which slows down the convergence of the global objective function. When adopting the normalized aggregation method, the server computes the direction of the global update by normalizing the sum of local updates, which is the same as the naive method. The global update norm is obtained by averaging the local updates. In this way, the client $p$'s contribution on the global update $\\theta$ grows from $\\lVert \\triangle_p \\rVert \\cos \\theta_p$ to $\\lVert \\triangle_p \\rVert$. Accordingly, the norm of the global update grows and the speedup the convergence. In order to compare the two aggregation methods, we replace Figure 2 with a new one (Figure 2 in attached pdf).\n\n    **- W3: It seems that the widely used CIFAR-10 benchmark is not involved for evaluation.**\n    - A3: We place more challenged results on CIFAR100 and TinyImageNet in the amin part of the paper. The results on CIFAR-10 are presented in Figure 1, Table 1 and 2 of the Supplementary Material. From these results, FedMRUR achieves the fastest convergence and the highest test accuracy. \n\n    **- W4: The experiments to validate the linear speedup property are missing.**\n    - A4: Thank you for the reminder about validating the linear speedup property. Because the whole dataset is fixed, increasing the number of clients changes the amount of data in the local data, which changes the entire optimization problem, we conduct the experiment under various settings of local intervals $K$ fixing the number of clients to verify the linear speedup property. From Figure 3 in the attached pdf, when $K$ increases to $8$, the algorithm performs $2.0\\times$ faster than $K=4$. When $K$ is increased to $\\mathcal{O} (ST)^{\\frac{1}{2}}$, the impact of the second term of (7) in Theorem 1 becomes greater and the first term becomes less. When $K$ increase from 8 to 16, the speedup of convergence is not obvious.    "}, {"review_id": "caWV6IRrSG", "reviewer": "Reviewer_HiX9", "summary": "The authors found the existing vanilla distillation in FL, the model inconsistency caused by the local data heterogeneity across clients results in the near-orthogonality of client updates, which leads to the global update norm reduction and slows down the convergence. Moreover, the authors argue previous works may fail to reflect the model inconsistency due to the complex structure of the machine learning model and the Euclidean space’s limitation in meaningful geometric representations. To resolve the above issues, they propose the FedMRUR algorithm for FL. By adopting a hyperbolic graph manifold regularizer and aggregating the client updates norms as the global update norm, the FedMRUR achieves a new state-of-the-art (SOTA) accuracy with less communication.", "questions": "1) Some notations are confusing. For example, in line 14 of Algorithm 1, what does the notation “$\\delta_i^{t}$” mean? \n2) In line 263, what should the setting of the Dirichlet coefficient be?\n3) The authors should discuss the effect of local interval $K$ on the performance of the algorithm.\n", "limitations": "Please find the weaknesses and limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "originality: Exploiting the manifold structures of the representations can reflect the model bias better than the parameters (or gradients) method, to significantly reduce the model inconsistency. Aggregating the client updates norms as the global update norm can mitigate the norm reduction caused by model inconsistency.\n\nquality: The paper is well written, with detailed experiments and ablation studies with other methods and the proposed variants.\n\nclarity: The paper consists of an illustration of the workflow and text explanations of the proposed FedMRUR algorithm.\n\nsignificance: The paper solves the problem of the model inconsistency across the clients in FL, reduces the model bias more effectively than its baseline, mitigates the norm reduction caused by model inconsistency, and improves the test performance.\n", "weaknesses": "- There are some unclear points and confusing notations.\n- More experiments can be conducted.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments which helps us improve the paper. We have prepared our responses to each of your questions.\n- Weakness   \n    **- W1: There are some unclear points and confusing notations.**\n    - A1: Thank you for the careful readings. We check the whole paper and correct all the typos. We eliminated the unclear points and confusing notations. We unify the notations in Figure 1, Algorithm 1 and Section 3.2. In Figure 1, we replace $w_0$, $F_0$ with $w_g$, $F_g$ to denote the global parameter and function. Please refer to Figure 1 in the attached pdf. In Algorithm 1, the local gradient is denoted by $\\nabla F_p(\\cdot)$, where $F_p(\\cdot)$ is the local loss function formulated by (5) in Section 3.2. We replace Figure 2 with a new figure (Figure 2 in the attached pdf) to present the \"Normalized Aggregation of Local Updates\" more clearly. We also fix some typos in the proof. The convergence rate is $O \\left(\\frac{1}{\\sqrt{SKT}}\\right) + O \\left(\\frac{\\sqrt{K}}{{ST}}\\right) + O \\left(\\frac{1}{\\sqrt{K}T}\\right)$ in Theorem 1. In this way, we clearly demonstrate the two components of FedMRUR: \"hyperbolic graph fusion\" and \"normalized aggregation of local updates\" and validate the effectiveness of the algorithm.   \n    \n    **- W2: More experiments can be conducted.**\n    - A2: Thank you for the advice about adding experiments. \n    We conduct the experiment on discussing the effect of local interval $K$ and validate the linear speedup property of FedMRUR. The results can be seen from the Figure 3 in the attached pdf. From the figure, when $K$ increases to $8$, the algorithm performs $2.0\\times$ faster than $K=4$. When $K$ is increased to $\\mathcal{O} (ST)^{\\frac{1}{2}}$, the impact of the second dominated term of (7) in Theorem 1 becomes greater and the first term becomes less. When $K$ increase from 8 to 16, the speedup of convergence is not obvious.    \n    We also extend the range of $\\gamma$ to study its hyper-parameter sensitivity by experiments.\n\n        | $\\gamma=0.001$ | $\\gamma=0.0005$ | $\\gamma=0.0002$ | $\\gamma=0.0001$ | $\\gamma=0.00005$ | $\\gamma=0.00002$ |     \n        | :------------: | :-------------: | :-------------: | :-------------: | :--------------: | :--------------: |                   \n        |     63.35      |      65.95      |       67.75     |       68.54     |      68.96       |      68.45       |\n\n        Form this table,  we find that $\\gamma$ has great impacts on the performance. From (3) and (5), a high $\\gamma$ slows down the local training process; a low gamma value can't support the regularization term to constrain model inconsistency. When $\\gamma=0.00005$, the algorithm achieves the highest test accuracy.   \n        To study the impact of $\\beta$ for hyperbolic graph manifold regularization on the performance, we conduct the experiment on CIFAR100 task with different $\\beta$ settings and present the final test accuracy as follows. \n\n        | $\\beta=0.1$ | $\\beta=0.5$ | $\\beta=1.0$ | $\\beta=5.0$ | $\\beta=10.0$ |    \n        | :---------: | :---------: | :------:    | :------:    | :-------:    |                  \n        |    68.67    |    69.03    |   68.96     |   68.26     |     69.10    |\n\n        From this table, we can find that $\\beta$ has a limited impact on the final performance of the algorithm.\n- Questions  \n    **- Q1: Some notations are confusing. For example, in line 14 of Algorithm 1, what does the notation “$\\delta_i^t$” mean?** \n    - A1: Sorry about the unclear presentation. \"$\\delta_i^t$\" means the accumulated update of the local parameter at client $i$ within $t-$ th round (i.e., $w_i^{t,K} - w_i^{t,0}$). We have fix this issue in Algorithm 1.\n    \n    **- Q2: In line 263, what should the setting of the Dirichlet coefficient be ?**\n    - A2: Thank you for the careful Experiments readings. The setting of the Dirichlet coefficient should be $\\{0.3, 0.6\\}$.\n    \n    **- Q3: The authors should discuss the effect of local interval $K$ on the performance of the algorithm.**\n    - A3: Thank you for the advice about experiments. We conduct the experiment with different local interval $K$ on CIFAR-100 dataset and plot the test accuracy in communication rounds. In a certain range, enlarging $K$ can accelerate the convergence and improve the performance. When $K$ is out of the range, larger $K$ means more updates on the local dataset, which forces the local parameter $w_{p}^{t,k}$ closed to local optimum $x_{p}^*$. This causes severe client drifts that degrade the performance. In addition, since the number of local datasets changes as the number of clients grows, we also test the linear speedup property by changing $K$. From the figure, when $K$ increases to $8$, the algorithm performs $2.0\\times$ faster than $K=4$. When $K$ is increased to $\\mathcal{O} (ST)^{\\frac{1}{2}}$, the impact of the second dominated term of (7) in Theorem 1 becomes greater and the first term becomes less. When $K$ increase from 8 to 16, the speedup of convergence is not obvious.  \n"}, {"review_id": "nzmFHAYMLu", "reviewer": "Reviewer_aqu4", "summary": "This paper studies the problem of model inconsistency across clients in federated learning (FL). The authors propose a method called FedMRUR, which uses a hyperbolic graph manifold regularization term and a normalized update aggregation scheme to alleviate the issues introduced by model inconsistency. Compared with the previous works, the proposed FedMRUR can reflect the model bias better in a low-dimensional subspace and mitigate the norm reduction of global updates caused by model inconsistency. The authors prove the convergence of FedMRUR for nonconvex objectives under partial client participation. They also run experiments to show that FedMRUR can achieve a new state-of-the-art (SOTA) accuracy with less communication.", "questions": "1.\tIt is not clear what is the “manifold structure of their representations”.\n\n2.\tWhat does it mean by “norm reduction” and why it is important?\n\n3.\tFor the non-i.i.d pathological-n setting on TinyImageNet, line 264 conflicts with Figure 3(b).\n\n4.\tDoes the setting of the parameter $\\beta$ for hyperbolic graph manifold regularization have an impact on the performance of the algorithm? \n\n5.\tIn the ablation study, the author should discuss the hyperparameters sensitivity of $\\gamma$ in a wider range.\n", "limitations": "Please refer to the Questions.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe work is written clearly and intuitively.\n\n2.\tThe considered problem is quite meaningful as the model inconsistency severely impairs the performance of FL and building an algorithm to solve this issue is a promising direction to improve FL.\n\n3.\tThe theoretical result is rigorous for non-convex settings under partial client participation.\n\n4.\tThere are experiments to support the theory as well as to show that the algorithm will be useful in practice.\n", "weaknesses": "See questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments which helps us improve the paper. We have prepared our responses to each of your questions.\n- Q1: It is not clear what is the “manifold structure of their representations”.\n- A1: In our paper, we design the hyperbolic graph fusion scheme to mitigate the model inconsistency among the clients. In this scheme, we map the data representations of the clients to the vectors of Lorentz model in hyperbolic space. The induced distance of the Lorentz model is used to measure model inconsistency in our method and taken as the regularization term in the objective function to constrain the model inconsistency for improving the performance of FL. Since that hyperbolic geometry is a Riemannian manifold with a constant negative curvature, the typical geometric property of hyperbolic space is that its volume increases exponentially in proportion to its radius, whereas the Euclidean space grows polynomially. The hyperbolic space has 2 benefits enabling it to well deal with the complex real-world FL applications. \n    1. Hyperbolic space exhibits minimal distortion and fits the hierarchies particularly well since the space closely matches the growth rate of tree-like data while the Euclidean space cannot.\n    2. Even though with a low-embedding dimension space, hyperbolic models are surprisingly able to produce high-quality representation, which makes it to be especially favorable in low-memory and low-storage scenarios.  \n\n    In realistic scenarios, there extensively exists many tree-like data structure, such as the hypernym structure in natural languages, the subordinate structure of entities in the knowledge graph, the organizational structure for financial fraud, and the power-law distribution in recommender systems. In FL, clients and the server logically also have a tree-like hierarchical topological relationship. \n    Therefore, adopting the Lorentz metric of hyperbolic space makes use of the hierarchical information in the datasets and the hierarchical relationship between the server an clients, which are helpful to group data samples further bring prediction gains.\n    We also conducts experiments using different geometric spaces over several seeds and the results are as follow. \n\n    |  Space    | Euclidean | Hyperbolic |  \n    | :-------: | :-------: | :--------: |  \n    | Test Acc. |    67.46(0.32)     |    68.99 (0.34)      |   \n\n    From this results, we can find that using the Lortenz metric of hyperbolic space can help the algorithm achieving the highest test accuracy. With the use of \"manifold structure of their representations\" in hyperbolic space, FedMRUR improves the performance of FL.\n- Q2: What does it mean by “norm reduction” and why it is important?\n- A2: In FL framework, because of the data heterogeneous among the clients, there exits great divergences among the gradients of the local loss function and the similarity between the local updates are low during FL training. From the perspective of geometry, the low similarity means the vectors are almost orthogonal (i.e. the $\\cos \\theta \\approx 0$, where $\\theta$ is the angle between the vectors). When adopting the naive aggregation method, the server takes an average of many  vectors as the global update, whose norms will be very small. Then the global update becomes small, which leads to slow convergence and diminishing returns. We also conduct experiments in Ablation Study to vertify that compensating the \"norm reduction\" can imporve the performance of the algorithm. Therefore, it is critical to compensate the \"norm reduction\" in FL.\n- Q3: For the non-i.i.d pathological-n setting on TinyImageNet, line 264 conflicts with Figure 3(b).\n- A3: Thank you for the careful reading. For the non-i.i.d pathological-n setting on TinyImageNet, the coefficient $n$ should be selected from $\\{40 ,80\\}$, we correct this typo in the paper.\n- Q4: Does the setting of the parameter $\\beta$ for hyperbolic graph manifold regularization have an impact on the performance of the algorithm? \n- A4: Thank you for the suggestion to study the effects of parameter $\\beta$ on the performance. We conduct the experiemnt with various settings of $\\beta$. \nHere, we provide the test accuracy after the $1600-$ th communication round.  \n    | $\\beta=0.1$ | $\\beta=0.5$ | $\\beta=1.0$ | $\\beta=5.0$ | $\\beta=10.0$ |    \n    | :---------: | :---------: | :------:    | :------:    | :-------:    |                  \n    |    68.67    |    69.03    |   68.96     |   68.26     |     69.10    |   \n\n   From this table, we can find that $\\beta$ has a limited impact on the performance of the algorithm.\n- Q5: In the ablation study, the author should discuss the hyperparameters sensitivity of $\\gamma$ in a wider range.\n- A5: Thank you for the suggestion about investigating the impact of $\\gamma$ on the perofrmance. We conduct the experiemnt with various settings of $\\gamma$. \nHere, we provide the test accuracy after the $1600-$ th communication round.  \n    | $\\gamma=0.001$ | $\\gamma=0.0005$ | $\\gamma=0.0002$ | $\\gamma=0.0001$ | $\\gamma=0.00005$ |    \n    | :------------: | :-------------: | :-------------: | :-------------: | :--------------: |                  \n    |     63.35      |      65.95      |       67.75     |       68.54     |      68.96       |   \n\n    From this table,  we find that $\\gamma$ has great impacts on the performance. From (3) and (5), a high $\\gamma$ slows down the local training process; a low gamma value can't support the regularization term to constrain model inconsistency. When $\\gamma=0.00005$, the algorithm achieves the highest test accuracy.\n"}, {"review_id": "7THoMznFGU", "reviewer": "Reviewer_jAXP", "summary": "The authors propose a novel Federated Learning Algorithm, FedMRUR that uses hyperbolic graph diffusion to reduce the effect of data heterogeneity and thereby model inconsistencies. The authors also propose a normalized aggregation scheme to achieve faster convergence. The algorithm FedMRUR achieves state-of-the-art performance on standard datasets.\n\n", "questions": "1. The authors should clarify notation in Figure 1 and Algorithm 1. In Figure 1, the global parameter is subscripted, w_0, but superscripted in the algorithm.\n2. The algorithm can be hard to follow as it:\n  a. Did not specify the global parameter \\eta_g in the parameters\n  b. Did not specify the gradients, g_{i}^{t,k}, though that is easily induced\n  c. More importantly did not specify how \\tilde{g}_{i}^{t,k} is derived. This is mentioned later in line 213, but would make the presentation clearer by linking it directly to the equations\n  d. They can also specify the normalized aggregation more clearly, and tie to the algorithm 1.\n3. The authors should discuss normalized aggregation in more depth, as it seems to contribute more than hyperbolic graph fusion (Table 3). In fact, it would be interesting to understand the effect of this operator in other baselines.\n4. In terms of comparisons to baselines, FedMRUR seems to outperform in terms of accuracy for various Dir(\\mu) settings versus the baselines. The closest competitor seems to be MoFedSAM. First, it would be useful to have test accuracy and test accuracy std to enable better comparison between comparably performing models. Second, how was Table 1 derived, as the accuracy numbers are quite different and higher than the comparable Table 5 in the MoFedSAM paper for CIFAR-100 with Dir().\n5. Similarly, the convergence table 2 seem different for MoFedSAM and FedAvg vs Table 5 in the MoFedSAM paper. Is this a setup difference?\n", "limitations": "The authors have addressed some limitations of their method", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "In terms of comparisons to baselines, FedMRUR seems to outperform in terms of accuracy for various Dir(\\mu) settings versus the baselines. The closest competitor seems to be MoFedSAM.", "weaknesses": "Finally, while convergence speed is quicker for FedMRUR, Table 3 in the Supplemental Materials actually has FedCM to have the quickest total time to achieve 60% test accuracy on CIFAR-100. Do the authors have comments or suggestions on when to use FedMRUR vs FedCM given the disparity between convergence speed and clock times?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments which helps us improve the paper. We have prepared our responses to each of your questions.  \n\n**- Weaknesses: Finally, while convergence speed is quicker for FedMRUR, Table 3 in the Supplemental Materials actually has FedCM to have the quickest total time to achieve 60% test accuracy on CIFAR-100. Do the authors have comments or suggestions on when to use FedMRUR vs FedCM given the disparity between convergence speed and clock times?** \n- A(Weaknesses): The FedCM achieves the fastest total time is due to the less time consumption compared with FedMRUR. Because of the operations of SAM and Hyperbolic Graph Fusion scheme need a lot of computation, FedMRUR takes more time to compute in one round. For FL, academia and industry are more interested in test accuracy vs communication rounds. Under the setting of FL, the connections between devices are often through Wi-Fi or cellular network, their bandwidth is limited, and local devices are often redundant. It is common to reduce the number of communication rounds by adding local updates. In this paper to be fair, we compare test accuracy vs communication rounds with the same setting of local epochs. From the perspective of communication rounds, FedMRUR achieves the fastest convergence and the highest test accuracy; From the perspective of wall clock time, FedMRUR lags behind FedCM at low accuracy, but at higher accuracy, FedCM will gradually lag behind FedMRUR.\n\n**- Q1: The authors should clarify notation in Figure 1 and Algorithm 1. In Figure 1, the global parameter is subscripted, w_0, but superscripted in the algorithm.**   \n- A1: Thank you for the careful reading. To avoid confusion, we replace '$w_0$' with $w_g$ to denote the global parameter. Please refer to Figure 1 in the attached pdf.\n\n**- Q2: The algorithm can be hard to follow as it: a. Did not specify the global parameter \\eta_g in the parameters b. Did not specify the gradients, g_{i}^{t,k}, though that is easily induced c. More importantly did not specify how \\tilde{g}_{i}^{t,k} is derived. This is mentioned later in line 213, but would make the presentation clearer by linking it directly to the equations d. They can also specify the normalized aggregation more clearly, and tie to the algorithm 1.**    \n- A2: For the answer, please refer to Q1 in the Author Rebuttal.\n\n**- Q3: The authors should discuss normalized aggregation in more depth, as it seems to contribute more than hyperbolic graph fusion (Table 3). In fact, it would be interesting to understand the effect of this operator in other baselines.**    \n- A3: For the answer, please refer to Q2 in the Author Rebuttal.\n\n**- Q4: In terms of comparisons to baselines, FedMRUR seems to outperform in terms of accuracy for various Dir(\\mu) settings versus the baselines. The closest competitor seems to be MoFedSAM. First, it would be useful to have test accuracy and test accuracy std to enable better comparison between comparably performing models. Second, how was Table 1 derived, as the accuracy numbers are quite different and higher than the comparable Table 5 in the MoFedSAM paper for CIFAR-100 with Dir().** \n- A4: Thank you for the advice and the careful Experiments reading.   \nFirst, we add some experiments to compare our method FedMRUR and MoFedSAM. The results on CIFAR100 dataset are as follow.\n\n    | Algorithm | $\\mu=0.6$ | $\\mu=0.3$ | $n=20$ | $n=10$ |  \n    | :-------: | :-------: | :-------: | :----: | :----: |  \n    | MoFedSAM  |67.51(0.37)|66.81(0.36)|64.23(0.34)| 61.25(0.40)|  \n    | FedMRUR   |69.23(0.39)|68.99(0.35)|66.72(0.31)| 63.50(0.42)|  \n\nIn this table, we record the averaged test accuracy and the std at the 1600-th communication round over 8 seeds. From these results, we can clearly see that FedMRUR can achieves better performance than MoFedSAM. \nSecond, since the authors of MoFedSAM paper don't have the open source code, we obtain the baseline based on our own implementation of the paper, and we reproduce their code as well as possible for a fair comparison. As for the experimental setup, our paper sets the customer participation rate to 0.1 and the number of local epochs to 5, while the MoFedSAM paper sets the customer participation rate to 0.2 and the number of local epochs to 10. In addition, when replicating the MoFedSAM algorithm, we employ the ASAM[1] optimizer to compute the gradient of the local loss function.  \n\n**- Q5: Similarly, the convergence table 2 seem different for MoFedSAM and FedAvg vs Table 5 in the MoFedSAM paper. Is this a setup difference?** \n- A5: Thank you for the carefully reading the experiments. The difference between Table 2 in our paper and Table 5 in the MoFedSAM paper comes from the different setup in these two papers. In the MoFedSAM paper, the client participation ratio is $0.2$ and the local interval $K$ is $10$; In our paper, the client participation ration is $0.1$ and the local interval $K$ is $5$. When the data heterogeneity is not severe (Dirichlet 0.6), more participating clients $\\lVert S \\rVert$ and more local intervals $K$ results in a faster convergence; when the data heterogeneity is severe (Dirichlet 0.3) and larger local interval $K$ forces the local parameter to close to the local optimum, which cause severe client drifts and degrade the performance.  \n\nReference:\n1. Kwon, et.al \"ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks\", International Conference on Machine Learning, PMLR, 2021."}, {"review_id": "s2eECZDR7L", "reviewer": "Reviewer_EAwW", "summary": "The authors proposed a new algorithm for federated learning based on a Lorentzian regularization. The proposed algorithm achieves desired sub-linear convergence with linear speed-up. Numerical experiment shows the efficacy of the proposed algorithm over existing federated learning algorithms", "questions": "Major:\n    \n1. The presentation of the paper is confusing: First, I cannot link the proposed Algorithm 1 with all the argument in Section 3.2. More specifically, how is the “mapped Lorentzian vectors” in (3) used in Algorithm 1? The authors claimed that the regularized problem (5) has a number of desired property but is Algorithm 1 solving this regularized problem? The argument in this section seems to be ending at nowhere; Second, I really cannot understand the usages of “hyperbolic” and “Lorentzian”. The hyperbolic space is defined as space with constant -1 curvature in mathematics, but here we are not dealing with curvatures at all. As for the Lorentzian regularizer defined in (4) and (5) (which again I don’t understand how it is used in Algorithm 1), why do we need Lorentz metric? What would happen if we just use Euclidean metric, or some Riemannian metric defined by a positive definite matrix? The entire section 3.2 is confusing. On the other hand, all the described operations in Section 3.2 are not well-reflected in Algorithm 1. For example, what’s $g_{i}^{t,k}$ in Algorithm 1?\n    \n2. As the authors claimed, the algorithm is built upon MoFedSAM[1]. How is the rate of convergence (in theory) of Algorithm 1 compared with existing works, especially MoFedSAM? If the theoretical rate is not improved, I would argue that this refined engineering over MoFedSAM might not be very interesting since (in my perspective) the only difference of Algorithm 1 with MoFedSAM is the $\\Delta_i$ which serve as a local update corrector.\n    \n3. I cannot understand what point Figure 2 try to make. To me Figure 2(b) is a rotation operation, not projection, also I don’t understand how is the operation used in Algorithm 1.\n\n\nMinor:\n1. In the abstract, the authors include a formula. I think it’s better to either explain the meaning of each of the variable in the formula, or just use words instead;\n\n2. What does the name “FedMRUR” exactly stand for?\n\n3. Typos: Line 227, Assumption 1, missed one $\\nabla$; Line 263 “Dirichlet… from {0.30.6}” missed a comma?\n\n\nReferences:\n[1] Qu, Zhe, et al. \"Generalized federated learning via sharpness aware minimization.\" International Conference on Machine Learning. PMLR, 2022.", "limitations": "The limitation is well stated in weakness and question sections.\n\nI’m not aware of any potential negative social impact of this work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The proposed method is theoretically well-rounded with convergence guarantee for smooth non convex situation, which is the most commonly setting in the literature.", "weaknesses": "(Please respond to the Questions section directly) The presentation of the paper is confusing. Also some technical details are not well-illustrated. The theory part of the paper is not clearly presented and seems to be abusing terminologies a lot.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments which helps us improve the paper. We have prepared our responses to each of your questions.\n\n**- Q1(major):The presentation of the paper is confusing: \n    - I cannot link the proposed Algorithm 1 with all the argument in Section 3.2. More specifically, how is the “mapped Lorentzian vectors” in (3) used in Algorithm 1? The authors claimed that the regularized problem (5) has a number of desired property but is Algorithm 1 solving this regularized problem? The argument in this section seems to be ending at nowhere; \n    - I really cannot understand the usages of “hyperbolic” and “Lorentzian”. The hyperbolic space is defined as space with constant -1 curvature in mathematics, but here we are not dealing with curvatures at all. As for the Lorentzian regularizer defined in (4) and (5) (which again I don’t understand how it is used in Algorithm 1), why do we need Lorentz metric? What would happen if we just use Euclidean metric, or some Riemann metric defined by a positive definite matrix? The entire section 3.2 is confusing. \n    - On the other hand, all the described operations in Section 3.2 are not well-reflected in Algorithm 1. For example, what’s $g_{i}^{t,k}$ in Algorithm 1?**   \n- A1(major): For the answer, please refer to Q1 in the Author Rebuttal.\n\n**- Q2(major): As the authors claimed, the algorithm is built upon MoFedSAM. How is the rate of convergence (in theory) of Algorithm 1 compared with existing works, especially MoFedSAM? If the theoretical rate is not improved, I would argue that this refined engineering over MoFedSAM might not be very interesting since (in my perspective) the only difference of Algorithm 1 with MoFedSAM is the $\\triangle_i$ which serve as a local update corrector.**  \n- A2(major): For the answer, please refer to Q2 in  the Author Rebuttal.\n\n**- Q3(major): I cannot understand what point Figure 2 try to make. To me Figure 2(b) is a rotation operation, not projection, also I don’t understand how is the operation used in Algorithm 1.**  \n- A3(major): Sorry for the unclear presentation of Figure 2. Figure 2 is a toy schema to compare the naive aggregation and normalized aggregation of the local updates. The main difference is the norm $\\lVert \\triangle \\rVert$ of the global update, which is the aggregation of the local updates $\\triangle_p$ from the participating clients. When adopting the naive aggregation method, the server directly takes the average of the local updates as the global update. Each client's contribution on the global update is $\\lVert \\triangle_p \\rVert \\cos \\theta_p$, where $\\theta_p$ is the angle between the local update $\\triangle_p$ and the global update $\\triangle$. When the data heterogeneity is significant, the divergence between the directions of local updates is large. Correspondingly, the divergence between the directions of each local update and the global update is large, and the global update norm shrinks which slows down the convergence of the global objective function. When adopting the normalized aggregation method, the server computes the direction of the global update by normalizing the sum of local updates, which is the same as the naive method. The global update norm is obtained by averaging the local updates. The normalized aggregation way is depicted in line 14 of Algorithm 1. In this way, the client $p$'s contribution on the global update $\\theta$ grows from $\\lVert \\triangle_p \\rVert \\cos \\theta_p$ to $\\lVert \\triangle_p \\rVert$. Accordingly, the norm of the global update grows and the speedup the convergence. In order to compare the two aggregation methods, we repalce Figure 2 with a new one (Figure 2 in the attached pdf).\n\n**- Q1(minor): In the abstract, the authors include a formula. I think it’s better to either explain the meaning of each of the variable in the formula, or just use words instead;**  \nA1(minor): Thank you for the advice about the abstract. We explain the meaning of variables $S$ (the number of participated clients in one communication round), $K$ (the local interval), $T$(the total number of communication rounds) in the formula. \n\n**- Q2(minor): What does the name “FedMRUR” exactly stand for?**   \nA2(minor): Thanks for the reminder. In the revision of the paper, we include the full name where FedMRUR first appears. The name \"FedMRUR\" stands for **Fed**erated Learning with **M**anifold **R**egularization and Normalized **U**pdate **R**eaggregation. In the name \"FedMRUR\", we want to emphasize two main components in our algorithm: the manifold model fusion scheme using the hyperbolic model to constrain the divergence between the local and global model and the normalized client updates aggregation way applied at the server to mitigate the norm reduction of the global update.\n\n**- Q3(minor): Typos: Line 227, Assumption 1, missed one $\\nabla$; Line 263 “Dirichlet… from {0.30.6}” missed a comma?**  \nA3(minor): Thank you for the careful proofreading. We add the missed $\\nabla$ in the Assumption 1 at Line 215 (the new draft). At Line 252, we add the missed comma and the sentence is \"Dirichlet coefficient $\\mu$ from {0.3 , 0.6}\".   \n"}]

Author rebuttal:
**Thank you for the constructive comments which helps us improve the paper. We have prepared our responses to the common questions.** 

**-Q1: Problem on hyperbolic space and presentation of algorithm. (EAwW;jAXP;mqsa)**  
- A1:Sorry for our unclear presentation, we describe our method more clearly as follows:  
    - The "mapped Lorentzian vectors" $L$ of (3) are the model representation $Z$ mapped in the hyperbolic space. In our method, we take the distance $R(w_p,w_g)$ of the mapped Lorentzian vectors in hyperbolic space defined by (4) and (5)  to measure the bias between model $w_p$ and $w_g$. This distance is added to the original loss function $f(\cdot)$ as a regularization term to constrain the model inconsistency between the client and server side. Algorithm 1 solves the new regularized $F(\cdot)$. To illustrate the connection between Algorithm 1 and Section 3.2 clearly, we fix the typos in Algorithm 1.
    - Since hyperbolic geometry is a Riemann manifold with a constant negative curvature, its typical geometric property is that the volume grows exponentially with its radius, whereas the Euclidean space grows polynomially.
    Such a geometric trait has 2 advantages:
        1. The hyperbolic space exhibits minimal distortion and it fits the hierarchies particularly well since the space closely matches the growth rate of tree-like data while the Euclidean space doesn't.   
        2. Even with a low-embedding dimensional space, hyperbolic models are surprisingly able to produce a high quality representation, which makes them to be particularly advantageous in low-memory and low-storage scenarios.

        In realistic scenarios, there exists many tree-like data structure, such as the hypernym structure in NLP, the subordinate structure of entities in the knowledge graph and the power-law distribution in recommender systems. In FL, clients and the server logically also have a tree-like hierarchical topological relationship, so adopting the Lorentz metric of hyperbolic space makes use of the hierarchical information in the datasets and the hierarchical relationship between the server and clients, which are helpful to group data samples further bring prediction gains. Using Euclidean metric, or some Riemann metric defined by a positive definite matrix is an interesting idea. Here, we show the results of experiments using different geometric spaces as follow.  

        |  Space    | Euclidean | Hyperbolic |  
        | :-------: | :-------: | :--------: |  
        | Test Acc. |    67.46(0.32)     |    68.99(0.35)        |   

        From these results, we can find that Lortenz metric of hyperbolic space can help the algorithm achieving the highest test accuracy.  The reason is that the traits of hyperbolic space make good use of the hierarchical information in the datasets, which is helpful to group data samples and further bring prediction gains. 

        The representations generated by the model have fewer dimensions than the data. Mapping the representations to the hyperbolic space introduces less computation overhead than mapping the data. We also conduct experiments mapping the original data to the hyperbolic space over 8 seeds. The results are as follow.  

        | Original data | Representations |   
        |     :---------:   |   :---------:   |                  
        | 69.05(0.57) |   68.99(0.35)   |  
            
        From the table, we can see that both methods achieve similar performance.Thus, considering the computation overhead and performance, we only map representations to hyperbolic space and do not treat the entire learning process in hyperbolic space.
    - Section 3.2 is the basis of Algorithm 1. The first component is **Hyperbolic Graph Fusion**. With this scheme, we obtain a new manifold regularization term $R(w_p, w_g)$ to constrain the model inconsistency and formulate the new objective function $F(\cdot)$ with the regularization term. All the gradient computations are about $F(\cdot)$, such as '$g_{i}^{t,k}$'. To depict our method more clearly, we replace '$g_{i}^{t,k}$' with '$\nabla F(\cdot)$' to denote the gradient.  
    The second component is **Normalized Aggregation of Local Updates**, a new global optimizer to compensate for the global update norm reduction, as described in line 14 of Algorithm 1.

**- Q2: Problem on Normalized Aggregation (EAwW;jAXP)**     
- A2: Thank you for the comment on "Normalized Aggregation of Local Updates" from the perspective of theoretical rate.
We check the mathematical part and correct some typos in the proof of Lemma 8 and Theorem 9. The main dominated term of convergence rate is ${(F^0 - F^*)}/{(C \alpha \eta_g \sum_{t=1}^{T} d_t)} + \Phi$ in Theorem 9 (new version), where $d_t = {\sum_{i \in S_t}\Vert \triangle_i^t \Vert }/{\Vert \sum_{i \in S_t} \triangle_i^t \Vert}$ is introduced by "Normalized Aggregation of Local Updates". Compared with the main dominant term in convergence rate of MoFedSAM (${(F^0 - F^*)}/{(C \alpha \eta_g T)}$, in Theorem D.7 of the paper), the convergence rate of FedMRUR is faster than MoFedSAM. Because $\forall t, d_t \ge 1$ derived from the triangle inequality. Therefore, From the theoretical view, we can conclude that the "Normalized Aggregation of Local Updates" can accelerate the convergence. In fact, using this operator in other baselines can also improve the performance. Here, we present the effect of the normalized aggregation method applied to FedCM.  
  | Algorithm | $\mu=0.6$ | $\mu=0.3$ | $n=20$ | $n=10$ |  
    |:---------:|:---------:|:---------:|:------:|:------:|  
    | FedCM+    |  66.23(0.45)   |   65.23(0.48)   |  61.97(0.43) |  59.27(0.42)  |  
    | FedCM     |   64.87(0.42)   |   63.50(0.44)   |  60.58(0.39) |  57.56(0.36)  |  
    | FedAvg+   |  55.24(0.43)   |   53.05(0.41)   |  46.89(0.38) |  42.14(0.37) |  
    | FedAvg    |  53.85(0.38)   |   51.20(0.37)   |  45.70(0.32) |  40.80(0.33) |  

Meta-review: All reviewers are unanimously positive about this work although some of them raised concerns. The main strength of this paper includes technical novelty, theoretical justification, and comprehensive evaluation. On the other hand, the reviewers generally agree that the presentation could be improved. Overall, since the strengths outweigh the weaknesses, this is good enough to be presented in NeurIPS this year.
Decision: Accept (poster)