Title: Federated Learning with Manifold Regularization and Normalized Update Reaggregation
Venue: NeurIPS 2023
Accepted: yes
Keywords: federated learning; manifold regularization; update reaggregation
Abstract: Federated Learning (FL) is an emerging collaborative machine learning framework where multiple clients train the global model without sharing their own datasets. 
In FL, the model inconsistency caused by the local data heterogeneity across clients results in the near-orthogonality of client updates, which leads to the global update norm reduction and slows down the convergence.  Most previous works focus on eliminating the difference of parameters (or gradients) between the local and global models, which may fail to reflect the model inconsistency due to the complex structure of the machine learning model and the Euclidean space's limitation in meaningful geometric representations.
In this paper, we propose FedMRUR by adopting the manifold model fusion scheme and a new global optimizer to alleviate the negative impacts.
Concretely, FedMRUR adopts a hyperbolic graph manifold regularizer enforcing the representations of the data in the local and global models are close to each other in a low-dimensional subspace. 
Because the machine learning model has the graph structure, the distance in hyperbolic space can reflect the model bias better than the Euclidean distance.
In this way, FedMRUR exploits the manifold structures of the representations to significantly reduce the model inconsistency.
FedMRUR also aggregates the client updates norms as the global update norm, which can appropriately enlarge each client's contribution to the global update, thereby mitigating the norm reduction introduced by the near-orthogonality of client updates.
Furthermore, we theoretically prove that our algorithm can achieve a linear speedup property $\mathcal{O}(\frac{1}{\sqrt{SKT}})$ for non-convex setting under partial client participation, where $S$ is the participated clients number, $K$ is the local interval and $T$ is the total number of communication rounds.
Experiments demonstrate that FedMRUR can achieve a new state-of-the-art (SOTA) accuracy with less communication.

Federated Learning with Manifold Regularization and Normalized Update Reaggregation

Abstract

Federated Learning (FL) is an emerging collaborative machine learning framework where multiple clients train the global model without sharing their own datasets. In FL, the model inconsistency caused by the local data heterogeneity across clients results in the near-orthogonality of client updates, which leads to the global update norm reduction and slows down the convergence. Most previous works focus on eliminating the difference of parameters (or gradients) between the local and global models, which may fail to reflect the model inconsistency due to the complex structure of the machine learning model and the Euclidean space's limitation in meaningful geometric representations. In this paper, we propose FedMRUR by adopting the manifold model fusion scheme and a new global optimizer to alleviate the negative impacts. Concretely, FedMRUR adopts a hyperbolic graph manifold regularizer enforcing the representations of the data in the local and global models are close to each other in a low-dimensional subspace. Because the machine learning model has the graph structure, the distance in hyperbolic space can reflect the model bias better than the Euclidean distance. In this way, FedMRUR exploits the manifold structures of the representations to significantly reduce the model inconsistency. FedMRUR also aggregates the client updates norms as the global update norm, which can appropriately enlarge each client's contribution to the global update, thereby mitigating the norm reduction introduced by the near-orthogonality of client updates. Furthermore, we theoretically prove that our algorithm can achieve a linear speedup property O ( 1 √ SKT ) for non-convex setting under partial client participation, where S is the participated clients number, K is the local interval and T is the total number of communication rounds. Experiments demonstrate that FedMRUR can achieve a new state-of-the-art (SOTA) accuracy with less communication.

1 Introduction

FL is a collaborative distributed framework where multiple clients jointly train the model with their private datasets [27, 28]. To protect privacy, each client is unable to access the other dataset [2]. A centralized server receives the parameters or gradients from the clients and updates the global model[46]. Due to the limited communication resource, only part of the clients is involved in the collaborative learning process and train the local model in multiple intervals with their own datasets within one communication round [23]. Due to the data heterogeneity, clients' partial participation and multiple local training yield severe model inconsistency, which leads to the divergences between the directions of the local updates from the clients and thus reduces the magnitude of global updates [17]. Therefore, the model inconsistency is the major source of performance degradation in FL [40, 15].

So far, numerous works have focused on the issues of model inconsistency to improve the performance of FL. Many of them [20, 16, 42, 1] utilize the parameter (or gradient) difference between the local and global model to assist the local training. By incorporating the global model information into local training, the bias between the local and global objectives can be diminished at some level. However, the parameter (or gradient) difference may fail to characterize the model bias due to the complex structure of modern machine learning model and the Euclidean space has limitations in providing powerful and meaningful geometric representations [10]. Meanwhile, incorporating the difference introduces extra high computation and communication costs because of the highdimensional model parameter, which is common in the modern machine learning area[25]. Some other works [22, 39, 44, 24] exploit the permutation invariance property of the neurons in the neural networks to align and aggregate the model parameters for handling the model inconsistency issues, but the extra computation required for neuron alignment may slow down the speed of FL. In addition, Charles et al. [3] demonstrate that after multiple rounds, the similarities between the client updates approach zero and the local update direction are almost orthogonal to each other in FL. If the server aggregates the local updates, each client's contribution is little, which reduces the global update step. Therefore, we need to reduce the model inconsistency and compensate for the global norm reduction introduced by the near-orthogonality of client updates.

In order to alleviate the model inconsistency and compensate for the global reduction, we propose a practical and novel algorithm, dubbed as FedMRUR ( Fed erated learning with M anifold R egularization and Normalized U pdate R eaggregation). FedMRUR adopts two techniques to achieve SOTA performance. i) Firstly, FedMRUR adopts the hyperbolic graph fusion technique to reduce the model inconsistency between the client and server within local training. The intuition is that adding the manifold regularizer to the loss function to constrain the divergence between the local and global models. Unlike the Euclidean space, the hyperbolic space is a manifold structure with the constant negative curvature, which has the ability to produce minimal distortion embedding[8] with the low storage constraints[30] for graph data. And the neural network, the most prevail machine learning model, has a graph structure[34, 24], we map the representations to the hyperbolic space and compute their distance to indicate the model bias precisely. Considering the numerical stability[6], we select the Lorentz model to describe the hyperbolic space and the squared Lorentzian distance[19] to indicate the representations' proximity. By adopting the hyperbolic graph fusion technique, FedMRUR can constrain model inconsistency efficiently. ii) Secondly, FedMRUR aggregates the client's local updates in a novel normalized way to alleviate the global norm reduction. In the normalized aggregation scheme, the server aggregates the local update norms as the global norm and normalizes the sum of the local updates as the global direction. Compared with directly aggregating local updates, the new aggregation scheme enables each customer's contribution to be raised from its projection on the global direction to its own size. As a result, the size of the global update becomes larger and compensates for the norm reduction introduced by model inconsistency, which improves the convergence and generalization performance of the FL framework.

Theoretically, we prove that the proposed FedMRUR can achieve the convergence rate of O ( 1 √ SKT ) on the non-convex and L-smooth objective functions with heterogeneous datasets. Extensive experiments on CIFAR-10/100 and TinyImagenet show that our proposed FedMRUR algorithm achieves faster convergence speed and higher test accuracy in training deep neural networks for FL than several baselines including FedAvg, FedProx, SCAFFOLD, FedCM, FedExp, and MoFedSAM. We also study the impact on the performance of adopting the manifold regularization scheme and normalized aggregation scheme. In summary, the main contributions are as follows:

We propose a novel and practical FL algorithm, FedMRUR, which adopts the hyperbolic graph fusion technique to effectively reduce the model inconsistency introduced by data heterogeneity, and a normalized aggregation scheme to compensate the global norm reduction due to the near-orthogonality of client updates, which achieves fast convergence and generalizes better.

We provide the upper bound of the convergence rate under the smooth and non-convex cases and prove that FedMRUR has a linear speedup property O ( 1 √ SKT ) .

We conduct extensive numerical studies on the CIFAR-10/100 and TinyImagenet dataset to verify the performance of FedMRUR, which outperforms several classical baselines on different data heterogeneity.

2 Related Work

McMahan et al. [27] propose the FL framework and the well-known algorithm, FedAvg, which has been proved to achieve a linear speedup property [43]. Within the FL framework, clients train local models and the server aggregates them to update the global model. Due to the heterogeneity among the local dataset, there are two issues deteriorating the performance: the model biases across the local solutions at the clients [20] and the similarity between the client updates (which is also known as the near-orthogonality of client updates) [3], which needs a new aggregation scheme to solve. In this work, we focus on alleviating these two challenges to improve the convergence of the FL algorithms.

Model consistency. So far, numerous methods focus on dealing with the issue of model inconsistency in the FL framework. Li et al. [20] propose the FedProx algorithm utilizing the parameter difference between the local and global model as a prox-correction term to constrain the model bias during local training. Similar to [20], during local training, the dynamic regularizer in FedDyn [1] also utilizes the parameter difference to force the local solutions approaching the global solution. FedSMOO [36] utilizes a dynamic regularizer to make sure that the local optima approach the global objective. Karimireddy et al. [16] and Haddadpour et al. [9] mitigate the model inconsistency by tracking the gradient difference between the local and global side. Xu et al. [42] and Qu et al. [31] utilize a client-level momentum term incorporating global gradients to enhance the local training process. Sun et al. [37] estimates the global aggregation offset in the previous round and corrects the local drift through a momentum-like term to mitigate local over-fitting. Liu et al. [26] incorporate the weighted global gradient estimations as the inertial correction terms guiding the local training to enhance the model consistency. Charles et al. [4] demonstrate that the local learning rate decay scheme can achieve a balance between the model inconsistency and the convergence rate. Tan et al. [38] show that the local learning rate decay scheme is unable to reduce the model inconsistency when clients communicate with the server in an asynchronous way. Most methods alleviate the model inconsistency across the clients by making use of the parameter (or gradient) difference between the local and global model.

Aggregation scheme. There are numerous aggregation schemes applied on the server side for improving performance. Some works utilize classical optimization methods, such as SGD with momentum [45], and adaptive SGD [5], to design the new global optimizer for FL. For instance, FedAvgM [13, 35] and STEM [17] update the global model by combining the aggregated local updates and a momentum term. Reddi et al. [32] propose a federated optimization framework, where the server performs the adaptive SGD algorithms to update the global model. FedNova [41] normalizes the local updates and then aggregates them to eliminate the data and device heterogeneity. In addition, the permutation invariance property of the neurons in the neural networks is also applied for improving robustness to data heterogeneity. FedFTG [48] applies the data-free knowledge distillation method to fine-tune the global model in the server. FedMA [39] adopts the Bayesian optimization method to align and average the neurons in a layer-wise manner for a better global solution. Li et al. [22] propose Position-Aware Neurons (PANs) coupling neurons with their positions to align the neurons more precisely. Liu et al. [24] adopt the graph matching technique to perform model aggregation, which requires a large number of extra computing resources in the server. Many deep model fusion methods [21] are also applied in the research field of FL, such as model ensemble [47] and CVAE [12]. The aforementioned algorithms utilize the local parameters or gradients directly without considering the near-orthogonality of client updates, which may deteriorate the convergence performance of the FL framework.

The proposed method FedMRUR adopts the hyperbolic graph fusion technique to reduce the model inconsistency and a normalized update aggregation scheme to mitigate the norm reduction of the global update. Compared with the previous works, we utilize the squared Lorentzian distance of the features in the local and global model as the regularization term. This term can more precisely measure the model bias in the low-dimensional subspace. For the update aggregation at the server, FedMRUR averages the local updates norm as the global update norm, which achieves to alleviate the norm reduction introduced by the near-orthogonality of the client updates.

3 Methodology

In this section, we first formally describe the problem step for FL and then introduce the FedMRUR and the two novel hyperbolic graph fusion and normalized aggregation techniques in FedMRUR.

client p

client p-1

……

server

𝑤𝑔

: global parameter

∆𝑔

: global update

①

Broadcast

②

Construct the manifold regularizer:

𝑹 𝒘𝒑,𝒘𝒈

𝑤𝑔

∆𝑝

:local update

④

Upload

⑤

Aggregation

⑥

Update

𝑭𝒑

𝒘𝒑

= 𝒇𝒑

𝒘𝒑

+𝛄×𝑹 𝒘𝒑,𝒘𝒈

③

train

𝒘𝒑

locally with:

∆𝑔

= 𝑎𝑣𝑔𝑝

∆𝑝

𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒 ෍∆𝑝

𝑤𝑝

data

𝐹 𝑔

𝐹 𝑝

Euclidian to

Lorentzian

𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒(𝐹 𝑔 , 𝐹

𝑝 )

3.1 Problem setup

We consider collaboratively solving the stochastic non-convex optimization problem with P clients :

where w is the machine learning model parameter and z is a data sample following the specified distribution D p in client p ; meanwhile l ( w,z ) represents the model loss function evaluated by data z with parameter w . f p ( w ) and f ( w ) indicate the local and global loss function, respectively. The loss function l ( w,z ) , f p ( w ) , and f ( w ) are non-convex due to the complex machine learning model. The heterogeneity of distribution D p causes the model inconsistency across the clients, which may degrade the performance of the FL framework.

Notations. We define some notations to describe the proposed method conveniently. ‖ · ‖ denotes the spectral norm for a real symmetric matrix or L 2 norm for a vector. 〈· , ·〉 denotes the inner product of two vectors. For any nature a , b , a ∧ b and a ∨ b denote min { a, b } and max { a, b } , respectively. The notation O ( · ) , Θ( · ) , and Ω( · ) are utilized to hide only absolute constants that don't depend on any problem parameter.

3.2 FedMRUR Algorithm

In this part, we describe our proposed FedMRUR algorithm (see Figure 1 and Algorithm 1) to mitigate the negative impacts of model inconsistency and improve performance. We add a manifold regularization term on the objective function to alleviate the model inconsistency. To eliminate the near-orthogonality of client updates, we design a new method to aggregate the local updates from the clients. Within one communication round, the server first broadcast the global model to the participating clients. During local training, the client takes the sampled data into the local and received global model and gets the representations. Then the client maps the representations into the hyperbolic space and computes their distance, which is used to measure the divergence between the local and global models. Next, the client adopts the distance as a manifold regular to constrain the model bias, achieving model fusion in the hyperbolic graph. After local training, the client uploads its local update to the server. The server aggregates the local update norms as the global update step

Algorithm 1 FedMRUR 1: Input: initial parameter w 0 ; momentum coefficient α ; perturbation radius ρ ; local interval K ; communication rounds T ; set of selected clients S t ; global and local learning rate η g , η l . 2: Output: Global parameter w t , ∀ t ∈ T . 3: Initialization: Initialize glyph[triangle] 0 = 0 and w 0 as the global parameter at the server. 4: For t = 0 , 1 , ..., T -1 do: 5: The server broadcasts parameter w t and global update glyph[triangle] t to the selected clients S t . 6: For client p ∈ S t in parallel do: 7: client p initialize the local parameter as w t, 0 p = w t . 8: For k = 0 , ..., K -1 do: 9: ˜ w t,k p = w t,k p + ρ ∇ F p ( w t,k p ) ‖∇ F p ( w t,k p ) ‖ . 10: v t,k +1 i = α ∇ F p ( ˜ w t,k p ) + (1 -α ) glyph[triangle] t . 11: w t,k +1 i = w t,k i -η l v t,k +1 i . 12: End for. 13: glyph[triangle] t p = w t,K p -w t, 0 p 14: End for 15: Aggregate glyph[triangle] t +1 = ∑ p ∈ St ‖glyph[triangle] t p ‖ | S t |‖ ∑ p ∈ St glyph[triangle] t p ‖ ∑ i ∈ S t glyph[triangle] t p . 16: Update global parameter w t +1 = w t -η g glyph[triangle] t +1 . 17: End for.

and normalizes the sum of the local updates as the global update direction. Utilizing the normalized aggregation scheme, the server can update the model with a larger step and improve the convergence.

Hyperbolic Graph Fusion. In FL, the Euclidean distances between parameters[11, 20] (or gradients[16, 42]) between the client and the server is utilized to correct the local training for alleviating the model inconsistency. However, the Euclidean distance between the model parameters can't correctly reflect the variation in functionality due to the complex structure of the modern machine learning model. The model inconsistency across the clients is still large, which impairs the performance of the FL framework. Since the most prevail machine learning model, neural network has a graph structure and the hyperbolic space exhibits minimal distortion in describing data with graph structure, the client maps the representations of the local and global model into the hyperbolic shallow space[29] and uses the squared Lorentzian distance[19] between the representations to measure the model inconsistency.

To eliminate the model inconsistency effectively, we adopt the hyperbolic graph fusion technique, adding the distance of representations in the hyperbolic space as a regularization term to the loss function. Then the original problem (1) can be reformulated as:

where R ( w p , w g ) is the hyperbolic graph fusion regularization term, defined as:

In (2) and (3), L p and L g are the mapped Lorentzian vectors corresponding to Z p and Z g , the representations from local model w p and global model w g . γ and σ are parameters to tune the impact of the model divergence on training process. β is the parameter of the Lorentz model and 〈 x, y 〉 L denotes the Lorentzian scalar product defined as:

where x and y are d +1 dimensional mapped Lorentzian vectors. The new problem (2) can be divided into each client and client p uses its local optimizer to solve the following sub-problem:

The regularization term R ( w p , w g ) has two benefits for local training: (1) It mitigates the local over-fitting by constraining the local representation to be closer to the global representation in the hyperbolic space (Lorentzian model); (2) It adopts representation distances in a low-dimensional hyperbolic space to measure model deviation, which can be more precisely and save computation.

∆𝑖

∆ 𝑖

′

∆

∆𝑖

cos 𝜃 𝑖

𝜃𝑖

′

∆ 𝑖

′

cos 𝜃

𝑖

′

(a) Vanilla Aggregation.

∆𝑖

∆ 𝑖

′

∆

∆𝑖

∆ 𝑖

′

(b) Normalized Aggregation.

Normalized Aggregation of Local Updates. According to [3], after a number of communication rounds, the cosine similarities of the local updates across clients are almost zero, which comes from the model(gradient) inconsistency between the server and client sides. In the vanilla aggregation way, the server takes the mean of local updates from participated clients as the global gradient. As shown in Figure 2(a), client i makes ‖glyph[triangle] i ‖ cos θ i contribution on the aggregation result. When the data heterogeneous is significant, the cosine similarities across the local updates are small. Correspondingly, the cosine similarities cos θ i between the clients and the server narrows down, and the global update norm shrinks which slows down the convergence. To alleviate the negative impact of near-orthogonal local updates, we propose a new normalized aggregation method to compute the global update. The direction of the global update can be acquired by normalizing the sum of the local updates and the result is the same as the one obtained by the vanilla way. For the norm, the server computes it by taking the average of norms of the received local updates. As shown in Figure 2(b), with the proposed normalized aggregation method, the client i 's contribution on the global update increases from ‖glyph[triangle] i ‖ cos θ i to ‖glyph[triangle] i ‖ . Accordingly, the norm of the global update ‖glyph[triangle]‖ grows and accelerates the convergence.

Our proposed FedMRUR is characterized by Figure 1 and the detailed training process is summarized in Algorithm 1. Concretely, firstly, the server broadcasts the global parameter and updates it to the selected clients S t . At the initialization stage of local training, client i utilizes the manifold regularization scheme to construct its own local loss function F i with the received global parameter w t . Then, client i adopts the Sharpness Aware Minimization (SAM) [7] optimizer to compute the gradient ˜ g i with data sampled randomly. The local updater v i consists of the stochastic gradient α ˜ g i and the momentum term (1 -α ) glyph[triangle] t , the received global update from the last round. Client i applies v i to perform multiple SGD and uploads the accumulated local update glyph[triangle] t i to the server. The server takes two steps to construct the global update: 1) aggregating and normalizing the accumulated local updates from the participated clients S t as the direction of the global update; 2) averaging the norms of accumulated local updates as the norm of the global update. Finally, the server utilizes the constructed global update to perform one step SGD and get a new global parameter.

4 Convergence Analysis

In this section, we provide the theoretical analysis of our proposed FedMRUR for general non-convex FL setting. Due to space limitations, the detailed proofs are placed in Appendix . Before introducing the convergence results, we first state some commonly used assumptions as follows.

Assumption 1. f p ( x ) is L -smooth and R ( x, x 0 ) is r -smooth with fixed x 0 for all client p , i.e.,

Assumption 2. The stochastic gradient g t,k p with the randomly sampled data on the local client p is an unbiased estimator of ∇ F p ( x t,k p ) with bounded variance, i.e.,

Assumption 3. The dissimilarity of the dataset among the local clients is bounded by the local and global gradients, i.e.,

Assumption 1 guarantees the gradient Lipschitz continuity for the objective function and regularizer term. Assumption 2 guarantees the stochastic gradient is bounded by zero mean and constant variance. Assumption 3 gives the heterogeneity bound for the non-iid dataset across clients. All the above assumptions are widely used in many classical studies [1, 43, 33, 42, 14, 16], and our convergence analysis depends on them to study the properties of the proposed method.

Proof sketch. To explore the essential insights of the proposed FedMRUR, we first bound the client drift over all clients within the t-th communication round. Next, we characterize the global parameter moving within a communication round, which is similar to the one in centralized machine learning algorithms with momentum acceleration technology. Then, the upper bound for the global update glyph[triangle] t is provided. Lastly, we use ‖∇ F ( x t ) ‖ the global gradient norm as the metric of the convergence analysis of FedMRUR. The next theorem characterizes the convergence rate for FedMRUR.

Theorem 1. Let all the assumptions hold and with partial client participation. If η l ≤ 1 √ 30 αKL , η g ≤ S 2 αL ( S -1) satisfying 3 4 -2(1 -αL ) KN -70(1 -α ) K 2 ( L + r ) 2 η 2 l -90 α ( L + r ) 3 η g η 2 l S -3 α ( L + r ) η g 2 S , then for all K ≥ 0 and T ≥ 1 , we have:

where

and d t = ∑ i ‖ glyph[triangle] t i ‖ ‖ ∑ i glyph[triangle] t i ‖ ≥ 1 . Specifically, we set η g = Θ( √ SK √ T ) and η l = Θ( 1 √ STK ( L + r ) ) , the convergence rate of the FedMRUR under partial client participation can be bounded as:

Remark 2. Compared with the inequality F 0 -F ∗ Cαη g T +Φ of Theorem D.7 in MoFedSAM paper[31], the second constant term in (7) is same and the first term is less than the first term in MoFedSAM paper, which validates FedMRUR achieves faster convergence than MoFedSAM.

Remark 3. From (8), we can find that when T is large enough, the dominant term O ( 1 √ SKT ) in the bound achieves a linear speedup property with respect to the number of clients. It means that to achieve glyph[epsilon1] -precision, there are O ( 1 SKglyph[epsilon1] 2 ) communication rounds required at least for non-convex and L-smooth objective functions.

0

200

400

600

800

1000 1200 1400 1600

0.0

0.1

0.2

0.3

0.4

0.5

Test accuracy

FedAvg

FedCM

FedExp

FedProx

SCAFFOLD

MoFedSAM

FedMRUR

0

200

400

600

800

1000 1200 1400 1600

0.0

0.1

0.2

0.3

0.4

0.5

Test accuracy

FedAvg

FedCM

FedExp

FedProx

SCAFFOLD

MoFedSAM

FedMRUR

0

200

400

600

800

1000 1200 1400 1600

0.0

0.1

0.2

0.3

0.4

0.5

Test accuracy

FedAvg

FedCM

FedExp

FedProx

SCAFFOLD

MoFedSAM

FedMRUR

0

200

400

600

800

1000 1200 1400 1600

0.0

0.1

0.2

0.3

0.4

0.5

Test accuracy

FedAvg

FedCM

FedExp

FedProx

SCAFFOLD

MoFedSAM

FedMRUR

0

500

1000

1500

2000

2500

3000

0.0

0.1

0.2

0.3

0.4

Test accuracy

FedAvg

FedCM

FedExp

FedProx

SCAFFOLD

MoFedSAM

FedMRUR

0

500

1000

1500

2000

2500

3000

0.0

0.1

0.2

0.3

0.4

Test accuracy

FedAvg

FedCM

FedExp

FedProx

SCAFFOLD

MoFedSAM

FedMRUR

0

500

1000

1500

2000

2500

3000

0.0

0.1

0.2

0.3

0.4

Test accuracy

FedAvg

FedCM

FedExp

FedProx

SCAFFOLD

MoFedSAM

FedMRUR

0

500

1000

1500

2000

2500

3000

0.0

0.1

0.2

0.3

0.4

Test accuracy

FedAvg

FedCM

FedExp

FedProx

SCAFFOLD

MoFedSAM

FedMRUR

5 Experiments

In this section, we validate the effectiveness of the proposed FedMRUR algorithm using the experimental results on CIFAR-10/100 [18] and TinyImageNet [18]. We demonstrate that FedMRUR outperforms the vanilla FL baselines under heterogeneous settings. We also present that both manifold regularization and the proposed normalized update aggregation can improve the performance of SGD in FL. The experiments of CIFAR-10 are placed in the Appendix.

5.1 Experimental Setup

Datasets. We compare the performance of FL algorithms on CIFAR-10/100 and TinyImageNet datasets with 100 clients. The CIFAR-10 dataset consists of 50 K training images and 10 K testing images. All the images are with 32 × 32 resolution belonging to 10 categories. In the CIFAR-100 dataset, there are 100 categories of images with the same format as CIFAR-10. TinyImageNet includes 200 categories of 100 K training images and 10 K testing images, whose resolutions are 64 × 64 . For non-iid dataset partitioning over clients, we use Pathologicaln (abbreviated as Path( n )) and Dirichletµ (abbreviated as Dir( µ )) sampling as [13], where the coefficient n is the number of data categories on each client and µ measures the heterogeneity. In the experiments, we select the Dirichlet coefficient µ from { 0 . 3 , 0 . 6 } for all datasets and set the number of categories coefficient n from { 3 , 6 } on CIFAR-10, { 10 , 20 } on CIFAR-100 and { 40 , 80 } on TinyImageNet.

Implementation Details. For all algorithms on all datasets, following [1, 42], the local and global learning rates are set as 0 . 1 and 1 . 0 , the learning rate decay is set as 0 . 998 per communication round and the weight decay is set as 5 × 10 -4 . ResNet-18 together with group normalization is adopted as the backbone to train the model. The clients' settings for different tasks are summarized in Table 1. Other optimizer hyperparameters are as follow: ρ = 0 . 5 for SAM, α = 0 . 1 for client momentum, γ = 0 . 005 , σ = 10000 . 0 and β = 1 for manifold regularization.

Baselines. To compare the performances fairly, the random seeds are fixed. We compare the proposed FedMRUR with several competitive benchmarks: FedAvg [43], the most widely used baseline,

firstly applies local multiple training and partial participation for FL framework; SCAFFOLD [16] utilizes the SVRG method to mitigate the client drift issue; FedProx [20] uses a proximal operator to tackle data heterogeneity; FedCM [42] incorporates the client-momentum term in local training to maintain the model consistency among clients; Based on FedCM, MoFedSAM [31] improves the generalization performance with local SAM [7] optimizer; FedExp [14] determines the server step size adaptively based on the local updates to achieve faster convergence.

5.2 Evaluation Results

Figure 3 and Table 2 demonstrate the performance of ResNet-18 trained using multiple algorithms on CIFAR-100 and TinyImageNet datasets under four heterogeneous settings. We plot the test accuracy of the algorithms for a simple image classification task in the figure. We can observe that: our proposed FedMRUR performs well with good stability and effectively alleviates the negative impact of the model inconsistency. Specifically, on the CIFAR100 dataset, FedMRUR achieves 55.49% on the Dirichlet-0.3 setups, which is 5.07% higher than the second-best test performance. FedMRUR effectively reduces the model inconsistency and the enlarged global update improves the speed of convergence.

Table 3 depicts the convergence speed of multiple algorithms. From [13], a larger µ indicates less data heterogeneity across clients. We can observe that: 1) our proposed FedMRUR achieves the fastest convergence speed at most of the time, especially when the data heterogeneity is large. This

validates that FedMRUR can speed up iteration; 2) when the statistical heterogeneity is large, the proposed FedMRUR accelerates the convergence more effectively.

5.3 Ablation Study

Impact of partial participation. Figures 4(a) and 4(b) depict the optimization performance of the proposed FedMRUR with different client participation rates on CIFAR-100, where the dataset splitting method is Dirichlet sampling with coefficient µ = 0 . 3 and the client participation ratios are chosen from 0 . 02 to 0 . 2 . From this figure, we can observe that the client participation rate (PR) has a positive impact on the convergence speed, but the impact on test accuracy is little. Therefore, our method can work well under low PR settings especially when the communication resource is limited.

0

200

400

600

800

1000 1200 1400 1600

0

1

2

3

4

5

Train loss

act_prob=0.02

act_prob=0.05

act_prob=0.1

act_prob=0.2

(a)

0

200

400

600

800

1000 1200 1400 1600

0.0

0.1

0.2

0.3

0.4

0.5

Test accuracy

act_prob=0.02

act_prob=0.05

act_prob=0.1

act_prob=0.2

(b)

0

200

400

600

800

1000 1200 1400 1600

0.0

0.1

0.2

0.3

0.4

0.5

Test accuracy

=0.005

=0.05

=0.5

=5.0

(c)

0

200

400

600

800

1000 1200 1400 1600

0

1

2

3

4

5

Train loss

=0.005

=0.05

=0.5

=5.0

(d)

Hyperparameters Sensitivity. In Figures 4(c) and 4(d), we compare the performance of the proposed FedMRUR with different hyper-parameters on the CIFAR-100 dataset. From the results, we can see that our algorithm achieves similar test accuracy and training loss under different γ within a certain range ( γ ∈ [0 . 005 , 0 . 5] ) and this indicates the proposed FedMRUR is insensitive to the hyperparameter γ . The hyperparameter γ represents the method the punishment on the model bias.

Impact of Each Component. Table 4 demonstrates the impact of each component of FedMRURonthe test accuracy for CIFAR-100 dataset on the Dirichlet-0.3 setups. For convenience, we abbreviate normalized aggregation as 'normalized(N)' and hyperbolic graph fusion as 'hyperbolic(H)', respectively. From the results, we can find that both the normalized local update

aggregation scheme and the hyperbolic graph fusion can improve performance. This table validates that our algorithm design and theoretical analysis are correct and effective.

6 Conclusion

In this work, we propose a novel and practical federated method, dubbed FedMRUR which applies the hyperbolic graph fusion technique to alleviate the model inconsistency in the local training stage and utilizes normalized updates aggregation scheme to compensate for the global norm reduction due to the near-orthogonality of the local updates. We provide the theoretical analysis to guarantee its convergence and prove that FedMRUR achieves a linear-speedup property of O ( 1 √ SKT ) . We also conduct extensive experiments to validate the significant improvement and efficiency of our proposed FedMRUR, which is consistent with the properties of our analysis. This work inspires the FL framework design to focus on exploiting the manifold structure of the learning models.

Limitations&Broader Impacts. Our work focuses on the theory of federated optimization and proposes a novel FL algorithm. During the local training, the representations of the global model must be stored locally, which may bring extra pressure on the client. This will help us in inspiration for new algorithms. Since FL has wide applications in machine learning, Internet of Things, and UAV networks, our work may be useful in these areas.

Acknowledgements. This work is supported by National Key Research and Development Program of China under SQ2021YFC3300128, and National Natural Science Foundation of China under Grant 61971457. Thanks for the support from CENI-HEFEI and Laboratory for Future Networks in University of Science and Technology of China.

References

Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N. Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Koneˇ cn` y, Stefano Mazzocchi, Brendan McMahan, et al. Towards federated learning at scale: System design. Proceedings of machine learning and systems , 1:374-388, 2019.

Zachary Charles, Zachary Garrett, Zhouyuan Huo, Sergei Shmulyian, and Virginia Smith. On large-cohort training for federated learning. In 34th Advances in Neural Information Processing Systems, NeurIPS 2021, virtual, December 6-14, 2021 , pages 20461-20475, 2021.

Zachary Charles and Jakub Koneˇ cn` y. Convergence and accuracy trade-offs in federated learning and meta-learning. In The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event , pages 2575-2583. PMLR, 2021.

Ashok Cutkosky and Róbert Busa-Fekete. Distributed stochastic optimization via adaptive sgd. Advances in Neural Information Processing Systems , 31, 2018.

Shanshan Feng, Lisi Chen, Kaiqi Zhao, Wei Wei, Xuemeng Song, Shuo Shang, Panos Kalnis, and Ling Shao. Role: Rotated lorentzian graph embedding model for asymmetric proximity. IEEE Transactions on Knowledge and Data Engineering , 2022.

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 , 2021.

Mikhael Gromov. Hyperbolic groups. In Essays in group theory , pages 75-263. Springer, 1987.

Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi. Federated learning with compression: Unified analysis and sharp guarantees. In The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event , Proceedings of Machine Learning Research, pages 2350-2358. PMLR, 2021.

William L Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. arXiv preprint arXiv:1709.05584 , 2017.

Filip Hanzely and Peter Richtárik. Federated learning of a mixture of global and local models. arXiv preprint arXiv:2002.05516 , 2020.

Clare Elizabeth Heinbaugh, Emilio Luz-Ricca, and Huajie Shao. Data-free one-shot federated learning under very high statistical heterogeneity. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023.

Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 , 2019.

Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Fedexp: Speeding up federated averaging via extrapolation. In 11-th International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 , 2023.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210, 2021.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. SCAFFOLD: stochastic controlled averaging for federated learning. In Proc. 37th Int. Conf. Mach. Learn. , pages 5132-5143, 2020.

Prashant Khanduri, Pranay Sharma, Haibo Yang, Mingyi Hong, Jia Liu, Ketan Rajawat, and Pramod K. Varshney. STEM: A stochastic two-sided momentum algorithm achieving near-optimal sample and communication complexities for federated learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 6050-6061, 2021.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report , 2009.

Marc Teva Law, Renjie Liao, Jake Snell, and Richard S. Zemel. Lorentzian distance learning for hyperbolic representations. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 915 June 2019, Long Beach, California, USA , Proceedings of Machine Learning Research, pages 3672-3681, 2019.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Proc. Mach. Learn. Sys. , 2020.

Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. 2023.

Xin-Chun Li, Yi-Chu Xu, Shaoming Song, Bingshuai Li, Yinchuan Li, Yunfeng Shao, and De-Chuan Zhan. Federated learning with position-aware neurons. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 10072-10081. IEEE, 2022.

Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ying-Chang Liang, Qiang Yang, Dusit Niyato, and Chunyan Miao. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys & Tutorials , 22(3):2031-2063, 2020.

Chang Liu, Chenfei Lou, Runzhong Wang, Alan Yuhan Xi, Li Shen, and Junchi Yan. Deep neural network fusion via graph matching with applications to model ensemble and federated learning. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , pages 13857-13869. PMLR, 2022.

Weibo Liu, Zidong Wang, Xiaohui Liu, Nianyin Zeng, Yurong Liu, and Fuad E Alsaadi. A survey of deep neural network architectures and their applications. Neurocomputing , 234:11-26, 2017.

Yixing Liu, Yan Sun, Zhengtao Ding, Li Shen, Bo Liu, and Dacheng Tao. Enhance local consistency in federated learning: A multi-step inertial momentum approach. 2023.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL USA , Proceedings of Machine Learning Research, pages 1273-1282. PMLR, 2017.

Dinh C Nguyen, Ming Ding, Pubudu N Pathirana, Aruna Seneviratne, Jun Li, and H Vincent Poor. Federated learning for internet of things: A comprehensive survey. IEEE Communications Surveys & Tutorials , 23(3):1622-1658, 2021.

Maximilian Nickel and Douwe Kiela. Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80, pages 3776-3785. PMLR, 2018.

Wei Peng, Tuomas Varanka, Abdelrahman Mostafa, Henglin Shi, and Guoying Zhao. Hyperbolic deep neural networks: A survey. IEEE Trans. Pattern Anal. Mach. Intell. , 44(12):10023-10044, 2022.

Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , pages 18250-18280, 2022.

Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Koneˇ cn` y, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295 , 2020.

Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Koneˇ cný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Tao Sun, Dongsheng Li, and Bao Wang. Decentralized federated averaging. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Yan Sun, Li Shen, Shixiang Chen, Liang Ding, and Dacheng Tao. Dynamic regularized sharpness aware minimization in federated learning: Approaching global consistency and smooth landscape. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202, pages 32991-33013. PMLR, 2023.

Yan Sun, Li Shen, Hao Sun, Liang Ding, and Dacheng Tao. Efficient federated learning via local adaptive amended optimizer with linear speedup. IEEE Transactions on Pattern Analysis and Machine Intelligence , in press:1-12, 2023.

Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the 36th AAAI Conference on Artificial Intelligence , pages 8432-8440, 2022.

Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris S. Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated optimization. arXiv preprint arXiv:2107.06917 , 2021.

Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Jing Xu, Sen Wang, Liwei Wang, and Andrew Chi-Chih Yao. Fedcm: Federated learning with client-level momentum. arXiv preprint arXiv:2106.10874 , 2021.

Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participation in non-iid federated learning. In 9th Int. Conf. Learn. Representations , 2021.

Fuxun Yu, Weishan Zhang, Zhuwei Qin, Zirui Xu, Di Wang, Chenchen Liu, Zhi Tian, and Xiang Chen. Fed2: Feature-aligned federated learning. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining , pages 2066-2074, 2021.

Hao Yu, Rong Jin, and Sen Yang. On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization. In International Conference on Machine Learning , pages 7184-7193. PMLR, 2019.

Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. A survey on federated learning. Knowledge-Based Systems , 216:106775, 2021.

Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. DENSE: data-free one-shot federated learning. In NeurIPS , 2022.

Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 10164-10173. IEEE, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a new algorithm for federated learning based on a Lorentzian regularization. The proposed algorithm achieves desired sub-linear convergence with linear speed-up. Numerical experiment shows the efficacy of the proposed algorithm over existing federated learning algorithms\n\nSTRENGTHS:\nThe proposed method is theoretically well-rounded with convergence guarantee for smooth non convex situation, which is the most commonly setting in the literature.\n\nWEAKNESSES:\n(Please respond to the Questions section directly) The presentation of the paper is confusing. Also some technical details are not well-illustrated. The theory part of the paper is not clearly presented and seems to be abusing terminologies a lot.\n\nQUESTIONS:\nMajor:\n    \n1. The presentation of the paper is confusing: First, I cannot link the proposed Algorithm 1 with all the argument in Section 3.2. More specifically, how is the “mapped Lorentzian vectors” in (3) used in Algorithm 1? The authors claimed that the regularized problem (5) has a number of desired property but is Algorithm 1 solving this regularized problem? The argument in this section seems to be ending at nowhere; Second, I really cannot understand the usages of “hyperbolic” and “Lorentzian”. The hyperbolic space is defined as space with constant -1 curvature in mathematics, but here we are not dealing with curvatures at all. As for the Lorentzian regularizer defined in (4) and (5) (which again I don’t understand how it is used in Algorithm 1), why do we need Lorentz metric? What would happen if we just use Euclidean metric, or some Riemannian metric defined by a positive definite matrix? The entire section 3.2 is confusing. On the other hand, all the described operations in Section 3.2 are not well-reflected in Algorithm 1. For example, what’s $g_{i}^{t,k}$ in Algorithm 1?\n    \n2. As the authors claimed, the algorithm is built upon MoFedSAM[1]. How is the rate of convergence (in theory) of Algorithm 1 compared with existing works, especially MoFedSAM? If the theoretical rate is not improved, I would argue that this refined engineering over MoFedSAM might not be very interesting since (in my perspective) the only difference of Algorithm 1 with MoFedSAM is the $\\Delta_i$ which serve as a local update corrector.\n    \n3. I cannot understand what point Figure 2 try to make. To me Figure 2(b) is a rotation operation, not projection, also I don’t understand how is the operation used in Algorithm 1.\n\n\nMinor:\n1. In the abstract, the authors include a formula. I think it’s better to either explain the meaning of each of the variable in the formula, or just use words instead;\n\n2. What does the name “FedMRUR” exactly stand for?\n\n3. Typos: Line 227, Assumption 1, missed one $\\nabla$; Line 263 “Dirichlet… from {0.30.6}” missed a comma?\n\n\nReferences:\n[1] Qu, Zhe, et al. \"Generalized federated learning via sharpness aware minimization.\" International Conference on Machine Learning. PMLR, 2022.\n\nLIMITATIONS:\nThe limitation is well stated in weakness and question sections.\n\nI’m not aware of any potential negative social impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel Federated Learning Algorithm, FedMRUR that uses hyperbolic graph diffusion to reduce the effect of data heterogeneity and thereby model inconsistencies. The authors also propose a normalized aggregation scheme to achieve faster convergence. The algorithm FedMRUR achieves state-of-the-art performance on standard datasets.\n\nSTRENGTHS:\nIn terms of comparisons to baselines, FedMRUR seems to outperform in terms of accuracy for various Dir(\\mu) settings versus the baselines. The closest competitor seems to be MoFedSAM.\n\nWEAKNESSES:\nFinally, while convergence speed is quicker for FedMRUR, Table 3 in the Supplemental Materials actually has FedCM to have the quickest total time to achieve 60% test accuracy on CIFAR-100. Do the authors have comments or suggestions on when to use FedMRUR vs FedCM given the disparity between convergence speed and clock times?\n\nQUESTIONS:\n1. The authors should clarify notation in Figure 1 and Algorithm 1. In Figure 1, the global parameter is subscripted, w_0, but superscripted in the algorithm.\n2. The algorithm can be hard to follow as it:\n  a. Did not specify the global parameter \\eta_g in the parameters\n  b. Did not specify the gradients, g_{i}^{t,k}, though that is easily induced\n  c. More importantly did not specify how \\tilde{g}_{i}^{t,k} is derived. This is mentioned later in line 213, but would make the presentation clearer by linking it directly to the equations\n  d. They can also specify the normalized aggregation more clearly, and tie to the algorithm 1.\n3. The authors should discuss normalized aggregation in more depth, as it seems to contribute more than hyperbolic graph fusion (Table 3). In fact, it would be interesting to understand the effect of this operator in other baselines.\n4. In terms of comparisons to baselines, FedMRUR seems to outperform in terms of accuracy for various Dir(\\mu) settings versus the baselines. The closest competitor seems to be MoFedSAM. First, it would be useful to have test accuracy and test accuracy std to enable better comparison between comparably performing models. Second, how was Table 1 derived, as the accuracy numbers are quite different and higher than the comparable Table 5 in the MoFedSAM paper for CIFAR-100 with Dir().\n5. Similarly, the convergence table 2 seem different for MoFedSAM and FedAvg vs Table 5 in the MoFedSAM paper. Is this a setup difference?\n\nLIMITATIONS:\nThe authors have addressed some limitations of their method", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of model inconsistency across clients in federated learning (FL). The authors propose a method called FedMRUR, which uses a hyperbolic graph manifold regularization term and a normalized update aggregation scheme to alleviate the issues introduced by model inconsistency. Compared with the previous works, the proposed FedMRUR can reflect the model bias better in a low-dimensional subspace and mitigate the norm reduction of global updates caused by model inconsistency. The authors prove the convergence of FedMRUR for nonconvex objectives under partial client participation. They also run experiments to show that FedMRUR can achieve a new state-of-the-art (SOTA) accuracy with less communication.\n\nSTRENGTHS:\n1.\tThe work is written clearly and intuitively.\n\n2.\tThe considered problem is quite meaningful as the model inconsistency severely impairs the performance of FL and building an algorithm to solve this issue is a promising direction to improve FL.\n\n3.\tThe theoretical result is rigorous for non-convex settings under partial client participation.\n\n4.\tThere are experiments to support the theory as well as to show that the algorithm will be useful in practice.\n\nWEAKNESSES:\nSee questions\n\nQUESTIONS:\n1.\tIt is not clear what is the “manifold structure of their representations”.\n\n2.\tWhat does it mean by “norm reduction” and why it is important?\n\n3.\tFor the non-i.i.d pathological-n setting on TinyImageNet, line 264 conflicts with Figure 3(b).\n\n4.\tDoes the setting of the parameter $\\beta$ for hyperbolic graph manifold regularization have an impact on the performance of the algorithm? \n\n5.\tIn the ablation study, the author should discuss the hyperparameters sensitivity of $\\gamma$ in a wider range.\n\nLIMITATIONS:\nPlease refer to the Questions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors found the existing vanilla distillation in FL, the model inconsistency caused by the local data heterogeneity across clients results in the near-orthogonality of client updates, which leads to the global update norm reduction and slows down the convergence. Moreover, the authors argue previous works may fail to reflect the model inconsistency due to the complex structure of the machine learning model and the Euclidean space’s limitation in meaningful geometric representations. To resolve the above issues, they propose the FedMRUR algorithm for FL. By adopting a hyperbolic graph manifold regularizer and aggregating the client updates norms as the global update norm, the FedMRUR achieves a new state-of-the-art (SOTA) accuracy with less communication.\n\nSTRENGTHS:\noriginality: Exploiting the manifold structures of the representations can reflect the model bias better than the parameters (or gradients) method, to significantly reduce the model inconsistency. Aggregating the client updates norms as the global update norm can mitigate the norm reduction caused by model inconsistency.\n\nquality: The paper is well written, with detailed experiments and ablation studies with other methods and the proposed variants.\n\nclarity: The paper consists of an illustration of the workflow and text explanations of the proposed FedMRUR algorithm.\n\nsignificance: The paper solves the problem of the model inconsistency across the clients in FL, reduces the model bias more effectively than its baseline, mitigates the norm reduction caused by model inconsistency, and improves the test performance.\n\nWEAKNESSES:\n- There are some unclear points and confusing notations.\n- More experiments can be conducted.\n\nQUESTIONS:\n1) Some notations are confusing. For example, in line 14 of Algorithm 1, what does the notation “$\\delta_i^{t}$” mean? \n2) In line 263, what should the setting of the Dirichlet coefficient be?\n3) The authors should discuss the effect of local interval $K$ on the performance of the algorithm.\n\nLIMITATIONS:\nPlease find the weaknesses and limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes a federated learning framework called FedMRUR to deal with the model inconsistency caused by the local data heterogeneity across clients and insufficient geometric representation ability. To do this, it adopts a hyperbolic graph manifold regularizer to ensure that the representations obtained by the local and global models are close in a low-dimensional subspace. Then it aggregates the client updates norms as the global update norm to mitigate the norm reduction introduced by the near-orthogonality of client updates. A linear speedup property is theoretically proved for the proposed algorithm.

STRENGTHS:
+ The motivation to deal with the model inconsistency and insufficient representation ability is clear.
+ It is interesting to adopt the hyperbolic graph fusion scheme in federated learning.
+ The theoretical guarantee is provided for the convergence and especially the proposed method exhibits a linear speedup.
+ The experimental results seem promising.

WEAKNESSES:
I'm not very familiar with federated learning. Here are some major concerns:
- There are many ways to exploit the manifold structure. What's the advantage of adopting the graph fusion scheme, especially in the hyperbolic space? If the hyperbolic space is a good choice, then why not let the whole learning process work in the hyperbolic space?
- Figure 2 is less informative since the readers cannot understand how the normalized aggregation works and what's the difference between these two operations in terms of their workflow.
- It seems that the widely used CIFAR-10 benchmark is not involved for evaluation.
- The experiments to validate the linear speedup property are missing.

QUESTIONS:
Please see the above part.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26