On the effects of similarity metrics in decentralized deep learning under distribution shift
Authors: Edvin Listo Zec
Tom Hagander
Eric Ihre-Thomason
Sarunas Girdzijauskas
Venue: Accepted by TMLR ()

Abstract: Decentralized Learning (DL) enables privacy-preserving collaboration among organizations or users to enhance the performance of local deep learning models. However, model aggregation becomes challenging when client data is heterogeneous, and identifying compatible collaborators without direct data exchange remains a pressing issue. In this paper, we investigate the effectiveness of various similarity metrics in DL for identifying peers for model merging, conducting an empirical analysis across multiple datasets with distribution shifts. Our research provides insights into the performance of these metrics, examining their role in facilitating effective collaboration. By exploring the strengths and limitations of these metrics, we contribute to the development of robust DL methods.

Published in Transactions on Machine Learning Research (01/2025)
On the effects of similarity metrics in decentralized deep learning under distributional shift
Edvin Listo Zec
divin.listo.co www.nnlisto.zec@vi...o
RISE Research Institutes of Sweden
KTH Royal Institute of Technology
Tom Hagander
Land University
Eric Thom-sno
Lund University
Sarunas Girdzijauskas
KTH Rogal Institule of Technology
Reviewed on OpenReview: https: openreview. net/ forum? id- Wpp 4kk
Abstract
Decentralized Learning (DL) enables privacy-preserving collaboration among organizations or users to enhance the performance of local deep learning models. However, model aggre- gation becomes challenging when client data is heterogeneous, and identifying compatible collaborators without direct data exchange remains pressing issue. In this paper, in- vestigate the effectiveness of various similarity metrics in DL for identifying peers for model merging, conducling empirical analysis across multiple datasets with distribut ion shifts. Our research provides insights into the performance of these metrics, examining their role in facilitating effective collaboration. By exploring the strengths and limitations of these metrics, we contribute to the development of robust DL methods.
1 Introduction
In recent years, there has been a significant increase in interest towards distributed machine learning ap- proaches, driven primarily by growing concerns over user privacy and data security. This shift is driven by regulations such the California Consuner Privacy Act (CCPA), General Data Protection Regulation (GDPR), and the Data Protection Act, alongside rising awareness among users about their data rights. These developments necessitate advancements in privacy-preserving methods. Additionally, many organiza- tions possess proprietary data that cannot be shared externally, further fueling the need for decentralized frameworks.
As machine learning models become more sophisticated and continue to scale, they exhibit superior perfor- mance across a diverse array of tasks. Yet, these models roly heavily on extensive datasets, often compiled from various online sources. This poses significant issues regarding data ownership and privacy. There- fore, decentralized approaches are critical in mitigating these issues by enabling collaborative model training without the necessity for a centralized dataset repository. In this framework, no single entity controls vast dataset. Instead, multiple stakeholders participate in model training through a decentralized and confidential approach.
Among these approaches, foderated learning (FL) has emerged as a leading framework for distributed deep learning, allowing the training of global model across multiple clients that maintain private datasets McMahan et al. 2017). Various studies have aimed to address client drift phenomenon occurring when

---

Published in Transactions on Machine Learning Research (01/2025)
clients pursue divergent optimization objectives due to distributional shifts between datasets, resulting in diminished performance upon averaging their model parameters. To counter this, several strategies introduce a regularization objective into the global training goal to minimize deviations between the global model and individual local models (Li et al. 2020; Karimireddy et al.) 2020 Li ct al. 2021). This approach is effective when clients share similar datasets, sampled from the same underlying data generating process. In these cases, a common global model is optimal. Conversely, in scenarios where clients' data originate from distinct generating processes, these methods can be suboptimal and, in some cases, detrimental. [hhosh] et al. (2020); Vardhan et al. (2024) address this issue within centralized federated learning by training multiple global sodells-specifiall
K models corresponding to the number of data clusters, with clustering determined by empirical loss to approximate data similarity. Similarly, /Sattler et al. (2020) employ cosine similarity on local gradients for clustering clients in FL. Although effective, this leads to a high computational cost as the similarity is computed at stationary solutions of the FL objective. Moreover, there is other notable work on clustered contralized FL which also consider cosine similarity or Euclidean distance between Ruan & Joe- Noe-Wong 2022).
parameters (Briggs et al. 2020 Duan et al. 2021
1.1 Technical challenges and related work
The aforementioned approaches necessitate a central server. In contrast, our work explores fully decen- tralized methods for identifying client clusters, which are more suitable for large-scale applications. While federated lcarning enables distributed training of a global model, its reliance on a central server can become a limiting factor and a bottleneck the number of clients grows. This limitation has spurred interest in fully decentralized systems, known decentralized learning (DI.), where clients interact within a peer-to-peer network, eliminating the need for central server. Decentralized learning is particularly advantageous for its scalability and inherent robustness against single-point failures or targeted attacks. An illustration of different machine learning frameworks is shown in Figure
One notable approach within decentralized learning is gossip learning, a peer-to-peer communication proto- col, which has been extensively studied in various machine lcarning contexts (Kempe al., 2003 Boyd al., 2006 Ormándi et al. 2013 Hegedüs et al. 2019) J Initial decentralized work using gossip- based optimization for non-convex deep learning tasks, specifically with convolutional neural networks (CNNs), was explored by Blot et al. (2016). However, gossip learning faces challenges in non independent and identically distributed (non-iid) settings where there is a distributional shift across client datasets. This is primarily due to the in- adequacy of gossip learning in selectively identifying beneficial peers during the training process. To address these challenges, recent work has addressed decentralizod learning within non-iid contexts. Onoszko et al. (2021): Li et al. (2022) introduced client selection algorithms based on empirical loss, employing distinct discovery and selections phases. These studies address challenges in decentralized learning within non-iid scttings but roquire extensive hyperparameter tuning, particularly regarding scarch length, and rely on hard cluster assignments for clients. Such hard clustering can be detrimental if the discovery phase is noisy. To mitigate these issues, Listo Zec et al. (2022) proposed a sampling strategy that converts the similarity scores into sampling probabilities, which are used each communication round to sample collaborators. This ap- proach identifies similar clients and facilitates model merging in proportion to their similarity. By adopting a soft clustering technique, enhances the efficiency of peer collaboration, circumventing the need for the computationally intensive and difficult-to-tune discovery phase required by hard clustcring methods. Addi- tionally, Listo Zec et al.] (2024) introduced framework using multi-armed bandits for client selection, where the selection process is modeled empirical rewards derived from validation accuracy.
Quasi-Globa Momentum (QGM) [Lin et al.) 2021). Neighbourhood Gradient Mcan (NGM) (Aketi et al. 2023) and Maaaa--eeeerogeneiy---vaaa Mixing (DHM) DDandi et al., 2022) are contemporary approaches that address the challenges posed by non-iid data in decentralized learning. While the methods focus on improving consensus optimization, our work diverges significantly in its objectives and assumptions. Specifically, we aim to tackle the problem of varying data distributions across clusters rather than optimizing for a shared consensus model.
2

---

Published in Transactions on Machine Learning Research (01/2025)
1.2
Our contributions
Previous research has predominantly employed empirical loss as a proxy for data similarity (Ghosh et al. Onoszko et al. 2021 Listo Zec el al. 2022 ; while some studies have investigated similarity using gradients (Li al. 2022 Sattler et al.; 2020): Despite significant progress, the impact of the chosen simi- larity metric on decentralized learning, especially under distribution shifts between client datasets, remains insufficiently explored. In this paper, fill this gap by thoroughly study different similarity metrics in decentralized learning through extensive empirical experiments on common benchmark datasets.
2019
In summary, critical question persists which we seek to answer: how does the choice of similarity metric inffuence client identification and the performance of decentralized learning?
This question is crucial because the effectiveness of decentralized learning heavily relies on accurately identi- fying and engaging similar peers. To address this gap, we present comprehensive empirical study exploring the impacts of four distinct similarity metrics: empirical loss, cosine similarity gradients, cosine similarity model weights, and L distance model weights.
Furthermore, we introduce novel aggregation method for decentralized learning called Federated Similar- ity Averaging (FedSim). Unlike the traditional Federated Averaging (FedAvg) approach, which aggregates models based on training set sizes, FedSim merges client models using a weighted average based on their similaritics. This approach allows clients reduce the influence of dissimilar peers on their local model, thereby mitigating the impact of incorrectly sampled clients. In our experiments, we compare FedSim against FedAvg across various similarity metrics and demonstrate that in several scenarios, FedSim significantly enhance performance.
This study not only fills a gap in understanding the role of similarity metrics in decentralized learning but also offers practical insights into optimizing collaboration in environments subject to distributional shifts. Our experimental results show that empirical loss can a noisy estimator of data similarity in many cases, especially when the number of samples is small, leading to subpar client identification and negatively affecting performance. At the same time, we show that there are other metrics that can be more robust. By systematically evaluating these metrics, provide deeper understanding of their influence the performance and efficiency of decentralized learning systems, hopefully guiding future research and application in this domain.
Modi
Contall Server
Tekeo
Local learning.
(b) Federated learning.
(c) Decentralized learning.
Figure Illustrations of different types of machine learning paradigms. (a) In traditional ML, model is trained on centralized dataset. (b) In FL, a central server orchestrates the training of global model across multiple private datasets. (c) In DL. there is central server; instead clients communicate and merge models in a peer-to- peer network.
3

---

In our problem setup, we assume that the number of unique data distributions P& is fewer than the total number of clients K, implying certain devices share data generation processes and form clusters. Thus, each client's task is identify peers within their respective clusters with whom they can merge models to optimize their local objectives while preserving data privacy. This task essentially becomes a combinatorial selection problem where each client must select peers from (K) possible combinations during each communication round. The objective for each client is to select the most appropriate peers each round to minimize its Published in Transactions on Machine Learning Research (01/2025)
2 Problem formulation
Consider a set of clients denoted by S, each possessing a local and private dataset D; = {T Vn bn)]nn with N2 samples, sampled from an underlying probability distribution Pr(x, y) for i E S. In decentralized learning, we have a network of = S] clients, each with a local optimization objective consisting of a loss function over model parameters w;- The primary goal for each client to learn an optimal model fi parameterized by wi, which minimizes the following objective:
R(w;)= - (((((w;)) -
(1)
Given that the true distribution P(x, y) is unobserved, direct optimization of equation 1 is not possible. Instead, each client performs empirical risk minimization (ERM) locally by minimizing
f((w)) B(E,)) [((f((1;;
(2)
to obtain w arg w.," R(w:).
However, the sample size N; of each local dataset D; is assumed to be insufficient for solving equation Consequently, clients need to collaborate by merging models with peers to solve their local optimization problems. For any client i, each communication round t starts with E local epochs of training where the local model updated using an optimizer, such as stochastic gradient descent (SGD). Then, client interacts with a random subset of clients M C S, merging models through foderatod averaging (McMahan et al., 2017). This process involves caleulating weighted average of the model parameters, resulting in an updated model:
2
W
(3)
where N Em= N₂ and m J. After this aggregation, local training resumes for E epochs, and the process repeats for T rounds. Tn summary, there K models in the communication network and each client uses this collaborative approach to enhance model performance locally while preserving data privacy,
In standard federated and decentralized learning, it is implicitly assumed that a single model can effectively fit all client data simultaneously. This assumption holds true in an ideal scenario where all clients share the same data generating process, i.e,, P. P;= P, for all i,j S. In such cases, decentralized learning with random communication (also known as gossip learning) is effective. However, when clients form clusters with distinct data generating processes, i.e,, P. * Py, merging models from clients with different distributions can significantly hinder convergence even be detrimental (Zhao al. 2018: Kairouz et al. 2021). This is because the objective function in equation [ varies between clients, and there may not be a single model wi that achieves zero risk across all distributions. Thus, the challenge for each client is to identify subsets of clients with similar data distributions to ensure that merging models is beneficial, all while preserving the privacy of the datascts.
2.1 Similarity based peer selection
local objective function. When all client datascts are sampled from the same joint probability distribution Px(x, u) and are partitioned independently and identically distributed (iid), random communication uniform sampling of M c S- is effective. However, the goal of semiilartty-basee sampling is to strategically select the set ' and merge models a manner that enables each client to minimize the objective specified in equation [ This approach aims to enhance the efficiency and performance of decentralized learning by leveraging the inherent. similarities between clients' data distributions.
i.e.,
4

---

Published in Transactions on Machine Learning Research (01/2025)
Since local datasets are private, traditional measures cannot be used to calculate similarities between them directly for clustering clients. Instead, we rely model parameters gradients to infer similar datasets. To maintain simplicity, we adopt the decentralized adaptive clustering (DAC) framework (Listo Zec et al, 2022 In this framework, each client stores a probability vector p K which is used to sample new clients M', where each element Vi represents the probability for client i to sample client j at time l. This probability vector calculated using a softmax function and a similarity metric S; every time step i, as follows:
Pr(S:)
(4)
Σk=1 CT
Here, T the inverse temperature parameter controlling the distribution of the softmax, i.e. T gives a uniform distribution and as J we reach the argmax function. This is hyperparameter that problem dependent and that needs to be optimized.
The selection of similarity metric is critical in the DAC framework as it influences how effectively similar clients are identified and, consequently, how efficiently the models converge. Listo Zec et al. (2022) used inverse empirical loss as a similarity metric, defined by evaluating client i's model w at time t on the training
data of client. k, expressed as s/xx (Ez..tEE), f(wf; (xa]: vx))= This metric be computed without sharing data samples, as only model parameters and similarity scores are exchanged bet ween clients i and k. The authors demonstrated that this method effectively identifies similar clients over time in decentralized learning. However, this metric may have drawbacks since its performance depends on sample size and quality. Additionally, the authors did not compare with other similarity metrics within this framework. Therefore, work aims to explore how different similarity metries impact client selection and performance in DL.
Similar to previous studies, this work assumes fully connected network where all clients can communicate directly with each other, facilitated by communication protocols such as those available via the internet. We also assume synchronous client communication, as our focus not on system heterogeneity. While exploring the impact of constrained network topologics or varying levels of client connectivity on performance is a compelling area for future research, it is beyond the scope of this paper.
Algorithm Decentralized learning protocol for a client
Input: model parameters wi, temperature т, learning rate n, dataset Fi, y/ Di- Initializo prior probabilities: Piy K-1 for each communication round = 1, ...... do Select K - clients M (cl, : without replacement, according to the probability distribution p Pf..................
Compute similarity scores st, between client and each sampled client ME 6: Update probability vector via softmax: P < ( T).
7: For FedSim, normalize the similarities: $ Σ 855
8:
Aggregate w/ with the models from the sampled clients:
merged
OyWy
1=1
12 for FedAvg, and 0g = 8y for FedSim. where a, N 9: Update the merged model locally for E epochs:
merged V w. f(u) merged
w
w.
10: end for
I1: return Final model parameters for client é: w
5

---

Published in Transactions on Machine Learning Research (01/2025)
3 Experimental setup
In this section, we describe our experimental setup for studying the effect of common similarity metrics on client identification within decentralized learning. Empirical loss and cosine similarity on gradients have
been previously employed (Onoszko et al. 2021 Li ct al. 2022 Listo Zec et al. 2022 Additionally, introduce the Euclidean distance (L²) between model parameters and cosine similarity between parameters (as opposed to gradients) as two additional baselines, which have not been tested before in DL for client identification. We define the similarity metrics between two clients and j as follows:
Inverse empirical loss = (E( ((w;),,))
Inverse L distance = I|u; W/ m1
Cosine similarity of weights
Tw my
Cosine similarity of gradients
9:
1s |
For cosine similarity of gradients, we define the gradient at time t as:
= w
(5)
where of represents the vector describing the direction from the initialized weights to the current weights of the network.
3.1 Computational cost analysis
When selecting similarity metrics for client identification in decentralized learning, it is crucial to consider their computational overhead, especially analyze the computational complexity of cach of the four similarity metrics each communccation round, focusing on the operations required to compute the similarity score itself, rather than the cost of training epochs.
resource-constrinne decentralized settings. this section, we
Inverse empirical loss. To compute the inverse empirical loss as a similarity metric between client and client j, we evaluate client i's model with parameters w; on every data point (x, y) in client j's local dataset Dj. This necessitates performing a forward pass through the model for each data point in Dj, resulting in a computational complexity of O(N, x Cý), where N, the size of the dataset D, and C2 the cost of single forward pass. This approach, which deterministically evaluates the loss over the entire dataset Dj, imposes significant computational burden, especially with large datasets complex models, making it less feasible in practice. While empirical loss could be stochastically estimated using a subset of DD (e.g., a mini-batch). we use the full dataset to obtain a more deterministic and potentially more reliable measure of functional similarity for guiding peer selection.
Inverse 2 distance. The inverse Euclidean distance between model parameters w; and wy is computed directly from the parameter vectors and has computational complexity of O(P), where P is the number of model parameters. This calculation is deterministic and scales lincarly with model size, remaining feasible even for large models.
Cosine similarity of weights and gradients. Both the cosine similarity of weights and the cosine similarity of gradients, as defined by equation using of w wo, have a computational complexity of O(P), similar to the inverse L distance. These metrics involve computing dot products and norms of vectors of size P, which deterministic operations manageable even for models with large number of parameters.

---

Published in Transactions on Machine Learning Research (01/2025)
Practical example. To illustrate the practical implications of these computational costs, consider a mod- erately sized CNN with P P= 1, 1,000,000 parameters and local dataset size of Nj- 10,000 samples per client. Assuming an idealized scenario with efficient hardware where the cost of a single forward pass Cj is approx- imately 10 milliscconds, computing the inverse empirical loss for similarity asscssment would require a total time of N, x Cy 10, 000 x 10 ms = 100, 000 ms = 100 s. This calculation focuses solely on computation time and does not account for potential communication overhead in a decentralized sctting.
In contrast, computing the inverse L distance cosine similarities involves approximately p= 1,000,000 basic vector operations (dot products, norms, subtractions). Assuming each operation takes approximately nanosccond, the total time is roughly P x 1 ns 1,000,000 ns = 1 ms. This example, while simplified, highlights the order of magnitude difference in computational cost between empirical loss and the other parameter-base similarity metrics, emphasizing the trade-offs between functional similarity measures and computational efficiency.
3.2 Distributional shifts in decentralized learning
When local client data generating processes are identical, meaning [] y) Vi,j, random commu- nication is optimal as all clients are learning the same model. However, this assumption often unrealistic. In practical scenarios, distribution shifts between clients are common, leading to client data heterogeneity (non-iid data). Given the variety of ways two probability distributions can differ, it is crucial to formally describe the type of shift to understand the effectiveness of decentralized learning solutions.
Consider the factorized form of the joint probability distribution:
P(a.v) P(x))) = P(yl=)
(6)
Using this factorization, can describe some common types of distribution shifts. In this paper, we investigate the following distributional shifts in the context of decentralized learning.
1. Covariate shift: This occurs when P(r) varies between clients while the conditional distribution P(y(æ) remains constant. For instance, different hospitals may use different x-ray machines, lead- ing to variations in the input features collected, while the diagnosis based on the x-rays remains consistent across hospitals.
2. Label shift: This shift is present when P(y) varies between clients but P(x|y) is shared. An example of this scenario is mobile phone users taking photos of different types of animals, resulting in a variation of the label distribution across users.
3. Concept shift: This type of shift occurs when P(y|æ) varies but P(r) stays the same. In this case, the same inputs result in different labels. This can occur when modeling user preferences or sentiment.
4. Domain shift: This represents a more general shift where both the conditionals P(y and the marginals P(x) differ simultaneously across clusters. For example, consider voice recognition system trained on data from adults speaking in quiet environments. If the system is deployed to recognize children's voices noisy environments like playgrounds, both the input fcatures x (voice characteristies and background noise) and the relationship between these features and the labels y (spoken words) change significantly, illustrating domain shift.
Understanding these shifts is essential for the development and evaluation of decentralized learning algo- rithms. We evaluate the performance of similarity metrics using ERM in decentralized setting four datasots in the presence of distributional shifts. In all cases, we evaluate each client model w; on test set drawn from its underlying data generating process P, y)- The first dataset is a synthetic one, while the three others are classic benchmark image datasets (MNIST, CIFAR-10 and (NIST). We create different clusters of the datasots in order simulate heterogencous data distributions among clients. The synthetic dataset is used to study concept shift. For the benchmark datasets, we study covariate shift, label shift and domain shift. A summary of our experimental setup is shown in Table
7

---

Published in Transactions on Machine Learning Research (01/2025)
Dataset
Shift
Model
No. of clusters
No. of clients
CIFAR-10
Label
CNN
2
100
CIFAR-10
Label
CNN
5
100
Synthotic data
Concept
Lincar
3
99
MasHion-MNISTT
Covariate
CNN
4
100
Fashion-MNIST + MNIST
Domain
MLP & CNN
2
100
CIFAR-100
Label
Pre-trained ResNet-18
3
52
Table 1: Summary of our experimental selup.
3.3 Model selection and hyperparameter tuning
In both federated and ralized learning environments, the process of model selection and the setting of hyperparameters are crucial yet often under-diseussed elements. A common challenge in the literature the lack of detailed reporting on hyperparameter choices docentralized scttings, which complicates the task of conducting fair and consistent methodological comparisons. Variations in results that stem from differing hyperparameter tuning practices can be mistakenly attributed to the intrinsic qualities of the algorithms themselves, thereby obscuring accurate evaluations. In practice, while some studies set, others may employ a local validation set specific to each client, and still others might not clearly disclose their hyperparameter selection strategy.
global validation
In our investigation, we strive for a rigorous comparison across all models and similarity metrics by using a local validation set for each client. We tune all hyperparameters using these validation sets, and employ early stopping on each client locally. During communication, some clients may reach early stopping before others, When this occurs, these clients continue to broadcast their models to other clients but cease updating their models through further training merging. This approach ensures that our evaluation is both relevant and tailored to the specific conditions of each client's 's data. Additionally, we introduce a comparison against an Oracle baseline, which has access to clients with the same data distribution and only communicates with those, providing a benchmark under idcal conditions, and a Local bascline which only performs local training of the available data on a single client.
After hyperparameters were selected, each method was run three independent times, and we report the means and standard deviations across these runs. The found hyperparameters for each problem is presented in Appendix A in Table together with more experimental details. For the synthetic dataset, linear models trained with SGD were used. In these experiments, 15 independent runs were made for each method. For the benchmark datascts, the models comprised neural networks with two convolutional layers followed by two linear layers. In the domain shift experiment, we also employed a single-layer MLP with ReLU activation. While these models do not represent the stato-of-the-art for these tasks, they provide sufficient complexity to discern meaningful differences between the methods under study. This setup not only highlights the capabilities of the similarity metrics within controlled experimental environment but also provides insights into their practical applicability in more realistic, decentralized scenarios.
3.4 Distributional shifts and datasets
Concept shift. We begin by studying decentralized learning problem using linear regression in synthetic setup. In this scenario, we assume that each cluster is defined by yı (zx 07) + &i, for client i and cluster j where E; denotes normally distributed noise independent of the data x. Essentially, each cluster has a underlying data generating process defined by 85 which a model for that cluster aims to learn from the data. If the 0f vary significantly across clusters, merging models from different clusters would lead to suboptimal models.
To construct the clusters, generate three distinct 83 Rd, each drawn from uniform distribution.
The feature matrix X E Raxd populated by entries sampled uniformly from the range [-10, 10), forming d dimensional vector with n samples for each client. Finally, the response variable y: computed as yi (z,, 07). We use mean squared error the loss function, setting d 10 and n 50 for each client. We
8

---

Published in Transactions on Machine Learning Research (01/2025)
create the clusters such that they contain equal number of clients. This synthetic setup enables controlled exploration of the efficacy of similarity metrics in setting characterized by distinct, non-overlapping data distributions. This method of data generation results in a concept shift between clients - where P(y varics, while P(x) remains constant.
Label shift. Using the CIFAR-10 dataset (Krizhessky et al., 2009); we create two more realistic and complex scenarios. The first scenario divides the dataset into two clusters based on content categories: an animal cluster, comprising four labels, and vehicle cluster, with six labels. These clusters consist of 40 and 60 clients, respectively. In the second scenario, create five equally large clusters of 20 clients where each cluster is defined by two unique, randomly sampled labels. In this way, each cluster has unique data generating process where P(y) varies while P(xly) remains constant. We also study large scale pre-trained models under label shift. For this, we use the CIFAR-100 dataset (Krizhessky et al. 2009). which we split into three distinct clusters of sizes 26, 13 and based on the superclasses.
Covariate shift. To study covariate shift, we use the Fashion-MNIST dataset (Xiao et al. 2017). We construct four clusters, where each cluster is characterized by specific rotation angle that defines their respective data generation process Ps (zz y). This setup not only tests the impact of image rotation on model performance but also incorporates varying cluster sizes to mimic realistic client distribution: 70, 20, 5, and 5 clients per cluster. The rotation degrees set for these clusters are 0º, 10", 180 and 350 This arrangement creates three clusters with relatively similar conditions and one notably distinct cluster the 180" rotation. In this way, each cluster has a unique data generating process where P(x) varies while P(y|z) remains constant.
Domain Shift. We also explore domain shift, common issue in the multi-task learning litcrature where single model attempts to learn multiple domains (or lasks). In this setting, both P(ylz) and P(x) vary between clusters. We use the MNIST (LeCun et al. 1998) and Mushion-MNISTT datasets, defining two clusters by partitioning these datasets into halves, each assigned to different clients.
3.5 FedSim: leveraging similarity metrics for weighted averaging
In empirical studies, we observed that the traditional FedAvg, which weights updates by the number of data samples per client, can lead to catastrophic forgetting when client's model is combined with a poorly performing model from another client due to sampling a client from a cluster with a different data distribution. To explore potential mitigations, we introduce a novel approach termed Federated Similarity Averaging (FedSim). In FedSim, each client's contribution to the global model weighted by a similarity metric rather than merely the data count. Specifically, for a single client, the update computed as:
W
swwg
where sı represents the similarity metric for client k. and w denotes the model parameters from client ks iteration t. To incorporate the model updates from the client itself, we assign its weight the maximum similarity observed among all &g. Subsequently, we normalize these weights to ensure they sum to one. This methodological adjustment aims to prevent the adverse effects observed with traditional federated averaging by prioritizing contributions from clients that have more aligned optimization problems.
Limitations of FedAvg under distribution shift The FedAvg aggregation does not yield an unbiased estimate of the target distribution Ps that a client associated with P& aims learn when merging models trained other distribution P. + Pk. Specifically, the expected value of the aggregated model using FedAvg is
N E [l
N N
E El =
Σ
=
W +
w
(8)
N
1=1
9

---

Published in Transactions on Machine Learning Research (01/2025)
where N₁ is the number of samples for client and N N, w is the optimal model parameters for distribution P., and wi the optimal model parameters for distribution P:. The last inequality in equation g arises because the combination of the optimal W from other distributions P. for = I,..., m does not,
in general, equal w.
To address this limitation, we propose using similarity-based averaging- Under the assumption that the target distribution Ps can be expressed as weighted combination of the sampled client distributions P., we can show that similarity-basse averaging provides and unbiased estimate of the optimal parameters for Pk- Assumption 1. The target Px can be expressed as a weighted combination of the sampled client distributions, iec. Px Ei 8 PX,, where 82.......... 8m) is vector of weights salisfying 8; and 2 =.
Under Assumption 1 we have the following proposition.
Proposition 1 (Unbiased estimator). Consider single round in the batch stochastic gradient selling with learning rate d) Let each sampled client i [m]] compute its local update Waat we 1/ Re(we); where (we) is the empirical risk on client Then, the aggregated update using the similarity weights s is u++1 = s,00,, Under Assumption 1 the expected aggregated update satisfies
E|wa+1 we] = Elwif] we].
where wfft = wi V Rr, (we) is the batch SGD update that would be oblained using a somple from the target distribution Pk.
the expected gradient of the aggregated update can be written as Proof sketch. Under Assumption
E El
wi] =
Σ
E
Wi =
Σ 84 wi nE ~ Ra Ru(w)))
i=1
(9)
s; V Rs (w:) = we TB (w wft
Wt
i=1
Under this proposition, the expected value of the model update FedSim with perfect similarity scores 8; is
Elw
B Σ s.w
(10)
Thus, similarity-based averaging with perfect similarity scores provides an unbiased estimate of the optimal model paramelers for the target distribution. We present results from our experimental work analyzing common similarity metrics used in decentralized learning in Section 4 both for sampling clients and for aggregation using FedSim.
4 Results
In this section, we illustrate the effectiveness of the similarity metrics. Unless otherwise stated, all results are means over three independent runs. The main results are presented Table g
4.1 Concept shift
Synthetic experiment. In the synthetic experiment, we generated three distinct clusters where the con- ditional distribution P(y|æ) varies between clusters while the marginal P(x) remains constant. In these experiments, all reported results are means over 15 independent runs. The findings, illustrated in Figure 3a demonstrate significant improvement in performance using FedSim compared to Fed especially
10

---

Published in Transactions on Machine Learning Research (01/2025)
Table 2: Performance comparison across methods and tasks on test sets, Mean squared error (MSE) used for the synthetic experiment, while test accuracy used for all other tasks. The reported values represent the averages across all clusters for each method and experiment. Best performing method (excluding the Oracle) is denoted with bold text.
Synthetic (U)
CIFAR-10 (t)
CIFAR-10 (t)
F/MNIST (t)
FMNIST (1)
CIFAR-100 (1)
Shift
Concept
Label
Label
Domain
Covariate
Label
1.2 FERAVG
21.10 + 2.51
59.44 + 0.66
84.04 + 0.63
90.49 + 0.12
84.17 + 0.21
55.53 1: 0.57
1,2 FEASSM
10.85 0.33
57.63 = 1.14
84.06 + 0.31
90.80 = 0.11
84.16 0.01
loea FIRAAG
31.69 3.23
81.94 = 0.53
0.22
81.83 0.14
54.23
Inv loss FEDSIM
14.82 0.33
61.08 = 1.15
85.80 + 0.20
90.67 = 0.U7
85 + 0.26
Cos grad FEDAVG
53.87
Cos grad FEaSSM
0.25
58.13 0.75
0.75
86.00 + 0.32
90.72 = 0.24
0.24
83.32 0002:100 0.58
Cos weight. FEDAVG
0.28
59.14 1.62
85,85 0.20
91.03 0.23
86.04
54.54 0.04
Cos weight FEDSIM
10.30 0.21
39.66 1.02
85.66 + 0.36
90.62 0.17
81.82 : 0.42
Random
1494.84 + 1973
59.71 = 1.13
82.73 0.33
90.53 = 0.05
0.05
83.70 0.10
54.88 0.27
Local
0.24
76.26
Oracle
9.43 E 0.33
62.26 E 0.47
87.30 E 0.20
91.62 E 0.08
86.65 3 0.11
166.37 0.17
for the inverse loss and L2 metrics. This highlights the challenges associated with merging models from different clusters for FedAvg, as further illustrated in Table which shows that random communication fails to address the problem effectively. Since P(y2z) differs between clusters, single model fails to adequately represent all the data. In this scenario, the Oracle method, represented by the green area in Figure [a performs the best. In contrast, the inverse loss metric underperforms relative other metrics, struggling identify the correct clusters due noisy sampling, where clients frequently sample incorrect peers, leading catastrophic forgetting during model merging. Notably, cosine similarities on gradients and weights ap- proach optimal performance with minimal standard deviation across runs. Figure 2a presents heatmap of client communication, clearly indicating that the inverse loss and 1,2 metrics exhibit more noise compared to cosine similarity metrics.
We also conducted an experiment to examine the effect of training sample size on the similarity metrics. As shown Figure 3b. we repeated the same experiment with an increased number of samples for each method. The results indicate that the number of training samples increases, the performance gap for inverse loss and 2 distance metries narrows. However, these metrics never match the performance of cosine similarity gradients and weights, which achieve a low test loss even with fewer training samples.
Sampling heatmaps: How many times ciient on the y-axis sampled client on the x axis
Clues 9i64
Cleet indee
Clock
Clent
Cleat
Gleet rde
Cevr Indin <
(a) Synthetic linear regression problem.
Sampling heatmaps: how many times client on the y-acis sampled a client on the x-axis
Clart volee
Cer vabe
Clent
(b) Pretrained CIFAR-100 label shift problem.
Figure 2: Heatmaps of client communication, indicating how often client communicated with client y for the four different similarity metrics.
11

---

Published in Transactions on Machine Learning Research (01/2025)
4.2 Domain shift
PashionMNIST and MNIST. We further investigate the impact of similarity metrics on domain shift in & nonconvex optimization problem using neural networks. In this scenario, we form two clusters: one consisting of clients with data from the MNIST dataset, and the other with data from the Fashion-MNIST dataset. Experiments were conducted using both Multi-Layer Perceptron (MLP) and Convolutional Neural Network (CNN) to assess the influence of model capacity on concept shift.
As illustrated in Figure da there is a noticeable performance gap between random communication and the Oracle method (red and green, respectively). In this setup, FedSim negatively impacts all similarity metrics except for the L distancc. This suggests that when the similarity metries effectively separate the clusters, FedSim may hinder learning by causing clients to underweight models from other clients within the same cluster. Similar results observed with the CNN, as shown in Figure 4b Notably, inverse loss performs significantly worse than the other metries in this context, indicating that it samples incorrect peers too often. Incorrect merging lowers performance of the local client models, which in turn makes empirical loss a worse similarity metric. In both the MTP and CNN settings, cosine similarity on gradients or weights outperforms the other metrics, exhibiting relatively low variance in test accuracies. This indicates that cosine similarity is a robust choice for docentralized learning in environments with distributional shifts, applicable to both model complexities.
12
Gracle tedfeg FedSin
25
127
E-atorrt 6ie
(b) Average loss over clusters as func- tion of trainset size for all methods (Fe- dAvg). The orange line depicting cos grad is hidden as follows the green line.
InV Kos
Cos grad
Cos wegit
(a) Average loss over clusters for each
method, comparing FedAvg and FedSim.
Figure Test loss performance for all methods on the linear regression problem with concept shift.
Fashion Fas MMST and MNST Combined with MLP
Pashion-AMMST and VIEST Combined, winh CNN
Orade
52.0
Nandem
$0.5
$6.0
59.3
Random
Feelivg
80.0
PedSi
grad
weight
(a) MLP.
(b) CNN.
Figure 4: Test accuracy for all methods on the domain shift problem, where half of clients have data from the MNIST dataset and the other half have data from the Fashion- MNIST dataset. Results for two different model architectures: (a) an MLP and (b) a CNN.
12

---

Published in Transactions on Machine Learning Research (01/2025)
4.3 Covariate shift
Fashion-MNIST. For the covariate shift experiments, observe that all similarity metrics outperform random communication, except for cosine similarity on gradients when using FedSim. This is illustrated in Figure 5a and Table - FedAvg consistently demonstrates more stable performance across different metrics compared to FedSim, suggesting that the similarity metrics effectively separate the clusters. Among the metrics, inverse loss, cosine similarity on gradients, and cosine similarity on weights exhibit relatively com- parable performance. However, the L distance metric underperforms relative the other metrics, despite still achieving better results than random communication. These findings underscore the elfectiveness of using FedAAg with appropriate similarity metries to handle covariate shifts in decentralized learning.
FisSion-MISST Covariate Shift
Protrined
$5.5
accuro
64.5
$5.0
64.0
54.5
63.5
Crudd
54.0
Cnocle
Pesking
55.5
Feeling
53.0
Inw loss
Dos gmad
Cos weight
IW lods
Cos gnd
Cos seight
12
(a) Rashion-MNIST and covariate shift.
(b) CIFAR- 100 and label shift, 3 clusters.
CIFARLO Arimalvehicl Label stift
CHARLC S-Cluster Lobel Shint
Oradi
62
Random
Pesdig
24
Rancom
Pealing
RedSin
grid
weight
(c) CIFAR-10 10 and label shift, 2 clusters.
(d) CIFAR-10 and label shift, 5 clusters
Figure 5: Test accuracy for all methods on different problems: Fashion-MNIST covariate shift (a), pre- trained CTF label shift (b), and the CIFAR-10 label shifts (c,d).
4.4 Label shift
CIFAR-10, 2 clusters. In the CIFAR-10 experiment with an animal cluster and a vehicle cluster, most metrics do not exceed the performance of random communccatio,, presented in Figure 5c and Table 2 Additionally, the gap between random communication and the Oracle method is small, suggesting that this specific label shift relatively straightforward for random communication to manage. Nevertheless, we observe performance incrcase for the inverse loss metric this setting as outperforms random communication and the other metrics.
CIFAR-10, 5 clusters. In the scenario with five clusters, where each cluster is defined by its two unique labels, we observe a larger performance gain across all metrics. This is presented in Figure 5d and Table 2 Here, the label shift is more pronounced as P(y) differs more significantly between the clusters. This means that merging models with wrong peers has a larger negative impact on performance. This is illustrated by the fact that random communication achieves even worse performance compared to the Oracle method in this setup, as compared to the previous setup of two clusters. The cosine similarity metrics, however, demonstrate robust performance with low variance across runs. In this setting, inverse loss does not match
13

---

Published in Transactions on Machine Learning Research (01/2025)
the performance of the cosine similarity metrics. Mcanwhile, using FedSim improves the performance of inverse loss, indicating that inverse loss is able to identify clusters, albeit. noisily. The I metric performs worse than all other metrics but still manages to outperform random communication.
CIFAR-100, 3 clusters, pre-trained ResNet-18. Previous research in federated learning has shown that starting with a pre-trained model can mitigate some of the adverse impacts of label shift (Nguyen et al. 2022). In our experiments, we observed that this approach makes the Random method relatively stronger. As depicted in Figure 5b] three out of the four similarity metrics did not outperform the Random method when using pre-trained model. For the inverse loss metric, the pre-trained model's robustness results in uniformly low loss across all clients, causing all clients to appear very similar. In the case of cosine similarity on gradients and weights, the similarity scores between clients were also too uniform, making it challenging to detect significant cluster structures. Notably, the L metric was the only method capable of accurately clustering clients and outperforming random communication this context. This further illustrated in Figure 2b which presents communication heatmaps and highlights the difficulties these methods encounter in correctly identifying clusters. In these experiments, we did not evaluate FedSim with pre-trained models due to the substantial computational resources required for training large-scale peer-to peer-to-pest teer systems with large models. Future work will address this limitation by exploring both FedSim and other aggregation methods for decentralized learning.
5 Conclusions
We have conducted thorough empirical study the effect of similarity metrics in decentralized learning for client identification. Our findings highlight the strengths and weaknesses of various similarity metrics, which are dependent on the problem lype and the nature of the distributional shift. Based on our results, we recommend that researchers and practitioners carefully explore similarity metrics in decentralized learning to identify the most suitable one for their specific problem. Our study indicates that the effectivencss of similarity metrics varies depending on the specific conditions and types distributional shifts encountered. Our key conclusions are as follow:
i The L distance metric:
The L metric, although outperforming random communication most of the time, is the weakest of the similarity metrics we studied. We hypothesize that this is due to the metric being sensitive to the scaling of parameters and highly susceptible to permutations of neurons layers, making it less reliable when comparing models with different weight configurations but similar functional behavior.
Inverse empirical loss: The inverse empirical loss provides a direct functional measure of similarity. However, it highly dependent on the specific training data used, with its reliability being contingent on the number of samples and their quality. Clients with inherently more difficult tasks or poorer initial models can have higher loss values, which can skew the inverse loss and mislead the merging process. If the sample size is large, and the local models are able to solve the local problem well, this metric is effective at identify beneficial peers for model merging.
ii Cosine similarity:
Our results indicate that cosine similarity on model parameters often outperforms other metrics, particularly the L method. This is likely due to the fact that cosine similarity is less sensitive to scaling compared to 1,2 distance and captures the similarity the direction of weights rather than their exact values, making it more robust permutations. Similarly, cosine similarity on gradients provides valuable insights into the learning dynamics between models, beyond their static states. Our findings suggest. that cosine similarity metrics are relatively robust. across various scenarios.
iv Concept shift: Clustering of clients in decentralized learning is especially useful in the presence of strong concept shift, where P(y|z) varies across clusters while P(z) remains constant. In these cases, random communication cannot solve the optimization problem, necessitating accurate cluster identification.
Random communication: Despite various distributional shifts our results demonstrate that with proper and robust model selection, random communication remains strong baseline.
14

---

Published in Transactions on Machine Learning Research (01/2025)
vi Pre-training: Pre-training can enhance the performance of random communication in scenarios with label shift. However, the use of robust pre-trained model tends to standardize similarity scores, making it challenging for various metrics to effectively identify distinct clusters.
vii Federated Similarity Averaging (FedSim): We introduced a new aggregation scheme, FedSim. While it is not a silver bullet, FedSim can enhance the performance of FedAvg, particularly in scenarios where averaging with incorrect models incurs high costs, such as in the presence of strong concept shift between clusters (Figure 3a
Overall, our sludy underscores the importance of selecting appropriate similarity metrics and aggregation strategies based on the specific context and challenges decentralized learning tasks.
6 Future work
There are several promising directions for future research, both theoretical and empirical. Developing theo- retical frameworks for measuring similarity between private datasets could provide better insights into the optimal selection of similarity metries.
The rescarch area of decentralized learning non-iid data is closely related to domain generalization and transfer learning. These fields involve understanding when it is possible to jointly train deep learning models across different probability distributions and determining the extent to which these models can extrapolate to new data. Bridging the gap between these areas of research is an interesting direction.
Additionally, investigating the intersection of similarity metrics and privacy is a compelling direction. Be- yond model-based similarity metrics like inverse empirical loss, Euclidean distance, and cosine similarity which assess similarity by comparing clients' model weights optimal transport theory offers framework for measuring similarity based on clients' data distributions (Peyré et al. 2019). The Wasserslein distance, fundamental concept in optimal transport, quantifies the minimal cost of transforming one probability dis- tribution into another. While the Wassersteen distance provides a nuanced and theoretically sound measure of data distribution similarity, its practical application hindered by computational complexity, Computing the exact Wasserstein distance for high-dimensional empirical distributions involves solving optimization problems with significant resource demands. Moreover, using the Wasserstein distance in decentralized learning introduces considerable privacy concerns. Since this metric relies detailed statistical information even direct access to empirical data distributions, it risks exposing sensitive information, thereby con- flicting with the preeeccpreeerving goals of decentralized systems. Future work should focus on addressing these computational and privacy challenges associated with advanced similarity metrics like the Wasserstein distance. Developing efficient approximations or privacy-onhancing techniques tailored to these metrics could significantly enhance their utility in decentralized learning, fostering more robust, generalizable, and privacy-conscious machine learning models.
References
Sai Aparna Aketi, Sangamesh Kodge, and Kaushik Roy. Neighborhood gradient mean: An efficient decen- tralized learning method for non-iid data. Transactions on Machine Learning Research, 2023.
Michael Blot, David Picard, Matthieu Cord, and Nicolas Thome. Gossip training for deep learning. anXiv preprint xxXto:1611.09796, 2016.
Stephen Boyd, Arpita Ghosh, Balaji Prabhakar, and Devavrat Shah. Randomized gossip algorithms. IEEE transactions on information theory, 52(())2508 2530, 2006.
Christopher Briggs, Zhong Fan, and Peter Andras. Federated learning with hicrarchical clustering of local updates to improve training on non-iid data. In 2020 international joint conference on neural networks (IJCNN), pp. 1-9. IEEE, 2020.
Yatin Dandi, Anastasia Koloskova, Martin Jaggi, and Sebastian Stich. Data--htterogeneity-aaee mixing for decentralized learning. arXiv preprint ar 2rx::22004.00 2022.
15

---

Published in Transactions on Machine Learning Research (01/2025)
Moming Duan, Duo Liu, Xinyuan Ji, Renping Liu, Liang Liang, Xianzhang Chen, and Yujuan Tan. Fedgroup: Efficcient federated learning via decomposed similarity-baser clustering. In 2021 IEEE Tull Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking JSPA//BDCooddSoccilCooo Com pp. 228- 237. TEEE, 2021.
Avishek Ghosh, Justin Hong, Dong Yin, and Kannan Ramchandran. Robust federated learning in a hetero- geneous environment. arXiv preprint xiv:
1xx:1906.000 906. 06629, 2019.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An elficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 11119586-19599 2020.
István Hegedús, Gábor Danner, and Márk Jelasity. Gossip learning as decentralized alternative to fed- erated learning. In Distributed Applications and Interoperable Systems: 19th IFIP WG 6.1 1 International Conference, DAIS 2019, Held as Part of the 14th International Federated Conference on Distributed Com- puting Techniques, DisCoTee 2019, Kongens Lyngby, Denmark, June 17-21, 2019, Proceedings 19, pp. 74-90. Springer, 2019.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and trends0 in machine learning, 14(1 2):1 210, 2021.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pp. 5132 5143. PMLR, 2020.
David Kempe, Alin Dobra, and Johannes Gehrke. Gossip-based computation of aggregate information. In 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings., pp. 482-491. JEEE, 2003.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278 22222224, 1998.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10713 10713-10722, 2021.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020.
Zexi Li, Jiaxun Lu, Shuang Luo, Didi Zhu, Yunfeng Shao, Yinchuan Li, Zhimeng Zhang, Yongheng Wang, and Chao Wu. Towards effective clustered federated learning: A peer-to-peer framework with adaptive neighbor matching. IEEE Transactions Big Data, 2022.
Tao Lin, Sai Pranceth Karimireddy, Sebastian Stich, and Martin Jaggi. Quasi-global momentum: Accelerat- ing docentralized deep learning on heterogencous data. In International Conference on Machine Learning, pp. 6654 6665. PMLR, 2021.
Edvin Listo Zec, Ebba Ekblom, Martin Willbo, Olof Mogren, and Sarunas Girdzijauskas. Decentralized adaptive clustering of deep nets beneficial for client collaboration. In International Workshop on Trust- worthy Federated Learning, pp. 59-71. Springer, 2022.
Edvin Listo Zec, Johan Östman, Olof Mogren, and Daniel Gillblad. Efffciintt node selection private personalized decentralized learning. In Northern Lights Deep Learning Conference, pp. 244 250. PMLR, 2024.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-offttie efffccient learning of deep networks from decentralized data. In Artificial intelligence and slatistics, pp. 1 1282. PMLR, 2017.
16

---

Published in Transactions on Machine Learning Research (01/2025)
John Nguyen, Jianyu Wang, Kshitiz Malik, Maziar Sanjabi, and Michael Rabbat. Where to begin? the impact of pre-training and initialization in federated learning. arXiv preprint orXin:2206. 1000006..3344 15387, 2022.
Noa Onoszko, Gustav Karlsson, Olof Mogren, and Edvin Listo Zec. Decentralized federated learning of deep neural networks non-iid data. 2021 ICML Workshop Federated Learning for User Privacy and Data Confidentiality, 2021.
Róbert Ormándi, István Hegedus, and Márk Jelasity, Gossip learning with linear models on fully distributed data. Concurrency and Computation: Practice and Experience, 25(4):556-571, 2013.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trendst in Machine Learning, 11(5-6):355 607, 2019.
Yichen Ruan and Carlee Joe-Wong. Fedsoft: Soft clustered federated learning with proximal local updating. In Proceedings of the AA AT conference arlificial intelligence, volume 36, pp. 8124-811 8131, 2022.
Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems, (2(8):3710-3725 2020.
Harsh Vardhan, Avishek Ghosh, and Arya Mazumdar. An improved federated clustering algorithm with model-based clustering. Transactions on Machine Leurning Reseurch, 2024.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image datasct for benchmarking machine learning algorithms, 2017.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. aix preprint arXiv: 00582, 2018.
17

---

Published in Transactions on Machine Learning Research (01/2025)
A Appendix
A.1 Hyperparameters and experimental details
In our study, we aimed for a rigorous and fair evaluation where we performed a grid search over hyperpa- rameters to optimize them for each experimental selup. We ensured that no selected hyperparameter was at the boundary of the grid. If this occurred, we expanded the grid size accordingly. All tuning was carried out using local validation times for the synthetic problem) with the optimized hyperparameters.
for each client. Each experiment was independently run three times (fiftoen
Table g presents the selected hyperparameters and other experimental details. Figure g illustrates the impact of in equation 4 on performance. Calibrating 7 appropriately is crucial for the effectivencss of hiillrity-baseed clustering, and the optimal value of t varies across different problems and methods.
A.2 Additional experimental results
In this section we present additional results. In Table 3 the results for the synthetic linear regression problem is presented for each cluster. Table shows the accuracies for each cluster in the domain shift problem. presents the results for all clusters in the covariate shift experiment with PaShion-MNIST. Table and summarizes the results on the label shift experiments for CIFAR-10.
In Figure we observe how the number of sampled clients affect accuracy the two-cluster (ani- mal/vehicles CIFAR-10 experiment. In this experiment have 200 training samples in each client, and total of 100 clients. Sampling lot of clients start give diminishing returns, but increases computational cost especially for inverse loss that has to perform forward passes on all sampled clients. Therefore it is important to tune this hyperparameter to ensure good trade-off between performance and computation.
57.5
55.0
Test accuracy
52.5
50.0
47.5
12
cosine cosine_grad
42.5
cosine weight
10.0
17.5
Number of sampled clients
Figure 6: Test accuracy vs number of sampled clients for all similarity metrics.
In Figure 7 heatmaps over client communication for the CIFAR-10 and Fashion-MNIST are presented.
18

---

Published in Transactions on Machine Learning Research (01/2025)
Sampling heatmape: How many
yeakes samnlled dlient x-axis
Clart Indee
Number SOFTEC
(a) CIFAR-10, 2 clusters, label shift.
hastmape: How many sampled dlient x-axis weight
Clarn 11CE
Chentindes
Numter
(b) CIFAR-10 10, 5 clusters, label shift.
Sampling
many grod
samplee
Dos vecgit
Numoe DIIORe
Dietk roler
Clent
Cleet
Chertirdes
60
(c) Fashion- MNIST, 4 clusters, covariate shift.
many Cos grod
client
the
Dos woggt
Number Simee STTOEe
Clart Bedex
Clett
Clant
Chartirdeo
(d) F/MNIST, clusters, domain shift.
Figure 7: Heatmaps of client communication, indicating how often client x communicated with client y for the four different similarity metrics.
Cluster
Cluster
Cluster 3
Mean
Baselines
FedAvg
FedSim
FedAvg
FedSim
FedAvg
FedSim
FedAvg
FedSim
Random
1196.04
1251.50
2036.90
1494.84
Oracle
9.54
9.29
9.46
9.43
Inv loss
32.91
14.87
30.40
14.58
31.77
15.01
31.69
14.82
Cos grad
10.19
10.40
10.32
10.22
10.44
10.29
10.32
10.30
Cos weight
10.29
10.37
10.25
10.41
10.48
10.13
10.34
10.30
L
21.97
10.85
19.77
10.92
21.56
10.78
21.10
10.85
Local
28.76
26.80
35.22
30.26
Table Test loss for each method on the synthetic linear regression problem.
19

---

Published in Transactions on Machine Learning Research (01/2025)
Cluster 1
Cluster 2
Mean
Baselines
FedAvg
FedSim
FedAvg
FedSim
FedAvg
FedSim
Random
97.01
84.05
90.53
Oracle
97.68
85.56
91.62
Inv loss
97.11
96.96
82.55
84.38
89.83
90.67
Cos grad
97.29
96.91
84.99
84.54
91.14
90.72
Cos weight L
97.22
96.72
84.83
84.52
91.03
90.62
97.05
97.04
83.92
84.56
90.49
90.80
Local
89.29
75.15
82.22
(a) CNN
Cluster 1
Cluster 2
Mean
Baselines
FedAvg
FedSim
FedAvg
FodSim
FodAvg
FedSim
Random
93.66
82.63
88.14
Oracle
95.38
84.36
89.87
Inv loss
94.19
94.73
83.96
82.02
89.07
88.38
Cos grad
94.79
94.54
83.46
82.96
89.13
88.75
Cos weight
95.15
95.18
83.39
82.86
89.27
89.02
L
94.83
95.32
81.73
82.22
88.28
88.77
Local
86.07
76.34
81.20
(b) MLP
Table 4: Test accuracy comparison of methods under domain shift with two clusters (MNIST and Fashion-
MNIST).
Cluster
Cluster
Cluster
Cluster 4
Mean
Basclines
FedAvg
PedSSi
FedAvg
FedSim
FedAvg
FedSim
FedAAg
FodSim
FedAvg
FedSim
Random
85.11
79.53
82.36
81.89
83.70
Oracle
85.19
82.60
85.55
Inv loes
85.49
85.51
83.73
84.72
82.44
82.64
82.31
83.02
84.83
85.08
Cos grad
85.61
85.32
83.68
77.02
82.57
82.11
81.59
81.85
84.87
83.32
Cos weight 1
85,81
85.24
83,50
84.67
82.74
82.34
82.35
82.13
85.04
84.82
85.54
85.37
80.17
80.87
82.66
82.21
82.43
82,26
84.17
84.16
Local
76.36
75.86
76.23
76.47
76.26
Table 5: Comparison of baselines across clusters and methods. Fashion-MNIST rotation
Cluster
Cluster
Mean
Baselines
FedAvg
FedSim
FedAvg
FedSim
FedAvg
FedSim
Random
52.88
69.94
59.71 71
Oracle
54.39
74.05
62.25
Inv loss
53.97
53.77
73.10
72.05
61.62
61.08
Cos grad
52.28
51.06
68.62
68.73
58.81
58.13
Cos weight
51.96
52.02
69.90
71.11
59.14
59.66
1
52.14
50.62
70.39
68.15
59.44
57.63
Local
30.14
49.21
37.77
Table 6: Test accuracy comparison of baselines across clusters and methods. Cifar-10, 2 clusters.
20

---

Published in Transactions on Machine Learning Research (01/2025)
Cluster
Cluster
Cluster
Cluster
Cluster
Mean
Baselines
FedAvg
FedSim
FodAvg
PedSim
FedAvg
FedSim
FedAvg
FedSim
FedAvg
FedSit
PedAvg
FedSim
Random
87.34
73.99
79.13
88.70
81.46
82.73
Oracle
91.66
78.64
83.25
92.2
91.06
87.36
Inv loss
88.1
90.86
77.83
76.77
82.14
81.36
88, 37
91.23
88, 26
88. 38.77
84.94
85.8
Cos grad
90.86
77.21
Cos weight
91.01
91.23
77.46
76.51
81.16
81.45
91.42
91.5
88.19
87.6
85.85
85.66
1.a
89.29
89.21
74.79
75.3
79.85
79.67
89.93
90.02
86.35
86.07
84.04
84.06
Local
82.31
66.93
74.07
83.84
78.25
77.08
Table 7: Test accuracy comparison of baselines across clusters and methods. Cifar-10, 5 clusters.
Validation accuracy Tau: CIFAR- 10 5 cluster label shift
Validation accuracy Tou: CIFAR-10 5 cluster label shift
91
---
Inv loss
90
Average validation accuraty
grad
Average validation accuracy
Cas weight
89
89
88
88
87
loss
Cos grad
86
Cos weight
10"
10
10a
10
10a
Tau
Tau
(a) CIFAR-10, 5 clusters, label shift (FedAvg).
(b) CIFAR-10, clusters, label shift (FedSim).
Validation accuracy vs. Tau: CIFAR-10 Animal/vehicle label shift
Validation accuracy vs. Tou: CIFAR-10 AnimalWehidle label shift
70
Average validation accuracy
Average validation accuracy
67
62
Inv loss
64
Inv loss +
60
grad
Cos weight
Cos weight
10P
10
10a
10a
10:
10a
(c) CIFAR-10, 2 clusters, label shift (FedAvg).
(d) CIFAR-10, 2 clusters, label shift (FedSim).
Figure& 8: Sensitivity analysis of how the choice of 7 effects performance for the different methods.
21

---

Published in Transactions on Machine Learning Research (01/2025)
Validation accuracy vs. Tau: Fashion MNIST MNIST Combined
Validation accuracy vs. Tau: Fashion MNIST MNIST Combined
94.5
loss
94.5
loss
Average validation accuracy
94.0
Cos grad
Average validation accurscy
94.0
Cos grad
weight
weight
93.5
93.5
92.5
92.5
92.0
92.0
91.5
91.5
10"
10a
10
10a
10a
10
10'
Tau
Tau
(a) F/MNIST, clusters, domain shift (FedAvg).
(b) CIFAR-10, 10, clusters, domain shift (FedSim).
Validation accuracy vs. Tau: Fashion-MNIST covariate shift
Validation accuracy vs. Tau: Fashion-MNIST covariate shift
89.0
89.5
Average validation accuracy
86.5
Average dalldatioo accuracy
89.0
88.0
80.5
87.0
Inv loss
Inv loss
06.5
87.0
86.0
Cos woight
Cos weight
10a
10a
10
10a
10
10a
10a
10a
10a
(c) Fashion-MNIST, clusters, covariate shift (Fe- dAvg).
(d) Fashion-MNIST, 4 clusters, covariate shift (Fed-
Sim).
Figure 9: Sensitivity analysis of how the choice of effects performance the different methods.
Validation accuracy vs. Tou: CIFAR-100 label shift
62
60
Average validatoo accuracy
58
56
loss
Cos weight
10a
10a
10'
Figure 10: Sensitivity analysis of how the choice of effects performance for the different methods the pre-trained CIFAR-100 experiment.
22

---

Published in Transactions on Machine Learning Research (01/2025)
Hyperparaneters
Experiment name
CIFAR- TO animal/
CIFATE cluster
Synthetic problem
Fashion- MNIST
Bushion-MNIS MNIST combined
CIPAR- TO0
vehicles label shift
label shift
covariate shift
label shift
Learning rate
0.003/0.008
0.001
No comm learning
0.0001
0.0001
0.008
10
-00
0.0001
No. of local epochs
1
1
1
1
1
No.
300
No. of clients
100
100
99
100
100
52
clients per cluster
40/60
20/20//20
/20220/20
33/33/33
70/20/5/5
50/50
26 1//33/13
Train data size per client
400
400
500
400
400
Validation data size per client
100
100
100
100
100
100
Early stopping rounds
No. of neighbors sampled
5
5
4
5
lahels
No. of runs per similarity Optimizer
3
3
15
3
3
3
Adam
Adam
Adam
Adam
Adam
Adam
Values
Iuv loss, FedAvg
10
10000
10
Cos grad, FedAvg
300
1000
140
2000
1
Cos weight, FedAvg
200
300
140
2000
5
30
1,2, FedAvg
30
30
10
30
Inv FedSim
5
5000
1
Cos grad, FedSim
200
200
140
300
5
Cos weight, FedSim
200
200
300
nA
L2. FedSim
5
10
30
30
Table 8: Summary of all experiment settings used, as well as the found optimal hyperparameters for each experiment.
23


Reviews:
[{"review_id": "mpZR59UpKv", "reviewer": "Reviewer_e9iP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. \n   The paper's objective of the formulation does not account for the fact that **heterogeneous data distributions** can make this local objective conflict with other clients’ objectives, particularly when \\(P_i(x, y) \\neq P_j(x, y)\\). Without explicit constraints or regularization terms in the formulation to align clients' objectives, **theoretical convergence of decentralized learning is not guaranteed**. A regularization term, as used in federated learning (e.g., penalizing the distance between local parameters \\(w_i\\) and a global parameter \\(w\\)), could help improve alignment. Thus, the soundness of assuming this formulation is suitable for decentralized non-IID settings may be questioned.\n\n2.\n   One similarity metric proposed in the paper is the inverse empirical loss:\n   \\[\n   s_{ik} = \\left( \\sum_{(x, y) \\in \\mathcal{D}_k} \\ell(w_i; x, y) \\right)^{-1}\n   \\]\n   There are concerns with this metric’s soundness:\n   - Noise Sensitivity: If \\(\\ell(w_i; x, y)\\) is small, \\(s_{ik}\\) could be excessively large, leading to **unstable client selection** due to the inverse relationship. Outliers in empirical loss values can disproportionately influence peer selection, causing clients to potentially choose suboptimal collaborators.\n   - Dependency on Data Quality and Size: \\(s_{ik}\\) depends on the dataset \\(\\mathcal{D}_k\\) used for evaluation. If \\(|\\mathcal{D}_k|\\) is small or not representative, then \\(s_{ik}\\) may not reliably reflect the true similarity between clients \\(i\\) and \\(k\\), leading to inaccurate cluster formation.\n   \n   For robustness, a **softmax scaling** or **bounded similarity metric** might improve stability, especially in low-data contexts.\n\n3. \n   The FedSim approach modifies the traditional FedAvg by weighting client updates based on similarity:\n   \\[\n   w^{t+1} = \\sum_{k=1}^m s_k w_k^t\n   \\]\n   Here, \\(s_k\\) represents a similarity weight. However:\n   -  The paper assumes normalization of \\(s_k\\) but does not analyze whether the weighted averaging preserves convergence properties (e.g., as in FedAvg with uniform weights). Without this analysis, there’s no guarantee that FedSim will converge to an optimal or even stable solution in non-IID settings.\n   - Impact of Misclassification in Similarity: FedSim may amplify errors if the similarity metrics misclassify clients, assigning high similarity to incompatible models, thereby affecting model quality. **Error propagation** in FedSim could be more severe than in FedAvg due to similarity-weighted averaging.\n\n4. \n   For selecting peers, the paper employs a softmax function with an inverse temperature parameter \\(\\tau\\):\n   \\[\n   p_i(s_i) = \\frac{e^{\\tau s_i}}{\\sum_{k=1}^K e^{\\tau s_{ik}}}\n   \\]\n   This choice raises several concerns:\n   - The effectiveness of similarity-based clustering is highly sensitive to \\(\\tau\\). For large \\(\\tau\\), the softmax approximates an \\(\\arg\\max\\) function, potentially causing instability if similarity estimates are noisy. For small \\(\\tau\\), similarity differences are minimized, reducing clustering effectiveness.\n   - The paper does not specify a mechanism to adapt \\(\\tau\\) across rounds, which could lead to **suboptimal peer selection** if data heterogeneity or the number of available clients varies over time. A theoretical study of optimal \\(\\tau\\) or an adaptive approach would improve the robustness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper titled \"On the Effects of Similarity Metrics in Decentralized Deep Learning Under Distributional Shift\" studies decentralized learning (DL) under heterogeneous client data distributions, aiming to enhance client collaboration without a central server and direct data sharing. Here’s a mathematical summary:\n\n   Each client \\(i\\) seeks to minimize an empirical risk over its dataset \\(\\mathcal{D}_i\\), represented as:\n   \\[\n   w_i^* = \\arg \\min_{w_i \\in \\mathbb{R}^d} \\mathbb{E}_{(x, y) \\sim \\mathcal{D}_i} \\left[ \\ell(f_i(w_i; x), y) \\right]\n   \\]\n   where \\(\\ell\\) is a local loss function, \\(f_i\\) the model, and \\(w_i\\) the parameters.\n\n   Clients in a fully decentralized network use similarity metrics to identify peers. Probability vector \\(p_i\\) for selecting a peer \\(j\\) is defined via:\n   \\[\n   p_i(s_i) = \\frac{e^{\\tau s_i}}{\\sum_{k=1}^K e^{\\tau s_{ik}}}\n   \\]\n   where \\(s_{ik}\\) is the similarity between clients \\(i\\) and \\(k\\), and \\(\\tau\\) controls selection sensitivity.\n\n   The paper evaluates four metrics, including:\n   - nverse Empirical Loss: \\(s_{ik} = \\left( \\sum_{(x, y) \\in \\mathcal{D}_k} \\ell(w_i; x, y) \\right)^{-1}\\)\n   - Inverse \\(L_2\\) Distance: \\(s_{ik} = \\|w_i - w_j\\|_2^{-1}\\)\n   - Cosine Similarity of Weights: \\(s_{ik} = \\frac{w_i \\cdot w_j}{\\|w_i\\| \\|w_j\\|}\\)\n   - Cosine Similarity of Gradients: \\(g_i^t = w_i^t - w_i^0\\) and \\(s_{ik} = \\frac{g_i \\cdot g_j}{\\|g_i\\| \\|g_j\\|}\\)\n\n   A proposed FedSim adjusts the FedAvg aggregation by weighing models by similarity:\n   \\[\n   w^{t+1} = \\sum_{k=1}^m s_k w_k^t\n   \\]\n   where \\(s_k\\) is the similarity of client \\(k\\), normalized to sum to one.\n   Testing these formulations on benchmark datasets (CIFAR-10, Fashion-MNIST, etc.), the study highlights that cosine similarity metrics consistently offer robust performance across scenarios with distributional shifts.\n\nThis formulation-based analysis highlights how different similarity metrics impact decentralized learning under non-IID conditions and supports the effectiveness of FedSim under complex, heterogeneous data environments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above discussion.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JSTrkYDQql", "reviewer": "Reviewer_nEmo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper evaluates the impact of 4 different similarity metrics under various non-IID distributions for decentralized adaptive clustering.\n2. Experiments are conducted on multiple small-scale datasets.\n3. The paper introduces a weighted averaging based on the clients' similarities referred to as FedSim.\n4. The paper is clearly written and is easy to follow.\n\nWeaknesses/questions:\n1. This line of work is similar to Federated Clustering Algorithms [1-4] but the related work or the baselines doesn't include any Federated Clustering Algorithms.\n2. Table 2 shows that FedSim doesn't have a significant benefit over FedAvg. \n3. For the domain shift, why did you consider two completely different tasks? The typical domain shift scenarios consider the same tasks such as MNIST and SVHN.\n4. For the label shift case where each cluster has a completely different set of classes, merging clients within the same cluster implies that the clients never learn about the classes present in other clusters. Is this intended behavior? In this case, the confusion matrix will be a better metric to report than accuracy.\n5. Even though averaging only requires each client to communicate with $m$ peers, the computation of similarity requires each client to communicate with all the clients increasing communication and compute overhead. \n6. For clarity: At the end of the training, there are $n$ unique models where $n$ is the number of clusters. Is the test accuracy averaged across all these models?\n7. For FedSim, if the similarity of a given client is computed with respect to all the clients then why is the model averaging restricted only to $m$ peers?  How does the performance change if we do a similarity-based weighted averaging by merging all the clients i.e., $\\sum_{k=1}^{K} s_k w_k^t$?\n\nReferences:\n1. Vardhan, Harsh, Avishek Ghosh, and Arya Mazumdar. \"An Improved Federated Clustering Algorithm with Model-based Clustering.\" Transactions on Machine Learning Research (2024).\n2. Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. IEEE Transactions on Information Theory, 68(12):8076–8091, 2022. shorter version in NeurIPS 2021.\n3. Yichen Ruan and Carlee Joe-Wong. Fedsoft: Soft clustered federated learning with proximal local updating. CoRR, abs/2112.06053, 2021.\n4. Duan, Moming, et al. \"Fedgroup: Efficient federated learning via decomposed similarity-based clustering.\" 2021 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom). IEEE, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on a decentralized adaptive clustering framework where $K$ clients train their local model for $E$ epochs and then each client merges its model with $m$ chosen peers through federated averaging. A random peer selection strategy works best when the data distribution across clients is IID. In the case of non-IID data, the convergence and performance of the model heavily depend on the peer selection. The main goal of this paper is to evaluate four different similarity metrics used for peer selection under a decentralized adaptive clustering framework. In particular, the paper considers the following four similarity metrics: 1. empirical loss, 2. cosine similarity on gradients, 3. cosine similarity on model weights, and 4. $L_2$ distance on model weights. The paper then presents FedSim which merges client models using a weighted average based on their similarities. The paper considers 4 different non-IID settings for the evaluation namely covariance shift, label shift, concept shift, and domain shift.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please add a pseudo code for the chosen setup. It will help the readers to better understand the decentralized adaptive clustering framework.\n2. Do the clients need to compute the similarity at every communication round? Or can this be reduced to every $h$ round? It would be interesting to evaluate how the performance changes are we reduce the frequency of similarity evaluation (cluster reassignment ). \n3. The paper deals with non-IID data but the experiments on conducted on the FedAvg algorithm which is meant for IID data. If the goal of the paper is to understand the impact of similarity metrics in non-IID settings then the analysis needs to extend beyond FedAvg. Consider decentralized learning algorithms meant for non-IID data such as QGM [1], NGM [2], etc. \n4. Please improve the related works section by adding the literature on Federated Clustering Algorithms and Decentralized Learning Algorithms for heterogeneous data. Additionally, [3] might be relevant as it focuses on Data-heterogeneity-aware mixing.\n5. It is useful to note the overheads of each metric. For example, inverse empirical loss evaluation requires additional $K$ forward passes through the model on the entire training dataset which is not feasible in a resource-constrained setting. \n\nReferences:\n1. Lin, Tao, et al. \"Quasi-global momentum: Accelerating decentralized deep learning on heterogeneous data.\" arXiv preprint arXiv:2102.04761 (2021).\n2. Aketi, Sai Aparna, Sangamesh Kodge, and Kaushik Roy. \"Neighborhood Gradient Mean: An efficient decentralized learning method for non-iid data.\" Transactions on Machine Learning Research (2023).\n3. Dandi, Yatin, et al. \"Data-heterogeneity-aware mixing for decentralized learning.\" arXiv preprint arXiv:2204.06477 (2022).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LKeHmEBYRs", "reviewer": "Reviewer_GnUc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n - The scope of the paper is clear\n - The experiments include several distribution shifts\n - A conclusion is drawn for each metric.\n\nWeaknesses:\n - The difference between metrics is marginal, yet the deviation of the experiments is not highlighted.\n - All datasets are derived from traditional ML datasets.\n - Lack of building on related research", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work is an empirical study using different distance metrics to identify similar clients for decentralized learning in non-IID data. The study investigates 4 metrics under 4 distribution shifts on 4 basic ML and a synthetic dataset distributed in a way common to literature.\n\nThe studied experiments have 2 steps: first, client selection: the clients select peers to help them build a better model. Secondly, the model updates from the peers are weighted. The client selection is based on the 4 similarity metrics. The weighting is either simple FedAvg (weighted by the dataset size) or FedSim, a weighted averaging method proposed by the study that considers the similarity.\n\nThe experimental results in summary: the choice of the metric is important. The effect of the new averaging technique is marginal.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Losses: 4 basic losses were used, I would recommend including in the discussion and the study optimal transport and Wasserstein distance as well.\n2. Clustering: hierarchical clustering is well-studied in Federated Learning eg. [1]. The current study's literature review also mentions works for client selection. I recommend implementing at least one of these and investigating the effect of changing the distance metric behind it.\n3. If someone studies non-IID data, I recommend using at least one more realistically decentralized dataset. See for example [2]\n4. The choice of label pairs in CIFAR-100 impacts the experiment highly, it shouldn't be random, see [3].\n5. The difference between the results in Table 2 is very small, I recommend adding the standard deviation of the 3 (15) runs.\n6. Table 2 results should be in order with the subsections of the 4. Results section.\n7. I think adding a discussion about the number of peers selected by clients (related to tau) would be good.\n\n\n\n[1] Briggs, C., Fan, Z., & Andras, P. (2020, July). Federated learning with hierarchical clustering of local updates to improve training on non-IID data. In 2020 international joint conference on neural networks (IJCNN) (pp. 1-9). IEEE.\n\n[2] Caldas, S., Duddu, S. M. K., Wu, P., Li, T., Konečný, J., McMahan, H. B., ... & Talwalkar, A. (2019). Leaf: A benchmark for federated settings. In Workshop on Federated Learning for Data Privacy and Confidentiality\n\n[3] Vahidian, S., Morafah, M., Chen, C., Shah, M., & Lin, B. (2023). Rethinking data heterogeneity in federated learning: Introducing a new notion and standard benchmarks. IEEE Transactions on Artificial Intelligence, 5(3), 1386-1397.", "broader_impact_concerns": "I do not have concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers’ opinions on this paper are divided.

One reviewer argued for acceptance, while two reviewers raised concerns about the limited scope of the empirical evaluation. Specifically, they noted that the synthetic scenarios (e.g., distribution shifts and network topology) address only a narrow range of cases. They also emphasized that, given the established nature of personalized learning, additional benchmarks would have been necessary to better assess the potential impact of this work.

Despite these concerns, I believe that the submission meets TMLR's acceptance criteria. The paper offers a focused contribution to the field of personalized learning, and I therefore recommend acceptance.


---
Minor revision:
- In the newly added Section 3.1, the computational cost of the metrics is discussed. However, it is unclear why stochastic quantities are being compared to deterministic measures. For example, cosine similarity could also be computed using full gradients, while the empirical loss could be stochastically estimated. Please clarify this discussion for the final version.
Decision: Accept (poster)