Collaborative Learning by Detecting Collaboration Partners
Authors: Shu Ding
Wei Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Collaborative learning

Abstract: Massive amounts of data are naturally dispersed over different clients in many real-world applications, collaborative learning has been a promising paradigm that allows to learn models through collaboration among the clients.   However, leveraging these dispersed data to learn good models is still challenging since data over different clients are heterogeneous.   Previous works mainly focus on learning the centralized model for all clients or learning a personalized model for each client.  When there are numerous clients, the centralized model performs badly on some clients, while learning a personalized model for each client costs unaffordable computational resources.  In this paper, we propose the collaborative learning method to detect collaboration partners and adaptively learn $K$ models for numerous heterogeneous clients.  We theoretically prove that the model learned for each client is a good approximation of its personalized model.  Experimental results on real-world datasets verify the effectiveness of our method.

Collaborative Learning by Detecting Collaboration Partners
Shu Ding, Wei Wang
National Key Laboratory for Novel Software Technology Nanjing University, Nanjing 210023, China {dings, vangw)@lamdee .nju edu. cn
Abstract
Massive amounts data are naturally dispersed over different clients in many real-world applications, collaborative learning has been a promising paradigm that allows to learn models through collaboration among the clients. However, leveraging these dispersed data to learn good models is still challenging since data over different clients are heterogeneous. Previous works mainly focus on learning the centralized model for all clients or learning personalized model for each client. When there are numerous clients, the centralized model performs badly on some clients, while learning personalized model for each client costs unaffordable computational resources. In this paper, we propose the collaborative learning method detect collaboration partners and adaptively learn K models for numerous heterogeneous clients. We theoretically prove that the model learned for each client good approximation of its personalized model. Experimental results on real-world datasets verify the effectiveness of our method.
Introduction
In many real-world applications, massive amounts of data are naturally dispersed over numerous clients and each client only has limited data. The model trained with local data on each client has poor performance since local data are insufficient. Leveraging these dispersed data on these clients to build well well-generalized models challenging. recent years, collaborative learning has been a promising paradigm that enables the clients collaboratively learn the models (Blum et al, 2017). Blum et al. (2117) formalized two settings of collaborative learning. i.e., personalized setting and centralized setting. The personalized setting allows return model for each client, while the centralized setting only allows to return one single model for all clients. They evaluated the efficiency collaborative learning with overhead, i.e., the ratio between the sample complexity of collaborative learning and its non-collaborative counterpart, and proved that the overhead of personalized setting is O N) and the overhead of centralized setting is O N). Later, Nguyen & Zakynthinou (018)) and Chen et al. (2018) proved that the overhead of centralized setting could be improved O(In N), which matches its lower bound 22(n N).
Generally, the data different clients come from heterogeneous distributions. For example, in the credit card fraud behavior detection task, the users' revenue and expenditure behavior and credit rating information To deal with heterogeneous data, Mohri et al. (2019) proposed Agnostic Federated Learning which optimizes the performance of the model for the single worst client with the min optimization scheme. Mansour et al. (220)) and Deng et al. (2020) advocated that combining local and global models could improve the generalization ability. Nevertheless, the learned single model may perform badly on clients whose distributions are different from the average distribution over all clients.
different among banks in regions with diverse socioeconomic characteristics.
Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

To achieve good performance, a personalized model should be learned for each client, but learning these personalized models is impractical when the number of clients N is very large since this costs unaffordable computational resources. Intuitively, it's better to return K (1 & K & N) appropriate models by analyzing the collaboration relationships among N heterogeneous clients. We expect that the performance of the returned model is comparable to that of each client's optimal personalized model. Based on this intuition, in this paper we propose the collaborative learning method to detect collaboration partners for each client and adaptively learn K models for N heterogeneous clients. We first estimate each client's contribution based on the derived generalization bound, and then partition the clients into K groups and return a model for each group. We also theoretically prove that the model returned for each client a good approximation of its optimal personalized model. Experimental results real-world datasets verify the effectiveness of our method.
2 Related Work
Blum et al. (017)) first theoretically studied collaborative learning and defined overhead as the ratio
between the sample complexity of collaborative learning and its non-collaborative counterpart. They
proved that the overhead of personalized setting is O(nNN) and the overhead of centralized setting is O(n2 N). Later, Nguyen & Zakynthinou (2018) and Chen et al. (018 proved that the overhead of centralized setting could be improved to OIn N) by using multiplicative weights methods, which matches its lower bound [[nn N)). When there are an fraction of malicious clients, Qiao (2018) proposed an algorithm that achieves Onn + In N) overhead, which is proved to be the worst-case optimal. Since the communication cost between clients and the central server is expensive, [Blmm] al. (20TT) developed algorithms by using distributed boosting to reduce the communication cost without compromising the sample complexity.
To deal with data heterogeneity. Mohri et al. (2019) proposed Agnostic Federated Learning, which optimizes the performance of the model for the single worst client with the min-max optimization
scheme. Mansour al. (2020) proved that one single model generalizes badly on the clients whose
distributions vary from the average distribution. Mansour et al.] (2020) and Deng et al. 2020)
advocated that the mixture of local and global models could improve the generalization ability.
et al.] (2020) highlighted the detrimental effects of heterogeneity by observing that heterogeneity
leads to poor convergence rate. They proposed FedProx to tackle heterogeneity and provided
convergence guarantee. |Wang et al. (2020b) proposed the normalized averaging method FedNova to eliminate objective inconsistency while preserving fast convergence rate. Dinh et al. (2220) proposed a personalized algorithm PFedMe by using Moreau envelops as regularized loss functions and theoretically showed it could obtain quadratic speedup. Huang al. (2020) proposed a novel framework FEDAMP which allows each client to collaboratively train its own model without using the global model and provided the convergence guarantee. These works mainly focus on learning the global model which performs well on the average distribution. However, it may perform badly concurrent work on the clients whose distributions are different from the average distribution.
(Even et al. 2022), the authors proposed methods to achieve the tradeoff between personalization and coordination, which tackles the same problem in our paper from different perspective.
Client selection has also been studied in some works. Considering that different groups of clients may have different learning objectives. Ghosh et al. (020) proposed IFCA which alternately estimates the cluster identities of the clients and optimizes the models for each cluster. Fraboni et al. 202T) introduced clustered sampling with better client representativity to reduced variance of the clients stochastic aggregation weights and provided two cluster approaches based on sample size and model similarity. [Toor et al. (2020) proposed method for selecting relevant data for model training, where they used the model trained on the benchmark dataset which is task-specific to evaluate the data relevance. Cui et al. (2022) studied a rational collaboration scheme called collaboration equilibrium where each client collaborates with certain members who maximally improve the model learning and isolates the others who make little contribution. However, these works are heuristic and lack
thcoretical support.
3 Preliminaries
Let X' be the instance space and y (0, is the label space. In the collaborative learning setting, there are N clients {CI...., Cx) } with distributions (Di...., DN J. In practice, client C, only has
2

---

access to m. examples S; = { (2). .....
} drawn from distribution D,, 1r.i=1,.... ...
Let M = -'=,, m, be the total number of examples. We consider the hypothesis space H with VC dimension d. A hypothesis h E H is a mapping h X' + V. Let / be the loss function and ((h(z),)) 3) the loss of hypothesis
H on example (x, v). We assume that is upper bounded by constant μ.. For client C₁, denote Lp. (h) the expected risk of hypothesis h H with respect to distribution D; and Ls, (h) is the empirical risk of hypothesis
H with respect sample S;
(1) = E [((((r))))) u),, Ls. = [(((()))))))
(a.v)>>;
For client C.. its goal to pursue the model h that minimizes the expected risk on its local distribution D,, ie., hū arg minGen LD. (h). Nevertheless, it is impractical to find hद since C, only has access to limited m, examples. Thus the key incentive of clients C₁ participating in collaboration is the desire of seeking the model that could generalize well on its local distribution D; with samples {St....,S4 from all the clients.
Our Work
In collaborative learning scenarios, the common strategy is to train the model over the weighted union of all samples, i.e., Sa = "=== a5Sz (McMahan et al. 2017 Mohri et al. 2019p. Here a (a1....,aN) E AN where AN is the N N-simplex, i.e., Σj=1 ag = 1 and aj > for = L...., The weighted empirical risk La (h) associated to hypothesis h e H. can be denoted as (h) = E/= az Ls, (h). In this way, the weighted empirical risk La, (h) for client C₁ can be denoted as (h) = Σك= aijãs,; (h) with the corresponding collaboration vector a; (aat...., oIN). The model for C₁ can be learned by minimizing La, (h). Let haa = arg min La, (h) be the hypothesis that minimizes the weighted empirical risk La, (h). We provide the generalization bound of ha, for client C₁ in the following Theorem
Theorem 1. Let H. be the hypothesis space with VC-dimension Let h = arg minKER CD, (h) be the hypothesis that minimizes the expected risk on Dj and hag = arg minnen La, (h) is the hypothesis that minimizes the weighted empirical risk. For any given ō E (0, 1) and vi E {1, (1....,N N). the following bound holds with probability at least 1 - 8:
N
N Σ m2
8 ((dlog(2M) + log 8))
LD. - -Lo &
Σ
(D; + 2/L 5=1
(1)
Here dD(D,, suPREM (h) LD, (h) is the Integral Probability Metrics (IPM) which measures the divergence between distributions Dj and Dj.
We leave the proof of Theorem [ in Appendix A due to space limitation. Theorem I reveals that for client C; the gap between ha, and h& depends on the weight a, the number of examples m. and the IPM divergence. further suggests that the optimal model for client C, can be obtained by choosing appropriate a to minimize the right-hand side of Eq. 1). The optimal aj (afi,..., for client C₂ can be derived with the following Theorem
Theorem For client C, let E dD,,,, where E [I.....,N, N}. Vi NJ. Given VC dimension d, total sample size M, constants i and ö, let X = HV 8(d (2M) + log 8) For client
Cr sort 1 1E....., ENN in ascending order get (=/ ,, "...,,,((N), [N)) iee,, =²(1) & ... < < (N(N)
where o (i) E (1...., N). The optimal a = = OiN for client Cj is given by
(in)
my(S-E) (((--))) E()
o7
Egen. mo(g) (S - =
Here [1+ =maz(.000 0). Ç is the larger root of equation Enso ma(a) (5-= = X2, and q: =
arg max { > =7(() ^ (Eose ((0)) > "Egsmoosy (Enge (E (E x2) x²)} }
3

---

We leave the proof of Theorem in Appendix B. Theorem 2 can be interpreted as follows. The optimal a makes the balance between the divergence and the number of examples. If of > then Cj is beneficial to C; in the collaboration process, i.e., C can utilize the data on Cj to relieve the scareity of its local data. ot represents the contribution that client C2 makes to C. We use the directed graph A = (V, E) to represent the collaboration relationships among clients, where I NN node i denotes client C; and the weight of edge from j to is oig. Here we provide an example Figure to give a more concrete interpretation.
there exists an edge from node j to node then C, needs the data from Cj. In Figure the contribution that C2 and C1 need from all the elients are more similar than that of 1 and Cs Le., the incoming edges of C2 and C1 are more similar than that of C₁ and Cg. We could probably return the same model for C2 and C4 while it is inappropriate to return the same model for C4 and Cs.
Figure 1: Illustrative example of the directed graph A. Node i denotes client C, and the weight of edge from j to is aty.
We refer ha; with respect to the optimal o as the personalized model for client C. When the number of clients N is large, learning personalized model for each client is impractical since this costs unaffordable computational resources. An alternate way is to learn K (1 & K < N) appropriate models which are expected to be comparable to the personalized models. Intuitively, if client C₁ and C, have similar incoming edges, then we can return the same model for them since they need similar contribution from other clients. In this way, C₁ and Cj are called collaboration partners. This motivates us to partition N clients into K groups based on the idea that collaboration partners should be in the same group. If we set K 2 in Figure 1 then the group structure is most likely to be (Ci, C2, Cs, Ca} and (Cs, Co C7, Cs} as shown in Higure
4.1 Collaboration with Modularity Maximization
To evaluate the incoming-edge similarity among clients, we construct matrix U which is defined as = D in where A is the adjacency matrix of the directed graph A, Din is the diagonal matrix of in-degrees in graph A and /3 is the discounting parameter. We set 1/ as suggested in Satuluri & Parthasarathy 201T.. Large UNj represents large similarity between clients, which provides the possibility of detecting the communities for the clients and partitioning collaboration partners into groups.
AATD. in
Modularity has been used as an objective function to evaluate the quality of group partitions by measuring the density of intra- group edges compared to inter-group edges (Newman & Girvan 2004), which is formulated as follows:
Q(G) W Σ W.G.
1
Σ (W C
4W2
Here G = .GI, Gx) } is the group partition. W & Wij is the sum of all edge weight.
W.C = Ei,jee, Wij is the total edge weight within group GR and WGG = E, wij is the total edge weight attached to nodes in group G5- Higher modularity implies better group partitions and modularity maximization has been widely used for detecting meaningful group partitions (Clauset jt al., 2004; Newman 2006a b). Although maximizing modualarity is Na-hard, approximation guarantees be obtained by introducing the technique of rounding semidefinite programs. We rewrite Modularity as follows:
1 2W
d,d 2W
Q(9)
wij
8(g. (,,.)))
(5)

---

where d, = E; Wiy is the degree of vertex i, and 9i is the group membership of client Cj where O(gi.gg) (2) 1 gi = 94 (i.e., client C₁ C2 belong to the same group) and otherwise. Let e; I k be the unit vector with 1 in the j-th coordinate and 0 elsewhere. Let be the variable that indicates the group membership of client C₁, i.e., V3 e, means C, belongs to group Gj. Then the modularity maximization problem can be formulated as the following vector programming:
E Wij d,d;
max
4W2 V VV)
(6)
2W
s.t. Us fei fei...... ext. Vi [1....,N..
Let M denote the modularity matrix with its elements My {(i,j) | My 0} and _M- = {(i.j) Mi < 0}. Noticing that Eij Myj = 0, the above vector programming can be relaxed to the following SemiDefinite Program (SDP) by adding a constant Em+ Mij to the objective and relaxing the constraints. The proof can be found in Appendix C.
wy
Aw Denote M
d.d.
2W
max
Misu Vj +
Σ
-Mi (1 1s
v2))) 12))
MI
M
8.1.
v =1, Vi (I,...,, N}.
V V. 7 O, Vi #i*1 J.
V1 KERK Vi [1, (L....,N3 NJ.
Let OPT QC)) QPT be the maximum value of modularity defined on the similarity matrix U. Dinh et al. (201)) provided the following result to guarantee the approximation property for the solution of the
SDP in Eq.77 Lemma limplies that we could find reasonable group partitions by solving the SDP
Eq. for maximizing modularity on U.
Lemma 1. Given matrix U. let Q()) be the modularity value of the group partition G obtained by solving the SDP in Eq. 7 using the rounding techniques in Charikar et al. (200)). Then Q(G)
satisfies Q(G)= > kOPT Q(g)- (1 (1-k) k). where 0.766 is the approximation factor.
After obtaining the group partitions, we train models for the groups according to their properties. Since we assign the clients into K groups, there may exist some clients which have small similarity with other clients in the same group. We should pay attention these clients.
We call edge EGj E U the weak edge if its weight satisfies Wi; < 1.- else strong edge otherwise. Given group partition G = [G1, Gx }. let Zin be the number of weak edges within all groups and N₂ is the number of clients group G3 where Σ N₂ = N. For client C2 in group Gk. we call C; the bad client if C; is isolated after removing all the weak edges within group G2. Generally, we keep the largest part when group is divided into several disjoint parts after removing all weak edges within this group, and clients that do not belong to the largest part are bad clients. Otherwise, we call it the good client.
Figure[ C1, C2. C3, C2 are the same group since they have relatively similar incoming edges, and Cg Co, C7 are in the same group for the same reason. Cs is most likely to be a bad client since it has less similar incoming edges with Cs, Co, C7 in the same group.
To guarantee the performance of the learned models, we pick out the bad clients since they have small similarity with other clients in the same group. We return haas for clients in group G& which is obtained by minimizing Lac, (h) where aG., G Eg,e,, O². We refer this method as Adaptive Collaborative Learning with Modularity Maximization (ACLMM). The whole process summarized Algorithm T where we use the rounding techniques introduced in Charikar et al. (003) to solve the SDP in Eq. for modularity maximization to obtain the group partition. Let B be the set of bad clients in the group partition returned by solving the SDP. We provide the following Lemma to bound the number of bad clients.
Lemma 2. Given the group partition G =
7 assume N > K}. Let Nmin = minH No. then [B] satisfies |B] < Nmin VNE -42.n
GK} returned by Algoritimm
2 Vk E {1,..... where Zin & 2(N-1)
N
NN_KN 2W ((K+ 1)OPT K-1) and 0.766 is constant.
K
The proof of Lemma 2 can be found in Appendix C. Lemma 2 indicates that the number of weak edges within all groups Zin depends on the maximum modularity OPTQ()) and the number of
5

---

Algorithm Adaptive Collaborative Learning with Modularity Maximization (ACLMM)
1: procedure TRAIN(K, A)
2: Compute similarity matrix U according to Eq.(3
3: Obtain G {G1,... .., Gx) with DIVIDE(K, U).
4: Pick out bad clients into set
5:
Compute OG,, G EC o for Gk- Train model h hack by minimizing Lacy (l)) and return hac, to all clients C in group Gk-
6:
7:
end procedure
8: function DIVIDE U)
9: Pick t random hyperplanes and use projection to divide the nodes into 2 groups.
10: Take the better solution of t = and t = 3 with respect to the SDP relaxation Eq. 7
return Group partition G .G G.].
12: end function
groups K. If each group Gj is not too small, i.e., N& > Zv Ak E { K). then the number of bad clients [B] is upper bounded by the number of weak edges.
We further theoretically analyze the generalization bound of the models returned for the clients. Given group partition g = [G..... Gx } returned by solving the SDP, we remove all the weak edges within all groups. For any good clients C, and Cj in the same group, there always exits shortest path Pij between them since good clients in the same group are connected through strong edges. Let 1 denote the maximum Púj between any good clients C and Cj group Gs. Ak (I...., K}. Denote e E F.., Wij where Ein is the set of strong edges within all groups. We provide Theorem to show that the model returned by Algorithm for each good client is an approximation of its personalized model.
Theorem 3. Let 9 (G1, Gk} ) be the group partition returned by solving the SDP hacy is the model returned by Algorithm D for client Ci in group Gs. Let upp(hace) ) be the upper bound of the expected risk of hao, on C₁ according to the right-hand side of Eq.el and upp(ha;) is the upper bound of the expected risk of the personalized model h bag on C2 according to the right-hand side of The following result holds except for the bad clients in B:
Eq. 47.
( (m0 1111-Ty-
N
upp(hac,) upp((a;)) <0
We leave the proof of Theorem S in Appendix due to space limitation. Theorem reveals that the generalization gap between the model returned by Algorithm for any good client and its personalized model relics on the minimum weight of strong edges 7 within all groups and the maximum length of the shortest paths between good clients in the same group 7). These are further determined by the intrinsic properties of with respect to modularity and the number groups K.
4.2 Collaboration with Clustering
Suppose we have the prior knowledge about the collaboration partnerships among the clients, i.e., there exists a potential partition P = [Pi..-. } In this case, we try to develop the method with tighter bound for the approximated models. We use (PP) to evaluate the quality of partitions which defined ((P)= -EK Ec,er, d d(a,t ār). where
.P,..... Pk) } the group partition, d(-s-) is the distance measure and Qt the center of group Ps. If partition is of high quality, then & (P) is small. Take Figure 2 for example where K = 2, the value & of the graph A in Figure is smaller than that in Figure since clients in the same group in Figure have more similar incoming edges. Thus the example in Figure 2 has better structures than that in Figure
minp ((P) be the minimum value of ((P) over all group partitions and Pt (Pi (P....., } is the corresponding optimal group partition. The distance between two group partitions P = (PI...., Pk} and p' = .P.....,kk)
Let OPT.;P
denoted as dist (P, P') mingesx N Σ I(Ps = -Po))) where 6 is the set of permutations, Balcan et al. 2009 proposed the (1 7 eyapproximation property to show that if given group partition

---

P satisfies (P) < (1 + T then we have dist(P, & This provides way to find good group partitions by optimizing (P)). We assume that {ai,..., aiyy satisfy the (1 + e--aproximation-stallliy property with respect (PP).
Figure 2: An example the directed graph A with potential structures.
In this case, there still exist some bad clients and we need to find them out. Let di (a;) ming d ( be the distance of a to its closest group center and dz(ax;) miny44 d (a) az) the distance of o to its second-closest group center. Denote d = N OPT as the average distance of clients. Given any constant where > 2, let d' ad
be the critical distance. We call C₁ the bad client if di (at) > d or dz(a - d1 (at) ) & _dd, else we call it the good client. It is easy to understand that good clients are those tightly coupled to clients in the same group and loosely coupled to clients in different groups
er
To guarantee the performance the learned models, we pick out the bad clients. We return hap, for clients in group Ps which is obtained by minimizing Eis (h) where px PAi EcERAA of We refer this method as Adaptive Collaborative Learning with Clustering (ACLC). The whole process is summarized in Algorithm 2 where 3 > and > 2 are given constants. We use the method in Balcan et al. (2009) to obtain the group partition, where the constant-factor K K-median approximation algorithm is done as that in Arya et al. (200)). Let B be the set of bad clients. We provide the following Lemma to bound the number bad clients [B] in the group partition P produced by Algorithm
property. Let P = (PI...., Px) be the group partition produced by Algorithm 0 Then the number of bad clients [B] satisfies [B] (6 + 4)BNN where t. and B > 1 are given constants.
Lemma 3. Assume that (aii...., o.yy satisfy the (1 +% e)apppoximationnttaii
The proof of Lemma B can be found in Appendix D. Lemma Cermma@reveate reveals that the number of bad clients related 7 and e, which reveals the degree of difficulty in finding approximate group partitions. We further theoretically analyze the generalization bound of the models returned for the clients in the following Theorem which shows that the model returned by Algorithm 2 for each good client approximation of its personalized model.
Theorem 4. Assume that (att...,, .'y} satisfy the (1 + D-Japroximation-sttbbiii property. Let P .I,,..., Pk} be the group partition produced by Algorithm hare is the model returned by Algorithm 2 for client Cs in group Pk. Let upp(hap.) be the upper bound of the expected risk of har, on O₁ according to the right-hand side of Eq. and upp(ha;) is the upper bound of the expected risk of the personalized model bay on C₂ according to the right- hand side of Eq. Then given any constants t >> 2 and 3 > 1, the following result holds except for the bad clients in B:
OOPT ((TT))) etN
upp(har,) -upp(ha;) <
We leave the proof of Theorem in Appendix D due to space limitation. Theorem reveals that the generalization gap between the model returned by Algorithm 2 for any good client and its personalized model relies on the potential structures of the clients' collaboration relationships, i.e., the optimal value of objective ((P). If the collaboration relationships have good structures, iee, OPTa(p) small, then the returned model for any good client is close to its personalized model.
7

---

Algorithm 2 Adaptive Collaborative Learning with Clustering (ACLC)
1: procedure TRAIN(K, A, constants c, Y Y,
2: Obtain p {Pi,..., .r.,, and d with DIVIDE(K, 6,
3: Compute the critical distance d'
nd
Pick out bad clients into set B.
Ste"
4: Compute P; PA Ec,EPAAB o for Pk.
5:
Train model han, by minimizing Lar, (h) and return har, to all clients C, in group Pk.
6: end procedure
7: function DIVIDE(K, e, 7,4,,)
8: Run the constant- factan K-median approximation algorithm on fai..... atyy to compute
(P).
9: Compute the value d =
(P) 3d.
10:
Construct the threshold graph G which connects all pairs (Ci, Cj) with d(a;; 0z) )
<
2nd
Dite
11:
for to K do
12:
Let os be the node with the highest degree in G. Select U; and its neighborhoods to get group Pk. Remove Pj from G. end for
13:
14:
15:
return P = (PI...., Pk} and d, where Pk = {at, .Ny (UK=)PP)
end function
5 Experiments
In the experiments, we use the following four datasets: (1) The MNIST dataset consists of 70K handwritten digit images in 10 classes, which has training set of 60K examples and a test set of 10K examples. (2) The CIFAR- 10/100 dataset consists of 60K color images in 10/100 classes, with 6000/600 images per class. They have 50K training images and 10K test images. (3) The Federated Extended MNIST (FEMNIST) dataset is built by partitioning the data in Extended MNIST based on writer of the digit/character. For MNIST, we use the same CNN model as that in McMahan et al. (017): for CIFAR- 100, we use the CNN model which consists of 2 convolutional layers followed by fully connected layers as that in Ghosh et al. (020); for CIFAR- 100, we use the ResNet- 18 model (He et al. 2016); for FEMNIST,
the same CNN model as that in LEAF (Caldas et al. 2018). In collaborative learning, using KL divergence to estimate the divergence between distributions is a
common practice (He al. 2020 Yang 2021 We use the right hand of Eq to estimate the IPM du(D,, in the experiments since
du(D,, < min { 1 dis(D.))D))
das dDs(
(8)
where das (D.))),))) is the Sensen-Shannee (JS) divergence (symmetrical KL divergence). The proof can be found in Appendix E.
First, we partition MNIST into clients using the same method as that in McMahan et al.
(017)) and
set client number N 10 where each client has 3 classes, total number of samples M
2000. We
partition CIFAR- 0/ 000 using the Dirichlet partition as that in Yurochkin et al. 2019) and Wang et al. 2020ab. We set N
and M 5000 for CIFAR- and set N 50 and
20000 for
CIFAR- 100. For FEMNIST, we use the default partition proposed in LEAF and choose N 20 clients randomly from the total 3550 clients (Caldas et al. 2018). We further set the number of groups K = 3 for MNIST, K 4 for CIFAR-I0, 10, K for CIFAR- 100 and K 4 for FEMNIST.
We divide N clients into groups and return model for each group according Algorithm I on the three datasets respectively, which is denoted as ACLMM. We use the centralized model as the baseline, where only one single model trained according to a;
A m, Vi E [[....,N] as that in McMahan et al. (2177). We also implement the personalized model, where each client will obtain the personalized model according to their o Since the client number on CIFAR- 100 is relatively large, we only perform the personulized model on MNIST, CIFAR-10 10 and FEMNIST. The results on four datasets are depicted in Figure From Figure g we find that the model learned with our ACLMM performs better than the centralized model and is comparable to the personalized model.

---

Possdlace
Penceioed
Clart Inde
(a) MNIST
(b) FEMNIST
Cerreland
ACUIMN
(c) CIFAR-10
(d) CIFAR-100
Figure 3: Comparisons of ACLMM with Personalized and Centralized models.
In order to verify the collaboration partners detected by our ACLMM, we take CIFAR-10 10 for example and visualize the group partition returned by ACLMM in Figure 4(a) where nodes with the same color are in the same group and thicker edge means larger similarity between clients. We also visualize the similarity matrix U color map in Figure 4(b where the (i,j)-th block characterizes the similarity between C; and Cj (the lighter the color is, the smaller the similarity is). Obviously,
clients in the group (having the same node color) in Figure 4(a) are those with large similarities
in Figure 4(b) which means they are collaboration partners.
0.20
030
20 1933
-0.20
Figure 4: (a) The group partition of CIFAR-10 based on ACLMM. (b) The color map associated to U CIFAR-10.
Next. we construct the clients that have intrinsic structures on MNIST and CIFAR-10/100 in the following way. For MNIST, we N 10 with K The first group contains clients with samples dominated by digit {0. 1, 2}. The second group contains 3 clients with samples dominated by digit {3, 4, 5). The third group contains 4 clients with samples dominated by digit {6. 7, 8. 9}. The number of samples on the clients satisfies the Dirichlet distribution as that in Yurochkin et al. (2099), and Wang et al. (2020a). We analogously set
20 with K = 4 for CIFAR-I0 50 with K = 7 for CIFAR-100 We set the total number of sample M 2000 2000/5000/0000 /2000 ANIST/CIFFA-IOOCIFAAR 100 respectively. Since the partition of FEMNIST have ambiguous clustering structures GGosh al. 2020), did not run clustering experiments FEMNIST.
set

---

K2C
(a) MNIST
(b) CIFAR-10
(c) CIFAR-100
Figure 5: Comparisons of ACLC with Personalized and Centralized models.
We divide N clients into groups and return model for each group according to Algorithm 2 on the three datasets respectively, which is denoted ACLC. We also implement the centralized model and the personalized model. We only perform the personalized model on MNIST and CIFAR-10 10. The results three datasets are depicted in Figure S From Figure 5 we can find that the model learned with our ACLC performs much better than centralized model and is comparable to personalized model. Furthermore, the gap between the model return by ACLC and the personalized model is small, which shows that our bound in Theorem is tight when the collaboration relationships have intrinsic clustering structures.
Conclusion
In this paper, propose the adaptive collaborative learning method to deal with heterogeneous data on the clients, which can detect the collaboration partners from numerous heterogeneous clients and adaptively learn K models for them. We also theoretically prove that the model learned with our method for each client a good approximation of its personalized model. Experimental results on real-world datasets verify the effectiveness of our method.
Broader Impact
Our work provides the possibility of lowering computational burden while maintaining comparable model performance in collaborative learning. We believe our work will be beneficial for collaborative applications, and do not have any negative societal impacts.
Acknowledgment
We would like thank Chiyu Cai for helpful discussions and anonymous reviewers for their valuable comments. This work is supported by the National Key Research and Development Program of China (2018AAA010) 100),, the National Science Foundation of China (61921006, 62276125), the Fundamental Research Funds for the Central Universities (0221 14380013), the project of HUAWEI- LAMDA Joint Laboratory of Artificial Intelligence, and the Collaborative Innovation Center of Novel Software Technology and Industrialization.
References
Arya, V., Garg, N., Khandekar, R., Meyerson, A., Munagala, K., and Pandit, V. Local search heuristics for k-median and facility location problems. SIAM Journal on Computing, 3343):544-562, 2004.
Balcan, M., Blum, A., and Gupta, A. Approximate clustering without the approximation. In Proceedings of the 20th Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 1068-1077, New York, NY, 2009.
Blum, A., Haghtalab, N., Procaccia, A. D., Qiao, M. Collaborative PAC learning. In Advances in Neural Information Processing Systems 30, pp. 2392-2401, Long Beach, CA, 2017.
10

---

Blum, A., Heinecke. S., and Reyzin, L. Communication aware collaborative learning. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, pp. 6786-6793, 2021.
Caldas, S.. Wu, P.. Li, T., Konečný, J., McMahan, H. B., Smith, V., and Talwalkar, A. LEAF: A benchmark for federated settings. 2018.
Charikar, M., Guruswami, V., and Wirth, A. Clustering with qualitative information. Proceedings of the 44th Symposium Foundations of Computer Science, pp. 524-532-3335 Cambridge, MA, 2003.
Chen, J., Zhang, Q., and Zhou, Y. Tight bounds for collaborative PAC learning via multiplicative weights, In Advances in Neural Information Processing Systems 31, pp. 3602-3611, Montréal, Canada, 2018.
Clauset, A., Newman, M. E., and Moore, C. Finding community structure in very large networks. Physical review E, 70(6):066111 2004.
Cui, S., Liang, J., Pan, W., Chen, K., Zhang, C., and Wang, F. Collaboration equilibrium in federated learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 241-251, Washington, DC, 2022.
Deng, Y., Kamani, M. M., and Mahdavi, M. 2003:3000.1341 2020.
Adaptive personalized federated learning.
Dinh, C.T. Tran, N. H., and Nguyen, T. D. Personalized federated learning with moreau envelopes. In Advances in Neural Information Processing Systems 33, 2020.
Dinh, T.N. Li, X., and Thai, M. T. Network clustering via maximizing modularity: Approximation algorithms and theoretical limits. Proceedings of the 15th International Conference on Data Mining, pp. 101-110, Atlantic City, NJ, 2015.
Even, M., Massoulié, L.,, and Scaman, K. On sample optimality in personalized collaborative and federated learning. In Advances in Neural Information Processing Systems 36, 2022.
Fraboni, Y., Vidal, R., Kameni, L., and Lorenzi, M. Clustered sampling: Low-variance and improved representativity for clients selection in federated learning. In Proceedings of the 38th International Conference on Machine Learning, pp. 3407-3416 2021.
Ghosh, A., Chung, J., Yin, D., and Ramchandran, K. An efficient framework for clustered federated learning, Advances in Neural Information Processing Systems 33, 2020.
He, C., Annavaram, M., and Avestimehr, S. Group knowledge transfer: Federated learning of large the edge. In Advances in Neural Information Processing Systems 33, 2020.
He, K., Zhang, X., Ren, S., and Sun, Deep residual learning for image recognition. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 770-778, Las Vegas, NV, 2016.
Huang, Y., Chu, I.., Zhou, Z., Wang, I.,, Liu, J., Pei, J., and Zhang, Y. Personalized federated learning: An attentive collaboration approach. arXiv:2007.03797, 2020.
Li, T., Sahu. A. K., Zaheer, M.. Sanjabi. M., Talwalkar, A., and Smith, V. Federated optimization in heterogencous networks. Proceedings of the 3rd Machine Learning and Systems, Austin, TX. 2020.
Mansour, Y. Mohri, M., Ro, J., and Suresh, A. T. Three approaches for personalization with applications to federated learning. arXiv:2002. 10619, 2020.
McMahan, B.. Moore, E.., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficcae learning of deep networks from decentralized data. Proceedings of the 20th International Conference Artificial Intelligence and Statistics, pp. 1273-1282, Fort Lauderdale, FL., 2017.
Mohri, M., Sivek, G., and Suresh, A. Agnostic federated learning. In Proceedings of the 36th International Conference Machine Learning, pp. 4615-4625, Long Beach, California, 2019.

---

Newman, M. E. Finding community structure in networks using the eigenvectors of matrices. Physical review E. 74(3):036104, 2006a.
Newman, M. E. and Girvan, M. Finding and evaluating community structure networks. Physical review E, 69(2):026133, 2004.
Newman, M. J. J. Modularity and community structure in networks. Proceedings of the National Academy of Sciences, 103((3):8577-8555) -582, 2006b.
Nguyen, H. I.. and Zakynthinou, I.. Improved algorithms for collahorative PAC learning. In Advances in Neural Information Processing Systems 31, pp. 7642-7650, Montréal, Canada, 2018.
Qiao, M. Do outliers ruin collaboration? In Proceedings of the 35th International Conference on Machine Learning, pp. 4177-4188 Stockholmsmässan, Stockholm, Sweden, 2018.
Satuluri, V. and Parthasarathy, S. Symmetrizations for clustering directed graphs. In Proceedings of the 14th International Conference on Extending Database Technology, pp. 343-354, Uppsala, Sweden, 2011.
Tuor, T., Wang, S., Ko, B., Liu, C., and Leung, K. K. Overcoming noisy and irrelevant data in federated learning. Proceedings of the 25th International Conference on Pattern Recognition, pp. 5020-5027, Milan, Italy, 2020.
Wang, H., Yurochkin, M., Sun, Y.. Papailiopoulos, D. S., and Khazaeni, Y. Federated learning with matched averaging. In Proceedings of the 8th International Conference on Learning Representa- tions, Addis Ababa, Ethiopia, 2020a.
Wang, J., Liu, Q., Liang, H., Joshi, G., and Poor, Tackling the objective inconsistency problem in heterogeneous federated optimization. In Advances in Neural Information Processing Systems 33, 2020b.
Yang, H. H-FL: A hierarchical communication efficcii and ated learning. In Proceedings of the 30th International Joint Conference Artificial Intelligence, pp. 479-485, 2021.
architecture for feder-
Yurochkin, M., Agarwal, M., Ghosh, S., Greenewald, K. H., Hoang, T. N., and Khazaeni, Y. Bayesian nonparametric federated learning of neural networks. In Proceedings of the 36th International Conference on Machine Learning, pp. 7252-7261, Long Beach, California, 2019.
12

---

Checklist
Did you include the license the code and datasets? [No] The code and the data are The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A]. You are strongly encouraged includ e a justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes]
proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 'rere using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "sn1gqcTGW8z", "reviewer": "Reviewer_VDbb", "summary": "This paper studied collaborative learning for heterogeneous clients. To achieve personalization while avoiding training personalized models for a large number of clients, the authors proposed methods to detect collaboration partners and adaptively learn models for client groups. The authors provide a theoretical guarantee that the expected risk of the learned group model is close to the personalized model. Empirical results on real-world datasets validated the theoretical result showing group model is a good approximation of the personalized model.", "questions": "Could the authors compare Theorem 3 and Theorem 4 under the same assumption, and discuss how much tighter Theorem 4 could be?\nWill ACLC perform better than ACLMM following the setting of Figure 5, especially with large $N$?", "limitations": "See weakness.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe motivation of learning models for client clusters to balance between computation and personalization is intuitive. The two algorithms, ACLMM which applies Modularity Maximization to estimate client group, and ACLC which assumes intrinsic client clustering, both are analyzed with generalization bounds.\n\nWeakness:\nI only have mostly minor concerns. While the two proposed algorithms are theoretically analyzed, they are not directly compared. It seems that with the $(1 + \\gamma, \\epsilon)$-approximation-stability assumption in Theorem 4, ACLC has a better generalization bound than ACLMM (when $OPT_$\\Phi(P)$ is small) and it would be interesting to compare the bounds in detail. It is also worth empirically testing ACLMM in Figure 5.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bSrMnJqBLAX", "reviewer": "Reviewer_XfiP", "summary": "This paper studies the collaborative learning problem, in which different clients could have non-i.i.d data. Compared with existing global or personalized methods, this paper proposes to detect collaboration partners to achieve a better collaboration. The authors formalize this problem as measuring the similarity between clients and finding an optimal partition of the collaboration federated network. The authors give various theoretical analysis and provide experimental results to verify the effectiveness of the proposed method. ", "questions": "Please see weaknesses.", "limitations": "The authors claim that they describe the limitations, but I did not found.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n1.This paper considers a challenging problem, the collaboration among multiple clients, which is very useful and significant in reality.\n\n2.This paper mainly focuses on theoretical analysis, including the similarity measurement between clients, the optimization of the partition, and the error bound of the approximated partition;\n\n## Weakness:\n### Motivation:\n1. The motivation in this paper is to find a partition for the collaboration network. It is reasonable and efficient. However, some related work and more discussions are missing, for example [1,2]. \n\n### Theory:\n1.Some theoretical results I’m very familiar with, e.g., theorem 1 and theorem 2. I’m afraid they are quite similar to existing theoretical work in domain adaptation and domain generalization. For the other theoretical conclusions, the authors also seems to refer to existing work. That being said, it’s not a big problem. More discussions about the difference between the theoretical analysis and existing work are necessary;\n\n### Experiments:\nThe experiments in this paper are seriously insufficient.\n1.the idea in this paper is clustering similar or helpful clients. More baselines are necessary, for example [2].\n2.the experiments focus on two image datasets. More other data are necessary.\n\n### Privacy:\nThis paper considers the problem of collaborative learning. The proposed method needs to measure the similarity between clients, which could cause privacy leakage in a federated learning setting. Could the authors discuss more about the privacy concerns?\n\n[1] Cui S, Liang J, Pan W, et al. Learning to Collaborate[J]. arXiv preprint arXiv:2108.07926, 2021.\n\n[2]. Ghosh A, Chung J, Yin D, et al. An efficient framework for clustered federated learning[J]. Advances in Neural Information Processing Systems, 2020, 33: 19586-19597.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "7-4TPYBmDEHw", "reviewer": "Reviewer_DG5y", "summary": "This paper introduces a novel approach to partition the clients into $K$ groups and build personalized model for each group. The approach leverage graph clustering method for learning purpose. The paper also provided a list of learning bounds. The experimental results demonstrate the proposed $K$ models can produce learning results close to personalized models. \n\n", "questions": "1. Can some greedy approach be applied here? like greedy clustering (merging the most similar pairs, or something along this line)\n   \n2. Is Lemma 1 essential? What about using other simple/alternative approaches for optimizing modularity (as this is a well studied problem) ", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. The problem is interestingly and timely. It can potentially be applied to many applications \n2. The approach is based on graph theoretical results\n3. The paper provides rather rigorous theoretical analysis\n\nWeaknesses: \n1. The writing can be improved and better organized\n2. Lack of base lines for comparison; the experimental results are rather weak.\n3. Some Figures are too small to read (like those in Figure 5) ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1EeJx6O61g", "reviewer": "Reviewer_br4T", "summary": "This paper studies the collaborative learning in personalized setting, which considers the heterogeneous distribution among clients. The paper proposes a method based on graph partition to learn personalized models for each partition group, which is shared by all clients within one partition group. The paper first provides the generalization bound for each client, from which the optimal personalized model is derived via optimizing on the contribution from the dataset of each client. Then, a contribution graph is constructed from the contribution weight, and similar clients are partitioned into one group. After filtering out bad clients, the personalized model for one group is learned based on the average weights of the clients within the group. The paper provides theoretically analysis on the increase of the upper bound of the excess risk due to the relaxation from optimal personalized model to group personalized model. Experiments demonstrate the group personalized model has similar performance with the optimal personalized model and justifies the proposed method.  \n", "questions": "1.One crucial step is to calculate the IPM between different data distributions. It is estimated by Eq 8, which involves the JS divergence between two data distributions. So how do you calculate the JS divergence between two datasets of clients.\n\n2.It seems that model training of one group needs the datasets from all clients. Does that mean clients need to transfer their local datasets? It seems that if sharing gradients, each client is required to keep all models of all groups and calculate gradients K(number of groups) times. \n\n\n", "limitations": "The paper does not discuss limitations of the work. It provides the discussion on social impact.\n", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper has several merits. First, the theoretical part is clear, rigorous, and sound. It justifies the effectiveness of the proposed framework. Second, the proposed method, which is based on graph partition of client contribution, is novel and interesting to me. \n \nHowever, I am not very familiar with this research topic. My major concern is on the writing of this paper, as it is sometimes hard to follow. For example, there are no subsections within Section 4 \"Our Work\", but to my understanding this section first talks about the optimal model, then provides graph partition based learning algorithm, and last gives tighter bounds with further assumptions. Thus, I think section4 can be separated into at least 3 subsections. The authors are encouraged to improve the presentation of this work. I have some more questions to ask the authors. Please see the comment below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept