Bayesian Coreset Optimization for Personalized Federated Learning
Authors: Prateek Chanda
Shrey Modi
Ganesh Ramakrishnan
Venue: ICLR 2024 poster (2024)
Keywords: federated learning
personalized federated learning
bayesian coreset
submodularity
variational inference
coresets
optimization
Primary area: general machine learning (i.e., none of the above)
TL;DR: The paper deals with utilizing a bayesian coreset on individual client's data in a federated learning setting that takes into account personalization at each client's side.

Abstract: In a distributed machine learning setting like Federated Learning where there are multiple clients involved which update their individual weights to a single central server, often training on the entire individual client's dataset for each client becomes cumbersome. To address this issue we propose CORESET-PFEDBAYES : a personalized coreset weighted federated learning setup where the training updates for each individual clients are forwarded to the central server based on only individual client coreset based representative data points instead of the entire client data. Through theoretical analysis we present how the average generalization error is minimax optimal up to logarithm bounds (upper bounded by $\mathcal{O}(n_k^{-\frac{2 \beta}{2 \beta+\boldsymbol{\Lambda}}} \log ^{2 \delta^{\prime}}(n_k))$) and lower bounds of $\mathcal{O}(n_k^{-\frac{2 \beta}{2 \beta+\boldsymbol{\Lambda}}})$, and how the overall generalization error on the data likelihood differs from a vanilla Federated Learning setup as a closed form function ${\boldsymbol{\Im}}(\boldsymbol{w}, n_k)$ of the coreset weights $\boldsymbol{w}$ and coreset sample size $n_k$. 
Our experiments on different benchmark datasets based on a variety of recent personalized federated learning architectures show significant gains as compared to random sampling on the training data followed by federated learning, thereby indicating how intelligently selecting such training samples can help in performance. Additionally, through experiments on medical datasets our proposed method showcases some gains as compared to  other submodular optimization based approaches used for subset selection on client's data.

Published as a conference paper at ICLR 2024
BAYESIAN CORESET OPTIMIZATION FOR PERSONALIZED FEDERATED LEARNING
Prateek Chanda
Shrey Modi
Department of Computer Science
Department of Computer Science Indian Institute of Technology Bombay, India 200020135eiitb ac, in
Indian Institute of Technology
Bombay, India prateekch@cse.c witiiiita......
Ganesh Ramakrishnan
Department of Computer Science
Indian Institute of Technology
Bombay, India ganesh@cse. witb.iit... attb.ac.in
ABSTRACT
In distributed machine learning setting like Federated Learning where there multiple clients involved which update their individual weights to a single central server, often training on the entire individual client's dataset for each client be- comes cumbersome. To address this issue we propose personalized coreset weighted federated learning setup where the training up- dates for each individual clients are forwarded to the central server based on only individual client coreset based representative data points instead of the en- tire client data. Through theoretical analysis we present how the average gen- eralization error is minimax optimal up to logarithm bounds (upper bounded by
22 Om 2344 log. (nz))) and lower bounds of O 2 ), and how the overall generalization error on the data likelihood differs from a vanilla Federated Learn- ing setup as a closed form function 3(w, n&a) of the coreset weights and core- set sample size ng.. Our experiments on different benchmark datasets based on a variety of recent personalized federated learning architectures show significant gains as compared to random sampling on the training data followed by federated learning, thereby indicating how intelligently selecting such training samples can help in performance. Additionally, through experiments on medical datasets our proposed method showcases mization based approaches used for subset selection on client's data.
gains compared other submodular opti-
INTRODUCTION
Distributed machine learning has wide variety applications in various domains like in recom- mendation systems (Zhang et al., (2022a), healthcare and other areas. Given the advantages of data- privacy, heterogeneity and resource efficiency, federated learning in particular stands out among all other learning methods. For instance in recommendation systems based work (Luo et al. 2022) (Zhang et al.) 2023) or healthcare (Lu al., 2022). personalization and privacy preserving based recommendations are currently most focused upon. However, often these require inference on larger datasets which is often computationally expensive given the fact that often these datasets only consist of subset of representative data points (Kaushal et al. 2018), Maheshwari et al. 2020).
Recently, researchers have tried to use Bayesian coresets for performing inference (Campbell & Beronov (ammpell & Broderick, 2018). Bayesian coreset refers to subset of the complete dataset which can be used to approximate the posterior inference as if was performed on the com- plete dataset. In Federated Learning, have distributed machine learning model where there set of clients each having their own share of data and a server. This is prevalent in many production level systems such recommendation systems machine learning models deployed for mobile

---

Published as a conference paper at ICLR 2024
apps. It is imperative for each client get the same user satisfiability and personalized experience with only a small chunk of data on their local devices. Instead of training on the entire user data for a particular client, what if we only learn based a representative set of the user data for each client and achieve near optimal accuracy? In this space (Huang et al., 2022) perform coresets optimization for vertical federated learning where empirically they show that coresets optimization help reducing the communication complexity among clients and server in the vertical FL setting.
Our contributions are as follows:
Proposal of new architecture to incorporate bayesian coresets optimization in the space of Federated Learning.
Proposal of multiple novel objective functions that take into account the optimization prob- lem of general Federated Learning in Bayesian coresets setting with particular focus on personalized coreset weights for each individual clients.
Theoretical analysis on the convergence rate shows our approach CORESET-PFEDDAYER achieves convergence rate within logarithmic bounds.
Experimental results across several benchmark datasets conducted with wide array of pre-existing baselines show promising results towards good performance in terms of model accuracy even with less data at each client's i end.
2 RELATED WORK
It is well known in literature that training datasets offer diminishing returns in terms of performance. has also been demonstrated that one can train and obtain better returns in performance and en- ergy efficiency by training models over subsets of data that are meticulously selected (Ghorbani & Zou 2019; Yoon et al., 2020: Katharopoulos & Fleuret 2018 Strubell et al.) 2019). This leads us to the problem of coreset selection that deals with approximating desirable quantity (e.g., gradi- ent of a loss function) over an entire dataset with a weighted subset of it. Traditional methods of coreset selection have used a submodular proxy function select the coreset and are model depen- dent (Wei et al., 2015; Kirchhoff & Bilmes| 2014, Kaushal et al., 2019: Har-Peled & Mazumdar) 2004 Mirzasoleiman et al.] (015)). Coreset selection with deep learning models has become pop- ular in recent years (Mirzasoleiman et al., 2020a b. Killamsetty et al. 2021 Coleman et al. 2019) Owen Daskin 1998). Coreset selection to approximately match the full-batch training gradient proposed in (Mirzasoleiman et al. (0200). Killamsetty et al. (2021) propose algorithms to select coresets that either match the full-batch gradient or the validation gradient. MMizzaolleiman et al.) (2020b) propose approach to select coreset that admits low-rank jacobian matrix and show that such an approach is robust to label noise. Most existing coreset selection approaches are pro- posed in conventional settings wherein all the data is available in one place. In FL, since client or the server gets a holistic view of the training dataset, coresets can at best approximate only local data characteristics and are thus inherently sub-optimal.
Coreset selection in federated learning has remained largely unexplored because of the intricacies involved due to privacy and data partition across clients. Federated Learning be modelled as cooperative game where it often uses Shapley values to select clients whose updates result in the best reduction of the loss validation dataset held by the server. One work that comes very close to ours is that of (Balakrishnan et al. 2021), that selects a coreset of clients[] whose update represents the update aggregated across all the clients. They apply facility location algorithm on the gradient space to select such a coreset of clients. In contrast to all these approaches, our proposal attempts to select a coreset of the dataset at each client and is thus more fine- -eeegra than such prior works. One another paradigm of FL that is in contrast to our setup is Personalized FL, where the aim is to train specialised models for each individual client (Collins et al. 2021 Fallah et al., 2020aa Marfoq et al.) 2022 Li et al. 2022; Jain et al. 2021). While personalized FL focuses on finetuning the model to match each client's data distribution, we build models to account for just the server's distribution, similar to (Karimireddy et al. 2020).
The algorithms that ensure privacy in FL include differential privacy Dwork et al., 2006 Kairouz et al. 2021), Homomorphic encryption (Segal et al. 2017 Li ct al. 2020), etc.; which is not the
As against selecting coreset of data instances
2

---

Published as a conference paper at ICLR 2024
Campare and find optimal
F FM (2)
(((2)
Family of digerbutions for the server
Client Objective Loss over coreas Data
Q N N(p
Global Model
(ps oz)
(pN m7
Lpes esl
Fumily
dittibutionsor
(lbe o.)
(A m.)
Mi o
negative personalization constraint - Reion log! Pl.s, (P")
Gient
Client
Chent
GientN
+cDnalo (0))::00)
wr D
wy D
07g D
END' D
Figure System Diagram Coreset Weighted Personalized Federated Learning model with pa- rameters under Gaussian assumptions. Each client uploads its updated distribution to the server based its corresponding coreset training data (each client i's data D* weighted by w.) and then the aggregrated global distribution is utilised from the server.
focus of work. However, note that all these methods be easily integrated with solution approach and can be used in conjunction.
3 PRELIMINARIES AND PROBLEM SETTING
We consider the problem setting from (Zhang et al. 2022b) as follows: Consider a distributed system that includes one server and N clients. Let the ith client's dataset be D {(xr), ( all N clients have the same sample size n) Further, let the i-th client satisfy regression model with random covariates as y = ((() + e), Vj E [n] where æ) E 1 y) R for Vi N. I,...,N. Here f'(o): IR + + denotes a nonlinear function (which is unknown and we want iid to estimate) and e) denotes a Gaussian noise independent of x i.c. et ~ NO o²) with variance o? We assume that (*) is 3-Hölder-smooth functions (Tkezawa)) 2005) and the intrinsic dimension of each of the client's 's data is A. We assume each client has the same fully-connected deep neural network (DNN), each having their individual neural network parameters. We denote the output of the neural network as fo(*) where 0 E R represents the network parameters. Analogously the output of the i-th client is denoted as Jg. Each neural network has 7. hidden layers, where the j-th hidden layer has 8y neurons and its corresponding activation functions a()). We denote S = {s: H the set all neurons the neural network. The assumption made that all the neural network parameters are bounded i.e., ||0|| < Ω for Ω 0.
(assuming
The main aim to architect a Federated Learning system that takes into consideration the coreset problem for each individual client along with personalization and tackling overfitting. Using the above problem setting we first formulate the client side objective and server side objective for the Federated Learning setting as a bi-level optimization problem.
3.1 FEDERATED LEARNING OBJECTIVES
The standard BNN model Jordan et al. 1999) aims to solve the optimization problem to find the closest distribution 9" (0) for the i-th client from the family of distributions Q; to match the posterior distribution π D*) of the given data D via minimizing the KL-divergence as follows.
Client Side Objective
reconstrnction error over 'D
rogularization term
Fi(2) A min (0)EER DRI (xL(((())==00)))
g" (0)E Ey log (D()) + DEL ( (
min

---

Published as a conference paper at ICLR 2024
Here r(0) denotes the prior distribution and Pe (D") denotes the likelihood and is a personaliza- tion constant that defines the weightage towards more regularization thus leading to more personal- ization.
Server Side Objective On the server side the global model tries to find the closest distribution in Q, to the client's distribution by minimizing the aggregrate KI. divergence from all the clients as follows:
min F(z) x 200)~0 Q
EF(())
3.2 BAYESIAN CORESETS OPTIMIZATION
We now introduce the notion of coreset weights i.e. we assign to each client i's data D a weight vector w; E R" that will act the corresponding coreset weight for the i-th client. In standard bayesian coresets optimization setting, the goal is to control the deviation of coreset log- log-likelihood from the true lig-likelihood via some sparsity (ng << nt). accordance to (Zhang et al. 202T) we utilise the following optimization objective for the i-th client:
('w.)) == (|P)) - Pa.w, (D') s.t. ||willo S rips Vie [N]
arg min W.E.
(3)
where the coreset weights w; considered over the data points D and (fr the distance metric is considered in the embedding Hilbert Space. Specifically, is the weighting distribution that has the same support as true posterior æ. The above equation can be further approximated to the following where 9j is Monte-Carlo approximation Monte-carlo samples (the derivation can be found Appendix
0i PD())) Eg Po (D())) for
min ('(w;) == WeRn
Σ
9J
s.t.
||1:lo S TRk Vi [N]
(4)
The above sparse regression problem non-convex due to the combinatorial nature of the con- straints. (Campbell& & Broderick] 2018) uses the 1n-oomm formulation which however results in less approximation accuracy compared to (Campbell & Beronoy 2019). As the authors in (Zhang et al.) 2021) point out both the above approaches have expensive computation cost and hence they propose a better alternative via accelerated iterative thresholding Appendix 10.7
METHODOLOGY CORESET-PPEDDAYEE
We now proceed towards formulating our problem for combining the coreset optimization problem in federated learning setting.
4.1 MODIFIED FEDERATED LEARNING OBJECTIVES INCORPORATING CORESET WEIGHTS
Modified Client Side Objective We now aim towards incorporating the coreset formulation from Eq: 3 in our federated learning setting Eq: Assuming the bayesian coreset weights setup for each client i from, we introduce modified client objective function
(2) min [o) log(Pe. w. (D")) + <DKz (0)122(()))
(0)~a.
where z0)) and (0) denote the global distribution and the local distribution for the i-the client that to be optimized respectively. Here, F. (o) and (:(+) indicates the coreset weighted i-th client objective and full data based client objective respectively.
In particular, let the family of distributions of the i-th client Q,and server @asatisfy
Q₂ ~ (u,, o2,)), 2s.t ~ NN(,,, 02.4)
k ==
(6)
where the above gaussian parameters correspond to the mean and variance for the k-th parameter of the i-th client respectively. Similar holds true for the server. This is valid assumption commonly used in literature (Blundell et al. 2015) that let's us simplify the evaluation of KLL divergence score Dex between 9¹ (0) and z(0), resulting in closed form solution.
4

---

Published as a conference paper at ICLR 2024
4.2 NOVEL OBJECTIVE ACHIEVING NEAR-OPTIMAL CLIENT DISTRIBUTION BASED ON OPTIMAL CORESET WEIGHTS
Let x" =0) be the optimal variational solution of the problem in Eq and q' (0) be its corre- sponding variational solution for the i'th client's objective in Eq: Now, for the same z (0), let 9¹ (0:0:u w) denote the corresponding variational solution for the weighted coreset client objective in Eq: 5 We want to ensure that the optimal distribution q' (f; w) which minimizes the coreset objective does not deviate too much from the original distribution g²) (which acts solution for Eq: D and hence we want to fixate on w accordingly. This intuition comes from the fact that we want to choose in such way that the performance of our federated learning system in terms of accuracy does not drop too further in the face of training on a small percentage data dictated by the coreset weights (in our problem setting we denote coresets weights as w. for each client i). From here forward, let F!! arg denote (0; w) and Fi( arg denote q' q'(O).
4.3 OBSERVATIONS AND MOTIVATION FOR A NEW ORJECTIVE
If we observe the client equation Eq: D closely, we will see that for each client i we are first fixing =0) in the equation. For fixed =0). we search for the optimal distribution g' (0) among the whole family of distributions local to that client º². This is similar also for Eq: for the coreset weighted client objective.
Our objective is the learned optimal local distribution for the general client optimization objective should be as close as possible to that of the weighted coreset based client optimization objective for the same value of z.
So want to formulate new objective function such that for each client minimize the di- vergence between the two optimal distributions resulting from the coreset and normal objective functions.
(w:"} } A arg min WKL (((2)))) (F)
arg min Deu (0) w
||w/llo S nh
w
4.4 INCORPORATING LIKELIHOOD INTO MODIFIED CLIENT OBJECTIVE
Although the above minimization objective Eq: 7 captures the intuition behind matching the near- optimal performance (accuracy) with only a small coreset of the original client's 's data to that by using the entire data, this approach does not take into account how close the likelihood of each of the client's 's coreset weighted data subset Po w. (D*) is to that of the original client's ' data PDDD"..
More specifically, we now want choose the optimal coreset weights (w, (personal coreset weights) by not only minimizing the KL divergence between the corresponding client distributions ( (0: w) and q² (0)) but also taking into account the closeness of the coreset weighted data likelihood to that of the original likelihood.
(w;") arg min (2
+ PDt))) Po. (12) P
||willo I ny
(8)
5 ALGORITHM
We showcase our complete algorithm for CORESET-PFEDBAVES in Table 1 below In line with (Zhang et al.) 2022b) we utilise a reparameterization trick for via variables lr and p i.e. ((vgg) g) where O. h (Um, 9m. 1m log (1 + exp (pm)) Jm. Im ~ NO, 1), where E
[1..... T]. For the first term in Equation 5 we use a minibatch stochastic gradient descent to get an
estimate for the i-th client as follows:
('(ww)
1 Σ Σ K
log Ph( (D)) + <IEK (gt.. (0; w (0))
5

---

Published as a conference paper at ICLR 2024
Algorithm I: CORESET-PHEDBAYER
Server side Objective & Coreset Update
Client Side Objective
Cloud server executes:
ClientUpdate (2, W; v²]
Input 7, R, S, 8,6, vd (14 x00) for f 0,1,...,2 T do
v o² for 0, u.l...... R do
parallel do wfth + CoresetO
1,2,.......................
D c sample a minibatch
size b from D
(i, e²)
D A. + sample a minihatch A with size 6 from my D)
S < Random subset of clients with size S
9r, +- Randomly draw K samples from [[0, (00,
o441 = (1 3) + Liest o/-
Q (w?) <- Use 9 g and of
V.S2 (vf)) ) +- Back propugation w.r. ot
== Po(D"), 4hw' Pa (D")
of < Update with V.St' (uf) using GD algorithms
Objective f(w) = Dez (1.llllll (19') ) 1lv 1w''l
Ω (u' Forward propagation w.r.
0,4, = 0, ug
V22 (v
Back propagation
wi 6- wigs f-
Update 12.++1 with VΩ2 (v2,.) using GD algorithms
repeat
Repeat the above 7 steps
' 7' V'..'. ClientUpdate (i, us, u²)
the weighted stochastic estimate
[(w) = (xu(dlll 1 owa
((z) 6- arg Q (vt..)) .9,(z) +- arg Qi (vh...)) v 2. (2),, (2)))
Accelerated-IIIT (w)) (Algo 10.7
until Stop criteria met
return v to the cloud server
where b and K minibatch size and Monte Carlo sample size, respectively.
Here R indicates the number iterations after which the clients upload the localized global models the server. Like (T Dinh et al., 2020). we use an additional parameter order to make the algorithm converge faster.
THEORETICAL CONTRIBUTIONS
Here we provide theoretical analysis related to the averaged generalization error for CORESET-PFEDBANE w.r.t our baseline PFEDBAYES. The main results and derivations of the proofs can be found in the Appendix 9 We first provide certain definitions here.
Definition 1. The Hellinger distance for a particular client between the estimate likelihood Pi
and the true likelihood p is defined follows d' Pt) = =Ex.
Let q¹ (0; w) be the corresponding variational solution for the i-th client's subproblem under the coreset weighted regime and let us define the following term Generalization Error Term:
Se d² (P) 0.a D'' (0; w)d0 as the i-th client's generalization error.
Theorem The difference in the upper bound incurred in the overall generalization error of CORESET-PFEDBANT
as compared w.r.t that of PFEDBAYES is always upper bounded by closed form positive function that depends on the coresel weights and coreset size- S(w, nz). generaliza- tion error in the original full data setup
N NI
Σ
/
1 Σ
P(P',.., (')
<Swwn
u.b.
u.b.
This indicates that the extra estimation error and approximation model incurred by the coreset weighted objective a direct function of the coreset weight and thus the coreset size and hence can be measured in closed form. Proof. in Appendix [
Theorem 2. The convergence rate of the generalization error under L norm of SURRSEE--FEEDBA.
is minimax optimal up to logarithmic term (in order n&) for bounded functions (3-HHölder-smooth functions) { {fa) and 3 iN where Cr. O3 and d' are
constants (defined in Appendix and A being the intrinsic dimension of each client's data:
CF N
N
28 1o (xi) (x*) 2 (0;w)dd < C2 log26 (nz)-

---

Published as a conference paper at ICLR 2024
and
CF N (1 (x') - - (x:) 2 Can
24
w1 sr inf {ir
20TA
s
N
where ny denotes the coreset size per client dataset and n denotes the original per client dataset
size and de (Po 18 (X---XX01
1-exp
B.7 & CF.
This indicates that the minimax optimality of the generalization error for CORESET-PFEDDAYES, is in logarithmic bounds w.r.t the coreset size ni. Proof. in Appendix 2
Theorem 3. The lower bound (Lb.) incurred for the deviation for the weighted coreset CORESET-PFEDBANE (5) generalization error always higher than the lower bound of that for
28
the original PFEDBAYES objective with a delta difference (Error EEror II) as O ( 254A
Σ fo.0 (x')-
- (x'))( (1 ( w)d
C.110 11 (x)
L.b.
Coreset weighted objective Generalization Error (Error
Viniila objective Generalization Error (Errur II)
This simply implies that the generalization error suffers in the case due limited coreset samples but that is bounded in closed form w.r.t. the coreset sample size. Proof. in Appendix: 3
Theorem 4. The lower bound incurred in the overall generalization error across all N clients of
CORESET-PFEDBAYE is always higher compared to that of the generalization error in the original full data setup
N Σ
N Σ
/
(') w)d0
L.b.
L.b.
Both Theorem I and 4 implies that the overall spread of the Generalization Error Term in case of is much more wider than that of the original
coreset weighted objective PFEDBAYES case. Proof. in Appendix 4
7
EXPERIMENTS
Here we perform our experiments showcase the utility of our method CORESET-PFEDBAVE compared to other baselines like PFEDBAYES and do an in-depth analysis of each of the components involved as follows.
7.1 UTILITY A-IHT IN VANILLA BAYESIAN CORESETS OPTIMIZATION
In Algorithm 1, since we are employing A-IHT algorithm during coreset updates, we first want to study the utility of applying A-IHT (Accelerated Iterative Thresholding) in a simplistic Bayesian coresets setting using the algorithm proposed by (Huang et al. (022). For analysis we test the same pr
on Housing Prices 2018 Pata. Figure 3 showeases the experiments done on the dataset using a riemann linear regression for different coreset sizes of the data (k 220, 260, 300). As it can be seen the radius capturing the weights w.r.t to the coreset points matches closely at k=300 with that of the true posterior distribution (extreme right), thereby indicating the correctness of approximation and recovery of the true posterior by the A-IHT algorithm.
7.2 EXPERIMENTS ON CORESET-PPEDDAATE AGAINST S.O.T.A FEDERATED LEARNING METHODS
Here, we compare the performance of the proposed method CORESET-PFEDDANE with a variety of baselines such as FedAvg (McMahan et al. 2017). BNFed (Yurochkin al., 2019, pFedMe

---

Published as a conference paper at ICLR 2024
Figure 2: Experiments on Bayesian reimann linear function regression for different settings of core- set size -220,,600,000 constructed by Accelerated IHT II. Coreset points are presented as black dots, with their radius indicating assigned weights. Extreme right showcases the true posterior distribu- tion
(T. Dinh et al. 2020), perFedAvg (Fallah et al. 2020b), PFedBayes (Zhang et al., 2022b) based on non-i.i.d. datasets. We generate the non-i.i.d. datasets based on three public benchmark datasets, MNIIT (Lecun et al. 1998), FMNIST (Fashion- MNIST) (Xiao et al. 2017) and CIFAR- 10 (Krizhevsky et al. (200)). For MNIST, FMNIST and CIFAR- 10 datasets, we follow the non-ii.d. setting strategy in (T Dinh et al. 2020). In our use case we considered total 10 clients each of whom holds a unique local data.
Table I: Comparative results of personal and global accuracies (in %) across all 7 methods
Method (Peroentage sampling fraction)
MNIST
FashionMNIST
CIFAR
Personal Model
Global Model
Personal Model
Global Model
Personal Model
Global Model
AedAAg (Full/ 50%)
92. 92.39(90.60 90.66)
85.42/83.00
79.05(56. 73)
BNFed (Full/ /50%)
82, (2.95(80.02)
70.1 68)
44.37(39.52)
PFedMe (Full
91.25(89.67)
92.02(84.71)
84.41(83.455
77.13666775
10.86(51.18)
perFedAvg (Full/ /50%%))
98.27
88.51184.90)
69 69.61 (52.98)
PFEDBAYES (Full 150%)
98.79(90.88)
97. (7.21(92.33)
93.01185.95
93.30882.33)
83.46(73.94)
64, (4x66660.84)
RANDOMSUDSET (50%)
88.4
61.35
CORESET- (k 50%)
92.48
96.3
89.55
92.7
69.66
71.5
(a) We report accuracies on both global and personal model for the current set proposed methods across major datasets like MNIST, CIFAR, FashionMNIST. Red indicates the highest accuracy column-wiss Similarly Orange and Magenta indicates the 2nd and 3rd best modelwise accuracy. (-) indicates no accuracy reported due to very slow convergence of the corresponding algorithm. Full indicates training on full dataset and 50% is on using half the data size after randomly sampling 50% % of the training set.
In Table 3 we showcase the accuracy statistics of the corresponding baselines discussed above with our method CORESET-PFEDDBAVEE across two different configurations Full Dataset Train- ing and 50% Training indicating half of the training samples were selected at random and then the corresponding algorithm was trained). As observed in almost all cases our method CORESET-PFEDBAYES. beats PFEDBAYES(random 50% data sampled) by the following margins +4.87% on MNIST, +8.61% % on FashionMNISTT, +9.71% and almost on other baselines (random 50% data sampled) and some baselines even when they were trained on full dataset (e.g. our method does better than PFedMe on FashionMNIST). In Appendix [0.2] we showcase the communication complexity of our proposed method.
7.3 EXPERIMENTS ON PERESEE-PFEEDBAES AGAINST SUBMODULAR SUBSET SELECTION
In order to showcase the advantage of model-centric subset selection methods over traditional data- centric(model-agnosiic) methods like our proposed work CoRESET-PFEDDANE PFEDBAYES which takes into account matching the client distribution under coreset setting g (0; w) to that in normal setting (0). Hence, we compare our proposed method CORESET- PFEDBAYE against submodular based functions (See Appendix7]oo definition) (specifically, diversity based submodular functions as the aim is to select a subset of data points that are most diverse). The discussion on some of the common diversity functions and their properties with regard to monotonicity and submodularity are provided in Appendix 10.5 Each of the medical datasets consists of 3 classes out of which 1 class (Normal is kept common) between clients and the data about the other two classes is distributed separately to the elients. Our aim here is to deploy our proposed method in this setting and compare against submodular based subset selection approaches.
8

---

Published as a conference paper at ICLR 2024
Divengence between the corset weighted client side distriuttinn normal dllent button
Divirganos Comvergunce Eflect on Randomlly Sumpleed Data vs ligeligentty Core set Wiigrtted Data
Dxz(of(0, (x.:(e((0.:)1l((0)
250030
KL. Lava Dverggeee
100000
of (60)
80000
250030
pesgoaensa Hoddii (Vin beng
00000
200000
coreseserepaaaee IOn Cand
40300
Spon 16e
(i)
100003
50003
Number Spectr
150
(a) The optimal distribution parameters learnt af- ter taking a coreset of the data tend to match up with parameters, when trained on the full dataset after few epochs resulting in decreasing KL diver- gence score. This is in line with our initial hypoth- esis 4.3
(b) We are comparing random subset selection of the data vs when a subset is selected using our method, the early convergence of the KL- divergence shows the better performance of method. When trained on random subset of data, the model takes more epochs to converge.
Figure 3: KL Divergence Plot over Number of Epochs (MNIST Dataset)
More details regarding the dataset description along with the experimental configurations can be found under Appendix. Note for each of the submodular function baselines, the data was sampled using a submodular function optimization strategy post which FedAvg was applied as the Federated Learning algorithm.
Table 2: Comparative results of classwise global accuracies of all 9 methods on 3 different medical datasets and 2 clients
Methed (Feorzage
C0V Matiegraphy Hataher
APTUS 2019 Blndraes Dntection
OCTSINIST Datnet
Nacmal
Lang Opecity
Nermal
Mud Duabeic Recirogulty
Scrocc Daabesc
Serual
Rettargaty
Vontlla Fed Kng (hrdl)
0314 IL167
16.80
IL39K:
11897
I:368 8.123
8:937 0.119
11.814
0.58
00.101
82992
Independest Learning (Fubb
BJ00E 9.911
0.869
0L356
0.882
IL945
IU826
03572 1 0.000
01300
11583
0.201
I1.73
01799 8.0175
9J41
Randendab Rolkrg (S86)
0,802 0.004
0.670
8659
0.583
+ 0/033
0915 + 0/44
0,835 2-0.09
0.552
9.60
0511 = 0.800
0.753
0.093
0,910
DiggSar Felliing (501g)
0.015
0.849
IU945
60845
0
8/085
[.86]
0.544
0.094
Dtephllin ho (937%)
03966 1100
0.700 1608
IU791
C2011 1 0.007
8.775
16033
11331 2 16001
00892 8.065
CoRiiN ioPiasss (3895)
0.919
0.771
0.825
0.894 0.829
0.806
11817
4.916 (1.42
0U900 00005
6.816 6/111
(a) We report classwise accuracies for the current set of proposed methods for all 3 medical datasets. Red indi-
cates the highest value in accuracy column-wis (i.c. for particular class for a dataset across all 9 baselines). Similarly Orange and Magenta indicates the 2nd and 3rd best classwise accuracy. Colors for Vanilla Fed,vg PFEDBAYES, , CORESET-PFEDDAVVS are grouped together to primarily compare against subset selection strategies
As observed in Table 2 CORESET-PFED performs better than submodular based approaches on average across Covid-19 19 and APTOS. This is promising as it indicates model centric subset selection much more useful in terms of performance than model agnostic subset selection methods.
8 CONCLUSION & FUTURE WORK
In this work we proposed several novel objective formulations that draw and synthesize from pre- vious works of two separate domains: coreset optimization and federated learning. Through our extensive experimentations, our proposed method showcases significant gains over traditional fed- erated learning approaches and submodularity based optimization functions followed by Federated Learning. We also showcased through theoretical analysis, how the average generalization error is minimax optimal upto logarithm bounds and how that estimation and approximation error compares against PFEDBAYES. In future we want to look into how client-wise data distribution affects the current scheme and how we can make the model more robust towards adversarial attacks from (say) skewed data distribution over client-side model parameters. Also, the interplay of how such coreset weights can affect model updates in a privacy- preseeriin manner is somewhat interesting to explore further.

---

Published as a conference paper at ICLR 2024
REFERENCES
Jincheng Bai, Qifan Song, and Guang Cheng. Efficient variational inference for sparse deep learning with theoretical guarantee. Advances in Neural Information Processing Systems, 33:466-4777 2020.
Ravikumar Balakrishnan, Tian Li, Tinyi Zhou, Nageen Himayat, Virginia Smith, and Jeff Bilmes. Diverse client selection for federated learning via submodular maximization. In International Conference on Learning Representations, 2021.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference machine learning, pp. 1613-1622. PMLR, 2015.
Trevor Campbell and Boyan Beronov. Sparse variational inference: Bayesian coresets from scratch. Advances in Neural Information Processing Systems, 32, 2019.
Trevor Campbell and Tamara Broderick. Bayesian coreset construction via greedy iterative geodesic ascent. In International Conference on Machine Learning, pp. 698-706. PMLR, 2018.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. arXiv preprint arXiv: 1906. /1229, 2019.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared repre- sentations for personalized federated learning. In International Conference on Machine Learning, pp. 2089-2099. PMLR, 2021.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. Theory of cryptography conference, pp. 265 Springer, 2006.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta- learning approach, 2020a. URL https://axxi..rry/b///00....7772
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with the- oretical guarantees: A model-agnostic meta-learning approach. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information P Processing Systems, Curran Associates. Inc., 2020b. URL httpps//proceedngss.errppppp Pappeep
volume 33, pp. 3557-3568.
paper
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pp. 2242-22511 PMLR, 2019.
Sariel Har-Peled and Soham Mazumdar. On coresets for k-means and k-median clustering. In Proceedings of the thirty-sixth annual ACM symposium on Theory of computing, pp. 291-300, 2004.
Lingxiao Huang, Zhize Li, Jialin Sun, and Haoyu Zhao. Coresets for vertical federated learning: Regularized linear regression and k-means clustering. arXiv preprint arXiv:2210. 14664, 2022.
Prateek Jain, John Rush, Adam Smith, Shuang Song, and Abhradeep Guha Thakurta. Differentially private model personalization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 29723-29735. Curran Associates, Inc., 2021. URL http:://pcceddng...euripp.cppppp
f0c08550000333000447900000022222 Paper.pdi
Michael Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 37:183-233, 1999.
Peter Kairouz, Ziyu Liu, and Thomas Steinke. The distributed discrete gaussian mechanism for federated learning with secure aggregation. In International Conference on Machine Learning, pp. 5201-5212. PMLR, 2021.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference Machine Learning, pp. 5132-5143. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2024
Angelos Katharopoulos and François Fleuret. Not all samples created equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2018.
PMLR,
Vishal Kaushal, Anurag Sahoo, Khoshrav Doctor, Narasimha Raju, Suyash Shetty, Pankaj Singh, Rishabh Iyer, and Ganesh Ramakrishnan. Learning from less data: Diversified subset selection and active learning in image classification tasks. arXiv preprint arXiv: 1805. 191, 2018.
Vishal Kaushal, Rishabh Iyer, Suraj Kothawade, Rohan Mahadev, Khoshrav Doctor, and Ganesh Ramakrishnan. Learning from less data: A unified data subset selection and active learning framework for computer vision. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1289-1299. IEEE, 2019.
Vishal Kaushal, Ganesh Ramakrishnan, and Rishabh Iyer. Submodlib: A submodular optimization library. arXiv preprint arXiv:2202. 10680, 2022.
Krishnateja Killamsetty, S Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pp. 5464-5474. PMLR, 2021.
Katrin Kirchhoff and Jeff Bilmes. Submodularity for data selection in statistical machine translation. In Proceedings of EMNLP, pp. 131-141, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient based learning applied to document recog- nition. Proceedings of the IEEE, 86(1 (6611)):227-22) 2324, 1998. doi: 10. 109/5.72679)
Shuangtong Li, Tianyi Zhou, Xinmei Tian, and Dacheng Tao. Learning to collaborate in decentral- ized learning of personalized models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9766-9775, June 2022.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-60, 2020.
Wang Lu, Jindong Wang, Yiqiang Chen, Xin Qin, Renjun Xu, Dimitrios Dimitriadis, and Tao Qin. Personalized federated learning with adaptive batchnorm for healthcare. IEEE Transactions Big Data, 2022.
Sichun Luo, Yuanzhang Xiao, and Linqi Song. Personalized federated recommendation via joint representation learning, user clustering, and model adaptation. In Proceedings of the 31st ACM International Conference Information & Knowledge Management, pp. 4289-4293, 2022.
Ayush Maheshwari, Oishik Chatterjee, Krishnateja Killamsetty, Ganesh Ramakrishnan, and Rishabh Iyer. Semi-super data programming with subset selection. arXiv preprint arXiv:2008.0988 2020.
Othmane Marfoq, Giovanni Neglia, Richard Vidal, and Laetitia Kameni. Personalized federated learning through local memorization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 15070- 15092. PMLR, 17-23 Jul 2022. URL httpss//procceedings.. www.mml...sss... paress/v16220 162mmarfq222aatmm
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-Efficii Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statisties, volume 54 of Proceedings of Machine Learning Research, pp. 1273-1282, PMLR, 20-22 Apr 2017. URL https://proceeddngg.. wwwpadiinnngmmg....emmmmm halr.prrss//54/mmahann 7a.html
Baharan Mirzasoleiman, Amin Karbasi, Ashwinkumar Badanidiyuru, and Andreas Krause. Dis- tributed submodular cover: Succinctly summarizing massive data. Advances in Neural Informa- fion Processing Systems, 28, 2015.
11

---

Published as a conference paper at ICLR 2024
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960. PMLR. 2020a.
Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovec. Coresets for robust training of deep neural networks against noisy labels. In NeurIPS, 2020b. URL https::/proceeednggs.s. wwt.::ss.nnurc//aaappa.pa
Abstract.html
Ryumei Nakada and Masaaki Imaizumi. Adaptive approximation and generalization of deep neural network with intrinsic dimensionality. The Journal of Machine Learning Research, 21(1):7018- 7055, 2020.
Susan Hesse Owen and Mark Daskin. Strategic facility location: review. European journal of operational research, 111(3):423 447, 1998.
Debdeep Pati, Anirban Bhattacharya, and Yun Yang. On statistical optimality variational bayes. In International Conference Artificial Intelligence and Statistics, pp. 1579-1588. PMLR, 2018.
Nicholas G Polson and Veronika Ročková. Posterior concentration for sparse deep learning. Ad- vances Neural Information Processing Systems, 31, 2018.
Aaron Segal, Antonio Marcedone, Benjamin Kreuter, Daniel Ramage, H Brendan McMahan, Karn Seth, KA Bonawitz, Sarvar Patel, and Vladimir Ivanov. Practical secure aggregation for privacy- preserving machine learning. 2017.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv: 1906. 02243,224433 2019.
Canh T. Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21394-21405. Cur- ran Associates, Inc., 2020. URL httpss//poceedinss..unppscoppppp wtw.r//eeseeeuuttcooooo...eoeeee 110
Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau en-
velopes. Advances in Neural Information Processing Systems, 33:21394-21405, 2020.
Kunio Takezawa. Introduction to nonparametric regression. John Wiley & Sons, 2005.
Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International Conference on Machine Learning, pp. 1954-1963. PMLR, 2015.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark-
ing machine learning algorithms, 2017.
Jinsung Yoon, Sercan Arik, and Tomas Pfister. Data valuation using reinforcement learning. In International Conference on Machine Learning, pp. 10842-10851. PMLR, 2020.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7252-7261. www.mmmr....r..... http:///wwm.mmmeeee/////////rroe (areroccchii999hhmll 19a.html
PMLR, 09-15 Jun 2019. URL
Chunxu Zhang, Guodong Long, Tianyi Zhou, Peng Yan, Zijian Zhang, Chengqi Zhang, and Bo Yang. Dual personalization on federated recommendation. arXiv preprint arXiv:2301.08144 2023.
Jacky Zhang, Rajiv Khanna, Anastasios Kyrillidis, and Sanmi Koyejo. Bayesian coresets: Revisiting the nonconvex optimization perspective. In International Conference on Artificial Intelligence and Statistics, pp. 2792- PMLR, 2021.
12

---

Published as a conference paper at ICLR 2024
Shijie Zhang, Hongzhi Yin, Tong Chen, Zi Huang, Quoe Viet Hung Nguyen, and Lizhen Cui. Pi- pattack: Poisoning federated recommender systems for manipulating item promotion. In Pro- ceedings of the Fifteenth ACM International Conference Web Search and Data Mining, pp- 1415-1423, 2022a.
Xu Zhang, Yinchuan Li, Wenpeng Li, Kaiyang Guo, and Yunfeng Shao. Personalized federated learning via variational bayesian inference. In International Conference on Machine Learning, pp. 26293-26310 10. PMLR, 2022b.
13

---

Published as a conference paper at ICLR 2024
Supplementary Material: Bayesian Coreset Optimization for Personalized Federated Learning
In this supplementary material we discuss extensively on the proofs. involved for the theoretical analysis for PORDSEE-PFEDBAYES along with more fine-gre ane-grained experimental details and corre- sponding baselines.
PROOFS
Here discuss the proofs involved with particular propositions and theorems specified the The- oretical Contributions of this paper. Utilising the assumptions taken in Zhang et al. (2022b), Polson Ročková (018)) we consider the analysis for width Bayesian Neural network.
Assumption 1: The widths of the neural network are equal width i.e. = M.
Assumption 2: Each individual client [N] has equal coreset size of samples ny n.
Assumption 3: Parameters so. (total client dataset size) ( coreset client dataset size). M, L (number of DNN layers as per Section 3 are large enough such that the sequence o is bounded as follows
T' T 02,
of
8n
where r = Ω2M and
= log (3) (27)-2(1++1) 2(1++1
(( (00 +1+ +
2
2
(2T)²
(2r 1)3
Here indicates the total number of parameters defined in Section Similarly, utilising the coreset regime, we have the following:
o
8m1k
Since ni << n, hence 22 >> o
Assumption 4: We consider I-Lipschitz continuous activation function (()
We also define here few terms defined in (Zhang et al., 2022b) which would be useful for our following proof proposals as well.
Definition 2. Preliminaries and Definitions required for theoretical proofs under PFEDBAYES
dP P') = P') =Fx(1= -
BO?
Tn ((L+ + (LL+1)T/n) log M + (T/n) log (so/n/T
c 0ce(L.5), 11 5Ω |1 1
inf
En n
(I+ (I+1) 1)glog M T Tlog (so (soVn/T) log
log"(m) (m) Vrn logo(o)))))
where 8
14

---

Published as a conference paper at ICLR 2024
Here In indicates the variational error incurred due to the Bayesian approximation to the true pos- terior distribution in Equation and Sn indicates the approximation error incurred during regression w.r.t the actual function to be learnt.
Similarly for the coreset size ny we define the following:
Definition 3. Preliminaries and Definitions required for theoretical proof' under CORESET- PFEDBAYES
( P') Ex:(1 - ((xx))))) 802
Smz 0E0(1,5))11011 5Q
inf
110, - --l
Tmu = -( 1|T/+ m)) log M (T/nz) log 80 /ovnn// nx/T)
Enu = / log M T log (so/ log" (mx) /rnn log (nx)
Lemma 1. The Hellinger Distance from Definition 7 is symmetrical in its arguments Pa and pi
Proof. It is easy show that,
(/b(xt) ("(x) 852
P') = =Ex: -
(10)
=xx:(1
(11)
d(Pii, Pbo)
(12)
PROOF. OF THEOREM
Theorem 1. The difference in the upper bound incurred in the overall generalization error of CORESET-PFEDBAYER as compared war that of PFEDBAYES is always upper bounded by closed form positive function that depends the coreset weights and coreset size- S(w, nz). generaliza- tion error in the original full data setup
N
N
1
Σ
/ e
((P)
1
d (0; wid0
S (ww, mu))
N
N
1=1
Proof. Let us define log ()( Pa Pt) (P Pt) + nd2 (P P
Using Theorem 3.1 of Pati et al. 2018) with probability at most Creee with high probability for CORES RT- PT-PFEDBAYES we have
where C is constant,
/ (PPb p²)==(())))))))))))))))))))))p (0) S Cuecin
(13)
Similarly with high probability at most e Cne? for the vanilla PFEDBAYES
/ n(Pb. P')2" (0)d0 < One
(14)
Using Lemma A.I from Zhang et al. (2022b) we know that for any probability measure and any measurable function with e E (1()),
log f (())
(M u(dr)) = sup / (Iimlodnn Dez(ellu)
15

---

Published as a conference paper at ICLR 2024
Further, we let 4n (Pi, Pb) is the log-likelihood ratio of Pt and Pi
Pi (
In (Pi, Pg) = loog
Po (D))
Hence,
Pt) = (n P')/C - log log
= (Pt, Po)/C log (P Po) since (PG, Pi) = dF(Pi, Pi) from Lemma
This follows from 9 Similarly, for the weighted likelihood based Hellinger Distance,
(Po.o: Pi) n In Ln (P)), log n (Po. Pt)
(15)
By using Lemma A.1 with h(n) = log (Po, Pt) .1 =0) and = (i), we obtain
/ d (Ps d'(0)d0 <
In (pr Pb) (O) + De (0))===²()) + log / 1 Pt) =0))d0)
1
La (P' (') + DEL (6 (0)(
+ +C
5
| dP wo (0) wid0 <
1 [ Tik [ / in (P', Po.w) (((0:))d + DeL w)))==
Ce
Utilising analysis under Supplementary in Bai et al. 2020), there exists an upper bound for the term
/ In (P', Pb) (0)d0 s c" (nr, + ngg))
(16)
Lemma 2 from (Zhang et al. 2022b) provides the upper bound for the KL divergence term
Del (0((0)12²²()) < c' (nrw)
(17)
16

---

Published as a conference paper at ICLR 2024
Therefore we write the following expression that captures the weighted Hellinger distance dis- placement given in our coreset framework PEEDBAYS as compared to PFEDBAYES
N Σ
N Σ
(Pb P ))
d (P's 0,w P p')g'(( (0, w)d0
N
N
+ +ce ( In (P', Pe) (0)00 DeL (0)) (0))
N 1=1
Σ 7
/ In (P', Po.w) (0; w) do Drr n² (0; w) ())=))ll=>0 [1)) 0
1
Ce
Using Eq: 17 and Eq: 16
C" N
S C Cs + C' C' +
Σ
5n
-2kk C' C'Crra
+
SnL
C"
$ <C(=, 2 + +5C' (nT -ngTm,)
+
(nEt nl
N
N
C (c ) +CC (nra mer
(ng) St
N
Extmaaion cmer Type Drift
emror Type Drift
Approximation Firor Drift
5(w, m4)
Where S(w, n&)
(ngt Cx. where each of the coefficients of the closed form function are constants related to 80, B, A, L, M,C and ni
C(e = CC nTTm)) + S EE
Using Lemma and with suitable assumptions the Approximation drift error such that that each of the indiiidual error terms are positive, there by indicating S(w, n&) 2 0
Lemma 2. The Estimation error Type Drift a positive quantity Le. nr, > DgTry-
Proof. By Definition,
Fn = 1og (T/n) log (soVn/T)
and
[ lo/ M + (T/nx) log sov nx/TT)
Fin
Hence
((L+ + 1)7) log M (T) log so sov nn/T
Tn
Bg X
Tak
((L+ 1)7) log M (T) 80 sovnn. /
(L 1++1 1)+1)ogg M + log 500/ /o/vT) + log (vri
Tnk
(1+1)log M + log 500/ vT) + log (VTLE)
Considering (L + 1) log M + log /00/ /T
as a constant & we have
17

---

Published as a conference paper at ICLR 2024
Tn
n
+ log v1)
x
Tnp
n
8 + log \Tt)
Thus,
6 + logg n))
+ log( /TL))
is clear since log( is an increasing function for > ny we have nr, > ngTne Lemma 3.
The Estimation error Type I Drift is a positive quantity i.e. E > en
Proof. From the definition under Assumption 3
Enz = 12 - (L 1)T log M Tlog (soV nx/TT log" = Tan dog"((14)
Hence e2 .nn (og)) Similarly, oh rn log 2 (n)
=2 In log Jog2010 (n)
log 29 (m))
nr. n
2?
I'n log (nx)
nyTT (mg))
From Lemma 2 we know that nr, ng hence
log 25 (12)
E
log 20 (ma)
ExL
This follows due to the increasing nature of the function.
PROOF. OF THEOREM 2
Theorem 2. The convergence rate of the generalization error under L norm of CORESET-PFEDDANYE is minimax optimal up to logarithmic term (in order n&d) for bounded functions (5- 3--Höder-smoott functions) { { and {fe ) where Cz. C3 and 8' are constants and A being the intrinsic dimension of each client's 's data:
CF N i=1
Σ
N
|1
(x') fr fr (xx
20 / (0; w)d s Con log2 (nz)-
and
N OF
inf (1 f x
Σ
/ |156 (x') f (x*)
2 Cyn x
N
1=1
where ny denotes the coreset size per client dataset and denotes the original per client dataset
2 (Pp.....
1-exp
ax22 807
size and
2
A CF-
(X-))
4F"
18

---

the approximation error is upper-bounded as Published as a conference paper at ICLR 2024
We present the choice of I for typical class of functions. Hölder-smooth functions (Definition (Nakada & Imaizumi 2020)) and the intrinsic dimension of data is A.
We already assumed that {f 3-
From our above theorem result from Theorem: we say the following:
C"
P' < Cen
C'ran
(18)
NC
Utilising Corollary 6 in (Nakada & Imaizumi 2020 follows
f I'l I CoT
where Co > 0 is constant related to so, and A
Thus from the above definitions'
2 and
3 we have the following
Sxz Sng
I Co C
A 1, 1,...,N N
Utilising the above upper bound in 18 and substituting T = Cnn Int we get
N (P
C" N
S C +C' +
CoT
N
NÇ
C" N
S Crai log 28 (nx) + C'rn +
Σ NÇ
CTT enn, = -nne log 28 (nx)
S C2ng
28
log
28'
[n+) substituting T in rns.
where 8' > 1, and C₁ C2 0 are constants related to sp. B, A, M, and 71g-
Similar to Theorem 1.1 from (Bai et al. 2020) and Theorem from (Zhang et al. 2022b) norm, we write the following
N
CF
1 (x') - (X') ('(0, w)d0
N
1=1
1
dl (Po wo Po) (0: w)
N
C2n
ToEK log 28
(nz))
Now, using the minimax lower bound under 2 norm in Theorem 8 of (Naaada & Imaizumi 2020). we see that for coreset regime the same formulation holds similar to our original setting as shown in (Zhang et al. 2022b)
inf
CF N
Σ i=1
|| (X') ff IX w0) 2 Can .
11 sF) sF ( (1r S
where C3 > 0 is constant.
Combining the above equations, the convergence rate of the generalization error of the coreset weighted objective is minimax optimal upto a logarithmic term for bounded functions { I. and [
19

---

Published as a conference paper at ICLR 2024
PROOF OF THEOREM
Theorem 3. The lower bound (L.b.) incurred for the deviation for the weighted coreset CORESET-PFEDBAYYR (5) generalization error always higher than the lower bound of that for
28
the original PFEDBAYES objective with a delta difference (Error Error II) as (
284A
N
N
Σ
1o. (x') S (xx) (0, w)dd
/ (x))
1.b.
Coreset weighted objective Generalization Error (Error I)
Vanilla objective Generalization Ervor (Error II)
Proof. As know n& hence Cyn2 23-A or ng), which therefore means that inequality holds the lower bound (l.b.) of the two expressions (shown by the previous proposition 2).
Cnn T*** (: C3 is constant independent of
Σ/
Σ
1o (x+) ff (
900 wide
Σ / (x') - (x+) 12= (0)
(x²)
L.b.
Let us denote A 1.b deviation as follows
N Σ
/ 1 - (x (0, w w)d0
Σ (x - (x') {x) ())
Lb deviation
l.b.
1.b.
And the A l.b. deviation term is given by C3nk
28 ス
23 Cz
28 2 O(,
25+ス
PROOF. OF THEOREM 4
Theorem 4. The lower bound incurred the overall generalization error across all N clients of CORESET-PFEDBATTE always higher compared that the generalization error in the original full data setup
N NΣ N
N
/
(P
(0; wd
N
t.b.
1=1
Proof. It is easy to show since from Theorem 3 and also since n 7& holds, hence rewrite follows:
we know the lower bounds for the individual terms
N Σ / (P)
(Pp (') w w
2 Czn мकィ
28 Czn
2 0
The implication of this proof states that the overall enror incurred due to coreset weighted deviation always more than that of the original deviation which can be measured approximately in order of nk, the coreset sample size.
Proposition 1. The grudient of the first term in Equation 7 i.e.
V.DDKI ((0:w) ())
is given by the following expression
20

---

Published as a conference paper at ICLR 2024
/ V Vu l +1 log (0) d0
where
V (w
(0:0) ee e'(0,.m Im (ur) + 9m(w)V V. Ie e (0sxx and (0; w)
I e' W)
Proof.
V..Dall w ())
V (0:w)
log (0; - (
= -V 1/
(():
-1 v (00::))og(((0:w))
(19)
V ())0:w)log 0 d0
(0: d0 / log (0: w dd0
log (
v V.. v w) log (0: + 1 log Io
d0
In order to compute the gradient V 4' w). the following objective can be utilized.
Let 2" (0) be the optimal variable solution Equation 5
FT" (2*)
0
g'(w:w)
(i)
log (D')
+ +vV Do) (0)
0
(((ww)
('(6:w
Second Part
First Part
For the first part,
- log Pa.. (D') (O))
((())
(6:w)
V (0)
-log log Po, (D" )r' (0)
d0
4r (6:w)
(20)
(0)V (0) log P (D') log Pe (D)) do
Modified First part
By the assumption that the distribution (0) satisfies mean-field decomposition i.e.
('( II NO z
(21)
T Ø (00m)
Y72
21

---

Published as a conference paper at ICLR 2024
Let us denote MM Po, Therefore, extract out the following portion from 20 V (())) log Po (D))
(0) log Po.w(() = Vor(o) Mu
(22)
We now consider the individual partial differentials here
1 2M dw dw
log Mw
(23)
Oe' ( (m
M.
(0.m))
Thus, we can rewrite 20 from the perspective of individual components of g' (0) as follows:
/11 (0) 1 M. dw (0,,)
log Po ( (Di)
d0
Mw Sw
(((0:)
1
(P dw
(24)
(0)
+ log Po (Di) d0
Po
(D')
Sw
0e' (0s.m)
(=(::)
Modified First
Now, we can rewrite the second part follows:
<V (0)0
q (d:w)
1/8 r² (' o' (0) log(== (0))d0
97 ('(ww)
= / V (o(()) (f (0) (0) log( ==" (0)) de
(25)
(((0:))
= c / log (0)) + log( = (0)) d0
('(w:)
Modified Second
Combining both the first and second part we get
/
OP (D") Ow
(0(
log (e do
8
Po.m (D')
Ow
Oe (0).m)
+ / log ( w) log(2" 10)) d0
(26)
5 l log (0: +1 11-++1-ggh +1 (0)) + log Po..v m (D)) d0
= ( (D)
8P (D*
Ow
Ow
0e (0
Let us assume without loss of generality that each of the individual components of the optimal coreset weighted client distribution q'(0; w)) can denoted as some function gww). More, specifi- cally,
22

---

Published as a conference paper at ICLR 2024
e (0i.ji w) (j(w) V e () = (
(27)
Thus reuse the above expression to simplify (26
(w)) = Se (0(00 w) (D!)
(P,...DD) the
d0
(28)
< Se log (0; +1 log(=*(0)) log Po. (D)) do
We now go back to utilizing the above derived expression in our main Eq. 19 to replace
V o (0; w)
V Vu: w)
T
00 w)
9h (w)
1=1
T II km
(29)
[m(w) V.
II
(2w
II g (e (w) (m V. 1 II (x) k=fm
9¹ (0: w)
0¹ (0 w) Im (w) + 9m (w)) V II k-mm
e e' w)
Thus, we now have a closed form solution to computing the gradient of the KI. divergence D(q (6; w) w)ll w.t. the coresct weight parameters w.
Proposition The grudient of the second term in Equation w.nl w i.e.
V (P Po. (D ||3
given by the following expression
-2P" P Pa w
where P =E
9y and P+ 02. ... Gnl
Proof. First, we reformulate the given expression in terms
Po(DD Po.. Po. )12
E (Po(D') Po. (D') 2
23

---

Published as a conference paper at ICLR 2024
We define 9j Po((D)) - Pe
As result the equivalent optimization problem becomes minimizing Eラ= 9j Ej= ووDر |
n.2
Further, using Monte Carlo approximation, given S samples {0,} 8g ~ n. the L² -nom can be approximated as follows
93
U
j=1
where
0j V8 (D.)) Po, -(.D))....,PPe P(D))] and P(D))
3 Po (D))
We can write the above problem in matrix notation as follows
((w) Iip Pa
where P Σド= Bj and Pe = [9t, $2.... gml
Thus we have the gradient w.r.t u as follows:
Vuf(w)= -2P3 p P
(30)
10 EXPERIMENTS
All the experiments have been done using the following configuration: Nvidia RTX A4000(16GB) and Apple M2 Pro 10 cores and 16GB memory.
10.1 PROPOSAL FOR A MODIFIED OBJECTIVE IN EQUATION g
[w;"] A arg min DKL (0 w) (0)) + P Po.w, [
||w;|lo S k
(31)
We discuss here the utility of our proposed modified client side objective function via ablation study where we want to gauge the inclusion of the first term in our objective function as just inlcud- ing the coreset loss.
Through experimental analysis, we find that just including the coreset loss optimization results in early saturation, possibly hinting towards getting stuck in local minima, but however inclusion the KL Divergence loss and forcing the coreset weighted local distribution of the client and the normal local distribution of the client to be similar leads better stability in the training loss and better convergence.
10.2 COMMUNICATION COMPLEXITY ANALYSIS FOR DIFFERENT CORESET SIZES
Here showcase analysis for different coreset sample size for different datasets and how it af- fects on the final accuracy and the total number of communication rounds the Federated Learning setting. This showcases cost-effectiveness of our approach where by using only small number of communication rounds our proposed approach is able to attain near-optimal performance as per the table below. In addition Fig: 5 substantissss the cost-effectiveness of our approach.
24

---

Published as a conference paper at ICLR 2024
300
Communiaaionn Founds
Figure 4: Ablation Study on using KL divergence between two local distribution w.r. just using coreset weights
Table 3: Comparative results of test accuracies across different coreset sample complexity
Methed (Peroentage sampling fraciio)
MNIST
CITAR
Tixt Acturacy
Communiction Rounds
Tee Accunay
Communcation Roand
Toxt Aucurac
Conounictioon Rounds
PFEDBAYES (Pall)
98.79
194
93.01
215
K3.46
RANDOMICESEE (50%)
30.2
135
87:12
172
43.31
(k 98%)
92.48
93
CORESE P-PRSEE-PEEBBAVRS 98%)
90.17
72
CoRKSEx-PFEnB.Ve
15%)
85
$5.06
ConESEv-PiEDBAYE (k 10%)
35.43
32
82.64
24
48.25
(a) We report test accuracies acToss different sample complexity for datasets like MNIST, CIFAR, Fashion- MNIST. Full indicates training full dataset and 50% is using half the data size after randomly sampling 50 the training
Figure 5: Communication Rounds across Different Sample Size Convergence analysis
10.3
COMPUTING LIKELIHOOD OBJECTIVE USING AIHT
Here, showcase how we utilised the Accelerated Iterative Hard Thresholding algorithm (A-IHT) for computing the likelihood.
10.4 MEDICAL DATASET EXPERIMENT DETAILS
Owing the rise of Federated Learning based approaches in the medical setting due privacy- preserving features, we chose to perform our experiments on medical datasets in addition to our main experiments.
For our Federated Learning setup, we considered the setting where we have only 2 clients and one global server.
For each of the 3 datasets in the medical dataset setting, we consider each client has X-ray images of symptomatic type A/ type B and Normal images. We perform classification task at each client.
10.5 BASLINE COMPARISONS: DIVERSITY BASED SUBMODULAR OPTIMIZATION FUNCTIONS
For our second set of experiments, we chose different diversity based submodular optimization functions, specifically the following functions whose definition have been provided here
25

---

Published as a conference paper at ICLR 2024
Definition 4. Log-determinant Function is diversity-based submodular function. It is non- monotone in nature. Let denote a positive semidefinite kernel matrix and Lg denote the subset of rows and columns indexed by set S. Log- degerreeertnnttttemme function f is specified as:
f(S) logdet(Ls)
(32)
The det function models diversity and is closely related to determinantal point process. Definition 5. Disparity Sum Function characterizes diversity by considering the sum of distances between every pair of points in subset S. For any two points i. SS let dij denote the distance between them.
f(S)
Σ de
(33)
The aim is to select subset S such that f(S) is maximized. Disparity sum is not submodular function.
Definition 6. Disparity Min Function characterizes diversity by considering the minimum distance between any Iwo non-similar points in subset S.
f(S) = min dij
(34)
(.jE8,i7)
The aim is to select a subset S such that ((S) is maximized. Disparity min is not a submodular function.
For the above experiments utilise the Submodlib library for our implementation Kaushal et al. 2002).
10.6
EXPERIMENT CONFIGURATION
10.6.1
MNIST EXPERIMENT CONFIGURATION
For both CORESET- CORESET-PFEDBANE PFEDBAYES and corresponding baseline PFEDBAYES, we use a fully con- nected DNN model with layers 180,100,100 MNIST dataset.
Learning rate hyperparameters: As per Zhang et al. (022b)'s proposal i.c. PFEDBAYES the learning rates for personalized (client model) and global model (m) 72) to 0.001 since these choices result in the best setting for PFEDBAYES. To compare against the stable best hyperparame- ters of PAED,AYES, we also fix the same for our proposal CORESET-PFEDDAYER
Personalization Hyperparameter: The Ç parameter adjusts the degree of personalization in the case of clients. Again for fair comparison against our baseline PYEDBAYES, for our proposal CORESET-PPEDD..AT (2022b) the authors tune Ç E (0.5, 1,5, 10, 20) and find that ( 10 results in the best setting. We, therelore, fix the personalization parameter = 10.
fix the ( parameter to the best setting given by the baseline. In ZZang et al.)
10.6.2 MEDICAL DATASETS EXPERIMENT CONFIGURATION
We discuss here the detailed configuration and models used for our further experiments.
Here specifically consider the setting where only have 2 clients and single global server. Each of the 2 clients are assigned with data from only 2 classes along with a shared class for classi- fication purpose.
For example, client 1 has class A and Normal (shared class) images while client 2 has class B and the remaining Normal images.
COVID-19 Radiography Database: Client has COVID-19 x-ray images while client 2 has lung opacity x-ray images. Normal X-ray images are shared across both clients. Fig. depicts the dataset distribution. For random subset selection, we randomly choose À = 0.1 fraction of samples on the client side. For diversity- -aase subset selection, we first convert each of the x99x299 images into a [512x1] 1] vector embeddings using a ResNet architecture. Diversity functions are then applied to
Submodlib decile library
26

---

Published as a conference paper at ICLR 2024
DOVB
Norel
Chard
A-1
Figure 6: Data distribution for Medical Datasets
these embeddings to retrieve final subset of diverse and representative embeddings. Eventually, we decode back to the original space using the chosen representative indices.
APTOS 2019 Blindness Detection: Unlike the COVID-19 19 radiography dataset, the APTOS dataset has RGB channels and higher resolution. We rescale the dimension of images to 299x299 for maintaining uniformity across all datasets. The same model configuration is followed as in the COVID-19 radiography dataset.
OCTMNIST: The OCTMnist dataset is a large dataset with single-channe images of a higher res- olution. We have resized the images to x99 299 resolution for our experiments. The Normal class has above 50,000 train images itself, with the other two classes having close to 10,000 train images. Due to this class imbalance, iments. Post which we again use a ResNet architecture to reduce the feature dimensions, which we then feed into the CORESET-PFEDDATE pipeline.
have randomly selected 8,000 images from each class for our exper-
Baseline Independent Learning In this scenario, each of the 2 clients solve the classification problem independently without any involvement of server opposed to federated learning. Thus there is no sharing of model weights to a common server as compared to the federated setting.
Baseline Independent Learning other client's test data In this scenario, similar to the inde- pendent learning setup, we report the metrics for particular client not only on its own test data but also on the other client's test data by training on the individual client's own training data.
For all the experiments for the medical dataset analysis across all the baselines, we report the class- wise accuracy Table 2
Definition 7. Submodular Functions are set functions which exhibit diminishing returns. Let V denote the ground-se set of n data points {a1, 32, 22.... where x; Rd. More formally, {xx) Let A C B where A, B C and v E V. A submodular function J 2 + R satisfies the diminishing returns property as follows:
f(AUe) f(A) 2 f(Uu)) (BB
(35)
10.7 ALGORITIM FOR ACCELERATED IHT
We first present the accelerated IHT algorithm proposed in Zhang al. 2021 in Algorithm 10.7
27

---

Published as a conference paper at ICLR 2024
Algorithm 2 Accelerated IHT (A-IHT) for Bayesian Coreset Optimization Input Objective ((w) |lv Sw] 2; sparsity
I: = 0, 20 O, wo = 0
repeat
3: Z = supp (z+)
S supp (TNVA (VI (2+)) U 2 where |S] - 3k
5: V. = v (zr)ss
Ie = arg min, f (2 (2- v. = 2 05 1
7: w++1 ney (24 14 f (24)
8: T++1 arg min, f (Wi+1 + T (W1+1 wL)
(v -u.-1, ( 1
9:
2++1 W/+1 + Te+1 (W4+1 W+)
10: t=t+1
11: until Stop criteria met
12:
return W/
The algorithm Accelerated IHT above is proposed by Zhang et al. (2021) We share a high level view of the algorithm include some of the important features.
Step Size Selection The authors propose that given the quadratic objective of the coreset optimiza-
tion, they perform exact line search to obtain the best step size per iteration. | 2
Momentum The authors propose adaptive momentum acceleration as is evident from line 8 of the pseudocode. At the end during the next update, Nesterov Accelerated Gradient is applied as shown in line 9.
11 CODE
We share our code GitHub at Link
28


Reviews:
[{"review_id": "01OnEveC0T", "reviewer": "Reviewer_YU5h", "summary": "The paper describes a method to use Bayesian coresets for each individual client in a federated learning setting. Bayesian coreset can be used as proxy for full data at each individual client to estimate client-side distribution. The authors describe objective functions to incorporate the Bayesian coresets with federated learning setting. The authors give an algorithm and also give theoretical guarantees for the generalization error and its convergence. The authors support their theoretical claims with empirical results comapring their proposed approach with a number of baselines.", "questions": "1. Why the prior $\\pi$ in equation 1 is replaced by $\\mathbf{z}$ in eq.6 - the modified client-side objective. Please clarify.\n\n2. The subsample size is 50%. Is it not quite large? Does it give significant computational time benefits when compared with full data? Other than figure 3, there are no experiments mentioning computational efficiency.\n\n3. Not a question but a suggestion. Algorithm 1 is not easy to follow for anyone unfamiliar with existing work or similar algorithms. How exactly is the coreset getting constructed? It would be good to give a high-level description of the same. \n\nOverall, the paper appears sound and I would be happy to raise my score once the doubts are cleared.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is, for the most part, well written. There is not much work in terms of coresets for federated learning and as such the paper will be of interest to the community.\n2. The authors have compared their method with a variety of baselines consisting of both - federated learning algorithms and also sampling strategies that incorporate diversity.  Their method performs well in most of the cases.\n3. The algorithm is backed with theoretical guarantees. I did not check the proofs, but the statements appear sound.", "weaknesses": "1. I am not sure what is the challenge in incorporating the Bayesian coreset framework in federated learning setting. It would be better to explain clearly why this is a significant contribution. Both the algorithm and proof techniques appear to be heavily inspired from Zhang 2022b. The only modification seems to be use of Bayesian coresets. \n\n2. There are minor grammatical errors. Please do a grammar check.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mgCGLlJuXp", "reviewer": "Reviewer_QB83", "summary": "This paper introduces an optimization framework for personalized federated learning by incorporating Bayesian coresets into the model proposed in [1]. The author want to ensure that the accuracy performance does not deteriorate when applying coresets. To achieve this, they have made modifications to the common coreset objective. Furthermore, they provide proof of the convergence rate of generalization error using their approach and evaluate the effectiveness of their method on a range of datasets.\n\n[1] Xu Zhang, Yinchuan Li, Wenpeng Li, Kaiyang Guo, and Yunfeng Shao. Personalized federated learning via variational bayesian inference.", "questions": "- What is the benefits to apply coreset in the personalized federated learning? I think one of the most important is that it can reduce the communication complexity. It would be valuable to investigate and quantify the extent to which the coreset approach reduces communication complexity in the specific optimization task addressed in this work. This can be done theoretically, by providing a complexity formula, and practically, by presenting numerical results from experiments that show the reduction in communication complexity achieved.\n- the intuition behind the new objective in eq. 9 is not very persuasive. If you could compute a coreset with a sufficiently small loss as defined in eq. 3, it is unecessary to add the term representing the “distance” between $\\hat{q}^i(\\theta, w)$ and $\\hat{q}^i(\\theta)$ since $\\hat{q}^i(\\theta, w)$ and $\\hat{q}^i(\\theta)$ will lead to closed losses; On the other hand, if you couldn’t make it under the constraint $\\| w \\|_0 \\leq k$, which means there is no such small coreset with ideal error, the coreset method could not work well. It would be beneficial to clarify the merits of the new objective, such as its robustness or any other advantages it offers. Experiments that demonstrate the effectiveness of the new objective would greatly strengthen the argument.\n- Does the modifications of eq. 6 consist of the following two parts: i) use the weighted likelihood. ii) replace prior distribution with global distritution. I am not sure for that.\n- is there any strategy for choosing the value of k in practice?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The integration of Bayesian coresets with federated learning is innovative.\n- In the context of personalized federated learning, this work presents new ideas and considerations for defining the objective in coreset computation, which differs from the commonly used coreset definition.", "weaknesses": "- The paper's content is a bit bloated, and the use of notations can be messy. For instance, sections 3.2 and 4 could be condensed to make them more concise. Additionally, there is potential to simplify the formulaic aspect.\n- It would be beneficial if the author could emphasize their novel contribution, distinguishing it from the techniques previously proposed by others. Currently, these ideas seem to be mixed within the intricate details of the interpretations.\n- The overall architecture, as well as certain smaller techniques and theoretical analysis methods, seem to be largely derived from previous work.\n- The contribution on the coreset construction is limited. Although the authors introduce a new coreset objective, they do not provide sufficient NEW optimization techniques for the new objective. I could only identify some techniques borrowed from previous work.\n- In my opinion, the primary contribution of this paper is the modified objective (eq. 9) tailored to personalized federated learning. However, the advantages of this modified objective are not adequately elucidated in the current presentation.\n\nsome minor problems\n\n- In section 3, there is a confusion of n and N. For example, n in Fig 1 should be N. \n- In section 3.2 , it should be $ g_j = \\mathcal{P}_\\theta(\\mathcal{D}_j^i) = E_{\\theta\\sim \\hat{\\pi}} P_\\theta(\\mathcal{D}_j^i) $.\n- The subscript of the bold variable should not be bolded if it is a scalar.\n- many other typos, e.g. missing equation references and confusing sentence like “For the first term in Equation 1, the authors we use a minibatch stochastic gradient descent …”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OBgnw6LrcV", "reviewer": "Reviewer_FxK1", "summary": "The work incorporated granular-level bayesian coresets optimization in Federated Learning. The proposed approach gave minimax convergence rate and showed good performance in empirical studies.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of incorporating coreset optimization in FL is new and well-motivated.\n2. Solid theoretical results are given.\n3. Some optimistic empirical studies are presented.", "weaknesses": "1. The major weakness is the lack of convergence comparison in the empirical part. One of the major concerns in FL is the communication cost. Thus the number of iteration rounds is crucial in FL. The reviewer suggests not only including the comparison of the final accuracy under (maybe different levels, not only 50%) of sample complexity, but also including the convergence speed, i.e., the communication cost comparison.\n\n2. How expensive it is to calculate the coreset samples/weights? Is there any empirical runtime results?\n\n3. How is \\hat{\\pi} defined in Eq. (3) and (4)?\n\n4. Some typo: first sentence in section 3.2 is incomplete. Different places for \\hat notation in q^i(\\theta, w), on q, or q^i, or q^i(theta, w).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)