QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning
Authors: Kaan Ozkara
Navjot Singh
Deepesh Data
Suhas Diggavi
Venue: NeurIPS 2021 Poster (2021)
Keywords: Personalized Federated Learning
Distributed Optimization
Model Compression
Knowledge Distillation

Abstract: Traditionally, federated learning (FL) aims to train a single global model while collaboratively using multiple clients and a server. Two natural challenges that FL algorithms face are heterogeneity in data across clients and collaboration of clients with diverse resources. In this work, we introduce a quantized and personalized FL algorithm QuPeD that facilitates collective (personalized model compression) training via knowledge distillation (KD)  among clients who have access to heterogeneous data and resources. For personalization, we allow clients to learn compressed personalized models with different quantization parameters and model dimensions/structures. Towards this, first we propose an algorithm for learning quantized models through a relaxed optimization problem, where quantization values are also optimized over. When each client participating in the (federated) learning process has different requirements of the compressed model (both in model dimension and precision), we formulate a compressed personalization framework by introducing knowledge distillation loss for local client objectives collaborating through a global model. We develop an alternating proximal gradient update for solving this compressed personalization problem, and analyze its convergence properties. Numerically, we validate that QuPeD outperforms competing personalized FL methods, FedAvg, and local training of clients in various heterogeneous settings.

QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning
Kaan Ozkara
Navjot Singh University of Califomia, Los Angeles navjotsingh@ucla..oo edu
University of California, Los Angeles kaanQucla.ed.
Deepesh Data University of California, Los Angeles datreesaalaaa@amaa.....m com
Suhas Diggavi
University of California, Los Angeles suhasSee. ucla.
Abstract
Traditionally, federated learning (FL) aims to train a single global model while collaboratively using multiple clients and server. Two natural challenges that FI. algorithms face are heterogeneity in data across clients and collaboration of clients with diverse resources. In this work, we introduce quantized and personalized FL. algorithm QuPeD that facilitates collective (personalized model compression) training via knowledge distillation (KD) among clients who have access to heterogeneous data and resources. For personalization, we allow clients learn compressed personalized models with different quantization parameters and model dimensions//sructures. Towards this, first we propose algorithm for learning quantized models through relaxed optimization problem, where quantization values are also optimized over. When each client participating in the (federated) learning process has different requirements for the compressed model (both in model dimension and precision), we formulate compressed personalization framework by introducing knowledge distillation loss for local client objectives collaborating through a global model. We develop an alternating proximal gradient update for solving this compressed personalization problem, and analyze its convergence properties. Numerically, we validate that QuPeD outperforms competing personalized FL methods, FedAvg, and local training clients in various heterogeneous settings.
Introduction
Federated Learning (FL.) is a learning procedure where the aim is to utilize vast amount of data residing in numerous (in millions) edge devices (clients) to train machine learning models without collecting clients' data [26]. Formally, if there are clients and ft denotes the local loss function at client then traditional FL learns single global model by minimizing
arg min (S(w): := Σ f((w)) WERd
(1)
It has been realized lately that single model may not provide good performance to all the clients in settings where data is distributed heterogeneously. This leads to the need for personalized learning, where each client wants learn its model [7, 8). Since locally learned client model may not generalize well due to insufficient data, in personalized FL process, clients maintain personalized models locally and utilize other clients' data via global model. Resource diversity among clients,
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

which is inherent to FL as the participating edge devices may vary widely in terms of resources, is often overlooked in personalized FL literature. This resource diversity may necessitate clients to learn personalized models with different precision as well as different dimension/architectur.. Systemati- cally studying both these resource heterogeneity together with data heterogeneity in personalized FI. is the primary objective of this paper.
In this work, we propose a model compression framework for personalized FL via knowledge distillation (KD) [14] that addresses both data and resource heterogeneity in a unified manner. Our framework allows collaboration among clients with different resource requirements both in terms of precision as well as model ciimension//tructuree for learning personalized quantized models (PQMs). Motivated by FL. where edge devices are resource constrained when actively used (e.g. when several applications are actively running on a battery powered smartphone) and available for training when not use (e.g., while charging and on wi-fi), we do training in full precision for learning compressed models to be deployed for inference time. For efficient model compression, we learn the quantization parameters for each client by including quantization levels in the optimization problem itself. First, we investigate our approach in a centralized setup, by formulating relaxed optimization problem and minimizing it through alternating proximal gradient steps, inspired by [3]. To extend this to FL for we employ our centralized algorithm locally
learning PQMs with different at clients and introduce KD loss for collaboration of personalized and global models. Although there exist empirical works where KD is used in personalized FL [20], we formalize it as an optimization problem, solve it using alternating proximal updates, and analyze its convergence.
Contributions. Our contributions can be summarized as follows:
the centralized case, we propose a novel relaxed optimization problem that enables optimization over quantization values (centers) as well model parameters. We use alternating proximal updates to minimize the objective and analyze its convergence properties.
More importantly, our work is the first to formulate a personalized FL optimization problem where clients may have different model dimensions and precision requirements for their personalized models. Our proposed scheme combines alternating proximal updates with knowledge distillation.
For optimizing non-convex objective, in the centralized setup, we recover the standard conver- gence rate of O(1/T) (despite optimizing over quantization centers), and for federated setting, we recover the standard convergence rate of O(1/T) (despite learning PQMs with different preci- sions/dimensions). In the federated setting, our convergence bound has an error term that depends on multiplication of two terms averaged over clients: one characterizing client's local model smoothness and the other data heterogeneity with respect to overall data distribution.?
We perform image and text classification experiments on multiple datasets in various resource and data heterogeneity settings, and compare performance of QuPeD against Per-FedAvg [8], PFedMe [7], FedAvg [26], and local training of clients. We observe that QuPeD in full precision outperforms all these methods on all the datasets that we considered for our experiments. Further, our results show that even with quantization, QuPel) outperforms the other methods in full precision on multiple settings and datasets, demonstrating the effectiveness of our scheme for FL. settings.
Our work should not be confused with works in distributed/ficerrtee learning, where models/gradients are compressed for communication efficiency [2. 17]. We also achieve communication efficiency through local iterations, but the main goal of our work is personalized quantization for inference.
Related work. To the best of our knowledge, this is the first work in personalized federated learning where the aim is to learn quantized and personalized models potentially having different dimenssion//sruucuurss for inference. Our work can be seen in the intersection of personalized federated learning and learning quantized models; we also employ knowledge distillation for collaboration.
Personalized federated learning: Recent works adopted different approaches for learning personalized models: (i) Combine global and local models throughout the training [6. 12, 25]; (ii) first learn a global model and then personalize locally [8]: (iii) consider multiple global models to collaborate among only those clients that share similar personalized models [9, 25, 31, 34]; (iv) augment the
'odde compression (MC) allows inference time deployment of compressed model. Though MC is a generic term comprising different methods, we will focus on its quantization (number of bits per model parameter) aspect.
An error term depending on data heterogeneity is commonly observed in personalized FL algorithms [7, 8j.
2

---

traditional FL objective via a penalty term that enables collaboration between global and personalized models [7. 11, 12].
Learning quantized models: There are two kinds of approaches for training quantized networks that are of our interest. The first one approximates the hard quantization function by using soft surrogate [5, 10, 24, 32], while the other one iteratively projects the model parameters onto the fixed set of centers [1, 15, 19, 33]. Each approach has its own limitation; see Section 2.1 for discussion. While the initial focus in learning quantized networks was on achieving good empirical performance, there are some works that analyzed convergence properties [1. 21, 331, but only in the centralized case. Among these, [1] analyzed convergence for relaxed/regullarieed loss function using proximal updates.
Knowledge distillation (KD): KD [14] is a framework for transfer learning that is generally used to train a small student network using the soft labels generated by a deep teacher network. It can also be used to train two or more networks mutually by switching teacher and students in each iteration [35]. KD has been employed in FL settings as an alternative simple aggregation which is not feasible when clients have models with different dimensions [23]. [20] used KD in personalized FL by assuming existence of public dataset. [27] used KD in combination with quantization in centralized case for model compression; in contrast, we do not use KD for model compression but for collaboration between personalized and global model. Unlike the above works which are empirical, our paper is the first to formalize an optimization problem for personalized FL training with KD and analyze its convergence properties. Our proposed scheme yields personalized client models with different precision/dimension through local alternating proximal updates; see Section 2.2 for details.
Paper organization: In Section 2, we formulate the optimization problem to be minimized. In Sections 3 and 4, we describe our algorithms along-with the main convergence results for the centralized and personalized settings, respectively. Section 5 provides extensive numerical results. Omitted proofs/details and additional experiments are provided in supplementary material.
2 Problem Formulation
Our goal in this paper is for clients collaboratively learn personalized quantized models (with potentially different precision and model sizes/types). To this end, below, we first state our final objective function that will end up optimizing in this paper for learning personalized quantized models, and then in the rest of this section we will describe the genesis of this objective.
Recall from (1), in the traditional FL setting, the local loss function at client i is denoted by fi. For personalized compressed model training, we define the following augmented loss function at client i:
Px Ci w) = (1 (1- + fi(0a, (AQc + AA((x,, c:)
+ Ap (I, (x, H. (. (x.))
Here, w E IRd denotes the global model, x; E R.d, denotes the personalized model of dimension d, at client i, c; E TR. denotes the model quantization centers (where m. is the number of centers), Qc, denotes the noft-quantization function with respect to (w.r.t.) the set of centers C₁, R(x;, c;) denotes the distance function, f.K denotes the knowledge distillation (KD) loss [14] between the two input models client i's dataset, À is design parameter for enforcing quantization (large À forces weights to be close to respective centers), and A, controls the weighted average of regular loss and KD loss functions (higher A, can be used when client data is limited). We will formally define the undefined quantities later in this section. Consequently, our main objective becomes:
min ".E
(F(w, (xi). (c.))
= n Σ Fi(x,,
Ci Fi(x,,
(3)
WERd
Thus, our formulation allows different clients to have personalized models X1.... . Xn, with different dimensions di....,d d, and architectures, different number of quantization levels m...... m,, (larger the m, higher the precision), and different quantization values in those quantization levels. Note that there are two layers of personalization, first is due to data heterogeneity, which is reflected in clients learning different models XI,..., Xn.
Xn, and second due to resource diversity, which is reflected in clients learning models with different sizes, both in terms in dimension as well as precision.
3

---

In Section 2.1, we motivate how we came up with the first three terms in (2), which are in fact about a centralized setting because the function fi and the parameters involved, ie.. x, Ct. are local to client i; and then, in Section 2.2, we motivate the use of the last two terms containing f in (2).
2.1 Model Compression in the Centralized Setup
Consider a setting where objective function f Rd+m R (which could be neural network loss function) is optimized over both the quantization centers с E R" and the assignment of model parameters (or weights) x E R those centers. There are two ways approach this problem, and we describe these approaches, their limitations, and the possible resolutions below.
Approach 1. A natural approach is to explicitly put constraint that weights belong to the set of centers, which suggests solving the following problem: minx,e fx) + 6e(x), where de denotes the indicator function for с E RT and for any c E IR" x E R define 6c(x) == 0 V j E [d] J; = e for some с E {ci...., Cm otherwise, define 6e(x) := oo. However, the discontinuity of (x(x) makes minimize this objective challenging. To mitigate this, like recent works [1, 33], we can approximate de (x) using distance function R(x, c) that is continuous everywhere (e.g., the didisttnc,,, R(x, c) == min fllz - x||1 : 24 E {ci, ... ,m. Vi)). This suggests solving:
min fx) + AR(x. c).
(4)
The centers are optimized to be close to the mean or median (depending on R) of the weights; however, there is no guarantee that this will help minimizing objective f. We believe that modeling the direct effect that centers have on the loss is crucial for a complete quantized training (see Appendix E for empirical verification of this fact), and our second approach is based on this idea.
Approach 2. We can embed the quantization function into the loss function itself, thus solving the problem: mitlx.e (h(x, c) := (x)),, where for every E Rd, RT the (hard) quantization function defined as Qx(x); == Ck. where k = arg min] {lar; cg1], which maps individual weights to the closest centers. Note that Qe(x) is actually a staircase function for which the derivative w.r.t. x is 0 almost everywhere, which discourages the use of gradient- based methods to optimize the above objective. To overcome this, similar to [10, 32], we can use a seft quantization function Qx(x) that is differentiable everywhere with derivative not necessarily O. For e.g., element-wise sigmoid or tanh functions, used by [32] and [10], respectively." This suggests the following relaxation:
min (h(x, c) == f()x(x))). X,e
(5)
Though we can observe the effect of centers on neural network loss in (5); however, the gradient w.r.t. x is heavily dependent on the choice of Q. and optimizing over x might deviate too much from optimizing the neural network loss function. For instance, in the limiting case when Qx(x) - Qc(x), gradient w.r.t. x is almost everywhere; hence, every point becomes a first order stationary point.
Our proposed objective for model quantization. Our aim is to come up with an objective function that would not diminish the significance of both x and in the overall procedure. To leverage the benefits of both, we combine both optimization problems (4) and (5) into one problem:
min (F)(x.c) x.C
f(x) f(āx(x)) + c)(x.))).
Here, the first term preserves the connection of x to neural network loss function, and the second term enables the optimization of centers w.r.t. the neural network training loss itself. As result, we obtain an objective function that is continuous everywhere, and for which we can use Lipschitz tools in the convergence analysis which previous works did not exploit. In fact, we compute the Lipschitz parameters for a specific soft quantization function Qe based on sigmoid in Appendix A.
Remark 1. It is important to note that with this new objective function, are able optimize not only over weights but also over centers. This allows us to theoretically analyze how the movements of the centers affect the convergence. As far as we know, this has not been the case in the literature of quantized neural network training. Moreover, we observe numerically that optimizing over centers improves performance of the network; see Appendix E.
2[1] and [33] proposed to approximate the indicator function 6<(x) using a distance function Re(x), where c is fixed, and unlike ours, it is not a variable that the loss function is optimized over.
IIn their setup, the quantization centers fixed. In contrast, we are also optimizing over these centers.

---

2.2 Towards Personalized Quantized Federated Learning: Knowledge Distillation
Note that the objective function defined in (6) can be used for learning a quantized model locally at any client. There are multiple ways to extend that objective for learning personalized quantized models (PQMs) via collaboratio.. For example, when all clients want to learn personalized models with the same dimension (but with different quantization levels), then one natural approach is to add an le penalty term in the objective that would prevent local models from drifting away from the global model and from simply fitting to local data. This approach, in fact, has been adopted in previous works [7, 12, 22] for learning personalized models in FL, though not quantized ones.' 5
In this paper, since we allow clients to learn PQMs with potentially different dimensions, the above approach of adding lg penalty term in the objective is not feasible. Observe that, the purpose of incorporating a Eg penalty in the objective is to ensure that the personalized models do not have significantly different output class scores compared to the global model which is trained using the data generated at all clients; this does not, however, require the global model to have the same dimension as that of local models and can be satisfied by augmenting the local objective (6) with certain knowl- edge distillation (KD) loss. In our setting, since clients' goal is to learn personalized models with different dimensions that may also have different quantization levels, we augment the local objective (x,, w) and J.KD (Qc, (x:), w), where the first one ensures that the behavior of X; E I24 is not very different from that of w E R and the second one ensures the same for the quantized version of x; and Formally, we define them using KL divergence fol- lows: P.D (x:, w) = D (s (w) (x)])s(xx))) and S.D (Qc, (x:). w) == DxI. (s." (w)]ls:(Qe., (x:))), where s and denote functions whose inputs are global and personalized models, respectively and data samples implicitly and outputs are the softmax classification probabilities of the network.
(6) with two separate KD losses:
We need to train x; and w mutually. Identifying the limitations of existing approaches for thcoretical analysis (as mentioned in related work in Section 1), we reverse KL. updates (i.e., taking gradient steps w.r.t. the first parameter in (x .)) to train the teacher network w from the student network x,. This type of update can be shown converge and also empirically outperforms [29] (see Section 5). We want to emphasize that though there are works [20, 23] that have used KD in FL and studied its performance (only empirically), ours is the first work that carefully formalizes it as an optimization problem (that also incorporate quantization) which is necessary to analyze convergence properties.
3 Centralized Model Quantization Training
In this section, we propose a centralized train- ing scheme (Algorithm 1) for minimizing (6) by optimizing over x E 12d (the model parame- ters) and E IRT (quantization values/centers). During training, we keep x full precision and learn the optimal quantization parameters c. The learned quantization values are then used to hard-quantize the personalized models to get quantized models for deployment in memory- constrained setting.
Algorithm 1 Centralized Model Quantization
Input: Regularization parameter X; initialize the full precision model x" and quantization cen- ters c"; penalty function enforcing quantization R(x, c); a soft quantizer Qe(x): and learning rates 71 7/2.
1: for t. = 0 to T do
2: Compute g' = (x')+ f(Q.
Description of the algorithm. We optimize (6) through alternating proximal gradient descent.
3:
x ++1 = prox AR (x* - migg)
4: Compute h' = Vet f( (x(+1))
The model parameters and the quantization vec- tor are initialized to random vectors x0 and co. The objective in (6) is composed of two parts:
5:
cl+1 prox
(c' 722hh)
6: end for
Output: Quantized model & = Qe Qx²(xx)
the loss function fxx) + f(Qx(x)) and a quan- tization inducing term R(x, c), which we control by a regularization coefficient A. At each I, we compute gradient g' of the loss function w.r.t. x' (line 2), and then take the gradient step followed by prox step for updating x' to x (line 3). For the centers, we similarly take gradient step and follow it by a prox step for updating c to c (line 4-5). These update steps ensure that we
We also analyze this approach in our setting (for learning PQMs but having the same dimension), and we call this version QuPeL: see Appendix D for problem setup and convergence results of QuPeL. In Section 5, we demonstrate that QuPeD (for the same task but using KD opposed to the &2 penalty) outperforms QuPeL.
5

---

learn the model parameters and quantization vector tied together through proximal mapping of the regularization function R. Finally, we quantize the full precision model x" using Qe (line 7).
Assumptions. We make the following assumptions on f:
A.1 (Finite lower bound of f): f(x) -00, Vx E Rd, which implies Fx(x,c) -0o for any
x E R P m E R.
A.2 (Smoothness of f): is L-smooth, i.e, for all x,y E R we have f(y) f(x)
V (x),, x) + 2 l|* y/1?.
A.3 (Bounded gradients of f): [[V](x)]2 S G EX
A.1 and A.2 are standard assumptions for convergence analysis of smooth objectives; and A.3 is commonly used for non-conver optimization, e.g., for personalized FL in [8]. To make the composite function fx))(x) smooth, we need additional assumptions on the soft quantization function Qx(x).
Our choice of Qx(xx) (see Appendix A) naturally satisfies these assumptions.
A.4 (Smoothness of the soft quantizer): We assume that (x) is lg, -Lipschizzz and -q w.r.t. x, i.e., for c E TR V x.y E TRd: ([ee(())x)) Qe(y)]l s le, ||x - y and [VVQe((x) vy s Lq ||x yll. We also assume (x) is 122 Lipschitz and LQN smooth w.r.t ie., for E Rd;
c, !.: [[@x(x) Qa(x)]l S lQ, (/c scc-dd dll and (x) vaQa(x))I S Lq dl|
A.5 (Bound on partial gradients of soft quantizer): There exists constants GQ Go < so such that: < Go. and [V)le(
( 1::dtmm:l IIr S GQ where X Xpq. denotes sub-matrix of X with rows between p and g. and -1- ||r is the Frobenius norm.
(x(xxx
Convergence result. Now we state our main convergence result (proved in Appendix B) for minimizing Fxx (x, c) in (6) w.r.t (x, c) Rd+m via Algorithm 1. This provides first-order guarantees for convergence of (x, c) to stationary point and recovers the O (4/T) convergence rate of [1.
Theorem 1. Consider running Algorithm for T iterations for minimizing (6) with qh 1/2 [lag)- ) For any I E [T], define G =
G/2(L= +Gq Mey) and 1/2
[Vxe1 Fx (x'+1, .c)) Vo (x'+1, (xt+),c++11+ c Then, under A.1-A.5 and for Linu
min m1 7p
}}, Tmmax max 1 1). we have:
T T-1 Σ 1=
12 L (Fa (xx, -Fx(x ce4))
IG
= o
T
Lmiin T
In Theorem 1, we see that gradient norm decays without any constant error terms. The convergence rate depends Lipschitz smoothness constants of f and f(Qe(-) through Tamax and Lmmnn Choosing a smoother Qe(-) would speed up convergence; however, if chosen too small, this could result in an accuracy loss when hurd-quantizing the parameters at the end of the algorithm.
Remark 2.
Remark 3 (Number of centers and convergence). The number of quantization levels m has direct effect on convergence through the soft quantization function Qx(x) and Lipschitz constants. Note that - Xc, have Qe(x) - x, Vx. In this case, Qe(x) is O-smooth and 1-Lipschitz w.r.t. all parameters. As a result, we would have Teuin = L. and Iamax = 2T. Note that the ratio U aggressive, and consequently, affects the convergence.
increases the quantization becomes
Personalized Quantization for FL via Knowledge Distillation
We now consider the FL setting where we aim to learn quantized and personalized models for each client with different precision and model dimensions in heterogeneous data setting. Our proposed method QuPeD (Algorithm 2), utilizes the centralized scheme of Algorithm locally at each client to minimize (3) over ((x,, c, ) w). Here, x,, C₁ denote the model parameters and the quantization vector (centers) for client i, and w denotes the global model that facilitates collaboration among clients which is encouraged through the knowledge distillation (KD) loss in the local objectives (2). "A short notation, use prox ARg to denote prox, nu AR) and prox m2A for prox AR

---

Algorithm 2 QuPeD: Quantized Personalization via Distillation
Input: Regularization parameters X, Api synchronization gap T; for client i [n], initialize full preci- sion personalized models xf. quantization centers c,, local model wº. learning rates m/ ?22 7s3 quantization enforeing penalty function R(x; c); soft quantizer Qx(x; number of clients to be sampled K.
1: for = 0 to T - do
2: if divides /. then
3: On Server do:
Choose subset of clients 1C2 S [n] with size K
4: Broadcast w' to all Clients
5: On Clients E K2 to n (in parallel) do:
6: Receive w from Server; set w! = w
7: end if
8: On Clients E Kr to (in parallel) do:
9: Compute gf = (1 {p)(Vxx + (xff((e((x())) + 2x(V fx// w!)
V x fi D (9g (x(), (x))
10:
1+1 prox 1/1 AR
(xf (6) g!)
11:
Compute hf
Xp) v ct (Qc (x +1 (K w!)
12:
cf -prxx (6) ARR 1+1
(c) 1/2 hp)
13:
W; ++I w - n3ppp V wtth :
(x} t+1 w/) + V w D ( (x, + w!))
if T divides then
15: Clients send w to Server
16:
Server receives (w(): computes w 1 Liek, w!
K
17:
end if
18:
end for
x = Qet (x²") for all E [n]
Output: Quantized personalized models (x]] 1
Description of the algorithm. Since clients perform local iterations, apart from maintaining x/, cf at each client i E [n).. also maintains a model w that helps in utilizing other clients' data via collaboration. We call {w]} local copies of the global model at clients at time t. At each iteration t server samples a subset of all indices Kc, S [n] with K S n elements. Client updates w! in between communication rounds based on its local data and synchronizes that with the server which aggregates them to update the global model. Note that the local objective in (2) can be split into the weighted + (Pf. (x;, W:) + F. (x.), w.))
average of loss functions (1 11---p (xx,)) ) + f( and the term enforeing quantization AR(x,, c.). At any iteration that not a communication round (line 3), if E Kr, client first computes the gradient g of the loss function w.r.t. xf (line 4) and then takes gradient step followed by the proximal step using R (line 5) to update from xf to x!+1 Then it computes the gradient hā of the loss function w.r.t. cf (line 6) and updates the centers followed by the proximal step (line 7). Finally, it updates w/ to w! by taking a gradient step of the loss function at w/ (line 8). Thus, the local training of x of also incorporates knowledge from other clients' data through w When & divisible by т, sampled clients upload (w() to the server (line 10) which aggregates them (line 15) and broadcasts the updated global model (line 16) to all the clients. At the end of training, clients learn their personalized models (xx" in and quantization centers {c] ) Finally, client quantizes x" using Qc? (line 19).
Assumptions. In addition to assumptions A.I A.5 (with A.3 and A.5 modified have client specific gradient bounds (GO), Gl(i) Q1" Q2 as they have different model dimensions") ), we assume:
A.6 (Bounded diversity): At any i E {O.... T - 1} and any client i [n]. the variance of the local gradient (at client i) w.r.t. the global gradient is bounded, i.e., there exists KG co, such that for every
{x] 1
E Idd, c m. i E [m]] and w' E Rd generated according to Algorithm 2, we have:
We keep smoothness constants to be the same across clients for notational simplicity. however, our result easily be extended to that case.
7

---

S Kg. This assumption is equivalent w')l cctt, w 1 و Fx .c 1+1
to the bounded diversity assumption in [7, 8); see Appendix A.
A.7 (Smoothness of D)
): We assume f. (x, w) is LD, -smooth w.I.t x, LD. -smooth w.r.t. for all i E [n]; as a result is Lm-smooth w.r.t. [x, w| where Lo = m LD,}. Furthermore, we assume f.K (Qc(()(()) (x) w) is LDq, -smooth w.r.t. x, LDQ,-smoone w.r.t с and LDq, -smooth w.r.t.w for all E [n]: as a result it T-Do-smoon w.r.t. [x, c, w where LDQ = max [LDQ,, I.DQu, L.Dq.)+ In Appendix A we discuss how this assumption can actually be inferred from previous assumptions.
Convergence result. In Theorem 2 we present the convergence result when there is full client participation, i.e. K = in Appendix C we discuss the modification in convergence result under client sampling. The following result (proved in Appendix C) achieves rate of O (1/V)) for finding a stationary point within an error that depends on the data heterogeneity, matching result in [7]:
Theorem 2. Under assumptions A.1-A.7, consider running Algorithm for T iterations for minimizing (3) with T S VT, 71 (5) = 1/2 -.ssL,,,=_ng)))) +Iop, (1 (p)(( ro, +G 1dq,)), f/2 (1 (p)(()))))) ( Los + + Heaal and 173 = 1/4(X, h,
(VCAA (CLVT) where La = =LD, + -DQqs
1+1 wT)T W (x(tt ,c, _oft",""") w"t/(T. Then Let Gi= (x 1+1 ct. w/,,")" Vott F((+ 1+1 o
Σ G O
F +
DF
7
C1
C2
+ +x).
VT
T
for some constants C1, C2. where Ar= = E Ei= (Tnnae) (Fi(x), oi w?) Fx of w))), R 4 Ei (Linax)<Kg Kg, CL 1 E (L and Linase depends on the smoothness parameters and the gradient bound at client i- the expression can be found in Appendix C.
R
(min, (min,(Lmiaa))
Remark 4 (Resource and data heterogeneity.). Firstly, our observation from Remark 3 holds here as well. Aggressive quantization has scaling effect on all the terms through Luax- Now the interesting question is: how does having different model structures across clients affect the convergence rate of Theorem 2? Note that in Assumptions A.3, A.5, we assume clients have different client-specific gradient bounds; this results in client specific [ and consequently R. which couples resource and data heterogeneity across clients. Here we make an important first observation regarding the coupled effect of data and resource heterogeneity on the convergence rate. Suppose data distributions are fixed across clients (i.e., 84's are fixed) and we need to choose mudels for each client in the federated ecosystem. Then, for a fast convergence, for the clients that have local data that is not representative of the general distribution flarge Ka), it is critical to choose models with small smoothness parameter (e.g., choosing a less aggressive quantization); whereas, clients with data that is representative of the overall data distribution (small Ka) can tolerate having less smooth model.
5 Experiments
In this section, we first briefly compare numerical results for our underlying model quantization scheme (Algorithm 1) in a centralized case against related works 33]. For major part of the section, we then compare QuPeD (Algorithm 2) against other personalization schemes [7, 8, 29] for data heterogeneous clients and demonstrate its effectiveness in resource heterogeneous environments. Additional experimental results on a language task are provided in Appendix E. Implementation details, and further experiments with modifications to settings in this section, are also in Appendix E.
Centralized Training: We compare Algorithm 1 with [1. 33] for ResNet-20 and ResNet-32 [13] models trained on CIFAR-10 10 [18] dataset.
Table 1 Test accuracy (in %) on CIFAR-10.
While both [1, 33] are limited to binary quantization, [1] can be seen as spe- cific case of our centralized method where the centers do not get updated. From Ta- ble 1, we see that updating centers (Algo- rithm 1) significantly improves the perfor- mance (0.48% increase in test accuracy).
Method
ResNet-20
ResNet-32
Full Precision (FP)
92.05
92.95
ProxQuant [1] (lbit)
90.69
91.55
BinaryRelax [33] (Ibit)
87.82
90.65
Algorithml 1 b1
91.17
92.20
Algorithml (2bits)
91.45
92.47
Allowing quantization with 2 bits instead of Ibit for Algorithm 1 further increases the test accuracy.
8

---

Table Test accuracy (in %) for CNNI model at all clients."
0.75
0.70
Method
FEMNIST
CIFAR- 10
0.65
0.60
FedAvg (FP)
94.92 + 0.04
61.40 + 0.29
0.55
QuPeD(2Bits)
Local Training (FP)
94.86 + 0.93
71.57 + 0.28
0.50
Local Training(28tr))
0.45
FodAve
Local Training (2 Bits)
93.95 = 0.23
70.87
Local Training (1 Bit)
93.00 = 0.50
6 0.13
0.40
0.80
100
150
200
250
QuPeD (FP)
97.31 0.12
75.06
0.75
QuPeD (2 Bits)
96.73 = 0.27
0.70
QuPeD Bit)
95.15 = 0.21
+
0.65
0.60
QuPeL (2 Bits)
96.10 = 0.14
=
0.55
DuPeD (FP!
Per-FedAvg
QuPeL (1 Bits)
94.06 = 0.28
71.01 0.32
0.50
PFedMe
pFedMe (FP) [7]
96.60 + 0.37
73.66 0.65
0.45
FML
Per-FedAvg (FP) [8]
97.16 + 0.21
74.15 + 0.41
0.40
50
100
150
200
250
Federated ML. (FP) [29]
96.32 + 0.32
74.34 + 0.30
Figure Test Acc. vs epoch (CIFAR-10)
Personalized Training: We consider an image classification task FEMNIST [4] and CIFAR-10
[18] datasets. We consider two CNN architectures: (i) CNN1 (used in [26]): has 2 convolutional and 3 fully connected layers. (ii) CNN2: this is CNNI with additional convolutional layer with 32
filters and 5 5 kernel size. For CIFAR-10 we choose a batch size of 25. For FEMNIST, we choose variable batch sizes to have 60 iterations for all clients per epoch. We train each algorithm for 250 epochs on CIFAR-10 and 30 epochs on FEMNIST. For quantized training, as standard practice [28]. we let the first and last layers of networks to be in full precision. We use last 50 epochs CIFAR-10. and 5 epochs on FEMNIST for the fine-tuning phase.
Data Heterogeneity (DHI): We consider simulate data heterogeneity on CIFAR-10, similar to [26]. we allow each client to have access to data samples from only 4 randomly chosen classes. Thus, each client has 1000 training samples and 200 test samples. On FEMNIST, we use a subset of 198 writers from the dataset and distribute the data so that each client has access to data samples written by 3 randomly chosen writers. The number of training samples per client varies between 203-336 and test samples per client varies between 25-40. Test samples are sampled from the same class/writer that training samples sampled from, in parallel with previous works in heterogeneous FL..
50 clients for CIFAR-10 10 and n 66 for FEMNIST. To
Table Test accuracy (in %) for CNNI model at all clients, with client sampling.
Method
MNIST K 0.1)
FEMNIST 3
FedAvg (FP)
92.87 + 0.05
91.30 0.43
QuPeD (FP)
98.17 + 0.32
94.93 = 0.25
QuPeD (2 Bits)
98.01 + 0.15
94.56 + 0.18
QuPeD (1 Bit)
97.58 + 0.23
92.52 0.64
pFedMe (FP) [7]
97.79 + 0.03
93.70 + 0.39
Per-FedAvg (FP) [8]
95.80 + 0.29
92.10 + 0.22
Federated MI. (FP) [29]
98.03 + 0.31
92.73 + 0.36
Resource Heterogeneity (RH): To simulate resource heterogeneity for QuPeD, we consider 4 settings: (i) half of the clients have CNNI in full precision (FP) and the other half CNN2 in FP, (ii) half of the clients have CNNI in 2 bits and the other half CNN2 in FP, (iii) half of the clients have CNNI in bits and the other in FP, (iv) half of the clients have CNNI in 2 bits and the other half CNN2 in 2 bits.
Results (DH): We compare QuPeD against FedAvg [26]. local training of clients (without any collaboration),, and personalized FL methods: PFedMe [7]. Per-FedAvg [8], Federated Mutual Learning [29], and QuPel. (Footnote 5). For all methods, if applicable,
set 10 local iterations, use learning rate decay 0.99 and use weight decay of 10-4; we fine tune the initial learning rate for each method independently, see Appendix E for details. The results are provided in Table 2 with full client participation (K = n²), plotted in Figure for CIFAR-IO, and in Table 3 with client sampling where we state average results over 3 runs; all clients train CNNI (see Appendix E for CNN2) and quantization values are indicated in parenthesis. Thus, we only consider model personalization for
'eere QuPeD (FP) corresponds to changing alternating proximal gradient updates with SGD update on model parameters in Algorithm 2, Local Training (FP) corresponds to SGD updates without communication.

---

Table Test accuracy (in %) on FEMNIST and CIFAR-IO for heterogeneous resource distribution among clients.
Resource Heterogeneity
FEMNIST
CIFAR-10
Local Training
QuPeD
Local Training
QuPeD
CNNI(FP) CNN2(FP)
93.41 = 0.82
97.44
72.81 + 0.03
75.50 + 0.25
C2 (CNN((P))
92.70 + 1.09
97.01 0.05
72.42 + 0.17
75.08 + 0.18
C2 (INS)))
93.56 + 0.38
96.96 0.15
71.23 + 0.08
74.84 + 0.30
C2 CNNN222 Bits)
91.11 + 0.23
96.64
12.15:0.477
74.64 0.27
data heterogeneity. In Table 2, we observe that full precision QuPeD consistently outperforms all other methods for both datasets. Furthermore, we observe QuPeD with 2-bit quantization is the second best performing method on CIFAR-10 (after QuPeD (FP)) and third best performing method on FEMNIST despite the loss due to quantization. Hence, QuPeD is highly effective for quantized training data heterogeneous settings. Since QuPeD outperforms QuPeL., we can also (empirically) claim that considering KD loss to encourage collaboration is superior to l2 distance loss. Lastly, we observe from Table 3 that QuPeD continues to outperform other methods under client sampling.
Results (DH+RH): We now discuss personalized FL setting with both data and resource heterogeneity across clients. Note that since FedAvg, pFedMe, and Per-FedAvg cannot work in settings where clients have different model dimensions, we only provide comparisons of QuPeD with local training (no collaboration) to demonstrate its effectiveness. The results are given in Table We observe that QuPeD (collaborative training) significantly outperforms local training in all cases (about 3.5% or higher on FEMNIST and 2.5% or higher on CIFAR-10) and works remarkably well even in cases where clients have quantized models without any significant loss in performance.
6 Discussion
In this work, we introduced QuPeD: an algorithm for training of quantized and personalized models in heterogeneous Federated Learning (FL) settings. Our proposed scheme tackles two of the major problems in practical FI. applications: resource and data heterogeneity among participating clients. QuPel) allows clients learn personalized models based on their local data distributions while simultaneously allowing these models to be different in architecture and weight precision. This is unlike existing personalized FL. algorithms in literature which only consider the data heterogeneity aspect of FL. Our theoretical results on the convergence rates recovers previous results, which are in simpler settings, when the clients have identical resources, i.e., same model size and precision. When clients have distinct resources, our result ties convergence speed to combination of resource and data heterogeneity revealing additional insights. Our experimental results demonstrate that QuPeD delivers superior empirical performance even under aggressive quantization.
Convergence results stated in Theorems 1, 2, use bounded gradients assumptions A.3, A.5 for non- convex objectives considered in this paper. Incorporating recent progress on weakening gradient assumptions in convergence analyses for FL (e.g.. [16, 30] and references therein) into our framework is part of future work. We remark that our current assumptions are fairly standard and applicable our new optimization formulation tackle data and resource heterogeneity of the clients, which is the principal focus and contribution of our work. Simulations consisting of tens of thousands (or more) of clients and lower client sampling ratios are also crucial, and are left for potential future investigations that can shed further light on our methods at massive scale.
Societal Impact: This paper considers collaborative personalization in learning, which can signif- icantly improve learning performance. However, such personalization is only as good as the data used for training, and if not used properly could lead to information bubbles and disjointed views for each client, e.g., clustering methods that put together clients with similar data statistics. We ameliorate that by not clustering similar data and models, but this is aspect that might need more examination for social consequences. We explicitly consider diverse resources in our designs, potentially enabling clients with fewer resources (devices with less capabilities) to benefit from richer resources; potentially a positive impact in using resources more equitably.
10

---

7 Acknowledgments
This work was partially supported by NSF grants #2007714 and #1955632, and by Army Research Laboratory under Cooperative Agreement
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on. We also thank the reviewers for detailed inputs that helped improve the presentation of our results.
References
[1] Yu Bai, Yu-Xiang Wang, and Edo Liberty. Proxquant: Quantized neural networks via proximal operators. In International Conference on Learning Representations, 2019.
[2] Debraj Basu, Deepesh Data, Can Karakus, and Suhas N. Diggavi. Qsparse-local-ser Dis- tributed SGD with quantization, sparsification and local computations. In Advances in Neural Information Processing Systems, pages 14668-14679, 2019.
[3] Jérôme Bolte, Shoham Sabach, and Mare Teboulle. Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program., (6(112):459-44 2014.
[4] Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv: 2018.
[5] Hassan Dbouk, Hetul Sanghvi, Mahesh Mehendale, and Naresh Shanbhag. Dbq: A differentiable branch quantizer for lightweight deep neural networks. European Conference on Computer Vision, pages 90-106. Springer, 2020.
[6] Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003. I 13461, 2020.
[7] Canh T. Dinh, Nguyen H. Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. In Advances in Neural Information Processing Systems, 2020.
[8] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta-learning approach. In Advances in Neural Information Processing Systems, 2020.
[9] Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. In Advances in Neural Information Processing Systems, 2020.
[10] Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li. Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full- full-precision and low-bit neural networks. Proceedings of the IEEE/CVF International Conference on Computer Vision,
pages 4852 4861, 2019.
[11] Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. Lower bounds and optimal algorithms for personalized federated learning. In Advances in Neural Information Processing Systems, 2020.
[12] Filip Hanzely and Peter Richtárik. Federated learning of mixture of global and local models. arXiv preprint arXiv:2002.05516 2020.
[13] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 000.00253,,10 2015.
[15] Lu Hou, Quanming Yao, and James T. Kwok. Loss-aware binarization of deep networks. In International Conference Learning Representations, 2017.
11

---

[16] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning, In International Conference on Machine Learning, pages 5132-5143, PMLR, 2020.
[17] Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. In International Conference on Machine Learning, pages 3252-3266 PMLR, 2019.
[18] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 11far-10 (canadian institute for advanced research). 2009.
[19] Cong Leng, Zesheng Dou, Hao Li, Shenghuo Zhu, and Rong Jin. Extremely low bit neural network: Squeeze the last bit out with admm. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[20] Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint 190.0311,,,3381, 2019.
[21] Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, and Tom Goldstein. Training quantized nets: A deeper understanding. In Advances in Neural Information Processing Systems, pages 5811-588 1-5221, 2017.
[22] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems 2020, MLSys, 2020.
[23] Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances Neural Information Processing Systems, 2020.
[24] Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Relaxed quantization for discretized neural networks. In International Conference on Learning Representations, 2019.
[25] Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002. 106799, 2020.
[26] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effon effficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273-1282. PMLR, 2017.
[27] Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. In International Conference on Learning Representations, 2018
[28] Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe. Binary neural networks: A survey. Pattern Recognition, 105: Sep 2020.
[29] Tao Shen, Jie Zhang, Xinkang Jia, Fengda Zhang, Gang Huang, Pan Zhou, Kun Kuang, Fei Wu, and Chao Wu. Federated mutual learning, arXiv preprint arXiv:2006. 167665,,06.665,, 2020.
[30] Navjot Singh, Deepesh Data, Jemin George, and Suhas Diggavi. Squarm-sgd: Communication- efficient momentum sgd for decentralized optimization. IEEE Journal on Selected Areas in Information Theory, 2(3):95,-969, 2021.
[31] Virginia Smith, Chao-Kai Chiang. Maziar Sanjabi, and Ameet S. Talwalkar. Federated multi-task learning. In Advances in Neural Information Processing Systems, pages 4434-4434, 2017.
[32] Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-sheng Hua. Quantization networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[33] Penghang Yin, Shuai Zhang, Jiancheng Lyu, Stanley J. Osher, Yingyong Qi, and Jack Xin. Binaryrelax: A relaxation approach for training deep neural networks with quantized weights. SIAM J. Imaging Sci., 11(4):200522225 2018.
12

---

[34] Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized federated learning with first order model optimization. In International Conference on Learning Representations, 2021.
[35] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4320-4328 4328, 2018.
13


Reviews:
[{"review_id": "imcnEH8xXf", "reviewer": "Reviewer_TyAu", "summary": "This paper proposes a new FL algorithm QuPeD for quantization and personalization in FL. \n\nQuPeD combines soft quantization surrogate and knowledge distillation to learn compressed personalized models with different quantization and model structures and uses local updates to improve communication efficiency. The idea is novel. \n\nThe paper well structured. It first explores a centralized training scheme (as well as its convergence property) and then extends it to FL setting where the mentioned three techniques are fused.\n\nTheir experiments show its effectiveness.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nQuPeD combines soft quantization surrogate and knowledge distillation to learn compressed personalized models with different quantization and model structures and uses local updates to improve communication efficiency. The idea is novel. \n\nQuality: \n\nThe method is technically sound. The proof seems correct.\n\nClarity:\nThe paper is well structured and clear. It first explores a centralized training scheme (as well as its convergence property) and then extends it to FL setting where the mentioned three techniques are fused. \n\nSignificance:\n\nTheir experiments show its effectiveness. It shows that QuPeD has better test accuracy on image classification than previous work.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author leaves the relaxation of assumption as future work.\n\nHowever, I have some other concerns:\n\n1. Why should we have equal weights for $f(\\mathbf{x})$ and $f(\\widetilde{Q}_{\\mathbf{c}}(\\mathbf{x}))$ in (6)? Intuitively, different weights might force the algorithm to focus on different goals, for better accuracy (in optimization) or better quantization (for reference). After all, (6) can be viewed as a way towards multi-tasks.\n2. Theorem 1 shows the gradient norm decays without any constant error term, while Theorem 2 didn’t. The author didn’t explain the reason and only say that we should choose the clients whose $\\kappa_i$ is small in Remark 4. From the previous analysis in FL, the data heterogeneity should not introduce additional error (e.g., see [1-2]). It will only slow convergence rather than introduce error. So why here we can’t obtain a similar result?\n\n[1] Khaled, Ahmed, Konstantin Mishchenko, and Peter Richtárik. \"Tighter theory for local SGD on identical and heterogeneous data.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2020.\n\n[2] Haddadpour F, Mahdavi M. On the convergence of local descent methods in federated learning[J]. arXiv preprint arXiv:1910.14425, 2019.\n\n3. Typically, if no local update is used (i.e., $\\tau=1$), we should recover the result of Theorem 1 by setting $\\tau=1$ in Theorem 2. However,  if we do that for line 314, I can’t recover the $O(1/T)$ convergence rate. What’s more, I can’t even remove the constant error term $\\bar{\\kappa}$, since it doesn’t depend on $\\tau$. With Point 1 and this, I guess the analysis of Theorem 2 might not be tight. \n4. The work didn’t consider partial participation, so it is unclear how can we select the clients whose $\\kappa_i$ is small in practice, as suggested in Remark 4. Besides, the work also didn’t consider the stochastic gradient variant. But, I think it is fine to defer them as future work since the main idea is well delivered already. \n5. In experiments, I think it is better to stress the importance of updating the centers (line 989) in the main text rather than in the appendix. Besides, the increased data heterogeneity will decrease the performance of (almost all) algorithms (from Table 3 and Table 5). It actually supports the theory and worth mentioning in the main text. \n\n\n\n===================\nI have read the author's response. The author almost handles all my concerns. I am willing to increase my score to 7.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "U7Lf-Mtmj2V", "reviewer": "Reviewer_USRi", "summary": "This paper investigates Federated Learning setup with several practical consideration including (i) personalization where each client aims to train a local model suited to its local dataset while benefiting from the overall data, (ii) compressed local models to facilitate memory-constraint applications, (iii) deferent model dimension and precision for different clients to address data and resource heterogeneity. To handle this aspects of FL, this work designs an alternative optimization problem, which, besides the standard loss of the models, also minimizes the loss of softly quantized models and knowledge distillation loss between local models and the global model, encourages the models to be close to quantization centers. To solve that optimization problem, a new method called QuPeD has been proposed and theoretical analysis for non-convex and smooth objectives has done. Convergence rates are complemented with extensive numerical evaluations on deep nets.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the presentation of the paper is clear and clean with necessary motivations. Personalization is an important aspect in FL and the relaxed optimization problem together with the proposed method seems interesting contribution in this direction (especially under data and resource heterogeneity). The paper models several practical aspects into a new optimization problem, provides solid convergence guarantees and extensive numerical experiments.\n\n\nI have one concern about knowledge distillation, which I am least familiar with.\n\nHow KD loss is defined exactly and how important is it for the theory ? It seems that the analysis only cares about smoothness of the loss and not the particular structure of it. As a reader not familiar with knowledge distillation much, description given in lines 193-195 does not explains what is the loss actually (e.g., functions $s_i^w$ and $s_i$) and whether it is only for neural networks or can be defined for other models too. I expected more details in the appendix, but could not find.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "KyVCutMKLwr", "reviewer": "Reviewer_n4oz", "summary": "The paper provides an algorithm (QuPeD) to learn personalized models in a federated learning (FL) setting where there is heterogeneity in data and computing resources. The method utilizes quantization and knowledge distillation (KD). The paper also formalizes KD in personalized FL as an optimization problem and proposes alternative proximal updates as a solution (with convergence analysis). Empirical evaluation is conducted on CIFAR-10 and FEMNIST, and compared against local training (no collaboration) + two quantization regimes and comparable models: Per-FedAvg, pFedMe, and FedAvg.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In the basic FL simulation setting, a given $M$ number of client devices (subset of total client devices) are sampled for training. However, I don't see this is the case for QuPeD. Algorithm 2 states \"On Clients $i = 1$ to $n$, where from Eq. (1) $n$ is total number of clients? Also, this leads to the questions of how the comparable methods: Per-FedAvg, pFedMe, and FedAvg were set up (since they use random sub-sampling of clients) to make the empirical evaluation comparable.\n\nLine 355,356: \"To simulate data heterogeneity on CIFAR-10, similar to [FedAvg],...\". But, the FedAvg paper states: \"since there isn’t a natural user partitioning of this data, we considered the balanced and IID setting\" under CIFAR experiments.\n\nThere are several hyper-parameters to be tuned for QuPeD (algorithm 2) and the theoretical analysis provides some insights. However, it would be more insightful to see the effects of hyper-parameters empirically.\n\nEmpirical analysis focuses on vision models and datasets, but what are the implication in applying the method to language based models and datasets?\n\nAlthough the paper states \"the main goal of our work is personalized quantization for inference.\" (line 70), it was not clear (at least not convincing) to me from the supporting motivation w.r.t. the paper on the use of quantization.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed limitations and potential negative societal impact of their work.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "1WAA0feAgS4", "reviewer": "Reviewer_y15P", "summary": "This paper tackles the problem of Federated Learning personalization, with both resource and data heterogeneity. The proposed method optimizes three objectives: (a) personalized FL loss, (b) staircase-quantization loss and explicit distance loss, and (c) Knowledge Distillation (KD) loss. This paper makes a lot of assumptions and tricks to make the above quantization-related objectives differentiable.\nThe major difference between this paper and other FL personalization and FL model compression works is that the proposed algorithm (QuPeD), aims at learning with full accuracy in the Federated Training phase (with the objective that it will be quantized in the adaptation phase), while heterogeneous quantization and knowledge distillation are only conducted in inference time. QuPed conducts alternative FL optimization, on full accuracy model, quantization centers, and local models one-by-one, which shows improved quantized personalized performance, on FEMNIST and Federated CIFAR-10 datasets.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "However, I do have some comments, which requires the authors to shed light on:\n\n(1) Paper organization: the main context ignores a lot of key materials due to the limitation of pages, which requires searching for the appendix to understand. For example, the \"prox\" step, which is very frequently used in the main context, is not shown in the main text. The notation in the main context actually makes it harder and more confusing to understand. Moreover, the soft quantization function is not illustrated clearly. I cannot understand how to implement the soft quantization function, by only reading the main context. The presentation should improve, so that FL engineers like me could understand how to implement the algorithms, without getting lost in details that I don't understand.\n\n(2) Proposed solution: The setup assumes that each client can do full accuracy training, but need different resources to conduct different levels of inference. The best usage in my opinion for the proposed algorithm, is FL on algorithms that will be deployed to IoT devices: Phone conduct full accuracy training, while the trained algorithm is deployed to IoT devices which require different dimension models and different levels of quantization. The proposed algorithm is not a direct fit, to current FL systems, where both training and inference are conducted on the same phone. \n\n(3) Empirical results: The empirical results support the claim that QuPed improves against its benchmarks. However, I find it hard to understand the theoretical convergence rate and the empirical results. Does the theoretical convergence rate mean the number of training epochs? Without showing the convergence rate empirically, the theoretical results seem to be a relative detour to the main context and could leave to the appendix.\n\nUnfortunately, I am not knowledgeable enough to evaluate the theoretical part of this work, other than understanding enabling differentiability and convergence speed. Due to the above reasons, I leave a borderline rating, and I am willing to read other reviewers’ comments and authors’ answers, to change my rating accordingly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers had a few concerns that were addressed in the authors' response.  The majority of the reviewers agreed that the paper provides an interesting contribution to FL. The paper is missing a lot of references, on the quantization side as well as other topics, and I recommend the authors to carefully review the literature and cite the related work properly.  
Decision: Accept (Poster)