QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning
Authors: Kaan Ozkara
Navjot Singh
Deepesh Data
Suhas Diggavi
Venue: NeurIPS 2021 Poster (2021)
Keywords: Personalized Federated Learning
Distributed Optimization
Model Compression
Knowledge Distillation

Abstract: Traditionally, federated learning (FL) aims to train a single global model while collaboratively using multiple clients and a server. Two natural challenges that FL algorithms face are heterogeneity in data across clients and collaboration of clients with diverse resources. In this work, we introduce a quantized and personalized FL algorithm QuPeD that facilitates collective (personalized model compression) training via knowledge distillation (KD)  among clients who have access to heterogeneous data and resources. For personalization, we allow clients to learn compressed personalized models with different quantization parameters and model dimensions/structures. Towards this, first we propose an algorithm for learning quantized models through a relaxed optimization problem, where quantization values are also optimized over. When each client participating in the (federated) learning process has different requirements of the compressed model (both in model dimension and precision), we formulate a compressed personalization framework by introducing knowledge distillation loss for local client objectives collaborating through a global model. We develop an alternating proximal gradient update for solving this compressed personalization problem, and analyze its convergence properties. Numerically, we validate that QuPeD outperforms competing personalized FL methods, FedAvg, and local training of clients in various heterogeneous settings.

QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning
Kaan Ozkara
Navjot Singh University of Califomia, Los Angeles navjotsingh@ucla..oo edu
University of California, Los Angeles kaanQucla.ed.
Deepesh Data University of California, Los Angeles datreesaalaaa@amaa.....m com
Suhas Diggavi
University of California, Los Angeles suhasSee. ucla.
Abstract
Traditionally, federated learning (FL) aims to train a single global model while collaboratively using multiple clients and server. Two natural challenges that FI. algorithms face are heterogeneity in data across clients and collaboration of clients with diverse resources. In this work, we introduce quantized and personalized FL. algorithm QuPeD that facilitates collective (personalized model compression) training via knowledge distillation (KD) among clients who have access to heterogeneous data and resources. For personalization, we allow clients learn compressed personalized models with different quantization parameters and model dimensions//tructures. Towards this, first we propose algorithm for learning quantized models through relaxed optimization problem, where quantization values are also optimized over. When each client participating in the (federated) learning process has different requirements for the compressed model (both in model dimension and precision), we formulate compressed personalization framework by introducing knowledge distillation loss for local client objectives collaborating through a global model. We develop an alternating proximal gradient update for solving this compressed personalization problem, and analyze its convergence properties. Numerically, we validate that QuPeD outperforms competing personalized FL methods, FedAvg, and local training clients in various heterogeneous settings.
Introduction
Federated Learning (FL.) is a learning procedure where the aim is to utilize vast amount of data residing in numerous (in millions) edge devices (clients) to train machine learning models without collecting clients' data [26]. Formally, if there are clients and ft denotes the local loss function at client then traditional FL learns single global model by minimizing
arg min (S(w): := Σ f((w)) WERd
It has been realized lately that single model may not provide good performance to all the clients in settings where data is distributed heterogeneously. This leads to the need for personalized learning, where each client wants learn its model [7, 8). Since locally learned client model may not generalize well due to insufficient data, in personalized FL process, clients maintain personalized models locally and utilize other clients' data via global model. Resource diversity among clients,
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

which is inherent to FL as the participating edge devices may vary widely in terms of resources, is often overlooked in personalized FL literature. This resource diversity may necessitate clients to learn personalized models with different precision as well as different dimension/architectur.. Systemati- cally studying both these resource heterogeneity together with data heterogeneity in personalized FI. is the primary objective of this paper.
In this work, we propose a model compression framework for personalized FL via knowledge distillation (KD) [14] that addresses both data and resource heterogeneity in a unified manner. Our framework allows collaboration among clients with different resource requirements both in terms of precision as well as model ciimension//tructuree for learning personalized quantized models (PQMs). Motivated by FL. where edge devices are resource constrained when actively used (e.g. when several applications are actively running on a battery powered smartphone) and available for training when not use (e.g., while charging and on wi-fi), we do training in full precision for learning compressed models to be deployed for inference time. For efficient model compression, we learn the quantization parameters for each client by including quantization levels in the optimization problem itself. First, we investigate our approach in a centralized setup, by formulating relaxed optimization problem and minimizing it through alternating proximal gradient steps, inspired by [3]. To extend this to FL for we employ our centralized algorithm locally
learning PQMs with different at clients and introduce KD loss for collaboration of personalized and global models. Although there exist empirical works where KD is used in personalized FL [20], we formalize it as an optimization problem, solve it using alternating proximal updates, and analyze its convergence.
Contributions. Our contributions can be summarized as follows:
the centralized case, we propose a novel relaxed optimization problem that enables optimization over quantization values (centers) as well model parameters. We use alternating proximal updates to minimize the objective and analyze its convergence properties.
More importantly, our work is the first to formulate a personalized FL optimization problem where clients may have different model dimensions and precision requirements for their personalized models. Our proposed scheme combines alternating proximal updates with knowledge distillation.
For optimizing non-convex objective, in the centralized setup, we recover the standard conver- gence rate of O(1/T) (despite optimizing over quantization centers), and for federated setting, we recover the standard convergence rate of O(1/T) (despite learning PQMs with different preci- sions/dimensions). In the federated setting, our convergence bound has an error term that depends on multiplication of two terms averaged over clients: one characterizing client's local model smoothness and the other data heterogeneity with respect to overall data distribution.?
We perform image and text classification experiments on multiple datasets in various resource and data heterogeneity settings, and compare performance of QuPeD against Per-FedAvg [8], PFedMe [7], FedAvg [26], and local training of clients. We observe that QuPeD in full precision outperforms all these methods on all the datasets that we considered for our experiments. Further, our results show that even with quantization, QuPel) outperforms the other methods in full precision on multiple settings and datasets, demonstrating the effectiveness of our scheme for FL. settings.
Our work should not be confused with works in distributed/ficerrtee learning, where models/gradients are compressed for communication efficiency [2. 17]. We also achieve communication efficiency through local iterations, but the main goal of our work is personalized quantization for inference.
Related work. To the best of our knowledge, this is the first work in personalized federated learning where the aim is to learn quantized and personalized models potentially having different dimenssion//sruucuurss for inference. Our work can be seen in the intersection of personalized federated learning and learning quantized models; we also employ knowledge distillation for collaboration.
Personalized federated learning: Recent works adopted different approaches for learning personalized models: (i) Combine global and local models throughout the training [6. 12, 25]; (ii) first learn a global model and then personalize locally [8]: (iii) consider multiple global models to collaborate among only those clients that share similar personalized models [9, 25, 31, 34]; (iv) augment the
'odde compression (MC) allows inference time deployment of compressed model. Though MC is a generic term comprising different methods, we will focus on its quantization (number of bits per model parameter) aspect.
An error term depending on data heterogeneity is commonly observed in personalized FL algorithms [7, 8j.
2

---

traditional FL objective via a penalty term that enables collaboration between global and personalized models [7. 11, 12].
Learning quantized models: There are two kinds of approaches for training quantized networks that are of our interest. The first one approximates the hard quantization function by using soft surrogate [5, 10, 24, 32], while the other one iteratively projects the model parameters onto the fixed set of centers [1, 15, 19, 33]. Each approach has its own limitation; see Section 2.1 for discussion. While the initial focus in learning quantized networks was on achieving good empirical performance, there are some works that analyzed convergence properties [1. 21, 331, but only in the centralized case. Among these, [1] analyzed convergence for relaxed/regullarieed loss function using proximal updates.
Knowledge distillation (KD): KD [14] is a framework for transfer learning that is generally used to train a small student network using the soft labels generated by a deep teacher network. It can also be used to train two or more networks mutually by switching teacher and students in each iteration [35]. KD has been employed in FL settings as an alternative simple aggregation which is not feasible when clients have models with different dimensions [23]. [20] used KD in personalized FL by assuming existence of public dataset. [27] used KD in combination with quantization in centralized case for model compression; in contrast, we do not use KD for model compression but for collaboration between personalized and global model. Unlike the above works which are empirical, our paper is the first to formalize an optimization problem for personalized FL training with KD and analyze its convergence properties. Our proposed scheme yields personalized client models with different precision/dimension through local alternating proximal updates; see Section 2.2 for details.
Paper organization: In Section 2, we formulate the optimization problem to be minimized. In Sections 3 and 4, we describe our algorithms along-with the main convergence results for the centralized and personalized settings, respectively. Section 5 provides extensive numerical results. Omitted proofs/details and additional experiments are provided in supplementary material.
2 Problem Formulation
Our goal in this paper is for clients collaboratively learn personalized quantized models (with potentially different precision and model sizes/types). To this end, below, we first state our final objective function that will end up optimizing in this paper for learning personalized quantized models, and then in the rest of this section we will describe the genesis of this objective.
Recall from (1), in the traditional FL setting, the local loss function at client i is denoted by fi. For personalized compressed model training, we define the following augmented loss function at client i:
Px Ci w) = (1 (1- + fi(0a, (AQc + AA((x,, c:)
+ Ap (5, (x, H. (0 (x.))
Here, w E IRd denotes the global model, x; E R.d, denotes the personalized model of dimension d, at client i, c; E TR. denotes the model quantization centers (where m. is the number of centers), Qc, denotes the noft-quantization function with respect to (w.r.t.) the set of centers C₁, R(x;, c;) denotes the distance function, f.K denotes the knowledge distillation (KD) loss [14] between the two input models client i's dataset, À is design parameter for enforcing quantization (large À forces weights to be close to respective centers), and A, controls the weighted average of regular loss and KD loss functions (higher A, can be used when client data is limited). We will formally define the undefined quantities later in this section. Consequently, our main objective becomes:
min ".
(F(w, (xi). (c.))
= n Σ Fi(x,,
Ci Fi(x,,
(3)
WERd
Thus, our formulation allows different clients to have personalized models X1.... . Xn, with different dimensions di....,d d, and architectures, different number of quantization levels m...... m,, (larger the m, higher the precision), and different quantization values in those quantization levels. Note that there are two layers of personalization, first is due to data heterogeneity, which is reflected in clients learning different models XI,..., Xn.
Xn, and second due to resource diversity, which is reflected in clients learning models with different sizes, both in terms in dimension as well as precision.
3

---

In Section 2.1, we motivate how we came up with the first three terms in (2), which are in fact about a centralized setting because the function fi and the parameters involved, ie.. x, Ct. are local to client i; and then, in Section 2.2, we motivate the use of the last two terms containing f in (2).
2.1 Model Compression in the Centralized Setup
Consider a setting where objective function f Rd+m R (which could be neural network loss function) is optimized over both the quantization centers с E R" and the assignment of model parameters (or weights) x E R those centers. There are two ways approach this problem, and we describe these approaches, their limitations, and the possible resolutions below.
Approach 1. A natural approach is to explicitly put constraint that weights belong to the set of centers, which suggests solving the following problem: minx,e fx) + 6e(x), where de denotes the indicator function for с E RT and for any c E IR" x E R define 6c(x) == 0 V j E [d] J; = e for some с E {ci...., Cm otherwise, define 6e(x) := oo. However, the discontinuity of (x(x) makes minimize this objective challenging. To mitigate this, like recent works [1, 33], we can approximate de (x) using distance function R(x, c) that is continuous everywhere (e.g., the didisttnc,,, R(x, c) == min fllz - x||1 : 24 E {ci, ... ,.. Vi)). This suggests solving:
min fx) + AR(x. c).
(4)
The centers are optimized to be close to the mean or median (depending on R) of the weights; however, there is no guarantee that this will help minimizing objective f. We believe that modeling the direct effect that centers have on the loss is crucial for a complete quantized training (see Appendix E for empirical verification of this fact), and our second approach is based on this idea.
Approach 2. We can embed the quantization function into the loss function itself, thus solving the problem: mitlx.e (h(x, c) := (x)),, where for every E Rd, RT the (hard) quantization function defined as Qx(x); == Ck. where k = arg min] {lar; cg1], which maps individual weights to the closest centers. Note that Qe(x) is actually a staircase function for which the derivative w.r.t. x is 0 almost everywhere, which discourages the use of gradient- based methods to optimize the above objective. To overcome this, similar to [10, 32], we can use a seft quantization function Qx(x) that is differentiable everywhere with derivative not necessarily O. For e.g., element-wise sigmoid or tanh functions, used by [32] and [10], respectively." This suggests the following relaxation:
min (h(x, c) == f()x(x))). X,e
(5)
Though we can observe the effect of centers on neural network loss in (5); however, the gradient w.r.t. x is heavily dependent on the choice of Q. and optimizing over x might deviate too much from optimizing the neural network loss function. For instance, in the limiting case when Qx(x) - Qc(x), gradient w.r.t. x is almost everywhere; hence, every point becomes a first order stationary point.
Our proposed objective for model quantization. Our aim is to come up with an objective function that would not diminish the significance of both x and in the overall procedure. To leverage the benefits of both, we combine both optimization problems (4) and (5) into one problem:
min (F)(x.c) x.C
f(x) f(āx(x)) + c)(x.))).
Here, the first term preserves the connection of x to neural network loss function, and the second term enables the optimization of centers w.r.t. the neural network training loss itself. As result, we obtain an objective function that is continuous everywhere, and for which we can use Lipschitz tools in the convergence analysis which previous works did not exploit. In fact, we compute the Lipschitz parameters for a specific soft quantization function Qe based on sigmoid in Appendix A.
Remark 1. It is important to note that with this new objective function, are able optimize not only over weights but also over centers. This allows us to theoretically analyze how the movements of the centers affect the convergence. As far as we know, this has not been the case in the literature of quantized neural network training. Moreover, we observe numerically that optimizing over centers improves performance of the network; see Appendix E.
2[1] and [33] proposed to approximate the indicator function 6<(x) using a distance function Re(x), where c is fixed, and unlike ours, it is not a variable that the loss function is optimized over.
IIn their setup, the quantization centers fixed. In contrast, we are also optimizing over these centers.

---

2.2 Towards Personalized Quantized Federated Learning: Knowledge Distillation
Note that the objective function defined in (6) can be used for learning a quantized model locally at any client. There are multiple ways to extend that objective for learning personalized quantized models (PQMs) via collaboratio.. For example, when all clients want to learn personalized models with the same dimension (but with different quantization levels), then one natural approach is to add an le penalty term in the objective that would prevent local models from drifting away from the global model and from simply fitting to local data. This approach, in fact, has been adopted in previous works [7, 12, 22] for learning personalized models in FL, though not quantized ones.' 5
In this paper, since we allow clients to learn PQMs with potentially different dimensions, the above approach of adding lg penalty term in the objective is not feasible. Observe that, the purpose of incorporating a Eg penalty in the objective is to ensure that the personalized models do not have significantly different output class scores compared to the global model which is trained using the data generated at all clients; this does not, however, require the global model to have the same dimension as that of local models and can be satisfied by augmenting the local objective (6) with certain knowl- edge distillation (KD) loss. In our setting, since clients' goal is to learn personalized models with different dimensions that may also have different quantization levels, we augment the local objective (x,, w) and J.KD (Qc, (x:), w), where the first one ensures that the behavior of X; E I24 is not very different from that of w E R and the second one ensures the same for the quantized version of x; and Formally, we define them using KL divergence fol- lows: P.D (x:, w) = D (s (w) (x)])s(xx))) and S.D (Qc, (x:). w) == DxI. (s." (w)]ls:(Qe., (x:))), where s and denote functions whose inputs are global and personalized models, respectively and data samples implicitly and outputs are the softmax classification probabilities of the network.
(6) with two separate KD losses:
We need to train x; and w mutually. Identifying the limitations of existing approaches for thcoretical analysis (as mentioned in related work in Section 1), we reverse KL. updates (i.e., taking gradient steps w.r.t. the first parameter in (x .)) to train the teacher network w from the student network x,. This type of update can be shown converge and also empirically outperforms [29] (see Section 5). We want to emphasize that though there are works [20, 23] that have used KD in FL and studied its performance (only empirically), ours is the first work that carefully formalizes it as an optimization problem (that also incorporate quantization) which is necessary to analyze convergence properties.
3 Centralized Model Quantization Training
In this section, we propose a centralized train- ing scheme (Algorithm 1) for minimizing (6) by optimizing over x E 12d (the model parame- ters) and E IRT (quantization values/centers). During training, we keep x full precision and learn the optimal quantization parameters c. The learned quantization values are then used to hard-quantize the personalized models to get quantized models for deployment in memory- constrained setting.
Algorithm 1 Centralized Model Quantization
Input: Regularization parameter X; initialize the full precision model x" and quantization cen- ters c"; penalty function enforcing quantization R(x, c); a soft quantizer Qe(x): and learning rates 71 7/2.
1: for t. = 0 to T do
2: Compute g' = (x')+ f(Q.
Description of the algorithm. We optimize (6) through alternating proximal gradient descent.
3:
x ++1 = prox AR (x* - migg)
4: Compute h' = Vet f( (x(+1))
The model parameters and the quantization vec- tor are initialized to random vectors x0 and co. The objective in (6) is composed of two parts:
5:
cl+1 prox
(c' 722hh)
6: end for
Output: Quantized model & = Qe Qx²(xx)
the loss function fxx) + f(Qx(x)) and a quan- tization inducing term R(x, c), which we control by a regularization coefficient A. At each I, we compute gradient g' of the loss function w.r.t. x' (line 2), and then take the gradient step followed by prox step for updating x' to x (line 3). For the centers, we similarly take gradient step and follow it by a prox step for updating c to c (line 4-5). These update steps ensure that we
We also analyze this approach in our setting (for learning PQMs but having the same dimension), and we call this version QuPeL: see Appendix D for problem setup and convergence results of QuPeL. In Section 5, we demonstrate that QuPeD (for the same task but using KD opposed to the &2 penalty) outperforms QuPeL.
5

---

learn the model parameters and quantization vector tied together through proximal mapping of the regularization function R. Finally, we quantize the full precision model x" using Qe (line 7).
Assumptions. We make the following assumptions on f:
A.1 (Finite lower bound of f): f(x) -00, Vx E Rd, which implies Fx(x,c) -0o for any
x E R P m E R.
A.2 (Smoothness of f): is L-smooth, i.e, for all x,y E R we have f(y) f(x)
V (x),, x) + 2 l|* y/1?.
A.3 (Bounded gradients of f): [[V](x)]2 S G VX
A.1 and A.2 are standard assumptions for convergence analysis of smooth objectives; and A.3 is commonly used for non-conves optimization, e.g., for personalized FL in [8]. To make the composite function fx))(x) smooth, we need additional assumptions on the soft quantization function Qx(x).
Our choice of Qx(xx) (see Appendix A) naturally satisfies these assumptions.
A.4 (Smoothness of the soft quantizer): We assume that (x) is lg, -Lipschizzz and -q w.r.t. x, i.e., for c E TR V x.y E TRd: ([ee(()xx)) Qe(y)]l s le, ||x - y and [VVQe((x) vy s Lq. ||x yll. We also assume (x) is 122 Lipschitz and LQN smooth w.r.t ie., for E R4;
c, !.: [[@x(x) Qa(x)]l S lQ, (/c scc-dd dll and (x) vaQa(x))I S Lq dl|
A.5 (Bound on partial gradients of soft quantizer): There exists constants GQ Go < so such that: < Go. and [V)le(
( 1::dtmm: IIr S GQ where X Xpq. denotes sub-matrix of X with rows between p and g, and -1- ||r is the Frobenius norm.
(x(xxx
Convergence result. Now we state our main convergence result (proved in Appendix B) for minimizing Fxx (x, c) in (6) w.r.t (x, c) Rd+m via Algorithm 1. This provides first-order guarantees for convergence of (x, c) to stationary point and recovers the O (4/T) convergence rate of [1.
Theorem 1. Consider running Algorithm for T iterations for minimizing (6) with qh 1/2 [lag)- ) For any I E [T], define G =
G/2(L= +Gq Mey) and 1/2
[Vxe1 Fx (x'+1, .c)) Vo (x'+1, (xt+),c++11+ c Then, under A.1-A.5 and for Linu
min m1 7p
}}, Tmmax max 1 1). we have:
T T-1 Σ 1=
12 L (Fa (xx, -Fx(x ce4))
IG
= o
T
Lmiin T
In Theorem 1, we see that gradient norm decays without any constant error terms. The convergence rate depends Lipschitz smoothness constants of f and f(Qe(-) through Tamax and Lmmnn Choosing a smoother Qe(-) would speed up convergence; however, if chosen too small, this could result in an accuracy loss when hurd-quantizing the parameters at the end of the algorithm.
Remark 2.
Remark 3 (Number of centers and convergence). The number of quantization levels m has direct effect on convergence through the soft quantization function Qx(x) and Lipschitz constants. Note that - oc, have Qe(x) - x, Vx. In this case, Qe(x) is O-smooth and 1-Lipschitz w.r.t. all parameters. As a result, we would have Teuin = L. and Iamax = 2T. Note that the ratio U aggressive, and consequently, affects the convergence.
increases the quantization becomes
Personalized Quantization for FL via Knowledge Distillation
We now consider the FL setting where we aim to learn quantized and personalized models for each client with different precision and model dimensions in heterogeneous data setting. Our proposed method QuPeD (Algorithm 2), utilizes the centralized scheme of Algorithm locally at each client to minimize (3) over ((x,, c, ) w). Here, x,, C₁ denote the model parameters and the quantization vector (centers) for client i, and w denotes the global model that facilitates collaboration among clients which is encouraged through the knowledge distillation (KD) loss in the local objectives (2). "A short notation, use prox ARg to denote prox, nu AR) and prox m2A for prox AR

---

Algorithm 2 QuPeD: Quantized Personalization via Distillation
Input: Regularization parameters X, Api synchronization gap T; for client i [n], initialize full preci- sion personalized models xf. quantization centers c,, local model wº. learning rates m/ ?22 7s3 quantization enforeing penalty function R(x; c); soft quantizer Qx(x; number of clients to be sampled K.
1: for = 0 to T - do
2: if divides /. then
3: On Server do:
Choose subset of clients 1C2 S [n] with size K
4: Broadcast w' to all Clients
5: On Clients E K2 to n (in parallel) do:
6: Receive w from Server; set w! = w
7: end if
8: On Clients E Kr to (in parallel) do:
9: Compute gf = (1 {p)(Vxx + (xff((e((x())) + 2x(V fx// w!)
V x fi D (9g (x(), (x))
10:
1+1 prox 1/1 AR
(xf (6) g!)
11:
Compute hf
Xp) v ct (Qc (x +1 (K w!)
12:
cf -prxx (6) ARR 1+1
(c) 1/2 hp)
13:
W; ++I w - n3ppp V wtt :
fKD (x} t+1 w/) + V w D ( (x, + w!))
if T divides then
15: Clients send w to Server
16:
Server receives (w(): computes w 1 Liek, w!
K
17:
end if
18:
end for
x = Qet (x²") for all E [n]
Output: Quantized personalized models (x]] 1
Description of the algorithm. Since clients perform local iterations, apart from maintaining x/, cf at each client i E [n).. also maintains a model w that helps in utilizing other clients' data via collaboration. We call [w]} local copies of the global model at clients at time t. At each iteration t server samples a subset of all indices Kc, S [n] with K S n elements. Client updates w! in between communication rounds based on its local data and synchronizes that with the server which aggregates them to update the global model. Note that the local objective in (2) can be split into the weighted + (Pf. (x;, W:) + F. (x.), w.))
average of loss functions (1 -1---p (xx,)) ) + f( and the term enforeing quantization AR(x,, c.). At any iteration that not a communication round (line 3), if E Kr, client first computes the gradient g of the loss function w.r.t. xf (line 4) and then takes gradient step followed by the proximal step using R (line 5) to update from xf to x!+1 Then it computes the gradient hā of the loss function w.r.t. cf (line 6) and updates the centers followed by the proximal step (line 7). Finally, it updates w/ to w! by taking a gradient step of the loss function at w/ (line 8). Thus, the local training of x of also incorporates knowledge from other clients' data through w When & divisible by т, sampled clients upload (w() to the server (line 10) which aggregates them (line 15) and broadcasts the updated global model (line 16) to all the clients. At the end of training, clients learn their personalized models (xx" in and quantization centers {c] ) Finally, client quantizes x" using Qc? (line 19).
Assumptions. In addition to assumptions A.I A.5 (with A.3 and A.5 modified have client specific gradient bounds (GO), Gl(i) Q1" Q2 as they have different model dimensions") ), we assume:
A.6 (Bounded diversity): At any i E {O.... T - 1} and any client i [n]. the variance of the local gradient (at client i) w.r.t. the global gradient is bounded, i.e., there exists KG co, such that for every
{x] 1
E Idd, c m. i E [m]] and w' E Rd generated according to Algorithm 2, we have:
We keep smoothness constants to be the same across clients for notational simplicity. however, our result easily be extended to that case.
7

---

S Kg. This assumption is equivalent w')l cctt, w 1 و Fx .c 1+1
to the bounded diversity assumption in [7, 8); see Appendix A.
A.7 (Smoothness of D)
): We assume f. (x, w) is LD, -smooth w.I.t x, LD. -smooth w.r.t. for all i E [n]; as a result is Lm-smooth w.r.t. [x, w| where Lo = m LD,}. Furthermore, we assume f.K (Qc(()(()) (x) w) is LDq, -smooth w.r.t. x, LDQ,-smoone w.r.t с and LDq, -smooth w.r.t.w for all E [n]: as a result it T-Do-smoon w.r.t. [x, c, w where LDQ = max [LDQ,, I.DQu, L.Dq.)+ In Appendix A we discuss how this assumption can actually be inferred from previous assumptions.
Convergence result. In Theorem 2 we present the convergence result when there is full client participation, i.e. K = in Appendix C we discuss the modification in convergence result under client sampling. The following result (proved in Appendix C) achieves rate of O (1/V)) for finding a stationary point within an error that depends on the data heterogeneity, matching result in [7]:
Theorem 2. Under assumptions A.1-A.7, consider running Algorithm for T iterations for minimizing (3) with T S VT, 71 (5) = 1/2 -.ssL,,,=_n))))) +Iop, (1 (p)(( ro, +G 1dq,)), f/2 (1 (p)(()))))))))))))))))))) ( Los + + Heaal and 173 = 1/4(X, h,
(VCAA (CLVT) where La = =LD, + -DQqs
1+1 wT)T W (x(tt ,c, _oft","")) w""/(T. Then Let Gi= (x 1+1 ct. w/,,")" Vott F((+ 1+1 o
Σ G O
F +
DF
7
C1
C2
+R) +x).
VT
T
for some constants C1, C2. where Ar= = E Ei= (Tnnae) (Fi(x), oi w?) Fx of w))), R 4 Ei (Linax)<Kg Kg, CL 1 E (L and Linase depends on the smoothness parameters and the gradient bound at client i- the expression can be found in Appendix C.
R
(min, (min.(Lmiaa))
Remark 4 (Resource and data heterogeneity.). Firstly, our observation from Remark 3 holds here as well. Aggressive quantization has scaling effect on all the terms through Luax- Now the interesting question is: how does having different model structures across clients affect the convergence rate of Theorem 2? Note that in Assumptions A.3, A.5, we assume clients have different client-specific gradient bounds; this results in client specific [ and consequently R. which couples resource and data heterogeneity across clients. Here we make an important first observation regarding the coupled effect of data and resource heterogeneity on the convergence rate. Suppose data distributions are fixed across clients (i.e., 84's are fixed) and we need to choose mudels for each client in the federated ecosystem. Then, for a fast convergence, for the clients that have local data that is not representative of the general distribution flarge Ka), it is critical to choose models with small smoothness parameter (e.g., choosing a less aggressive quantization); whereas, clients with data that is representative of the overall data distribution (small Ka) can tolerate having less smooth model.
5 Experiments
In this section, we first briefly compare numerical results for our underlying model quantization scheme (Algorithm 1) in a centralized case against related works 33]. For major part of the section, we then compare QuPeD (Algorithm 2) against other personalization schemes [7, 8, 29] for data heterogeneous clients and demonstrate its effectiveness in resource heterogeneous environments. Additional experimental results on a language task are provided in Appendix E. Implementation details, and further experiments with modifications to settings in this section, are also in Appendix E.
Centralized Training: We compare Algorithm 1 with [1. 33] for ResNet-20 and ResNet-32 [13] models trained on CIFAR-10 10 [18] dataset.
Table 1 Test accuracy (in %) on CIFAR-10.
While both [1, 33] are limited to binary quantization, [1] can be seen as spe- cific case of our centralized method where the centers do not get updated. From Ta- ble 1, we see that updating centers (Algo- rithm 1) significantly improves the perfor- mance (0.48% increase in test accuracy).
Method
ResNet-20
ResNet-32
Full Precision (FP)
92.05
92.95
ProxQuant [1] (lbit)
90.69
91.55
BinaryRelax [33] (Ibit)
87.82
90.65
Algorithml 1 b1
91.17
92.20
Algorithml (2bits)
91.45
92.47
Allowing quantization with 2 bits instead of Ibit for Algorithm 1 further increases the test accuracy.
8

---

Table Test accuracy (in %) for CNNI model at all clients."
0.75
0.70
Method
FEMNIST
CIFAR- 10
0.65
0.60
FedAvg (FP)
94.92 + 0.04
61.40 + 0.29
0.55
QuPeD(2Bits)
Local Training (FP)
94.86 + 0.93
71.57 + 0.28
0.50
Local Training(28tr))
0.45
FodAve
Local Training (2 Bits)
93.95 = 0.23
70.87
Local Training (1 Bit)
93.00 = 0.50
0.13
0.40
0.80
100
150
200
250
QuPeD (FP)
97.31 0.12
75.06
0.75
QuPeD (2 Bits)
96.73 = 0.27
0.70
QuPeD Bit)
95.15 = 0.21
+
0.65
0.60
QuPeL (2 Bits)
96.10 = 0.14
=
0.55
DuPeD (FP!
Per-FedAvg
QuPeL (1 Bits)
94.06 = 0.28
71.01 0.32
0.50
PFedMe
pFedMe (FP) [7]
96.60 + 0.37
73.66 0.65
0.45
FML
Per-FedAvg (FP) [8]
97.16 + 0.21
74.15 + 0.41
0.40
50
100
150
200
250
Federated ML. (FP) [29]
96.32 + 0.32
74.34 + 0.30
Figure Test Acc. vs epoch (CIFAR-10)
Personalized Training: We consider an image classification task FEMNIST [4] and CIFAR-10
[18] datasets. We consider two CNN architectures: (i) CNN1 (used in [26]): has 2 convolutional and 3 fully connected layers. (ii) CNN2: this is CNNI with additional convolutional layer with 32 filters and 5 5 kernel size. For CIFAR-10 we choose a batch size of 25. For FEMNIST, we choose variable batch sizes to have 60 iterations for all clients per epoch. We train each algorithm for 250 epochs on CIFAR-10 and 30 epochs on FEMNIST. For quantized training, as standard practice [28]. we let the first and last layers of networks to be in full precision. We use last 50 epochs CIFAR-10. and 5 epochs on FEMNIST for the fine-tuning phase.
Data Heterogeneity (DHI): We consider simulate data heterogeneity on CIFAR-10, similar to [26]. we allow each client to have access to data samples from only 4 randomly chosen classes. Thus, each client has 1000 training samples and 200 test samples. On FEMNIST, we use a subset of 198 writers from the dataset and distribute the data so that each client has access to data samples written by 3 randomly chosen writers. The number of training samples per client varies between 203-336 and test samples per client varies between 25-40. Test samples are sampled from the same class/writer that training samples sampled from, in parallel with previous works in heterogeneous FL..
50 clients for CIFAR-10 10 and n 66 for FEMNIST. To
Table Test accuracy (in %) for CNNI model at all clients, with client sampling.
Method
MNIST K 0.1)
FEMNIST 3
FedAvg (FP)
92.87 + 0.05
91.30 0.43
QuPeD (FP)
98.17 + 0.32
94.93 = 0.25
QuPeD (2 Bits)
98.01 + 0.15
94.56 + 0.18
QuPeD (1 Bit)
97.58 + 0.23
92.52 0.64
pFedMe (FP) [7]
97.79 + 0.03
93.70 + 0.39
Per-FedAvg (FP) [8]
95.80 + 0.29
92.10 + 0.22
Federated MI. (FP) [29]
98.03 + 0.31
92.73 + 0.36
Resource Heterogeneity (RH): To simulate resource heterogeneity for QuPeD, we consider 4 settings: (i) half of the clients have CNNI in full precision (FP) and the other half CNN2 in FP, (ii) half of the clients have CNNI in 2 bits and the other half CNN2 in FP, (iii) half of the clients have CNNI in bits and the other in FP, (iv) half of the clients have CNNI in 2 bits and the other half CNN2 in 2 bits.
Results (DH): We compare QuPeD against FedAvg [26]. local training of clients (without any collaboration),, and personalized FL methods: PFedMe [7]. Per-FedAvg [8], Federated Mutual Learning [29], and QuPel. (Footnote 5). For all methods, if applicable,
set 10 local iterations, use learning rate decay 0.99 and use weight decay of 10-4; we fine tune the initial learning rate for each method independently, see Appendix E for details. The results are provided in Table 2 with full client participation (K = n2), plotted in Figure for CIFAR-IO, and in Table 3 with client sampling where we state average results over 3 runs; all clients train CNNI (see Appendix E for CNN2) and quantization values are indicated in parenthesis. Thus, we only consider model personalization for
'eere QuPeD (FP) corresponds to changing alternating proximal gradient updates with SGD update on model parameters in Algorithm 2, Local Training (FP) corresponds to SGD updates without communication.

---

Table Test accuracy (in %) on FEMNIST and CIFAR-IO for heterogeneous resource distribution among clients.
Resource Heterogeneity
FEMNIST
CIFAR-10
Local Training
QuPeD
Local Training
QuPeD
CNNI(FP) CNN2(FP)
93.41 = 0.82
97.44
72.81 + 0.03
75.50 + 0.25
C2 (CNN((P))
92.70 + 1.09
97.01 0.05
72.42 + 0.17
75.08 + 0.18
C2 (INS)))
93.56 + 0.38
96.96 0.15
71.23 + 0.08
74.84 + 0.30
C2 CNNN222 Bits)
91.11 + 0.23
96.64
12.15:0.477
74.64 0.27
data heterogeneity. In Table 2, we observe that full precision QuPeD consistently outperforms all other methods for both datasets. Furthermore, we observe QuPeD with 2-bit quantization is the second best performing method on CIFAR-10 (after QuPeD (FP)) and third best performing method on FEMNIST despite the loss due to quantization. Hence, QuPeD is highly effective for quantized training data heterogeneous settings. Since QuPeD outperforms QuPeL., we can also (empirically) claim that considering KD loss to encourage collaboration is superior to l2 distance loss. Lastly, we observe from Table 3 that QuPeD continues to outperform other methods under client sampling.
Results (DH+RH): We now discuss personalized FL setting with both data and resource heterogeneity across clients. Note that since FedAvg, pFedMe, and Per-FedAvg cannot work in settings where clients have different model dimensions, we only provide comparisons of QuPeD with local training (no collaboration) to demonstrate its effectiveness. The results are given in Table We observe that QuPeD (collaborative training) significantly outperforms local training in all cases (about 3.5% or higher on FEMNIST and 2.5% or higher on CIFAR-10) and works remarkably well even in cases where clients have quantized models without any significant loss in performance.
6 Discussion
In this work, we introduced QuPeD: an algorithm for training of quantized and personalized models in heterogeneous Federated Learning (FL) settings. Our proposed scheme tackles two of the major problems in practical FI. applications: resource and data heterogeneity among participating clients. QuPel) allows clients learn personalized models based on their local data distributions while simultaneously allowing these models to be different in architecture and weight precision. This is unlike existing personalized FL. algorithms in literature which only consider the data heterogeneity aspect of FL. Our theoretical results on the convergence rates recovers previous results, which are in simpler settings, when the clients have identical resources, i.e., same model size and precision. When clients have distinct resources, our result ties convergence speed to combination of resource and data heterogeneity revealing additional insights. Our experimental results demonstrate that QuPeD delivers superior empirical performance even under aggressive quantization.
Convergence results stated in Theorems 1, 2, use bounded gradients assumptions A.3, A.5 for non- convex objectives considered in this paper. Incorporating recent progress on weakening gradient assumptions in convergence analyses for FL (e.g.. [16, 30] and references therein) into our framework is part of future work. We remark that our current assumptions are fairly standard and applicable our new optimization formulation tackle data and resource heterogeneity of the clients, which is the principal focus and contribution of our work. Simulations consisting of tens of thousands (or more) of clients and lower client sampling ratios are also crucial, and are left for potential future investigations that can shed further light on our methods at massive scale.
Societal Impact: This paper considers collaborative personalization in learning, which can signif- icantly improve learning performance. However, such personalization is only as good as the data used for training, and if not used properly could lead to information bubbles and disjointed views for each client, e.g., clustering methods that put together clients with similar data statistics. We ameliorate that by not clustering similar data and models, but this is aspect that might need more examination for social consequences. We explicitly consider diverse resources in our designs, potentially enabling clients with fewer resources (devices with less capabilities) to benefit from richer resources; potentially a positive impact in using resources more equitably.
10

---

7 Acknowledgments
This work was partially supported by NSF grants #2007714 and #1955632, and by Army Research Laboratory under Cooperative Agreement
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on. We also thank the reviewers for detailed inputs that helped improve the presentation of our results.
References
[1] Yu Bai, Yu-Xiang Wang, and Edo Liberty. Proxquant: Quantized neural networks via proximal operators. In International Conference on Learning Representations, 2019.
[2] Debraj Basu, Deepesh Data, Can Karakus, and Suhas N. Diggavi. Qsparse-local-ser Dis- tributed SGD with quantization, sparsification and local computations. In Advances in Neural Information Processing Systems, pages 14668-14679, 2019.
[3] Jérôme Bolte, Shoham Sabach, and Mare Teboulle. Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program., (6(112):459-44 2014.
[4] Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv: 2018.
[5] Hassan Dbouk, Hetul Sanghvi, Mahesh Mehendale, and Naresh Shanbhag. Dbq: A differentiable branch quantizer for lightweight deep neural networks. European Conference on Computer Vision, pages 90-106. Springer, 2020.
[6] Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003. I 13461, 2020.
[7] Canh T. Dinh, Nguyen H. Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. In Advances in Neural Information Processing Systems, 2020.
[8] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta-learning approach. In Advances in Neural Information Processing Systems, 2020.
[9] Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. In Advances in Neural Information Processing Systems, 2020.
[10] Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li. Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full- full-precision and low-bit neural networks. Proceedings of the IEEE/CVF International Conference on Computer Vision,
pages 4852 4861, 2019.
[11] Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. Lower bounds and optimal algorithms for personalized federated learning. In Advances in Neural Information Processing Systems, 2020.
[12] Filip Hanzely and Peter Richtárik. Federated learning of mixture of global and local models. arXiv preprint arXiv:2002.05516 2020.
[13] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 000.00253,,10 2015.
[15] Lu Hou, Quanming Yao, and James T. Kwok. Loss-aware binarization of deep networks. In International Conference Learning Representations, 2017.
11

---

[16] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning, In International Conference on Machine Learning, pages 5132-5143, PMLR, 2020.
[17] Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. In International Conference on Machine Learning, pages 3252-3266 PMLR, 2019.
[18] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 11far-10 (canadian institute for advanced research). 2009.
[19] Cong Leng, Zesheng Dou, Hao Li, Shenghuo Zhu, and Rong Jin. Extremely low bit neural network: Squeeze the last bit out with admm. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[20] Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint 190.0311,,,3381, 2019.
[21] Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, and Tom Goldstein. Training quantized nets: A deeper understanding. In Advances in Neural Information Processing Systems, pages 5811-588 1-5221, 2017.
[22] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems 2020, MLSys, 2020.
[23] Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances Neural Information Processing Systems, 2020.
[24] Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Relaxed quantization for discretized neural networks. In International Conference on Learning Representations, 2019.
[25] Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002. 106799, 2020.
[26] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effin learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273-1282. PMLR, 2017.
[27] Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. In International Conference on Learning Representations, 2018
[28] Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe. Binary neural networks: A survey. Pattern Recognition, 105: Sep 2020.
[29] Tao Shen, Jie Zhang, Xinkang Jia, Fengda Zhang, Gang Huang, Pan Zhou, Kun Kuang, Fei Wu, and Chao Wu. Federated mutual learning, arXiv preprint arXiv:2006. 167665,,06.665,, 2020.
[30] Navjot Singh, Deepesh Data, Jemin George, and Suhas Diggavi. Squarm-sgd: Communication- efficient momentum sgd for decentralized optimization. IEEE Journal on Selected Areas in Information Theory, 2(3):95,-969, 2021.
[31] Virginia Smith, Chao-Kai Chiang. Maziar Sanjabi, and Ameet S. Talwalkar. Federated multi-task learning. In Advances in Neural Information Processing Systems, pages 4434-4434, 2017.
[32] Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-sheng Hua. Quantization networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[33] Penghang Yin, Shuai Zhang, Jiancheng Lyu, Stanley J. Osher, Yingyong Qi, and Jack Xin. Binaryrelax: A relaxation approach for training deep neural networks with quantized weights. SIAM J. Imaging Sci., 11(4):200522223 2018.
12

---

[34] Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized federated learning with first order model optimization. In International Conference on Learning Representations, 2021.
[35] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4320-4328 4328, 2018.
13


Reviews:
[{"review_id": "bRa6LzL9olU", "reviewer": "Reviewer_hK95", "summary": "This paper proposes a novel optimization objective to address the heterogeneity issue in the setting of federated learning. The heterogeneity comes from the fact that different local clients may have different data distribution and resource constraints. The proposed method allows every client to learn a quantized model with possibly different sizes. Locally, each client optimizes for a loss function containing two major terms: 1) loss of the local quantized model; 2) the distance between the quantized model and the global model.\n\nThe authors propose an alternating proximal algorithm, and provide convergence analysis under standard assumptions. The authors then empirically show that the proposed algorithm outperforms related works on CIFAR-10 and FEMNIST.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The paper is logically well structured. The design idea behind the proposed loss function is clearly stated in Section 2. All the theoretical results are followed by remarks that explain the implications, which greatly improves the readability of this paper.\n- The idea of allowing every client to learn a personalized quantized model of different size seems new in the federated learning literature. \n- The paper provides convergence analysis of the proposed algorithm.\n\nWeakness:\n- This paper is motivated by the cross-device federated learning (as stated Section 1), but the proposed algorithm may be difficult to use in the cross-device setting. Federated learning has two settings (see Table 1 in [1]): \n   - Cross-device: in this setting, a client can be a mobile device (e.g., smartphones), and there can be millions of clients in total, only a few of them (say hundred) participate in each training round, and each client usually participates *at most once* in the entire training session.\n    - Cross-silo: in this setting, a client can be an organization (e.g., banks or hospitals), and there can be hundreds of clients in total, and it is ok to assume that all clients are available and participate in every round.\n\nThe proposed algorithm (Algorithm 2 in the paper) requires that all clients participate in every training round, and that every client maintains a local model and is responsible to update and carry it from this round to the next round. Furthermore, the proposed algorithm does not support inference on new clients (i.e., clients that do not appear in the training). These constraints make it difficult to use the proposed algorithm in the real-world cross-device federated learning applications.\n\n- The paper does not compare with a natural baseline algorithm FedAvg + fine-tuning [2], i.e., each client fine-tunes the global model (learned by FedAvg) locally to create a personalized model. This simple algorithm can be used in both cross-device and cross-silo settings. If needed, one can add an extra quantization step after local fine-tuning to create a quantized personalized model.\n\n- The hyperparameters are not properly tuned for the baseline algorithms compared in Section 5. For example, in Section 5, the authors said that “For all methods, if applicable, we set τ = 10 local iterations, initialize learning rate of 0.1...”. It is unclear why the authors used the same learning rate for all baseline algorithms, and it would be more convincing if the authors can tune the learning rates (and also epsilons if adaptive optimizer like Adam is used) for different algorithms. \n\n[1] Kairouz et al., “Advances and Open Problems in Federated Learning”, arXiv: 1912.04977, 2019.\n\n[2] Jiang et al., “Improving Federated Learning Personalization via Model Agnostic Meta Learning”, arXiv: 1909.12488, 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors pointed out the potential societal impact in Section 6.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "_mLCzIfiybY", "reviewer": "Reviewer_UdzD", "summary": "The paper addresses the important problem of quantized personalization in Federated Learning. The paper introduces knowledge distillation across heterogeneous clients requiring different precision/architectures. To that effect a loss function is formulated, and the algorithm QuPeD is proposed. Theoretical converge of QuPed is provided, along with experimental validation. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The difference between model x_i and quantization center c_i should be emphasized. I would like to see an example of such quantization with particular values of c_i. \n\nMotivate the choice of R(x,c). l_1 norm is used, with is suited for proximal algorithms used in QuPeD, but such choice should be motivated well. what not l_2. Is there any sparsity concerns?\n\nAs mentioned by the authors themselves, the bounded gradient assumption is too strong.\n\nFurther explanation needed on Assumptions A.4 and A.5. Appendix A does not provide any values of the lipschitz/smoothness parameters. Since, the authors are using sigmoid/tanh, I would like to see the parameters derived for such functions. Only A.6 is justified in the appendix, by connecting it to Fallah, et.al. Moreover, A.7 seems very strong. Overall I believe that the set of assumptions are too strong, and more justifications are required. Either show standard instances where these thing hold, or argue that these are standard by citing papers using the same assumption.\n\nTheorem 2 is very hard to parse. There are several parameters, and scaling of those are not clear, so the result right now is non-intuitive. Also, after the theorem, I would request the authors to add implications/remarks about the theorem, maybe get back to some standard results.\n\nI have a question about the 1/\\sqrt{T} rate on gradient-norm-squared. In standard distributed learning, with full gradient (which is same as this paper), a 1/T rate on gradient norm squared can be obtained (see https://arxiv.org/pdf/1911.09721.pdf, https://arxiv.org/pdf/1803.01498.pdf with no Byzantine setup). I would request the authors to comment on this. Why is the rate slower? Is this because of the proximal operator? Is it optimal? Why do we need a proximal operator if it hurts the rate of conergence?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Mentioned briefly. For further limitations, see above for detailed comments.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "C2tCvof06Tf", "reviewer": "Reviewer_JdrY", "summary": "This paper proposes a QuPeD framework personalized FL via knowledge distillation (KD) to alleviate both data and resource heterogeneity. The authors first present a relaxed formulation and an algorithm under the centralized setting. Next, they extend the previous formulation to FL setting to enable different model dimensions and precision requirements for clients. The authors further provide detailed convergence arguments and empirically evaluate the proposed methods on practical datasets.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Merits:\n-The paper considers both data and resource heterogeneity, which is a fundamental issue in FL. The paper is well written and organized.  Algorithms and their corresponding convergence results are clearly interpreted. Empirical evidence is sufficient and the results seem promising. \n\n-The use of KD is natural (but not new) as each client may hold models with different dimensions and precision. To formalize the use of KD as an optimization problem is original and important as it facilities the theoretical characterizations. \n\n-The implications of the coupled effect of data and resource heterogeneity on the convergence rate from Theorem.2 are interesting.\n\nWeakness/potential improvements\n-The client's loss function (i.e., Eq.2) seems contrived since it simply combines a compressed objective and an integrated KD loss. Is there a principled way for each client to choose $\\lambda$ and $\\lambda_p$? \n\n-Further evaluations on more challenging tasks e.g. Stack Overflow are needed.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have explicitly addressed the limitations (appendix) and potential negative societal impact (last section in the paper).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "2uKjJ82lnp4", "reviewer": "Reviewer_g4tj", "summary": "This paper studies a quantized and personalized algorithm for faster inference and adapting to data/system heterogeneity in federated learning.\nPersonalization is facilitated by knowledge distillation. Model compression is achieved by end-to-end training to optimize for quantized models and quantization parameters.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nPros:\n\n* The problem of jointly consider personalization and compression is interesting.\n\n* In terms of quantization alone, the formulation (6) is novel to me (to the best of my knowledge), and demonstrates improvements compared with (4) or (5), although it is a direct combination of existing works.\n\n* The observation from Theorem 2 that clients with more representative data can tolerate more aggressive compression is interesting.\n\n* Experiments clearly show the effectiveness of both the quantization method, and the main contribution QuPeD. QuPeD also outperforms existing personalization methods, while allowing for heterogeneous model architecture and precision.\n\nConcerns:\n\n\n* (a) One bottleneck of federated learning is the computation and memory constraints of on-device training. The proposed method only reduces inference time, but adds much training overhead, which is very expensive. Other directions of compression methods (beyond teacher-student knowledge distillation) that can reduce training cost (computation or memory) while producing a smaller model could be more desirable for federated learning, for example, training factorized layers. (b) If only considering inference, there are other natural and simpler baselines, such as first performing regular personalization training (via KD or other techniques) and then pruning the models as a post-processing step. How does this work empirically?\n\n* The convergence analyses do not support partial client participation.  \n\n* I think the experiments are lacking results on text data. As the gradient distribution of image data and text/language data are very usually different., it not clear if the proposed method could still yield improvements on the new type of data, and how much. \n\n* Both the quantization method and the personalization method (KD) have been studied by previous literature, and the proposed optimization objective (2) combines the two ideas. I think discussing how potentially these two parts interact with each other (i.e., how to design better personalization methods that are inherently suitable for quantization) would be more interesting.\n\nMy major concerns are the first two points.\n\n\n## update\nI have updated my score (see detailed comments in my response below). ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the limitations of their work, and I don't think there is potential negative societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper discusses how to properly design personalized models in Federated Learning. A novel aspect of the paper is to  introduce knowledge distillation across heterogeneous clients, and require that different precision/architectures can be used across different network users.  Theoretical converge of the proposed QuPed is provided, along with extensive experimental validation. Overall the paper is well-written, and the authors have made some interesting observations, such as  how data and resource heterogeneity can affect the convergence rate, and that clients with more representative data can tolerate more aggressive compression are interesting to the community. On the downside, the KD for (personalized) federated learning has already been studied, and the optimization formulation for the proposed algorithm seems to be a straightforward combination of different aspects of the modeling (quantization and KD). 
Decision: Accept (Poster)