FedLite: Improving Communication Efficiency in Federated Split Learning
Authors: Jianyu Wang
Hang Qi
Ankit Singh Rawat
Sashank J. Reddi
Sagar M. Waghmare
Felix Yu
Gauri Joshi
Venue: Submitted to ICLR 2023 (2023)

Abstract: In classical federated learning, clients contribute to the overall training by communicating local updates for the underlying model on their private data to a coordinating server.  However, updating and communicating the entire model becomes prohibitively expensive when resource-constrained clients collectively aim to train a large machine learning model. Split learning provides a natural solution in such a setting, where only a (small) part of the model is stored and trained on clients while the remaining (large) part of the model only stays at the servers. Unfortunately, the model partitioning employed in split learning significantly increases the communication cost compared to the classical federated learning algorithms. This paper addresses this issue by compressing the additional communication cost associated with split learning via a novel clustering algorithm and a gradient correction technique. An extensive empirical evaluation on standard image and text benchmarks shows that the proposed method can achieve up to 490x communication cost reduction with minimal drop in accuracy, and enables a desirable performance vs. communication trade-off.

Under review as a conference paper at ICLR 2023
FEDLITE: IMPROVING COMMUNICATION EFFICIENCY IN FEDERATED SPLIT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In classical federated learning, clients contribute the overall training by communi- cating local updates for the underlying model on their private data to coordinating server. However, updating and communicating the entire model becomes pro- hibitively expensive when resource-come conpurrccoontr clients collectively aim to train a large machine learning model. Split learning provides a natural solution in such a setting, where only (small) part of the model is stored and trained on clients while the remaining (large) part the model only stays at the servers. Unfortunately, the model partitioning employed in split learning significantly increases the commu- nication cost compared to the classical federated learning algorithms. This paper addresses this issue by compressing the additional communication cost associated with split learning via a novel clustering algorithm and a gradient correction tech- nique. An extensive empirical evaluation on standard image and text benchmarks shows that the proposed method can achieve up to 490x communication cost reduction with minimal drop in accuracy, and enables desirable performance vs. communication trade-off.
INTRODUCTION
Federated learning (FL) is emerging field that collaboratively trains machine learning models on decentralized data (Li et al., 2019; Kairouz et al., 2019; Wang et al., 2021). One major advantage of FL that does not require clients to upload their data which may contain sensitive personal information. Instead, clients separately train local models on their private datasets, and the resulting locally trained model parameters are infrequently synchronized with the help of coordinating server (McMahan et al., 2017) 1).. While the FL. framework helps alleviate most of existing FL algorithms critically assume that the clients have enough compute and storage resources to perform local updates on the entire machine learning model. However, this assumption does not necessarily hold many modern applications. For example, classification problems with an extremely large number of classes (often in millions and billions) commonly arise in the context of recommender systems (Covington et al., 2016), information retrieval (Agrawal et al., 2013), and language modeling (Levy & Goldberg, 2014). Here, the classification layer of neural network itself is large enough that a typical FL. client, e.g., a mobile or IoTT device, cannot even store and locally update this single layer, let alone the entire neural network.
concerns for distributed training,
Split learning (SL) is a recently proposed technique (Vepakomma et al., 2018; 10; Thapa el al., 2022) that naturally addresses the above issue of FL. It splits the underlying model between the clients and server such that the first few layers are shared across the clients and the server, while the remaining layers are only stored the server. The reduction of resource requirement the clients is particularly pronounced when the last few dense layers constitute a large portion of the entire model. For instance. common convolutional neural network (Krizhevsky, 2014), the last two fully connected layers take 95% parameters of the entire model. In this case, if we allocate the last two layers to the server, then the client-side memory usage can be reduced by 20x. Nonetheless, one major limitation of SL is that the underlying model partitioning leads to increased communication cost for the resulting framework. Specifically, to train the split neural network, the activations and gradients at the layer where the model is split (referred to as cut layer) need to be communicated between the server and clients at each iteration. The additional message size is in proportion to the mini-batch size as well as the activation size. As result, the communication cost for the model training can become prohibitive

---

Under review as a conference paper at ICLR 2023
Syschinaizztion
of server
codewwrds
Server side model My
ofeiient clsta
zini barch
Quantizer
Decoder
Loss
Gind Conrector
Clients
Server
Gradient
quantiaa actisstion
Vector Quantization
(by clustering similar activations)
Figure I: Overview of the proposed algorithm: FEDLITE. In order to reduce the additional communi- cation between the clients and the server, we propose to cluster similar client activations and send the cluster centroids to the server instead. This is equivalent to adding a vector quantization layer in split neural network training. The success of our method relies on a novel variant of product quantizer and gradient correction technique. We refer the reader to Section 4 for further details.
whenever the mini-batch size and the activation size of the cut layer large (e.g., in one of our experiments, the additional message size can be 10 times larger than that of the client-side model).
In this paper, aim make the split neural network training communicaioon-effeccentt as enable its widespread adoption for FL. resource-constraine the critical observation that, given a mini-batch of data, the client does not need communicate per-example activation vectors if the activation vectors (at the cut layer) for different examples in the mini-batch exhibit enough similarity. Thus, we propose a training framework that performs clustering of the activation vectors and only communicates the cluster centroids to the server. Interestingly, this is equivalent to adding vector quantization layer in the middle of the split neural network (see Figure for an overview of our proposed method). Our main contributions are follows.
settings. Our proposed solution is based on
We propose communication--fficiee method for federated split learning (cf. Section 4). The approach employs novel compression scheme that leverages product quantization to effectively compress the activations communicated between the clients and server.
After applying the activation quantization, the clients can only possibly receive noisy gradients from the server. The inaccurate client-side model updates lead to significant accuracy drops in our experiments. In order to mitigate this problem, we propose gradient correction scheme for the backward pass, which plays a critical role achieving a high compression ratio with minimal accuracy loss.
We empirically evaluate the performance of our approach on three standard FL datasets (cf. Sec- tion 5). Remarkably, we show that our approach allows for up to 490x communication reduction without significant accuracy loss.
We present a convergence analysis for the proposed method (cf. Section 4.3), which reveals trade-off between communication reduction and convergence speed. The analysis further helps explain why the proposed gradient correction technique is beneficial.
Here we note that our proposed method has potential applications beyond federated split learning as can be applied in any learning framework that can benefit from a quantization layer. For instance, it can be used to reduce the communication overhead in two-party vertical FL. (Romanini et al., 2021), where the model is naturally split across two institutions and the data labels are generated on the server. It also worth mentioning that the proposed method does not expose any additional client-side information to the server than vanilla split neural network training approach, such as SPLITFED (Thapa et al., 2022). Thus, our method can also leverage existing privacy preserving mechanisms such as differential privacy (Wei et al., 2020) to provide formal privacy guarantees, though this is orthogonal our work.
2

---

Under review as a conference paper at ICLR 2023
2 BACKGROUND AND RELATED WORKS
Federated Learning (FL). Suppose we have a collection of M clients I = 21, M..
Each client i E I has a local dataset D; and corresponding empirical loss function (u ESED, f(w;
where w denotes the model parameters, and n; = [Dy denotes the number of samples of the local dataset. The goal of FL is to find a shared model w that can minimize the averaged loss over all clients, defined as follows:
M
F(w) = Σ P.F((w), 1=1
(1)
where p; = -n:/Ei ny is the relative weight of local loss F,. Motivated by the data privacy concerns, under an FL framework, the clients perform local training and only communicate the resulting models to coordinating server as opposed to sharing their raw local data with the server (McMahan et al., 2017). In current FL algorithms, the model size is limited by the resource constraints of clients. Consequently, these algorithms are not feasible when dealing with large machine learning models that cannot fit in clients memory or require large compute. Compared to these vanilla FL algorithms, split learning-basse based approaches enables learning on resource-constrained clients at a cost of additionally exposing the label information of clients to the server.
Large Model Training in FL. To deploy large machine learning models on resource-come resource-constrained clients, a few recent works proposed methods to reduce the effective model size on clients. For example, Diao et al. (2020) varied the layer width on clients depending on their computational capacities; and Horvath et al. (2021) proposed ordered dropout to randomly removing neurons in the neural network. These methods are effective in reducing the width of the intermediate layers; however, they still place the full classification layer (which is in proportion to the size of the output space) each client to compute the the client update. This may not be feasible for many clients as the parameters in the classification layer of a model can dominate the total number of model parameters (Krizhevsky, 2014).
Split Learning (SL). Split learning (SL) is another way of minimizing (1) without explicitly sharing local data on clients (Vepakomma al., 2018). In partieular, SL splits the neural network model into two parts on a layer basis. The first few layers (called client-side model parameterized by w.) are shared across clients and the server, while the remaining layers (called server-side model parameterized by w.) are only stored and trained on the server. Under this splitting, the original loss function for data sample Ç can be re- re-iritten as follows:
fww; {) huw, u(u (wwr: E&
where u is the client-side function mapping the input data to the activation space and h is the server- side function mapping the activation scalar loss value. Training both the client- and server-side models requires communicating client activations (i.e., the output of the client-side model, also called smashed data in some literature) between the clients and the server. We further elaborate concrete SL algorithm as a baseline in Section 3.
Communication-effcienn SL and Model Parallel Training. Recently, He et al. (2020); Han et al. (2021) proposed to add a classification layer on the client-side model in SL so that each client can locally update its parameters. This can effectively reduce the communication frequency between the clients and the server. However, this kind of method is not suitable for the settings where the classification layer dominates the entire model size such that the clients may not have enough memory space to store the entire classification layer. This paper focuses on reducing the communication amount at each round, and thus complementary the above works.
Another relevant approach is reducing communication cost in model parallel (MP) training (Dean et al., 2012). In MP training the worker nodes (dedicated machines instead of resource-constrainee client devices) also need to transfer activations of intermediate layers with each other. Gupta et al. (2020) proposed to sparsify the activations according to the magnitude of each element and achieved up to 20x compression ratio (uncompessed//coppeseel) However, in practice it might be tricky to set the threshold for sparsification. Our proposed method explores an orthogonal dimension of sparsification by clustering similar activations within the mini-batches and communicating only the cluster centroids. Empirical results show that the proposed method can achieve up to 490x compression ratio without significant drop in the model performance.
3

---

Under review as a conference paper at ICLR 2023
Product Quantization. Product quantization (PQ) is compression method and has been widely used in approximate nearest neighbor search (Jegou et al., 2010; Ge et al., 2013b). Given a batch of vectors, PQ divides each vector into multiple chunks and performs K-means clustering separately each chunk. Now, the resulting cluster centroids construct the codebook and the closest codeword will be used to represent each vector. Instead of directly computing the distance between two high- dimensional vectors, computing the distance between two codewords is orders of magnitude more efficient and faster. Note that the previous works used PQ compress either learned features after training (see, e.g., Ge et al., 2013a) or the parameters of a neural network (see, e.g., Chen et al., 2020). In this paper, we show that PQ also effective in compressing the output of intermediate layers and present a simple technique to enable backpropagation through the corresponding quantization layer.
BASELINE: THE SPLITFED ALGORITHM
In this section, we review representative federated split learning algorithm, namely SPLITFED (Thapa et al., 2022). Each iteration of SPLITFED contains four steps detailed as follows. Wherever it simplifies the presentation, we assume that the mini-batch size on each client 1 as the derivations hold true for arbitrary batch sizes.
1. Client Forward Pass:
Let S be a randomly selected subset of clients. For E S, compute the output of the client-side model z; u Ww: {), where E ID; a randomly chosen training sample, and 2, E F denotes the activations of the last layer of the client-side model. Then, each selected client sends 2, (together with corresponding label if necessary) to the server.
2. Server Update: The server treats all the activations (also referred to as smashed data) (z,)es as inputs to perform one step of gradient descent on the server-side model: Aw, = is Eies Pt V ww,h(uu hw, 2,)/ Eies Pt. where 7)s denotes the server learning rate.' In addition, the server also computes the gradient with respect to the activations V h(w,;; 2,) and sends it back to the corresponding client i.
3. Client Backward Pass: Each selected client computes the gradient with respect the client-side model using the chain rule: g: e w. V h(w,; z;) V w, u(w:: E)-
4. Client-side Model Synchronization: At last, the client-side model is updated by synchronizing client gradients Aw. T/ Eies P:g:/ Eies Pi, where Ve denotes the learning rate for the client-side model.
One can easily validate that, with the above four steps, SPLITFED is equivalent to mini-batch stochastic gradient descent (SGD) with total batch size of B (S). No matter how the network is split, the performance of the algorithm remains unchanged and has the same iteration complexity as mini-batch SGD.
Besides, note that, a selected client needs to upload both the activations during "Client Forward Pass" and the gradients of the client-side model during "Client-side Model Synchronization". Thus, the communication size per client per iteration can be written as |wel + Bd. Since the up-link communication bandwidth is quite limited for clients in federated learning, when the mini-batch size B or the dimension of the activation d gets large, the up-link communication may become the main bottleneck that limits the scalability of SPLITFED. message from clients to the server (Reisizadeh et al., 2019; Li et al., 2019). Next, we present novel vector guantization based training method that addresses this limitation of SPLITFED to enable commuuicaation-ffficiiee training in federated split learning.
critical to compress the uploaded
PROPOSED METHOD: FEDLITE
We now introduce our proposed method, FEDerated spLIT learning with vEctor quantization (FEDLITE), which can drastically reduce the up-link communication cost clients in the SPLITFED algorithm. The key idea of our method to compress the redundant information activations for each mini-batch. For instance, if we assume that all the activation vectors within mini-batch are nearly identical to each other, then instead of sending all activation vectors, the
the client
TTee server learning rate here has different definition from previous FI. literature, e.g., Reddi al. (2020).
4

---

Under review as a conference paper at ICLR 2023
(i) Divide each vector into ( subvectors
(ii) Stack subvectors into R groups
(iii) Assign each subvector to a cluster centroid
Original activations
Quantized activations Z
1.00 1.0.0.100
K-means
g R
Codebook
Codewords
U.0.1.1,0.
K- K-means
0.0.1.0.11
B
gB/R
Communicated
B
Figure 2: Illustration of our proposed quantizer. Given mini-batch of activations Z RR there are three steps: (i) divide each activation vector into q subvectors; (ii) stack subvectors into R groups based on their indices; (iii) perform K-means clustering to get I, centroids for each group. Each subvector can be represented by the index the closest centroid in its corresponding group. On the server, by simply rearranging centroids, we can get the quantized activations Z.
client needs send only one representative vector to the server, thereby, reducing the communication cost by a factor of B.
Building on the above observation, we let each client imput their activations into clustering algorithm to get L centroids. Then, each activation vector is represented by the index of its closest centroid. Instead of sending a mini-batch of vectors, only I. centroids and the cluster assignments are transmitted to the server. This procedure is equivalent to vector quantization. Here, a simple choice of the clustering algorithm K-means (Krishna Murty, 1999). However, as we observe Figure vanilla K-means clustering leads to a high quantization error with very limited compression ratio. It non-trivial to design a proper clustering algorithm that can minimize the communication between clients and server, while maintaining a low quantization error. In Section 4.1, design novel variant product quantizer that can flexibly operate this trade-off.
While the above approach is appealing, also introduces new challenges. First, note that, in the backward pass, due to the additional quantization layer, the server no longer knows the true client activations; thus, constraining the quantized activations. Furthermore, in order to update the client-side model parameters, the clients need receive the gradients with respect to the original activations, which In Section 4.2, we introduce gradient correction technique to mitigate this gradient mismatch, which critical for our method achieve high compression ratio with minimal accuracy loss.
compute the gradients with respect the (possibly noisy)
available any more.
In the following sections, we present the key components of our proposed method and highlight how they address the aforementioned challenges. In addition, a convergence analysis is provided in Section 4.3. Unless otherwise stated, for the ease of exposition, we will focus on the operations specific client and omit the client index. However, it is important to keep in mind that the client-side operations happen in parallel on all selected clients.
4.1 FORWARD PASS: COMPRESSING ACTIVATIONS WITH PRODUCT QUANTIZATION
In this subsection, we first present our proposed qunatizer and then elaborate on its advantages. A visual illustration provided in Figure
The Proposed Compression Scheme. Suppose a client has computed batch of activations
Z = [z(1) (...,,((B)) 2( E d using the client-side model we, where d is the dimension of each activation and B denotes the mini-batch size on each client. Our proposed quantizer first divides each activation vector z(), E [1, B] into g subvectors with equal dimension d/g. We denote the s-th subvector of 20) as 20.0) Rd/2, Then, the quantizer stacks all subvectors into
5

---

Under review as a conference paper at ICLR 2023
Qvantization eroo
Durs (1-4R-co)
0.10
nerozse
(3<Reeg)
K-meens (Amqu=1
0.08
0.06
0.04
0.02
Decreaso
0.00
10a
10'
10'
Compression ratio
Figure 3: Quantization error (lower is hetter) versus compression ratio (larger is better). Our proposed quantizer achieves a better quantization error vs. compression ratio trade-off as compared vanilla product quantization (PQ) and K-means. In this example, the activation size is d = 9216, and the mini-batch size B 20. The activations are trained via two-layer CNN on Federated EMNIST. In each curve, we vary the number of clusters L. For green curves, the number of subvectors takes value in {288, 1152, 4608); for red curves, the number of groups R takes value in (2304, 1152, 384, 1}. and y is fixed as 4608.
R groups based on their indices. For example, the first group g() contains all subvectors with indices (j, 1), (j. (j.g/ Rj [1, B]; the second group g(2) contains all subvectors with indices (j.g/R) R + 1), (i. + 2)...., (j. 2g/ R), Vj E [1, B). and so forth. At last, K-means clus- tering is performed on subvectors within each group ((r), Vr [1, R) to find L cluster centroids ((r) = (c(t.1) ((rd), C Rd/9. Now each subvector 2 201)) can be approximated by its closest.
centroid in the corresponding group. Formally, if 2(1,)) E g(r), then
g(io) A clr,))))) where 1(d.0) arg min -c(0)p
[E[1,)]
In the above equation, g(i+) represents the quantized version of 2 2(1+)) By concatenating all quantized subvectors, server can get the quantized activations Z and use it input update the server-side model. In order to obtain the quantized subvectors, each client needs to send all cluster centroids (C(r) [r (referred to the codebook) and the cluster assignments (1!5-)) (j) (referred as the codewords) the server. Assuming that each floating-pon number occupies & bits?, the transmitted message size per client is reduced to @dRL/4 + Bg logg L from ødB.
The Benefit of Subvector Division: Reduced Quantization Error. The first key component of our proposed quantizer is subvector division, i.e., setting the number of subvectors q to be greater than 1. When q = 1, the quantizer reduces to vanilla K-means. In this case, the codebook size is odL and each activation vector have L possible choices. When we set = R > 1, the codebook size is still ødL. However, each activation vector now has L possible choices, as each of q subvectors has L choices. Thus, the number of quantization levels becomes exponentially larger without any increase in memory usage and computation complexity. As illustrated by the green lines in Figure 3, having more quantization levels or effective centroid choices can significantly lower the quantization error.
The Benefit of Subvector Grouping: Improved Compression Ratio. By using subvector division, although the quantization error can be reduced, the codebook size ødL is still pretty large. This is because subvectors within one vector are quantized separately using different codebooks. To further reduce the communication size, propose subvector grouping and force subvectors within one group to share the same codebook. As a result, the total codebook size can be reduced to gdL.R/g When q R > 1, there be order of magnitude increase in the compression ratio, illustrated by the red lines in Figure 3. One can also observe that, by changing the value of R, our proposed quantizer (red línes) achieves much better waao--vvrsu--opprssiiioe trade-off than K-means and vanilla production quantization scheme.
Why not Reuse the Codebooks from Previous Iterations? In our proposed scheme, the codebook is reconstructed at each iteration on clients. This is necessary, since in FL, only few stateless clients are selected to participate in training each iteration and they have non-IID data distributions (Kairouz et al., 2019). Previous codebooks can be stale and other clients' codebooks may not be suitable.
When computing the compression ratio in this paper, always assume Φ 64.

---

Under review as a conference paper at ICLR 2023
Now that we have introduced our product quantization- based compression scheme, next address the issue of performing backpropagation in the presence of the additional quantization layer.
4.2 BACKWARD PASS: GRADIENT CORRECTION
As mentioned at the beginning of Section 4, there is a gradient mismatch problem in the client backward pass. While client needs Vz, h( w' zi) to update the client-side models, it can only receive Vg, h x,) from the server. A naive solution is to treat Vz,h as an approximation of V h and update the client-side model by V h(w,, V V m. u(un;; E) = (a. (00//22))a (0z, However, due to gradient mismatch, this approach can lead to significant performance drop, as observed in our experiments (cf. Section 5). In order to address this issue,
propose gradient correction technique. In particular, we approximate the gradient Vz, h W z;) by its first-order Taylor series expansion: Va, h( w.; _,) + V h( w.,; _ (z; E.). While the higher-order derivative may be expensive to compute, we use scalar parameter AI > 0 to replace V3 h (w; 3),. Consequently, the gradient of the client-side model is defined as follows:
ah(w,: x
(u(wi:)))
g:
A(z: 2i)
(4)
0z:
Ow.
In Section 4.3, we will provide convergence analysis that can help explain the effects of gradient correction; in Section 5, we will empirically show that setting a strictly positive is crucial for the success of the proposed method. In the following discussion, provide an intuitive explanation of the correction to further motivate our approach.
The Effect of Regularization. We show that the client-side gradient in (4) can be considered gradient of the following surrogate loss (proof is provided in the Appendix):
1
2 2 2
(5)
where u(wei {) is the activation of the client-side model. Besides, 2, 2 -v,h-u, //2 and the quantized activation & are fixed vectors that do not have derivatives when computing the gradient. Setting À > O is equivalent to having regularization term, as per (5). The regularizer encourages the client-side model w. to move in a direction that can decrease the quantization error ||z; ž: Interestingly, using larger value of À may let clients put more emphasis on minimizing the quantization error and lead to quantization--rient
activations. However, one cannot set a arbitrarily large value for X because the client-side model may output the same activation for all inputs and fail to minimize the original loss function.
4.3 CONVERGENCE ANALYSIS
In this subsection, we provide a convergence analysis for FEDLITE. The analysis will highlight how the quantization error influences the convergence and how the gradient correction technique helps. In particular, the analysis is conducted under standard assumptions for mini-batch SGD. We assume the objective function F(w) F(ww,: ux,)) is T--Lipschitz smooth (i.e., F(w) F( < L. 1|w vl)), and stochastic gradient (ww) has bounded variance [ V F < / where B is the mini-batch size per client and S is the number of selected clients per iteration. Under these assumptions, we have the following theorem.
Theorem (Convergence of FEDLITE). If the client-side model and server-side model update using the same learning rate a = = BS/T where T is the number of total iterations, then the expected gradient norm of the function ming (VF(((()))) II? can be bounded by
4(F(w(()) Fin)
4Lo²
BS T +2
A
+ (A2 (A2
(6)
VBST
VBST
Opt. mini batch SGD
Addul. caused quantization
where k. is the maximal quantization error max || -1I. X is the tunable parameter in our gradient correction scheme, Fing the lower bound of function value, and constants A1, A2, Ag are the largest eigenvalues of matrices 82 hiw,, z)/dziw, 82t Ph =)/ Du(w,; E)/ur, ww, respectively.
7

---

Under review as a conference paper at ICLR 2023
Theorem guarantees that the proposed algorithm FEDLITE converges to neighbourhood around the stationary point of the global function. And the size of this neighborhood in proportion to the maximal quantization eπror k during the training. When there is quantization, the error bound (6) recovers that of mini-batch SGD. Moreover, one can observe that setting a positive À can help to reduce the additional error caused by adding the quantization layer.
5 EXPERIMENTS
We implement the proposed method FEDLITE using FedJAX (Ro et al., 2021) and evaluate its effectiveness three standard federated datasets provided by the Tensorflow Federated (TFF) package (TFF, 2021): ) (i) image classification on FEMNIST, (ii) tag prediction on StackOverflow (referred as SO Tag) and (iii) next word prediction on StackOverflow (referred as SO NWP). On FEMNIST, the same model architecture as (Reddi et al., 2020) is adopted. We place two convolutional layers and two dense layers on the clients and the server, respectively. With this splitting, the client- side model only has about 1.6% trainable parameters of the entire model. On SO Tag, both the client- and server-side models contains only one dense layer. On SO NWP, we place one LSTM layer and one dense layer on the clients, and another dense layer on the server. The ratios between the client-side model size and entire model size are 83% and 79% on SO Tag and SO NWP, respectively. Here, we note that even though the client-side model sizes for SO Tag and SO NWP do not correspond to an ideal setting for the federated split learning approaches, we include these two datasets to showcase the utility of our proposed method for anguage-based tasks.
For each task, we select the learning rate that is best for the baseline SPLITFED algorithm. Although separately tuning the learning rate for proposed method can further improve its performance, using the same learning rate as SPLITFED already demonstrates the advantages of our method. Unless otherwise stated, the number of groups R in our proposed quantizer is trade-off in Figure 3 and experiments.
as it exhibits the best
Effectiveness of FedLite. As discussed in Section 4.1, there is a trade-off between the final performance and compression ratio. Although the additional quantization layer helps reduce the communication cost, also causes performance drop. While utilizing the quantization layer, important to understand the range of the compression ratio that does not degrade the final performance too much. To this end, we vary the number of clusters L and number of subvectors q in our proposed method and report the resulting accuracy vs. compression trade-off in Figure 4. Note that we define the compression ratio to be the ratio between the original activation size and compressed message (codebook codewords) size.
Main Results:
One can observe that the proposed method can achieve at least 10x compression ratio with almost no accuracy loss. Furthermore, if we allow for 5% relative loss compared to SPLITFED, then our method can achieve a 490x compression ratio on FEMNIST. This suggests that 99.8% information is redundant and can be dropped during the up-link communication. Surprisingly, SO Tag, the Recall @ 5 even improves when adding the quantization layer. We conjecture that compressing the outputs of a intermediate layer might have regularization effects similar dropout.
Overall Communication and Computation Efficiencies. Here, we provide a concrete example showing how the proposed method improves the communication efficiencies over previous works. On FEMNIST, setting : = 1152 and L = 2 amounts to 490x compression ratio, which amounts to a 490x reduction in the additional communication introduced by the network splitting. Compared to SPLITFED, the overall up-link commutation cost (including the client-side gradients synchronization) about 10x smaller. Compared FEDAYG, the up-link communication cost per round is reduced by 62x, with 64x less trainable parameters on the clients. We further compare the training curves with respect to total communication costs of FEDLITE against SPLITFED and FEDAVG in Figure in Appendix. Besides, the overall wall-clock time saving may depend on the characteristies of the training system. One can get an estimate using the analytical model in (Agarwal et al., 2021).
Effectiveness of the Gradient Correction. From Figure 4, it is easy observe that the gradient correction technique (i.e., >> 0) is crucial for improving performance. Without correction, the algorithm even diverge in the high compression ratio regime. While À is separately tuned for each point (i.c., each (g. L) pair) in Figure 4, we found that fixing À for all (q. L) pairs still leads to significant improvements. In Figure 5, report the performance on FEMNIST for various
8

---

Under review as a conference paper at ICLR 2023
Accuracy)
Recallies
50
Accuracy
70
15
65
Outs
Ours
Ours
Ours w/o grad. car.
30
Ours weo grad.
10
Ours we grad. cor.
SplitFed
SplitFed
SpiitFad
<5% acc.
<5% drop
5
<5% acc. drop
10a
16a
10a
10
10a
10-
10a
107
10a
10
Compression ratio
Compression ratio
Compression ratio
(a) FEMNIST
(b) Tag
(c) SO NWP
Figure 4: Trade-off between the accuracy and compression ratio. Our proposed method can achieve up to x90x, 247x, and 51x communication reduction with less than 5% accuracy drop on FEMNSIT, SO Tag, and SO NWP, respectively. Each curve in the figures corresponds to one value of g (number of subvectors); and each point on a curve corresponds to a specific value of L (number of clusters). For our method, the number of groups R is fixed as 1.
Accuracy
Accuracy
4608
3608
Subvectors
2300
83.5
82.7
Subvectors
2304
83.7
63.5
83.5
62.9
61.4
1152
83.5
82.0
1253
82.6
0
82
80.9
576
82.1
81
80.5
78.7
576
75.4
80
Vanilla PQ
Ours
2b18
81.4
79.7
79.8
79.3
77.4
78.1
75.7
68.3
55.7
5.1
78
wio quanteation
144
80.7
79.7
78.5
72.8
42.1
124
76.7
74.2
63.0
5.1
5.1
76
16
2
32
4
10a
Compression ratio
10:
10'
10a
Clusters
Clusters
(a) À = 5 10
(b) A= =0
(c) Quantizer comparison
Figure 5: Ablation studies on FEMNIST. (a) and (b): Validation accuracy on FEMNIST when fixing À and varying g and L. Setting small positive value for À improves accuracy for almost all (g. L) pairs; (c) Thanks to subvector grouping, our proposed quantizer can achieve an order magnitude larger compression ratio as compared to vanilla product quantization scheme.
choices of q and L. In particular, with g 288, the accuracy improvement ranges from 72%. Furthermore, we found that small À ranging from 10 to 10-3 works well on all training tasks in our experiments. Setting larger leads to near-zero quantization error. But the model may have poor performance, as it tends to output the same activation for all inputs and becomes meaningless.
Effectiveness of the Proposed Quantizer: In Section 4.1, we already show that stacking subvectors into groups (i.e., setting R < 9) is critical to reach a high compression ratio. But how does it affect the model performance? To answer the question, we run the proposed method with = g > (vanilla PQ), and report the results in Figure 5c. Observe that the proposed method significantly improves the compression ratio with minimal loss of accuracy.
CONCLUSION
In this paper, we studied the problem of improving the communication efficiencyin federated split learning. Due to limitations on storage and/or compute capacity, the clients cannot locally optimize the entire model. This prohibits the usage of previous federated learning algorithms. Federated split learning is promising approach to address this issue, as it only assigns the first few (small) layers of a neural network to the clients. However, the network splitting incurs additional communication. In order to make split neural network training to be communication-effiee
we propose algorithm FEDLITE that can compress the additional communication by up to 490x with minimal loss of accuracy. The success of our method relies variant of product quantization scheme and gradient correction technique. We perform theoretical analysis as well as extensive experiments on both vision and language tasks to validate its effectiveness.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Saurabh Agarwal, Hongyi Wang, Shivaram Venkataraman, and Dimitris Papailiopoulos. On the utility of gradient compression distributed training systems. arXiv preprint arXiv:2103.000 005400 2021.
Rahul Agrawal, Archit Gupta, Yashoteja Prabhu, and Manik Varma. Multi-label learning with millions of labels: Recommending advertiser bid phrases for web pages. In Proceedings of the 22nd international conference on World Wide Web, pp. 13-24, 2013.
Ting Chen, Lala Li, and Yizhou Sun. Differentiable product quantization for end-to-end embedding compression. In International Conference Machine Learning, pp. 1617-1626. PMLR, 2020.
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference recommender systems, pp. 191-198, 2016.
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoe V Lee et al. Large scale distributed deep networks. In Advances in neural information processing systems, pp. 1223-1231, 2012.
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient feder- ated learning for heterogeneous clients. In International Conference on Learning Representations, 2020.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. Optimized product quantization for approximate nearest neighbor search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2013a.
Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. Optimized product quantization. IEEE transactions pattern analysis and machine intelligence, 36(4):744-755, 2013b.
Vipul Gupta, Dhruv Choudhary, Ping Tak Peter Tang, Xiaohan Wei, Xing Wang, Yuzhen Huang, Arun Kejariwal, Kannan Ramchandran, and Michael W Mahoney. Fast distributed training of deep neural networks: Dynamic communication thresholding for model and data parallelism. arXiv preprint ar 0rXiv:2010.08890 2020.
Dong-Jun Jun Han, Hasnain Irshad Bhatti, Jungmoon Lee, and Jackyun Moon. Accelerating federated learning with split learning on locally generated losses. ICML'21 workshop on federated learning for User Privacy and Data Confidentiality, 2021.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large cnns the edge. Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020.
Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. arXiv preprint arXiv:2/02. 13451, 2021.
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search.
IEEE transactions pattern analysis and machine intelligence, 33(1): 17--188, 2010.
Peter Kairouz. H Brendan McMahan, Brendan Avent, Aurélien Bellet. Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint 1rXiv::91222 92499777 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.660 6980, 2014.
K Krishna and M Narasimha Murty. Genetic k-means algorithm. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 29(3):433-439) 1999.
10

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky, One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.597,, 2014.
Omer Levy and Yoav Goldberg, Neural word embedding as implicit matrix factorization. Advances in neural information processing systems, 27:2177-2185, 2014.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. CoRR, 2019.
H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. Communication-efficient learning of deep networks from decentralized data. Proceedings of the 20th International Conference Artificial Intelligence and Statistics, 2017.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumnr, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint M0Xi::2003.00295. 2020.
Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: A communication-efficceet federated learning method with periodic averaging and quanti- zation. arXiv preprint arXiv: 1909. / 13014, 2019.
Jae Hun Ro, Ananda Theertha Suresh, and Ke Wu. Fedjax: Federated learning simulation with jax. arXiv preprint arXiv:2108.021177 2021.
Daniele Romanini, Adam James Hall, Pavlos Papadopoulos, Tom Titcombe, Abbas Ismail, Tudor Cebere, Robert Sandmann, Robin Roehm, and Michael A Hoeh. Pyvertical: A vertical federated learning framework for multi-headed splitnn. arXiv preprint arXiv:2104.0m N.NY:2204400049,, 2021.
TFF. TensorFlow Federated: machine learning on decentralized data. https: /www. 2021. Accessed: 30 September 2021.
tensorflow
Chandra Thapa, Pathum Chamikara Mahawaga Arachchige, Seyit Camtepe, and Lichao Sun. Splitfed: When federated learning meets split learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8485-8492 2022.
Pranecth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learning for health: Distributed deep learning without sharing raw patient data. arXiv preprint arXiv:1812.00560 2018.
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly. Deepesh Data, et al. A field guide to federated optimization. arXiv preprint arXiv:2107.06917 2021.
Kang Wei, Jun Li, Ming Ding, Chuan Ma Howard H Yang, Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security, 15:3454-3469, 2020.

---

Under review as a conference paper at ICLR 2023
A DETAILED EXPLANATION FOR THE EFFECTS OF REGULARIZATION
In Section 4.2, we argue that the client-side gradient is gradient of the following surrogate loss function
s(w.) - 2 2 1z
1 -
where 2; - Vz, h(w,; /2 and the quantized activation ス, fixed vectors that do not have derivatives when computing the gradient. Here, we are going to provide a formal proof to support this argument. Directly taking the derivative, we have
@s(w.) Om,
0zi
dzi
-2 21) Sw.
+ A(zi - zzi)
(8)
Sw.
Ozi
(V h 2,) A(z; 2,)) ow. The above equation exactly the same as the client-side gradient we defined in (4).
(9)
B PROOF OF THEOREM
B. PRELIMINARIES
Without loss of generality, we define w (wee wr). Then, the gradient of the original loss function can be decomposed into two parts:
F (w) V m. F(w); F..FF(w))
(10)
Similarly, we denote the stochastic gradients as follows:
g(w) = (w): (w))
(11)
where [w)] = Vw, F(w) and Elg. (w)) = w., F(w). In addition, we assume the stochastic
gradients with mini-batch size of BS have bounded variance:
E [lg(()) - [F(w)]l BS
(12)
That is equivalent to
o²
E (g)
w. F(w)11 + (gg((w) w F(w))l S
(13)
BS
Then, we denote the client- and server-side gradients in the presence of the quantization layer as follows:
[.(w) (w) + (,w))
(14)
(w) = g. + (w) The concrete expressions of 6, and 8, will be derived later.
(15)
B.2 MAIN PROOF
With the above notation, according to the Lipschitz smoothness of the loss function, we have
B [[F(w((+1)) - (0) < V Vw, ((w (w (ww (() (..
a² E ()))
||²
aLL E
p.. (1o(t)))
(16)
For the first term on the right-hand side of (16), we have
(1) V.n. (9) + ((8.
< V (z) /1)
V Vw. F(w
+ 2
13 18.8. (w
(17)
< (1 1 4 m. (1))
2
+ S
5 (20
(18)
2e
2
V (20(00)
||
+ 2
1
(w (1)
(19)
12

---

Under review as a conference paper at ICLR 2023
where (17) uses Young's Inequality, and 0 is arbitrary positive number. We set right after (17). For the third term on the right-hand side of (16). we have
E ()) + )
S2E ge
(")
||
+ 2E 6e (2(((()))
(20)
For the server-side gradients Is. repeat the same process. that
Combining them together, it follows
[ - a 2
/| + t ())
| |(
+ 122aL(x + x
2 g + xxx)
(21)
where
X max E |8.(w))1
(22)
2 max E [6,(1))
(23)
Using the assumption that the stochastic gradient has bounded variance, obtain that
B[ ((w(()) S - --a al 1 - ab) / Lo
BS
+2a2 xx2 + 2 (e + X2).
(24)
When aL S 1/4, we have
+ a²
E F (++2)) - (1 5 a
VF(w
+ 2a L(x + xx))
BS
"2 (xe + x2)
a
(25)
With minor rearranging and taking the total expectation, it follows that
(0)) ||
4E - ((w
4aLo²
V F
BS + (2 + 4aL) +x3).
(26)
Taking the average from t 0 T obtain that
T-1
((F(w(()) F(w)))
4aLo²
+ (2 (aL) +
(27)
aT
BS
0=
If v BS/T.
then
T-1
1)) ()) FF(w"))
4Lo²
4VBSL VT
+
+2
(x. (x.+x x3). (28)
/BST
VBST
Next we are going to show how Xe, Xs relate to the quantization error. Suppose that ((O) denotes the randomly sampled mini-batch on client Then according to the definition of server-side gradient, we have
Sh(w,; Q(u(we: E)))
ge (w)
(29)
S(0)
[
Ow,
(ES
SEB.
Oh ah(w, Q(u(wa:()) Ow,
ah (w.; ul (wei {)) Ow,
(g. +
Σ
[S()
|BB())
(Est))
SE
(30)
13

---

Under review as a conference paper at ICLR 2023
where Q represents the quantizer and maps the original activation to its quantized version. As consequence,
ah(w, Q(u(wr::) E)))
(h(w,; (w,; uwwc: E))
(,())
(31)
|
18,
Ow,
Ow.
iEste
SEB!
and
dh( Wsi Q(u(w,:E)) ah(w, u(w::
[[6,(w)]'
max {ED,JEZ
(32)
Ows
Ow,
For the ease of writing, we denote we have
u(we {) and E = Quu(w,) ()). Using mean value theorem,
Sh(ws, u)
[[)((w)] S max
(z (z-2) 2)
(33)
Sudw,
SApnk
(34)
where to + t) for some t 1, constant A₁ is the largest eigenvalue of matrix 02 (2w u//duw,, and x = max 2 =1 denotes the maximal quantization error.
Using the same technique, for the client-side gradients, we have
Oh(un: ((u(wr:()) Oh (ug u uwwe E)) (Q(u(we:E))
Ōu(we: E)
6c(w))
Σ
|800
(B(0))
Ou(wo: E))
Sw.
(ESt))
(EB(1)
@u(weik) Owc
Σ X(u(we):) Q(u((nc:
(35)
[St0]
(B((1) SEB!")
VES())
Accordingly,
Öh(w,;
@h(wsiz)
Oz
[1,,(w))
max
+A(z 2)
(36)
0z
Oz
Omc
[0Ph(ws;ur u)
2) (3 (3- z)
Oz
maax
(37)
Ou
Dur.
<Aa
A3
(38)
where A2 and Ag are the largest eigenvalues of matrices h(w, u)/Ou² and 02/Ow, respectively. Substituting the above bounds (34) and (38) on 82, 5, into (28), we have
T-1 Σ
((((w()) (0) F(w"))
4Lo²
VBST
VBST
AVBS BS
+
+2 (A? + (A2 - 2)2 max ||> - 2112. (39)
VT
Here complete the proof.
C MORE EXPERIMENTAL DETAILS
C.I ADDITIONAL RESULTS
We additionally report the training curves of FedAvg, FedLite and SplitFed on EMNIST in Figure It can be observed that in terms of communication cost, FedLite is significantly faster than other two baselines. We also need to highlight that FedLite and SplitFed have lower memory and computation requirements clients than FedAvg.
14

---

Under review as a conference paper at ICLR 2023
Accuracy
FedAvg
SplitFed
10a
10a
10a
10*
Communication size (MB)
Figure 6: Training curves for different algorithms on FEMNIST. Although split Haaming-hased approaches (SplitFed and ours FedLite) communicate every iteration, they communicate much less than FedAvg
C.2 HYPER- PAREERRR CHOICES
Here, summarize all the hyper-parameteed three training tasks.
FEMNIST
Best learning rate: 10-1.5 115
Optimizer: SGD
Mini-batch size per client B: 20
Activation size d: 9216
Number of clients per iteration: 10
Ranges of g: {4608, 2304, 1152, 576, 288, 144]
Ranges of [32, 16, 8, 4,
Ranges of X: {0, 10 110-0.555 10 1,-04,,, 10-4)
Client-side model: Same the first five layers of the neural network used in (Reddi et al.,
2020): Conv2d + Conv2d + MaxPool2d + Dropout + Flatten. Model size: 18, 816 x 64 bits.
Server-side model: Same the last three layers the neural network used in (Reddi et al., 2020): Dense Dropout Dense. Model size: 1, 187, 774 x 64 bits.
SO NWP
Best learning rate: 0.01
Optimizer: Adam (Kingma & Ba, 2014)
Mini-batch size per client B: 128. Each sample contains 30 words. So the effective batch size is 3840.
Activation size d: 96
Number of clients per iteration: 50
Ranges of q: {48, 24, 6,3)
Ranges of L: {960,, 480, 240, 120, 60,
Ranges of X: {0, 5 10
10 10
Client-side model: Same as the first three layers of the neural network used in (Reddi et al., 2020): Embedding + LSTM + Dense. Model size: 3, 680, 360 64 bits.
Server-side model: Same as the last layer of the neural network used in (Reddi et al., 2020):
Dense. Model size: 970, 388 x 64 bits.
SO Tag
Best leaming rate: 10 -0.0.
15

---

Under review as a conference paper at ICLR 2023
Optimizer: AdaGrad (Duchi al., 2011)
Mini-batch size per client B: 100.
Activation size d: 2000
Number of clients per iteration: 10
Ranges of q: (1000, 500, 250, 200, 125, 25)
Ranges of L: (100, 60, 40, 20, 10)
Ranges of A: {0, 10-3,55550 x 10 3, 10 2,5 10 10
Client-side model: One dense layer. Model size: 5000 x 2000 x 64 bits.
Server-side model: One dense layer. Model size: 2000 x 1000 x 64 bits.
16


Reviews:
[{"review_id": "lL4ElnbCt7I", "reviewer": "Reviewer_fQmd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel method for minimizing communication overheads incurred in the setting of split learning when training large models. Specifically, the authors propose a quantization scheme by dividing the activations into sub-vector and applying k-means clustering and use the resulting centroids as a compact representation for these activations. Finally, authors introduce a gradient correction technique to avoid the mismatch introduced due to the approximation. Empirically, authors verify that the proposed method is able to provide 490x reduction in communication cost with minimal drop in accuracy.", "summary_of_the_review": "Overall I feel the paper addresses an important challenge in federated learning and the proposed ideas can alleviate bottlenecks in large model split learning. However, I feel the core ideas presented in the method still need some more justifications (overheads of k-means step, role of hyper-parameters involved in gradient correction, experiments on some frequently used benchmark datasets).", "strength_and_weaknesses": "Strengths:\n- The presentation in the paper is good and clear to understand.\n- 490x compression without any loss of accuracy looks impressive and appealing to large-scale model training of today.\n- It seems the proposed idea will translate to more general model parallel settings (e.g. all-reduce architecture used in many large deep learning frameworks used today), but I will leave it to authors to comment on this.\n\nWeaknesses:\n- Related work seems to have no discussion on hashing and other gradient compression / quantization techniques. Encourage authors to make this section more thorough by covering more recent works in this area.\n- I am curious about how much overhead the k-means step poses? Getting a good quality clustering is key to the proposed method. I do not see much discussion regarding this in the paper. Have the authors done any evaluation to understand the time/accuracy tradeoffs involved in the quality of clusters obtained? Also, how was the clustering quality determined and k chosen?\n- I feel that while the empirical section has some essential experiments to validate the correctness and efficiency of proposed method, it could be made more compelling by adding some evaluations on real-world benchmark datasets perhaps from the standard computer vision, language model literature. This will help put the contribution of this work in perspective.\n\nSome additional questions:\n- Here obviously parameter-server framework is used. But can authors comment on how their techniques can be applied in other general model parallel scenarios (e.g. all-reduce in DeepSpeed Zero-2,3 etc)?\n- Product Quantization - how is this comparable to Hashing based approaches (for e.g. using LSH) for hashing activations?\n- Avoiding communication of redundant activations is a great idea. I am curious to what extent is this solution data independent or dependent? If we assume a really adversarial dataset where there is no redundancy in activations, how does the proposed algorithm perform?\n- In clustering into L clusters, how is the L decided? What is the impact of choosing different values for L?\n- Seems like tuning \\lambda is crucial to good performance of the method, can authors comment on how to systematically choose this value for a given setup?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "goHDHCnJreL", "reviewer": "Reviewer_SnmU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "FedLite is a framework for split FL that addresses on of the main problems this variant of FL has: high communication costs that result from sending the activations generated by the last layer (the split layer) that runs on the client side to the server AND downloading the gradients from the server which are needed to update the client-only layers and complete a training step (i.e. forward+backward through the whole model). The Authors propose a variant to Product Quantization (PQ) to compress the activations to be sent to the server and a gradient correction step to later on update the client-only layers with the \"approximated\" gradients received from the server. FedLite demonstrates over an order of magnitude reduced communication costs for three federated datasets. ", "summary_of_the_review": "As I stated earlier, I think this work is proposing something interesting and that delivers good results (but without other works as baselines/context). Product quantization is a known technique and, given that in the end the Authors use `R=1` (as stated at the end of the second paragraph in Sec 5), the only novelty of this work is the correction mechanism for the incoming gradient from the server side training.", "strength_and_weaknesses": "I don't have that many comments about this work. Product quantization has been around for a while and used extensively to compress models (among other use-cases). Splitting the vectors into sub-vectors is not novel but grouping them (so to reduce the number of centroid sets) is -- although very marginal contribution in my opinion. But then experiments are done with `R=1` (at least those in Figure 4 -- what I would consider as the main results in this work -- suggest so)?\n\n### Strengths\n*    The proposed gradient error-correction mechanism works well. That being said, a 5% performance drop is quite a large margin and I don't think it is right to label that as \"minimal loss in accuracy\", as done in various point in the paper. \n\n\n### Weaknesses\n\n*    Why do the Authors assumed a floating-point value requires 64 bits (e.g. a `double`) instead of, the much more common, 32 bits (e.g. a `float`)?\n*    Lack of comparison to other compression baselines. For example, how would `INT4` or binary compression of the activation sent to the server perform? (that would be a `8x`, `32x` compression though). \n*    From the paper it is not clear to me if the server also applies PQ to the computed gradients before sending them to the clients. Is this the case?\n*    Could the Authors elaborate how would someone use this approach in a real FL setup (where it is likely you'd not normally run the experiment multiple times)? Could the Authors suggest a procedure to automatically (maybe at each round) determine how much we can afford compressing the activation sent by the clients? I would imagine one way to asses how \"compressable\" the activation are would be to look into the distribution of the centroid assignments: if uniform, it is likely that a higher quantization error would be introduced.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TlYAKUEBQO", "reviewer": "Reviewer_2ofa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an extension to the existing SplitFed algorithm to address the high communication costs imposed by transmitting gradients during each backprop. They propose using product quantization to reduce the communication costs, followed by a Taylor series correct to correct the returned gradients from the server (which were computed using the quantized uploaded gradients). The author provide a convergence analysis for the error introduced by the quantization + correction, showing that the scale of the newly introduced error term can be made to have the same rates as the error from SGD. ", "summary_of_the_review": "The method is interesting and quite novel. However without comparison with some baseline and much more simple compression methods it's hard to gauge the value of the method. It seems that it provides upload communication savings in the order of 10-100x compression compared with FedAvg with minimal impact on quality. However these savings are on the same order of magnitude as other much more simple methods, and so without close comparison with these simple methods, we cannot tell if the added complexity is adding value. ", "strength_and_weaknesses": "Strengths\n- Strong convergence analysis which shows the error introduced is not too big (same rates in T as SGD error), and gives guidance on setting the value of the tuning parameter \\lambda (even if that guidance is in terms of Evalues that in reality will never be known).\n\nWeaknesses\n- Experimental results are not compared with standard simple communication reduction methods, making it hard to fully assess whether this new method is any good. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "83ZE_RTIJMj", "reviewer": "Reviewer_9KPW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper builds on Split learning [Thapa et al. 2022], where the first few layers of a neural network model are stored and shared across clients and the server, while the remaining layers are only stored and trained on the server. Since the communication cost is high to \"make the connection\" between the split layers, the paper proposes to quantize the forward pass outputs so that centroids are communicated instead. The paper further proposes a gradient correction technique for clients to fix the gradient mismatch problem.", "summary_of_the_review": "The novelty is on the low side and the empirical results are not compelling.", "strength_and_weaknesses": "Main concerns:\n1. The quantizing technique proposed by the paper uses $k$-means as part of the quantization process. Question is how big are the $R$ groups and what is value of $R$? This could be potentially computationally expensive since it is done during each iteration. Also, is this done on the client-side? There is a bolded computation efficiencies in Section 5 Experiments, but I do not see it being addressed.\n2. To follow-up 1., the centroids are sent from client to server, but the way I am reading the paper, the server is sending to the clients all the so-called \"mismatched\" gradients?\n3. It appears the technique increases the computation burden on the clients.\n4. The technique is specifically limited to split learning.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject