Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper builds on Split learning [Thapa et al. 2022], where the first few layers of a neural network model are stored and shared across clients and the server, while the remaining layers are only stored and trained on the server. Since the communication cost is high to "make the connection" between the split layers, the paper proposes to quantize the forward pass outputs so that centroids are communicated instead. The paper further proposes a gradient correction technique for clients to fix the gradient mismatch problem.
strength_and_weaknesses: Main concerns:
1. The quantizing technique proposed by the paper uses $k$-means as part of the quantization process. Question is how big are the $R$ groups and what is value of $R$? This could be potentially computationally expensive since it is done during each iteration. Also, is this done on the client-side? There is a bolded computation efficiencies in Section 5 Experiments, but I do not see it being addressed.
2. To follow-up 1., the centroids are sent from client to server, but the way I am reading the paper, the server is sending to the clients all the so-called "mismatched" gradients?
3. It appears the technique increases the computation burden on the clients.
4. The technique is specifically limited to split learning.

clarity,_quality,_novelty_and_reproducibility: The paper is straightforward to read. The empirical results do no strongly convey the novelty of the proposed technique.
summary_of_the_review: The novelty is on the low side and the empirical results are not compelling.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel method for minimizing communication overheads incurred in the setting of split learning when training large models. Specifically, the authors propose a quantization scheme by dividing the activations into sub-vector and applying k-means clustering and use the resulting centroids as a compact representation for these activations. Finally, authors introduce a gradient correction technique to avoid the mismatch introduced due to the approximation. Empirically, authors verify that the proposed method is able to provide 490x reduction in communication cost with minimal drop in accuracy.
strength_and_weaknesses: Strengths:
- The presentation in the paper is good and clear to understand.
- 490x compression without any loss of accuracy looks impressive and appealing to large-scale model training of today.
- It seems the proposed idea will translate to more general model parallel settings (e.g. all-reduce architecture used in many large deep learning frameworks used today), but I will leave it to authors to comment on this.

Weaknesses:
- Related work seems to have no discussion on hashing and other gradient compression / quantization techniques. Encourage authors to make this section more thorough by covering more recent works in this area.
- I am curious about how much overhead the k-means step poses? Getting a good quality clustering is key to the proposed method. I do not see much discussion regarding this in the paper. Have the authors done any evaluation to understand the time/accuracy tradeoffs involved in the quality of clusters obtained? Also, how was the clustering quality determined and k chosen?
- I feel that while the empirical section has some essential experiments to validate the correctness and efficiency of proposed method, it could be made more compelling by adding some evaluations on real-world benchmark datasets perhaps from the standard computer vision, language model literature. This will help put the contribution of this work in perspective.

Some additional questions:
- Here obviously parameter-server framework is used. But can authors comment on how their techniques can be applied in other general model parallel scenarios (e.g. all-reduce in DeepSpeed Zero-2,3 etc)?
- Product Quantization - how is this comparable to Hashing based approaches (for e.g. using LSH) for hashing activations?
- Avoiding communication of redundant activations is a great idea. I am curious to what extent is this solution data independent or dependent? If we assume a really adversarial dataset where there is no redundancy in activations, how does the proposed algorithm perform?
- In clustering into L clusters, how is the L decided? What is the impact of choosing different values for L?
- Seems like tuning \lambda is crucial to good performance of the method, can authors comment on how to systematically choose this value for a given setup?
clarity,_quality,_novelty_and_reproducibility: - The paper is written well and the problem statement, motivation, method and evaluation sections are reasonably clear to me. I do think that the paper can be strengthened further by providing a more exhaustive relation work (brief contrast of other compression, quantization, related hashing techniques etc).
- I also feel the novelty factor of the paper is average.
- The authors have presented some details to help reproduce the results (e.g. Appendix has hyper-parameter values for the training algorithm). However, I do not see much details on how k-means was used. Would appreciate if the authors considered adding details on this step.
summary_of_the_review: Overall I feel the paper addresses an important challenge in federated learning and the proposed ideas can alleviate bottlenecks in large model split learning. However, I feel the core ideas presented in the method still need some more justifications (overheads of k-means step, role of hyper-parameters involved in gradient correction, experiments on some frequently used benchmark datasets).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: FedLite is a framework for split FL that addresses on of the main problems this variant of FL has: high communication costs that result from sending the activations generated by the last layer (the split layer) that runs on the client side to the server AND downloading the gradients from the server which are needed to update the client-only layers and complete a training step (i.e. forward+backward through the whole model). The Authors propose a variant to Product Quantization (PQ) to compress the activations to be sent to the server and a gradient correction step to later on update the client-only layers with the "approximated" gradients received from the server. FedLite demonstrates over an order of magnitude reduced communication costs for three federated datasets. 
strength_and_weaknesses: I don't have that many comments about this work. Product quantization has been around for a while and used extensively to compress models (among other use-cases). Splitting the vectors into sub-vectors is not novel but grouping them (so to reduce the number of centroid sets) is -- although very marginal contribution in my opinion. But then experiments are done with `R=1` (at least those in Figure 4 -- what I would consider as the main results in this work -- suggest so)?

### Strengths
*    The proposed gradient error-correction mechanism works well. That being said, a 5% performance drop is quite a large margin and I don't think it is right to label that as "minimal loss in accuracy", as done in various point in the paper. 


### Weaknesses

*    Why do the Authors assumed a floating-point value requires 64 bits (e.g. a `double`) instead of, the much more common, 32 bits (e.g. a `float`)?
*    Lack of comparison to other compression baselines. For example, how would `INT4` or binary compression of the activation sent to the server perform? (that would be a `8x`, `32x` compression though). 
*    From the paper it is not clear to me if the server also applies PQ to the computed gradients before sending them to the clients. Is this the case?
*    Could the Authors elaborate how would someone use this approach in a real FL setup (where it is likely you'd not normally run the experiment multiple times)? Could the Authors suggest a procedure to automatically (maybe at each round) determine how much we can afford compressing the activation sent by the clients? I would imagine one way to asses how "compressable" the activation are would be to look into the distribution of the centroid assignments: if uniform, it is likely that a higher quantization error would be introduced.
clarity,_quality,_novelty_and_reproducibility: The paper is well written in general and Figure 2 is good. I would have like to see more interesting content in Figure 3 or at least make it more self-contained. 
summary_of_the_review: As I stated earlier, I think this work is proposing something interesting and that delivers good results (but without other works as baselines/context). Product quantization is a known technique and, given that in the end the Authors use `R=1` (as stated at the end of the second paragraph in Sec 5), the only novelty of this work is the correction mechanism for the incoming gradient from the server side training.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose an extension to the existing SplitFed algorithm to address the high communication costs imposed by transmitting gradients during each backprop. They propose using product quantization to reduce the communication costs, followed by a Taylor series correct to correct the returned gradients from the server (which were computed using the quantized uploaded gradients). The author provide a convergence analysis for the error introduced by the quantization + correction, showing that the scale of the newly introduced error term can be made to have the same rates as the error from SGD. 
strength_and_weaknesses: Strengths
- Strong convergence analysis which shows the error introduced is not too big (same rates in T as SGD error), and gives guidance on setting the value of the tuning parameter \lambda (even if that guidance is in terms of Evalues that in reality will never be known).

Weaknesses
- Experimental results are not compared with standard simple communication reduction methods, making it hard to fully assess whether this new method is any good. 
clarity,_quality,_novelty_and_reproducibility: The method is well described and the writing makes it easy to follow, even though the method is somewhat complex.

The experimental results could be presented to make it more clear what the overall benefit (in terms of total communication) from the method is. Currently the results are focused on comparison with the original SplitFed method, however what is most interesting is comparison with FedAvg, as well as other simple communication cost reduction methods (e.g. simple masking, or simple quantization). Comparison with FedAvg is briefly discussed in the results section, but it would be much better for data to make it easy to see that comparison, rather than comparison with SplitFed. 

nit: 
- A little proofreading required (e.g. 'efficiencyin' in the conclusion)
summary_of_the_review: The method is interesting and quite novel. However without comparison with some baseline and much more simple compression methods it's hard to gauge the value of the method. It seems that it provides upload communication savings in the order of 10-100x compression compared with FedAvg with minimal impact on quality. However these savings are on the same order of magnitude as other much more simple methods, and so without close comparison with these simple methods, we cannot tell if the added complexity is adding value.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a split-learning approach to perform model partition in federated learning with large model. The idea of using a clustering idea with gradient correction is proposed. On some dataset the communication cost is reduced by 490x 

The proposed algorithms is not well described and it seems a lot of details are hidden. For example, it is not clear if k-mean clustering for quantization is even feasible in each epoch. Also, the method rely heavily on the quality of the cluster and hence a very cheap clustering is unlikely to work. The paper also did not discuss many hashing and other ideas solving the problem of training large models. 


justification_for_why_not_higher_score: The practicality of a good clustering is questionable and there was no rebuttal. 
justification_for_why_not_lower_score: NA
Decision: Reject