Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed FedLite, which extends a specific federated learning setting called split-learning to be more transmission efficient by a vector quantization scheme. In split federated learning, the classification layer that has a larger parameter set is saved and learned in the server, while the client learns the relatively lightweight feature layers. In FedLite, the activation outputs of the cut layer are compressed by vector quantization before sending to the coordinate server. To alleviate the issue of noisy gradients caused by compressed activations, a regularization term is introduced into the gradient update step which is derived from the Taylor expansion. Empirical results show the effectiveness of the proposed approach, which largely reduces transmission costs without significant performance loss. 
main_review: Strengths:
* The approach is well motivated to reduce the transmission cost in split federated learning. 
* The gradient correction scheme is a principled solution to tackle the noisy gradients issue and has solid theoretical implications. They also analyzed the theoretical error bound of their proposed approach which further echos the role played by the gradient correction term.
* Their approach shows empirical effectiveness compared with the Split learning baselines. The importance of a positive lambada is clearly analyzed by ablation studies.
 
Weakness / Questions:
* This paper addresses a specific federated learning scenario, i.e. the split learning setting. However, there are other FL approaches towards efficient parameter transmission or computation cost. I feel that the narrow problem setting that FedLite aims to tackle somehow weakened its contribution. It is unclear to me how FedLite may or may not benefit other FL approaches.
* The experiment section is not extensive enough, as only one splitFed baseline is compared. I am curious to see how the asymptotic performance of the FedLite is dropped compared with non-split learning baselines, especially FedAvg. 
summary_of_the_review: This paper is a solid extension of the Split Federated Learning work, which improves transmission efficiency with an acceptable performance drop. The approaches are well motivated and theoretically verified, however the contribution scope of this work is kind of limited and the empirical study part can be further enriched by discussing and comparing with more related work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces FedLite, a compression method built on top of Split Learning (SL) for better communication efficiency. Under the cases where the size of the intermediate output is large, SL can introduce a communication bottleneck in model training. FedLite tackles that issue via compressing the intermediate output via a vector quantization-based method. A gradient correction scheme is proposed to further enhance the proposed compression method. Convergence analysis and experimental results are provided to justify the effectiveness of FedLite.
main_review: The paper proposes to use vector quantization to enhance the communication efficiency of SL, a popular paradigm to conduct federated learning. Both theoretical and empirical results are provided to show the effectiveness.


Pros:
1. The paper is well written, and the research direction of enhancing the communication efficiency of federated learning is potentially impactful. 
2. The proposed method is analyzed under both theoretical and empirical lenses. 
3. The detailed hyper-parameters and training schedules are provided, which gives good evidence of the reproducibility of the experiments. 

Cons: 
1. The main concern I have for the current version of the draft is that the proposed FedLite method is not compared against any baseline methods (except for SL itself). Let’s assume we want memory efficient and communication efficient federated learning method, an immediate idea is to use EfficientNet/MobileNet (which are both small and computationally efficient) with FedAvg. How does FedLite look like compared against that vanilla baseline? 
2. The paper proposes to use vector quantization without motivation. It is pointed out that the sparsification-based method requires setting the targeted sparsity, but other than the sparsification method, there are still a series of methods to compress tensors, e.g., [1-5]. The authors are expected to at least motivate the proposed method a bit more. 
3. I wonder if the communication inefficiency of SL is only caused by the CNN and LSTM architectures. In Transformers, splitting Encoder blocks (which is widely used in pipeline parallelism [6-7]) usually leads to quite good communication efficiency. If so, does it mean FedLite can only attain marginal gains on Transformer based NNs? 
4. I’m highly curious about the convergence curve of FedLite compared against FedAvg. It seems FedLite is a FedSGD-ish method, which requires a lot more rounds of communication compared against FedAvg (as stated in the very first FedAvg paper). It is widely known that the communication cost depends on message size and latency (which is related to the round of communication). I wonder if FedLite does lead to significant wall-clock time-saving. 
5. One potential approach to reduce the required number of communication rounds for FedSGD-like methods is to increase the batch size. However, since FedLite aims at low-memory training, it limits the space of using large-batch training. 
6. A recent technical report indicates that the super-high compression ratio is actually meaningless with reasonable bandwidth [8]. In federated learning applications, the bandwidth is usually limited. Thus, it would be useful to show the actual communication time saving of FedLite. 
7. It is stated that FedLite can be combined with existing privacy-preserving methods. But that aspect is not discussed in the paper at all. It’s important for the authors to discuss how to make FedLite private since the vanilla version clearly discloses the users’ privacy. 

[1] https://arxiv.org/abs/1802.04434

[2] https://arxiv.org/abs/1610.02132

[3] https://arxiv.org/abs/1905.13727

[4] https://arxiv.org/abs/1806.04090

[5] https://arxiv.org/abs/1705.07878

[6] https://arxiv.org/abs/1806.03377

[7] https://arxiv.org/pdf/2104.04473.pdf

[8] https://arxiv.org/pdf/2103.00543.pdf
summary_of_the_review: Frankly, I think the paper was rushed out. The idea is sweet but clearly requires more effort to show its effectiveness. The current draft only comes with very preliminary experimental results without comprehensive comparisons against various baseline methods. The motivation of the proposed vector quantization method is also not clear. Overall, I do not think the current draft deserves to appear for a presentation at ICLR 2022.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method to compress the intermediate activations of a neural network that is split across a client and server in the Federated Learning setting. In this 'Split Learning' scenario, transmitting activations and corresponding gradients incur significant communication costs. The authors discuss FedLITE, a method to compress activations during the forward pass and to correct the gradient to account for the quantization during in the forward pass. Their paper is augmented with an interpretation of the gradient correction term as minimising a surrogate loss function, as well as a convergence analysis of their approach and empirical evaluations. 
main_review: I truly enjoyed reading this paper. I had thought about the problem before and was excited to read a paper that addresses the communication of split activations in the Split Learning framework. The proposed solution to the problem is presented clearly and the experimental evaluations are thorough. 
Regardless, there are a few questions I would like to see answers, as well as some issues I have with the paper, the removing of which would make it a lot stronger.

Introduction & Motivation:
- While Split Learning allows to 'outsource' the final layers' heavy computations to a server, this inherently reduces the privacy compared to FedAvg based approaches that communicate weights-only. Especially since labels have to be communicated for the loss-computation. While I understand that there is a need for SplitLearning regardless of this disadvantage, I would encourage the authors to mention this significant downside to SL in either introduction or discussion.

Background/RW:
- The authors mention model parallel training in some places in their paper. However they do not focus at all on it, provide no experiments for it and ignore questions that arise when using their approach in the model parallel training setting. More specifically, in model-parallel training the additional latency of the quantisation approach cannot be ignored and would presumably cannibalise on any advantages that the reduced communication brings with it. Either the authors include a proper analysis of Model Parallel Training or they should remove the discussion around it. 
- There is large body of literature that studies quantised training of neural networks. Some works focus on weight-quantisation, others on activation-quantization (and the combination of both) for the purpose of efficient inference on resource-constrained devices. While many different and sometimes complicated approaches exist, the simplest approaches consider stochastic scalar quantization in combination with the Straight-Through estimator. I am not so familiar with the Split Learning literature, but I would expect this to be a trivial baseline for quantising the activations of a single layer's activations in the forward pass (as considered in this paper). In order to be applicable for sped-up inference in these settings the quantization grid needs to be fixed (optionally learned) at the end of training. In the split learning setting, this grid could be constructed arbitrarily for any chunk of the b x D activation matrix, similarly to the authors' discussion in this paper. Naturally the split-learning approach is not constrained to uniform grids (as would be for inference acceleration on e.g. in8 inputs), so you might even consider quantising a transformation of the activations with e.g. the Normal CDF (assuming activations are normally distributed). I would ask the authors to either consider including experiments for these quantization approaches in their paper or elaborate why this would be out of scope for Split Learning, maybe I am missing something...


Method
- The authors mention that client-weights $w_c$ need to be synchronised every iteration. Is that the default in SplitLearning or would you also consider averaging every E local epochs, as is done in FedAvg traditionally? 
- You mention that you assume 'each float-number' to require 64bits. In the context of FedLITE that is relevant for the code-book entries. Is that a requirement for your method to function well? Since NN activations are usually 32bits (16bits are often enough), I am surprised by that assumption. 
- Computational complexity: Can you elaborate on how expensive it is to compute the quantised activations for a mini-batch of data? Ideally you would provide timing numbers of forward/backward pass and quantization on a given hardware and discuss the impact on resource-constrained clients. 
- You discuss the infeasibility to share codebooks from previous iterations. Did you consider sharing codebooks between data-points of given class, assuming that activations especially higher up in the network share many similarities conditioned on the label? Since you have to transmit the label anyways, decoding would come for free. 
- Your discussion surrounding Eq. (5) is not entirely clear to me: What is the benefit from considering Eq (4) as a gradient of the loss in (5)? The first term involving $\hat{z}_i$ is not intuitive to me. 
- One of the core insights of Figure 3 seems to be that decreasing R to be =1 seems to have the most favourable scaling properties. Maybe that insight could be reinforced and pointed towards when setting up that hyper-param in the experiment section. 
- I did not analyse the provided convergence proof in detail

Experiments
- Hyperparameter selection: How did you end up at the hyper parameters chosen for the base SplitFed algorithm? 
- At the last paragraph of 5.1, the authors compare against FedAvg, which assumes a certain number of local epochs E (unless the authors compare against FedSGD). What is the assumption here?
- Does the optimal lambda change significantly with different hyper-parameters L and q? (Figure 4)
- Codebook-design: I would be interested to see the consequences of quantising the codebook to less than $\phi$ bits. Is that something sensible to consider? Additionally, it would be interesting to study the entropy over the selected code-words: Is the method making use of the available code-words? If not, Entropy-coding of the code-words might be a viable approach for further communication reduction. 

Small things:
- In the introduction you make a claim about AlexNet, however it sounds like you claim a general CNN architecture.
- Typo above Eq (2): 'while and the remaining layers'
- I understand the need for an acronym, but the letter-selection seems random... :D On the other hand I have no better suggestion. 

summary_of_the_review: This is a promising paper and discusses a relevant problem. Some detailed questions remain and some changes I believe are necessary. If the authors address my concerns I will improve my rating of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose end to end training framework that relies on a novel vector quantization.

The authors claim the following contributions:
- End-to-end communication-efficient neural network splitting-based FL approach for resource-constrained clients.
- New gradient correction scheme for backward pass.
- Reducing communication rounds up to 490X without compromising the accuracy.
- Providing 
main_review: **Pros**:
- The paper is well written and structured.
- The paper provides theoretical insight about FedLite convergence rate.
- Interesting and novel idea aim to reduce the communication cost.

**Cons**:
- The authors should cite additional previous works that tried to address the problem of client's device heterogeneity [1,2,3].
- My main concern is that under supervised learning setup FedLite sends the labels to the server to obtain $\nabla_{z_i}h\(w_s;z_i\)$. Sending the labels to the main server compromises privacy (for example in healthcare systems).
- I think the authors should add more challenging experiment to check the trade off between compression and accuracy. (Celeb a from LEAF dataset for example).

**References**
[1] Shamsian, Aviv, Aviv Navon, Ethan Fetaya, and Gal Chechik. "Personalized Federated Learning using Hypernetworks." arXiv preprint arXiv:2103.04628 (2021).
[2] Achituve, Idan, Aviv Shamsian, Aviv Navon, Gal Chechik, and Ethan Fetaya. "Personalized Federated Learning with Gaussian Processes." arXiv preprint arXiv:2106.15482 (2021).
[3] Cho, Yae Jee, Jianyu Wang, Tarun Chiruvolu, and Gauri Joshi. "Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer." arXiv preprint arXiv:2109.08119 (2021).
summary_of_the_review: Questions were raised during the review process.
review_title: null

Meta-review: The paper introduces a compression method for distributed Split Learning for better communication efficiency, by compressing the intermediate output between client and server model by vector quantization. Convergence analysis and experimental results are provided. 
Unfortunately consensus among the reviewers remained that it remains slightly below the bar after the discussion phase. Main remaining concerns were the variety of baselines and benefits from split learning setup in experiments, compared to other FL approaches, quantization approaches, architecture splits. Reviewers also missed a discussion of latency requirements of model-parallel training in FL as opposed to data parallel which allows less frequent communication compared to here (e.g. discussing the split layer size vs latency trade-off, here of quantized intermediate layers compared to regular FL). The newly added Figure 6 does not specify or vary the number of local steps (or batch size) in FedAvg.
We hope the detailed feedback helps to strengthen the paper for a future occasion.
Decision: Reject