Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the communication bottleneck in the scalability of federated learning. A linearity constraint on the decompression operator is introduced, together with some popular scalar quantizations to to reconcile security and communication efficiency. The proposed algorithm reports the best results on LEAF benchmarks. 
strength_and_weaknesses: Strength: This paper is well written. The proposed model to reconcile security and communication efficiency is interesting and efficient. 

Weakness:The proposed model may suffer the applicability to privacy models such differential privacy.

clarity,_quality,_novelty_and_reproducibility: The paper is of good quality and clarity. The idea is novel as far as I know.
summary_of_the_review: Overall, I think that the studied model in this paper is well-motivated. The proposed algorithm is sound and evaluated with better performance compared to previous related works. Therefore, I recommend “weak accept”.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper addresses the problem of compressing the client-to-server model updates in a Federated Learning (FL) setup implementing secure aggregation (SecAgg) protocols. 

First, the paper discusses the incompatibility of standard compression methods (scalar quantization,  pruning, and product quantization) with secure aggregation protocols. 

Next, a solution to address this incompatibility has been proposed, which primarily relies on using common/shared information across all clients. This common/shared information includes scalar quantization parameters (scale $s$, zero point $z$, bit-width $b$ and SecAgg bit-width $p$ with $p > b$), pruning masks, and product quantization parameters (block size $d$, codebook size $k$, and codebook C). 

A "Secure Indexing" (SecInd) protocol has also been proposed for product quantization, which leverages the linearity of the computations (since the eventual goal is to obtain sum of the weight matrices, one can easily compute this by counting the number of occurrences of a code at a specific location and multiplying it with the value of the code).   

Experiments show that product quantization with SecInd achieves the highest compression rate at iso-accuracy. 
strength_and_weaknesses: Strengths: 
1.	The motivation is clear and well-defined.
2.	The paper is well-organized and easy to follow.

Weaknesses: 

1.  There is very little technical novelty in the proposed work. As discussed in the summary, the core idea is to select common parameters for the compression methods across all the clients. This appears to be an obvious and naive solution. The proposed SecInd method is also fairly obvious, involving only re-arranging some computations. 

The claim that "operation to take assignments to produce an aggregated update is not linear" is not a valid one. For example, suppose that we want to select the j-th element of a vector v. This can be made linear by computing the dot product between v and an indicator vector i, where the vector i has zero everywhere except the j-th element, which is set to 1. This is exactly the idea used in this work - since an outer summation is also involved, one can sum up the indicator vectors to obtain an "histogram" before computing the dot product.   

2. The whole concept of using common parameters across all clients is unlikely to work well in the non-iid setting (statistical heterogeneity). No effort has been made to investigate the effectiveness of the proposed compression methods in the non-iid scenario.  
clarity,_quality,_novelty_and_reproducibility: The paper is clear and well-written. There are no concerns about the reproducibility. The main limitation is the lack of any novel ideas.
summary_of_the_review: A well-written paper that address a valid problem, but the proposed solutions are obvious, straightforward, and lack novelty.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper provides an efficient approach on how to integrate scalar quantization, product quantization and random pruning into existing secure aggregation algorithms. The paper makes extensive experiments to show that the proposed method can reach a good balance between communication efficiency and accuracy.

strength_and_weaknesses: Strengthen:
1. The paper proposes a novel mechanism for the application of compression techniques to secure aggregation, and such compression is bi-directional.
2. Based on the experiment, the method can reach a high accuracy with a high compression rate.
​

Weakness:
1. Compression masks also contain certain information about the clients, I am not sure whether adding m_i only in the compressed domain will affect the security. The authors didn’t emphasize it. 
2. It’s only designed for uplink compressions, which in my opinion, can only reduce up to half of the overall communication bits
3. To ensure that this framework could go through, the authors need to have the same compression mask (e.g., pruning mask) in each round. It might work in the iid setting but in the non-iid setting, such limitation may lead to a huge downgrade in performance.  

​

clarity,_quality,_novelty_and_reproducibility: 1. Fig.1 is a bit hard to follow now. It’s better to give an illustration on the uplink and downlink communications
2. It’s not clearly stated why adding the mask will not affect communication efficiency. The authors should emphasize that.

summary_of_the_review: The paper gives an efficient approach on how to integrate scalar quantization, product quantization and random pruning into existing secure aggregation algorithms, and the experiment on the proposed experiment is extensive. However, it is only for uplink compression and the non-iid setting is not considered.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 1. The authors target the communication bottleneck and scalability of multi-client federated learning (FL). They formalize and tackle the client-to-server model updates compression during training under Secure Aggregation (SECAGG) primitive, which is a core component of Federated Learning pipelines that allows the server to aggregate the client updates without accessing them individually.

2. The key techniques they adopt to make SECAGG efficient and scalable include: scalar quantization, pruning, and Secure Indexing (SECIND, a variant of SECAGG to enable product quantization). 

3. They claim to achieve a 40x compression ratio in uplink communication on LEAF benchmarks without meaningful loss.

strength_and_weaknesses: strength

1. It aggressively saves the uplink communication bandwidth and the authors claim that there is no loss in security.


2. The key approach is to modify compression techniques to share hyperparameters globally across all clients so that aggregation can be done by uniformly combining responses from all clients. This seems to be portable to existing FL works without much implementation efforts.

weakness

1. The global sharing strategy, which is their key idea, seems to be quite straightforward and lacks novelty.
2. I guess the performance drop introduced by shared mask (pruning) and shared quantization parameters (quantization) is depending on the data variance across clients. I suppose this could potentially limit the usability of the work.
3. Compatibility with DP as they stated in section 6.

clarity,_quality,_novelty_and_reproducibility: 1. It would be great to be specific about what “meaningful loss” means in the abstract.
2. I am not an expert in this field, and from an non-expert point of view, it would be good to show the breakdown for FL workloads, demonstrating the uplink communication is indeed the system bottleneck.
3. The demonstration is based on a linear layer, and is this method adoptable on convolutional layers and other types of layers? If so, what is the difference in terms of accuracy, privacy and compression ratio?
4. Is aggregation overflow the only reason why scalar quantization is not compatible with SECAGG? If so, why must the server use the same bit width as the clients? Can the server use more bits to avoid overflow? I suppose another reason is the different quantization parameters are not compatible for aggregation, but this is not mentioned in the “Challenge” part of 3.2.1.
5. Is the global hyperparameter sharing strategy for compression a possible source for information leakage? If not, what’s the price for adopting your solution besides more download time (given the no-free-lunch principle)?
6. In Figure.1, DP is omitted for clarity. Just want to be clear, is g_i already calculated by data after adding the DP noise? Is the experimental results in Figure 2 considering DP?
7. Plan for open source (ignore if I missed the link in the paper) ?

summary_of_the_review: Overall I enjoyed reading this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper focuses on the problem of compressing the uplink model updates in a Federated Learning (FL) setup implementing secure aggregation protocols. It first  argues about the incompatibility of standard compression methods (scalar quantization, pruning, and product quantization) with secure aggregation protocols, and then it proposes a method to this incompatibility based on using common/shared information across all clients.  A "Secure Indexing" (SecInd) protocol has also been proposed for product quantization, which leverages the linearity of the computations. Experimental results using the LEAF data/platform are also provided. 

The reviewers had a number of concerns, some of which were addressed, but several major ones have remained. In particular, the reviewers still felt that the novelty of the work and the proposed methods should better be demonstrated. They also felt that more thorough investigation is needed to verify the effectiveness of the proposed methods especially in the non-iid setting (while they acknowledge that the results of figure 2 are obtained in the non-iid setting). Please see the (updated) reviews for more details. 

Overall, once the above concerns are addressed, the revised version of the paper could be a great addition to the literature of FL. 
justification_for_why_not_higher_score: the majority of the reviewers are in favor of rejecting the paper. 
justification_for_why_not_lower_score: -
Decision: Reject