Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces FedPara, a low-rank based method for achieving communication efficient federated learning. On the client-side, low rank factorized models are trained. Different from the traditional approaches to finding low rank factorized networks, FedPara introduces a novel low rank factorization strategy, which attains a higher maximal rank for the weight matrices. The paper also identifies that FedPara can be used to enhance personalized federated learning (by introducing a variant of FedPara, i.e., pFedPara). Both theoretical analysis and experimental results are provided to show the effectiveness of FedPara. 
main_review: ======================= Post Rebuttal ======================

After reading the authors' response. Part of my concerns are addressed. Thus
I would like to increase my overall evaluation to 6. And I believe that this work
can make a decent contribution to the efficient FL training area.

========================================================


The paper proposes to train low rank factorized model on the client side to improve the communication efficiency of federated learning while introducing a novel low rank factorization method. 

Pros:
1. The paper is well written, and the research direction of enhancing the communication efficiency of federated learning is potentially impactful. 
2. The proposed low rank factorization method is novel. 
3. Both theoretical analysis and empirical results are provided to justify the effectiveness of FedPara. 

Cons: 
1. The main concern I have for the current version of the draft is that the proposed FedPara method is only compared against the traditional low rank based model factorization method, e.g., vanilla low rank model or Pufferfish. It is not clear if FedPara is also better compared against other communication efficient federated learning methods, e.g., [1, 2] or recently proposed low rank based communication efficient FL method [3]. 
2. By looking at Algorithms 1 and 2, it is not clear which weights are kept on the client side and which weights are kept on the server side. It says “Optimize(W)”, does it mean only $X_1, Y_1$ and $X_2, Y_2$ are optimized? Also for aggregate $X_1, Y_1$, and $X_2, Y_2$, does it require to transform them back to W before aggregation across clients’ updates? The authors are supposed to clarify this. 
3. How to choose $\gamma$ is not clear, a fixed rank ratio seems to make sense. But it may also be possible that different layers can contain different redundancy. Is there a chance to adaptively choose $\gamma$ for different layers? 
4. It is recently observed that the model weights of the Transformer are not low rank. Does it mean FedPara can not be applied over the Transformer architectures? 

[1] https://arxiv.org/pdf/1610.05492.pdf

[2] http://proceedings.mlr.press/v108/reisizadeh20a/reisizadeh20a.pdf

[3] https://arxiv.org/pdf/2104.12416.pdf

[4] https://openreview.net/forum?id=_sSHg203jSu
summary_of_the_review: Overall I think the paper introduces a valid research idea on improving the communication efficiency of federated learning. However, several concerns exist (please refer to “Cons” in “Main Review”). If my major concerns are addressed, I will consider increasing my overall evaluation score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a communication-efficient parameterization for federated learning by using low-rank weights. Several numerical experiments are conducted to demonstrate the effectiveness of the proposed FedPara and pFedPara.
main_review: ======================= After Rebuttal ======================

I have read the authors' responses. Part of my concerns is alleviated. I would like to increase my overall evaluation to 6. 

========================================================

Strengths:
This paper is well organized and easy to follow. The authors provided comprehensive experiments.

Questions:
1. As for Figure 1, does the low-rank parameterization $W=XY^{\rm T}$ always leads to the Hadamard product of two low-rank inner matrices $W=W_1\circ W_2=(X_1Y_1^{\rm T})\circ(X_2Y_2^{\rm T})$? What if the low-rank weight matrix $W$ cannot be decomposed into the Hadamard product of two low-rank inner matrices?
2. Proposition 1 only provides the upper bound for the rank(W). It does not necessarily mean that $rank(W)=r_1r_2$. How could the constructed matrix always span a full-rank matrix when $r_1r_2\ge \min(m,n)$?
3. In the numerical results with varying $\gamma\in[0.1,0.9]$, what is the corresponding number of parameters?
4. The accuracy differences between the low-rank method and FedPara in the non-IID case are smaller than the differences in the IID case. Is that any explanation for this phenomenon?
5. It would be better to move the details of proof to the appendix and add more discussions in the main manuscript.
summary_of_the_review: The paper is mostly sound, while the contributions are marginally significant.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a communication-efficient parameterization methodology for federated learning tasks through the Hadamard product of low-rank weights. Such parameterization has better model expressiveness in terms of minimal parameters to achieve a maximal rank, compared with conventional low-rank approaches. Using such parameterization, the work designs the corresponding communication-efficient federated learning framework which only uploads a low-rank part of parameters of each level to the server. The work presents empirical studies of the proposed algorithm and validates its effectiveness in terms of accuracy vs communication costs.
main_review: The paper proposes a low-rank parameterization method to help improve the communication efficiency of federated learning. The work provides theoretical justification of the effectiveness in terms of the maximal rank the parameterization can express. The work shows extensive empirical studies of the proposed method on multiple FL datasets. The results indicate the superiority of the proposed method in terms of model accuracy and communication cost.

Questions and comments:
1. In terms of expressiveness, what is the key limitation of the Hadamard product parameterization compared with general unconstrained parameterization?
2. The paper does not discuss any related work about the Hadamard product parameterization. Is this a new idea? Or what is the difference compared to other literature involving Hadamard product parameterization?
3. In terms of computational cost, how does training the Hadamard product parameterization compare to previous low-rank approaches?
summary_of_the_review: The work proposes an effective low-rank parameterization that shows the superiority of communication efficiency in federated learning tasks. The idea is relatively interesting to the reviewer. The overall presentation of the idea and empirical results is clear.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces FedPara, which is a low-rank parametrization method for the neural networks aiming to reduce the total number of communication bits while preserving the model accuracy in the federated learning scenario. Besides FedPara, which is designed for federated learning applications, the paper also generalizes FedPara into pFedPara, which is designed for personalized federated learning.

The novelty of this paper includes using the Hadamard product in the low-rank approximation. Compared with directly using low-rank approximation, using Hadamard produce can increase the expressiveness of the parametrization.

The paper also includes empirical results to show the effectiveness of FedPara/pFedPara in the federated learning setting.
main_review: I am not very familiar with reparametrization, and I am not able to comment on the significance of this paper.

I think that the idea of this paper is interesting, using the Hadamard product of two low-rank matrices may only increase the model size by a factor of 2, but may drastically increase the expressiveness of the parametrization.

As for the experiments, I think the experiment design is generally enough. The experiment compares FedPara with original training methods with no reparametrization, and low-rank reparametrization without the Hadamard product on different machine learning models, e.g. VGG and LSTM.

Below are some suggestions and questions:
1. In the experiment, the authors compare FedPara with original low-rank reparametrization and without reparametrization, and the optimization algorithms are chosen to be the same, e.g. FedAvg, SCAFFOLD, etc. I wonder what are the results between FedPara and gradient compression algorithms, for example, quantization. Or if the result can become better if using gradient compression schemes in FedPara instead of local steps. It may also be good to add some experiment results with compression.

2. FedPara converges much faster than the original models in terms of the communication bits. How about the final accuracy? What is the tradeoff between communication efficiency and the testing accuracy?

3. Although the low-rank constraints reduce the number of communication bits during the training procedures, it may increase the local computational cost since the gradient of the new model may become more complicated. In your experiment, what is the local training cost, e.g. the local training time between two communications, or the number of flops for one backpropagation?
summary_of_the_review: I am not very familiar with reparametrization and its corresponding related works, and I am not able to comment on the significance. However, I think that the technique used in this paper is interesting and simple, which can be applied to real applications. The experiments shown in the paper are enough. Some questions and small suggestions are listed in the main review part.
review_title: null

Meta-review: Dear Authors,

The response you have provided, based on the main concerns of reviewers, have answered most of the questions raised. 
As far as I understand from the added experiments you have provided, the proposed methodology shows resilience in being at least as good as state of the art approaches, while at the same time it is a mathematically interesting approach. 

Your response has covered concerns like comparison to other communication techniques (the comparison list is not complete, but yet your effort is appreciated), adding discussion on the rank parameter, add comments on expressiveness and the connection with low-rank parameterization, etc. 

These efforts cannot be overlooked, and for that reason I suggest acceptance (poster).

Best

AC
Decision: Accept (Poster)