Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors have considered the problem of designing weights for federated learning problem via bilevel optimization. The problem is interesting and analyzed theoretically with extensive experiments. But there are certain limitations as provided next. 
strength_and_weaknesses: *Strength*
The problem is interesting, paper is well written withe extensive experiments. 

*Weakness*

- The contribution is limited and incremental. 
- Once the problem is formulated in the form of P*, I cannot see any additional challenge to solve it. There are multiple algorithms already existing in the literature which can directly solve this problem. 
- It is limited to consider the lower level problem as strongly convex, because even the simplest FL problem where we train NN would be non-convex. For instance, all the experiments are definitely considering non-convex lower level, therefore theoretical results are not consistent with the experiments. 
- Assuming the lower level to be strongly convex makes the setting quite straightforward to analyze and provide guarantees. It would be nice to specifically discuss the additional challenges, theoretically, authors had to address to do the analysis.
-  
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, novelty is limited. 
summary_of_the_review: The problem is interesting, but the contribution is incremental.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a node reweighting strategy for federated learning based on bilevel optimization. Specifically, this the proposed adaptive reweight method learns the weight assigned to each node by considering the “weighted aggregation” on the server as the inner problem, and minimizes the empirical risk on an in-distribution validation dataset. Theoretical results are given to verify the effectiveness of the bilevel method.
strength_and_weaknesses: Strength:

The paper seems to be the first one to formulate node reweighting for FL into a bi-level optimization problem, and theoretical results are given to support the superiority of training a model locally and to federated learning with static and evenly distributed weights.

Weaknesses:

1. It is not surprising that FL with properly learnt reweighting performs better than local training and evenly distributed weights, which is the main theoretical contribution of this paper.
2. The proposed bi-level reweighting method does not seem to be applicable in practice. The algorithm needs S X T global communication rounds, just to acquire the properly learnt node reweighting. This is obviously not acceptable, since global communication is often the main bottleneck in the application of FL. In addition, in the inner loop of Local-SVRG, the model seems to be re-initialized every time, this does not seem to match the real scenario, since we cannot ask the clients to re-initialize their models now and then.
3. It seems that the weights are assigned to each specific client node, will it be problematic when the participation ratio is not 1 when running FL? This is often the case in practice, and this paper does not discuss about it.
4. The bilevel method needs an in-distribution validation set,  and the size of such a dataset will affect the performance. An ablation study is required to clarify this.
5. The algorithm is built based on Local-SVRG, which should serve as a baseline method in the experiment, so that we can see how much performance gain the reweighting actually brings.


clarity,_quality,_novelty_and_reproducibility: The paper is well written, with good clarity.
Even though they are the first to adopt a bi-level formulation for FL, this approach is quite straightforward and does not present enough novelty. Besides, some major limitations of bilevel formulations, such as expensive inner-loop evaluation, are not addressed in this paper. The increase in global communication rounds also introduces more difficulty in applying this method.

summary_of_the_review: This paper has certain contribution, but I do not think it is enough for getting accepted by ICLR. Some major limitations need to be addressed for the applicability of the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on Federated learning (FL) with heterogeneous data distributions. In this case, a globally shared model may not achieve a good local generalization performance. So, the author put forward a method that each node can still exploit global data through FL but, at the same time identify and collaborate only with the nodes whose data distributions are similar or identical to its local distribution. Furthermore, the author mentioned, it is also a bilevel optimization (BO) problem where the inner problem is federated learning with weighted nodes, and the outer problem optimizes the weights. In addition, the theorem also shows the bound on the statistical distance, dominated by the $O(1/\sqrt{N_{valid}})$, which is the same as the generalization bound achieved by leaning locally. Finally, the algorithm in this article is well-consistent with the analysis of BO since it solves the inner problem before the outer problem. As a result, the algorithm performs better than the baseline when the data is heterogeneously distributed where higher test accuracy can be reached. 
strength_and_weaknesses: Pros:

1. Motivation of this paper is reasonable to me and the weighted node formulation seems new in the literature of federated learning although it is not new in other multi-task or device applications such as meta-learning or life-long learning (Chen et al., 2021). In addition, the bilevel optimization based formulation provides a principled way to improve the generalization. 

2. Generalization performance of the proposed weighted node is provided, with several modifications of the analysis in Chen et al., 2021. Some new characterizations such as identical neighbors and error bound conditions are also included. 

3. Convergence analysis is also provided under local-svrg and some accelerated bilevel methods as in Ghadimi & Wang, 2018. 

Cons:

1. There are too many notations and it is a little hard to find the definition among words. I recommend having a brief summary of all notations, which is helpful for reading both the article and the proof. 

2. The discussion of communication rounds is not conspicuous although the author strengthened the algorithm is communication-efficient as this is important in FL. I recommend having a table or a figure to show the communication rounds of different algorithms so that readers can intuitively get this. 

3. For the generalization analysis, Assumption 2 is strong although some simple examples and Assumption 2’ are provided. This assumption requires $p_0$ is identical to at least one distribution in $p_1,…,p_K$. First, for the practical consideration,  the training-validation separation trick does not hold here, and the generalization may be affected as well. In addition, in terms of the technical novelties, the improved analysis by the error bound condition and the  identical neighbors (Assumption 2’) seems to be not that significant given the framework in Chen et al., 2021, and no tightness results are provided. 

4. The algorithm seems not to be new enough. For example, it seems to be a combination of local-SVRG, accelerated bilevel update in  Ghadimi & Wang 2018, and FedAvg types of aggregations. The algorithm also looks complicated with many hyperparameters, and its application may have some difficulty in more complex ML applications. I wonder if the acceleration step truly provides practical acceleration? Could you please include such an illustration in the experiments? 

5. Some important related works are missing:

Distributed bilevel optimization: 

1. Ataee Tarzanagh, Davoud, et al. "FEDNEST: Federated Bilevel, Minimax, and Compositional Optimization." arXiv e-prints (2022): arXiv-2205.
2. Chen, Xuxing, Minhui Huang, and Shiqian Ma. "Decentralized bilevel optimization." arXiv preprint arXiv:2206.05670 (2022).

3. Li, Junyi, Feihu Huang, and Heng Huang. "Local Stochastic Bilevel Optimization with Momentum-Based Variance Reduction." arXiv preprint arXiv:2205.01608 (2022).

Bilevel optimization and analysis:

4. Ji, Kaiyi, Junjie Yang, and Yingbin Liang. "Bilevel optimization: Convergence analysis and enhanced design." International Conference on Machine Learning. PMLR, 2021.

5. Grazzi, Riccardo, et al. "On the iteration complexity of hypergradient computation." International Conference on Machine Learning. PMLR, 2020.
clarity,_quality,_novelty_and_reproducibility: The quality, clarity and originality of the work need to be improved. 
summary_of_the_review: Overall, this paper provides an interesting bilevel formulation, which is shown to improve the local generalization performance. The analysis involves several new treatments. However, I have some concerns regarding analysis, related work and writing. I think this is a borderline work, and my evaluation will be adjusted based on the other reviewers’ comments and the authors’ response.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a bilevel optimization-based federated learning method that can assign different nodes different weights to optimize the performance on a validation dataset. The authors present thorough theoretical analysis of the problem and verify that learning dynamic weights is better than using evenly distributed weights. Experimental analysis also shows the superiority of the proposed method over several baseline methods.
strength_and_weaknesses: Strengths:
- The proposed method is interesting.
- The theoretical analysis provides additional guarantees.
- The results show the effectiveness of the proposed method.

Weaknesses:
- The bilevel optimization highly depends on the size, quality, and distribution of the validation dataset. In many real-world scenarios, high-quality validation data is not available on local devices (e.g., personal computers or smartphones).
- Since this paper aims to bolster local model performance, it is required to compare several more baseline methods for personalized federated learning. 
- The experiments are conducted only on image classification datasets. I suggest the authors consider using some recommendation datasets (e.g., Movielens) since the user data in these datasets are naturally heterogeneous (and more realistic than simulated data).
- The real communication costs of the proposed method and existing methods are not reported. It is suggested to include the results to confirm that the proposed method is "communication-efficient".
clarity,_quality,_novelty_and_reproducibility: This paper has good quality, clarity, and contributions in both theoretical and empirical aspects.
summary_of_the_review: This paper has good contributions and the paper is clearly written, but the experiments are not very solid and certain claims are not fully verified. Thus, my recommendation is a weak accept.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a bilevel optimization-based federated learning method that can assign different nodes different weights to optimize the performance on a validation dataset. The authors present thorough theoretical analysis of the problem and verify that learning dynamic weights is better than using evenly distributed weights. Experimental analysis also shows the superiority of the proposed method over several baseline methods.

There remains concerns on the proposed approach being straightforward, and the contribution / novelty may not be significant enough for ICLR. We thus unfortunately recommend rejection.
justification_for_why_not_higher_score: The contribution and novel are not sufficient.
Decision: Reject