Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the training dynamics in federated deep learning from the perspective of mini-batch SGD. Client coherence and global weight shrinking regularization are introduced in the proposed FedAWO framework. The corresponding parameters are learned in the server side with a proxy dataset. FedAWO’s effectiveness is demonstrated empirically across several datasets and model architectures. Interestingly, FedAWO shows the ability to filter out corrupted clients.
strength_and_weaknesses: Overall, this paper is well written and clearly presented. 

The core components (client coherence and global weight shrinking regularization) of the proposed framework are carefully analyzed independently to each other before presenting the resulting FedAWO. 

Related work on federated learning is partially missing; it is necessary to move key related work to the main text. As a result, the paper is not appropriately positioned in literature in a certain sense: Is it possible to introduce more aggregation parameters on the server side in a similar adaptive manner, except for the proposed $\gamma$ and $\lambda$? Are there any previous work adopting a similar adaptive aggregation approach?

The concept of “number of local epochs” is confusing. Intuitively, it is about how $g^t_i$ is derived locally for each client. However, since the preliminary about FL is not introduced in detail, it is not sufficiently readable.
clarity,_quality,_novelty_and_reproducibility: Overall, this paper is well written and clearly presented. 

More materials should be provided to help evaluate novelty (see Strength And Weaknesses).

The code should be made public to validate reproducibility owing to the fact that implementation details are missing in the text.
summary_of_the_review: See “Strength And Weaknesses” and “Clarity, Quality, Novelty And Reproducibility”.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of optimizing for aggregation weights assigned to client model updates along with weight decay so as to aid generalization. In particular, the authors take the viewpoint of seeing the aggregation and server side update analogous to that of mini-batch SGD in centralized training and propose an algorithm named FedAWO. In particular, the proposed algorithm optimizes for weight decay and aggregation weights by making use of a proxy dataset at the server. Experiments on different datasets and models demonstrate the efficacy of the approach.
strength_and_weaknesses: Strengths:
+ The paper is well written and the problem under consideration is well motivated. The proposed algorithm is light weight, i.e., doesn't add any computational overhead for the devices but instead adds it to the server which makes the algorithm attractive to use. The algorithm is easy to use and doesn't add any additional hyperparameters.
+ The authors provide various new insights in terms of the effects of gradient coherence and heterogeneous coherence and its interplay with global weight shrinking and critical points to better explain the training dynamics in FL. 
+ Experimental results are comprehensive in terms of different datasets, models, different levels of heterogeneity both in terms of non-iidness and the number of samples on clients.

Weaknesses:
- It seems success of the approach is tied to the proxy dataset being a representative sample of the entire data distribution. No matter how small or how large the proxy dataset is, the server gets access to the true data distribution of all the clients. This is a major limitation of the approach. Having a proxy dataset at the server is something entirely possible, but the distribution of it being represenative of the true data distribution is a strong assumption. It is not clear how the performance of the algorithm is affected, when the proxy dataset is not distributionally similar to the data distribution of all the clients as a whole. It would especially strengthen the paper, as to how much of the benefit is attributable to the distribution of the proxy dataset and how much of it is from adaptively optimizing for $\gamma$ and $\lambda_{i}$'s.
- The proposed approach is devoid of any theoretical guarantees and is based on observations based on SGD being the local solver. In particular, the observations concerning gradient coherence wouldn't carry over if each client undergoes multiple local steps. 
- The experiments are missing some details regarding the number of local epochs considered for different baselines and the performance of FedAWO in settings involving multiple local steps.  
clarity,_quality,_novelty_and_reproducibility: The paper is very well written and clear to understand. The paper brings forth a lot of new insights in terms of the training dynamics of FL. However, a lot of benefits of the proposed algorithm seems to come from the usage of proxy data at the server which has been used before in other papers.
summary_of_the_review: The paper solves a well motivated problem with a simple, practical and well examined solution. The proposed algorithms introduces no additional hyperparameters which makes the solution particularly effective. However, the benefits claimed seems to be tightly tied to the similarity of the proxy dataset with that of the sum total data distribution of the clients. The paper could be strengthened by studying the robustness of the approach in terms of benefits still intact even when the data at the server is distributionally different from the data distribution of the sum total of all clients.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The author proposes to optimize the aggregation weight of each client, and find that the aggregation weight is closely related to gradient coherence. The proposed method improves the heterogeneity coherence. The author also proposes global weight shrinking to improve the training performance. Moreover, the author makes many definitions of new concepts and try to explain how the training dynamic are affected in FL. Empirical experiments show improvement over existing methods.
strength_and_weaknesses: Strength:
1. The paper is well-organized and easy to follow.
2. The empirical improvement is significant.

Weaknesses/questiones:
1. How is the proxy dataset selected? Proxy dataset does not seem to be a common practice in most FL settings, and it is not fair to compare it with other methods not using proxy dataset.
2. How is Eq(3) and Eq(4) optimized? The default aggregation weight in FL is used in Eq(4), instead of the proposed $\lambda$. Why not jointly optimize $\gamma$ and $\lambda$?
3. In Eq(2), why is the first order Tyler expansion $-\eta\langle g^t,g^t\rangle$? Shouldn't it be $-\eta\langle g^t, \nabla \mathcal{L}(w^t)\rangle$?
4. After Eq(2), why do you assume all gradients have the same norm? This does not seem to be realistic.
5. Regarding cosine similarity/gradient coherence, it is not a new concept. For example, check [1] (there are also many other works). Moreover, it is actually the core problem in FL studied by many prior works as it contributes to the client drift issue.
6. The global weight shrinking does not seem novel to me. It is applying weight decay to the pseudo-gradient (local update of a client) and I do not think it is new.
7. How do you compare GWS with server learning rate both theoretically and empirically?
8. How do you compare methods employing server optimizers, which apply SGD/Adam variants to pseudo-gradient/local update from clients, both theoretically and empirically?
9. Optimizing aggregation weights is not new. There has been more advanced methods to deal with it, e.g. reinforcement learning [2,3]. While the method proposed is very simple by using a proxy dataset.

References:
[1] Yin, Dong, et al. "Gradient diversity: a key ingredient for scalable distributed learning." International Conference on Artificial Intelligence and Statistics. PMLR, 2018.
[2] Auto-FedRL: Federated Hyperparameter Optimization for Multi-institutional Medical Image Segmentation. ECCV 2022.
[3] Robust federated learning through representation matching and adaptive hyper-parameters. arXiv preprint arXiv:1912.13075 (2019)
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. However, I think most of the findings are not new or surprising for a reader with experience in FL.
summary_of_the_review: The author made a lot of new definitions and propose to optimize the aggregation weight and gradient weight shrinking. However, the use of proxy dataset make it less practically appealing, and the proposed techniques do not seem novel. Most of the findings are not new to me, too.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper investigates the training dynamics in federated deep learning from the perspective of mini-batch SGD. Client coherence and global weight shrinking regularization are introduced in the proposed FedAWO framework. The corresponding parameters are learned in the server side with a proxy dataset. FedAWO’s effectiveness is demonstrated empirically across several datasets and model architectures. Interestingly, FedAWO shows the ability to filter out corrupted clients.

Issues include: 

1. As an expert in FL, I find it difficult to appreciate the research question posed by the authors. I believe it is fundamentally misguided - the ideas look ad hoc, and are a combination of ideas that already appeared in the literature (often without citing the original or prior sources). In summary, I do not see much novelty here. 

2. Further, I do not believe that local SGD should be studied using the lens of minibatch SGD; the methods are fundementaly different. Recent results studying the nature of local SGD theoretically have shown that local steps can be seen as a communication acceleration mechanism (without any homogeneity assumptions). This is something that minibatch SGD cannot do. Similarly, recent results from a different line of works have shown that local steps in SGD can be provably linked to generalization (with some structured homogeneity/similarity assumptions). This is unrelated to the phenomena the authors study here. In short, I do not believe the questions the authors are asking are worth asking. Moreover, they are not sufficiently well justified. Relevant recent literature which gives deep insights into what local SGD does is largely ignored by the authors. The mixture of FL related phenomena the authors study seems very ad-hoc from the point of view of were the literature on local SGD stands now. Large swaths of very relevant literature are omitted.

3. The paper contains no theoretical results. This is not required, of course, but empirical works should come with a particularly strong empirical component, which I do not see here.

4. It seems success of the approach is tied to the proxy dataset being a representative sample of the entire data distribution. This was criticized by some reviewers, and I concur. 

5. I have read the positive reviews, and I am unconvinced by the strength of the arguments used to praise the paper. That is, I see a disconnect between the scores and the wording used to justify it.

In summary, I believe the work does not raise to the bar required of an ICLR paper.

AC

justification_for_why_not_higher_score: I believe the work does not raise to the bar required of an ICLR paper. Explanation included in my metareview.
Decision: Reject