Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper makes an interesting observation in learning from heterogeneous data sources by empirically illustrating that representations tend to reside in a lower-dimensional space.  To overcome this issue, authors propose FEDDECORR algorithm that encourages the Frobenius norm of the correlation matrix of representations at each client (penultimate layer of a neural network) to be small.

strength_and_weaknesses: The paper makes an interesting observation and has some merits, but 

On the empirical side, 

1. There is a discrepancy between experiments the observation made from and proposed algorithmic solution: the way the heterogeneity is introduced among clients to infer empirical observation include discrepancy among both covariates and labels, while the proposed algorithm only tackles representations by regularizing  covariance matrix of the representations. This left me wondering about the contribution of each source of heterogeneity on dimensional collapse. 

2. Also, in the results reported in Table 4, it seems as the number of local updates increases, the proposed algorithm does not show significant improvement over baseline (FedAvg). Since in federated settings the number of local updates could be much larger than 20, it seems the proposed regularization becomes less beneficial (it is conjectured by authors that the dimensional collapse of the global model stems from local models). 

3. Finally, it would be nice to see how the observed  phenomena manifest itself in other algorithms where a drift/diversity mitigation schema is employed. This could potentially better illustrate the role of heterogeneity in dimensional collapse and the effectiveness of the proposed method. 

On the theoretical side, the analysis is limited to linear neural networks, does not take into account the local updating and aggregation schema, and more importantly does not clearly illustrate how covariance regularization can improve accuracy as it analyzes the gradient flow of optimization which does not necessarily translate to a good generalization in the presence of data heterogeneity.

clarity,_quality,_novelty_and_reproducibility: The paper is overall well-written and enough details about experiments in main body and appendix are included for reproducibility purpose.
summary_of_the_review: The paper makes an interesting observation and algorithmic solution, but the theoretical results and empirical results need to be strengthened to resolve  issues discussed above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper considers the federated learning problem with data heterogeneity. From experiments, the authors find an interesting phenomenon, dimensional collapse. The global model suffers from dimensional collapse in the feature space such that the covariance matrix of the representations of the global model becomes approximately low-rank. The model collapse also happens in local models. The authors claim that the model collapse is one of the main reasons that degrade the FL performance. To remedy this problem, they propose FedDecorr, which applies a Frobenius norm regularization term during local training, making the distributions of singular values more uniform. FedDecorr outperforms baselines on standard benchmark datasets.

strength_and_weaknesses: Strengths

- This paper is easy to follow.

- The proposed algorithm is well-motivated and performed very well.

- The dimensional collapse is an interesting observation.

Weakness

- More recent FL algorithms should be discussed and compared. 

- More experiments with other data sets and neural net architectures might be required.
clarity,_quality,_novelty_and_reproducibility: The observation and the proposed algorithm are novel and clear. However, the source code is not provided.

summary_of_the_review: This paper proposes a simple yet efficient algorithm for FL with a good justification.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper identifies a key bottleneck in highly heterogeneous federated systems, namely dimensional collapse. The authors propose a simple but novel technique for mitigating dimensional collapse by adding a regularization term during local training. The paper shows both theoretical and empirical analysis to support their results.
strength_and_weaknesses: The paper identifies a key issue in data heterogeneous federated learning. The overview of dimensional collapse in FL is a significant stepping stone for future research. The proposed algorithm is simple yet effective. The paper also shows theoretical and experimental results which makes their arguments strong. I do have some suggestions.

1. I believe it would be a good to see how the algorithm behaves when we vary the client sampling rate in addition to the number of clients.
2. Add a convergence analysis for the algorithm if tractable.
3. An ablation study studying how the added regression evolves over time is important.
4. Add more tasks besides image classification.
clarity,_quality,_novelty_and_reproducibility: The paper is clear in its arguments and is of good quality. The approach to the best of my knowledge is novel. I have several questions, however, regarding the experimental results.

1. In table 4 on Page 9, the results for different client sampling rates are shown. Can the authors explain why performance improves when the local epoch is increased or cite other sources as counter examples? As per my understanding, increasing local epochs increases client drift and should degrade performance.[1]
2. Why does correlation matrix work better than covariance matrix?  
3. How stable is the added regression term? 
4. Can the proposed algorithm work for other tasks like natural language processing, eg masked language modeling?


[1] Wang, Jianyu, et al. "A field guide to federated optimization." arXiv preprint arXiv:2107.06917 (2021).
summary_of_the_review: The paper points out a key problem in highly heterogeneous federated systems and proposes a novel approach to solving it. The study is a good direction for future research as well. The authors back up their arguments with both rigorous theoretical and experimental results on image classification tasks. I do have reservations regarding the stability of the algorithm and its effectiveness in other domains. Nevertheless, the results are more or less rigorous and show it is an effective technique.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies dimension collapse in federated learning with label distribution shift, and methods on more to mitigate it. With the simple observation that the singular values of features are decaying fast for heterogeneous cases, the authors propose to minimize the corresponding variance, or equivalently, the Frobenius norm of the correlation matrix. This F-norm regularization has been tested on vision datasets with several baseline comparisons.

--post rebuttal--

Thanks to the authors for showing extra experiments on new data splits and datasets as well as comparison with new baselines. From the discussion, this paper is primarily experimental, and there is not much theoretical analysis on the explanation of $\beta$, or which algorithm  is the best when combined with DeCorr. These are some future directions. 
strength_and_weaknesses: Strengths:
1) Data heterogeneity is common in FL and this paper studies this problem from the perspective of singular values of features. This perspective is interesting and has not been studied before. The paper is well-motivated.
2) The proposed method is easy to understand, by equalizing the variance of singular values of the covariance matrix, with the Frobenius norm of the correlation matrix, the authors propose to use the F-norm as a regularization method. This is easy to implement and can be adapted to most existing FL methods, such as FedAvg, FedProx, MOON, etc.
3) This method has been tested on a variety of vision datasets, with different levels of data heterogeneity.

Weaknesses:
Although this paper proposes an interesting idea for mitigating the label shift problem, I still feel it needs some improvement.

1) Sec 3.3 seems unnecessary and is detached from the main paper. The linear network assumption seems too restrictive and does not provide us with new insights. One can simply obtain eq. (6) from the observation in Figure 2, and there is no need to use gradient flow dynamics.
2) Only Dirichlet partition has been considered. More testing needs to be done on other splits. Also, realistic partitions should be tested, such as FEMNIST, Shakespeare and StackOverflow.
3) The current method seem more like heuristics and it needs more understanding. For example, why is $\beta = 0.1$ more desirable? Why increasing $\beta$ can sometimes decrease the performance? 
4) About baseline comparison, the authors should add more competitive baselines such as FedAdam [1] and FedLC [2] that aims to address the same data heterogeneity problem. 


[1] Reddi, Sashank J., et al. "Adaptive Federated Optimization." International Conference on Learning Representations. 2020.
[2] Zhang, Jie, et al. "Federated Learning with Label Distribution Skew via Logits Calibration." International Conference on Machine Learning. PMLR, 2022.

clarity,_quality,_novelty_and_reproducibility: Overall, this is an interesting experimental paper that uses simple observation of dimension collapse to propose new algorithms for mitigating data heterogeneity in FL. This paper is well-written with many supporting baselines and 3 vision datasets. It is well written but the code is not shared. Could the authors share the code in the supplementary, if possible? In this way people can confirm the reproducibility. 

The idea is novel compared to existing frameworks, but it still needs more theoretical understanding.
summary_of_the_review: In summary I would recommend weak accept as this paper identifies the problem of dimension collapse in FL and proposed a simple yet effective method to solve it. There are still some remaining issues with this method, such as baseline comparison, and deeper understanding beyond linear dynamics. The authors may start by proving faster convergence rate or better generalization error for FedDecorr, which could further improve the theoretical understanding. Also, it remains unclear to me why for TinyImageNet with $\alpha = \infty$ FedDecorr still has moderate improvement. Moreover, without FedDecorr, all the baseline algorithms do not beat FedAvg on CIFAR-100 and TinyImageNet by a large margin (1%), which raises questions for the correctness of the implementation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies federated learning on the realistic case of heterogeneous data, where client drift happens. In this setting of drift, it identifies undesired representation collapse into low-dimensional subspaces, and proposes to alleviate this with an algorithm based on regularizing representations for small correlations. Reviewers have consensus that they appreciate the interesting observations and simple new algorithm, while some questions remained if it would generalized to non-image data modalities too.

We hope the authors will incorporate the several points mentioned by the reviewers in the final version. 

As another small comment I would also suggest to discuss on the phenomenon not happening for very small numbers of local updates (Table 4), as if only a single local batch is used then there will be no drift at all. Currently it is not super clear from which level of drift the phenomenon starts appearing (it is shown that it does after a full local epoch though)
justification_for_why_not_higher_score: We're mostly positive but theory or cross-modality would need to be improved for reaching oral-level contribution.
justification_for_why_not_lower_score: No major concerns remained after feedback
Decision: Accept: poster