Venue: NeurIPS 2022 Workshop Federated_Learning

Reviews:

final_score: rating: null
review_content: This paper visualizes the loss landscape and training trajectory of federated learning. Experiments on CIFAR-10 showing the 2D counter of loss landscape, projected trajectory for FedAvg under different non-IID partitioning, and local updates on different clients. 

It is interesting to see such a study. I have a few suggestions,
(1) The major concern is that the experiments are only on CIFAR-10, which is a small dataset with artificial non-IID created by Dirichlet sampling of labels. I would encourage authors to consider realistic dataset with natural heterogeneity. 
(2) The insights that heterogeneity causes different final solutions have been extensively studied in the literature. I would be interested to know if there are new insights from visualization? 
(3) As visualization is an approximation, I am not sure if the trajectory is caused by heterogeneity, or other factors. For example, the IID curve is far from the non-IID curves, but I suspect we could get similar observation if we vary other hyperparameters like learning rate, number of local epochs etc. 

review_title: Interesting research that can benefit from more realistic settings 

final_score: rating: null
review_content: 
1. Interesting paper. The authors could study the loss landscape when the neural is trained on other popular datasets (such as CIFAR-100, FMNIST, and MiniImageNet) as well as different architectures.
2. More details on the experimental section should be provided in the appendix.
3. Other aspects including the effect of the local epoch, batch size, training rounds, total number of clients, etc. could be studied which makes the paper more comprehensive.



review_title: A pilot study on how statistical data heterogeneity and model pre-training in FL can impact loss landscape and the training trajectory.

final_score: rating: null
review_content: The paper conducts a qualitative study of the convergence properties of FL in various regimes by visualizing 2D projections of loss landscapes. The main regimes used are IID vs. non-IID data, pre-training vs. random initialization, and all clients vs. client sampling.

Quality:
There are numerous typos and grammatical problems throughout the paper.

Clarity:
The paper's objective is simple and clear.

Originality:
To my knowledge there have not previously been extensive visualizations of the loss landscapes in FL for non-IID settings.

Significance:
The results of this paper are as one would expect, with little novel information gained. The visualizations confirm results that have already been established quantitatively. For instance, visualizing that non-IID data disributions across clients make convergence more difficult does not lead to any novel conclusions.

Pros:
 - Visualization can help provide intuition in complicated scenarios.
 - The conclusions drawn are consistent with prior quantitative work.
 - The regimes studied are the most important regimes in FL.

Cons:
 - The intuitions shown in this paper via visualization are already well known in the community.
 - It is unclear if we can extrapolate observations made from visualizations, and it is difficult to know when intuition can be relied on vs. when it will fail.
 - Using random 2D projections of the loss landscape may lead to conclusions that do not hold in the full dimensional space.
 - Linear interpolation between models to determine the existence of a barrier neglects the non-linear, non-convex shape of the loss function. Some form of geodesic path that minimizes the weighted loss along the path would be a better measure.

I recommend that the authors supplement their qualitative conclusions with more quantitative conclusions, i.e what are the final accuracy and loss metrics obtained by the variants depicted in each plot.

review_title: Visualizing loss landscapes in FL

Decision: Accept (Poster)