Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: his paper proposes a federated learning method to address a specific non-IID challenge over clients, i.e., when some clients are only available during the daytime and others are only available at the night, and these two types of clients' data are drawn from a mixture of two corresponding distributions. They study a multi-branch model, similar to multi-task learning, such that all clients share the same backbone network to extract data representations but a different branch is applied on top of the backbone network for different distributions. Only the clients with the same data distribution use the same branch to produce the final prediction. Under the assumption that the data representation produced by the backbone network follows a mixture of isotropic Gaussian distributions, this paper proposes an EM algorithm to assign clients to different clusters: the E-step determines the (soft) membership of each client belonging to each cluster and assigns it to the branches, while the M-step estimates the mean and variance of the representation for each Gaussian component in the mixture. Some additional heuristics are shown to be important for this method to achieve promising performance, e.g., label smoothing, a temporal prior of the ratio of clients from day and night, linear/cosine/soft schedule of this ratio, one-hot (hard) assignment of clients to clusters, etc. In experiments, they simulate the day-night distribution shift on EMNIST, CIFAR, and Stackflow datasets in FL by splitting each dataset into two parts with different data types. They evaluate the proposed method under different shifting settings (different p) with a few baselines proposed in this paper. They also empirically studied the effects of label smoothing and different period lengths.
main_review: Detailed comments:

(1) The contribution is incremental and lacks sufficient novelty. The multi-branch idea has been fully explored in both multi-task FL and heterogenous/personalized FL. The idea of client clustering has also been studied by several recent works in FL. The proposed solution, i.e., the EM algorithm, is also common for client clustering. These works should be discussed and compared in the paper. Label-smoothing seems helpful here but is also not new. 

(2) The newly proposed heuristics, e.g., temporal prior, different shifting schedules (cosine, linear, soft), one-hot cluster assignment, etc., lack rigorous justification. They are too hand-waving. It is not clear why they can be directly integrated into the EM algorithm without hurting its convergence or causing any inconsistency. Moreover, the performance seems to be sensitive to the application of these heuristics. 

(3) The datasets/benchmark evaluated in the paper do not have real periodic shifting distributions. The periodic shifting is artificially added but it is not clear why the realistic shifting in practice should follow one of the three schedules and p. 

(4) In experiments, when generating the dataset with periodic shifting distributions, q(t) is used to define the periods. However, the algorithm cannot have access to the ground truth q(t) defining the shifting schedule in practice: the application of $\tilde q(t)$ looks like cheating.

(5) There exist many FL methods specifically designed for addressing non-IID distributions across clients and better-personalized models for different clients. However, none of them has been compared in the experiments. "min-loss" is a baseline modified by this paper. Hence, it is hard to justify the advantage of the proposed method.

(6) Fig. 1 is hard to interpret: why do the curves for models trained only during daytime and the ones trained only during nighttime reach peak accuracy at the same time?

(7) The assumption that day-time (night-time) data representation follows a single-mode isotropic Gaussian distribution is too strong and may not hold in practical data. 
summary_of_the_review: The problem of FL with periodically shifting distributions is well-motivated by practical needs. The idea of tackling the non-IID problem in this case by assigning different clients to different branches also makes sense. However, this paper then reduces the problem to FL with client clustering, which is a crowded topic with several similar methods published recently. Given these works, the EM algorithm in this paper is not novel. Moreover, there are a great number of works in the FL community studying the statistical heterogeneity problem and they are all likely to be directly applied here since the day-night shifting is a special case of it. However, the experimental comparison ignores all of them and only compares to baselines developed(modified) in this paper, which fails to provide convincing evidence of the advantage of the proposed method over existing heterogeneous/personalized FL methods. No real day-night periodic shifting exists in any of the three datasets. Knowing the shifting parameter p is somehow cheating. The proposed heuristics are not sufficiently justified whether they are compatible with the EM algorithm and the Gaussian mixture model. Hence, I think there are currently many non-trivial problems that need to be fixed for this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper handles the distributional shift observed in data in clients in a federated learning production setting. In particular, the distributional shift is modeled as a mixture of distributions.Furthermore, a multi-branch network is used to encapsulate the shifting distribution and a Federated Expectation-Maximization algorithm enhanced by Temporal priors of the shifting distribution (FedTEM) is proposed. Experiments for image classification on EMNIST and CIFAR datasets, and next word prediction on the Stack Overflow dataset show the efficacy of the proposed algorithm.
main_review: Strengths:
- The paper proposes a flexible, intuitive and practical algorithm for being able to model distributional shift and incorporate the temporal information through a mixture distribution.
- The multi-branch based network ensures feature parity across different distributions being considered in this paper.
- While the paper is centered around two different distributions, it can be easily extended to multiple distributions to finer granularities depending on the need.
- Experimental results across all the vision and language tasks show an impressive improvement performance.

Cons:
- It seems the authors make an implicit assumption that a client at any given time consists of data from either of the distributions at any given time. Also, at any given time how does one ensure clients are selected from only one of the distributions being considered? As with FL, one can't have any information about the distribution of data on one client.
- The choice of the smooth transition function seems a little ad-hoc. What is the rationale behind selecting such functions? What other functions were considered?
- The performance improvement comes at the cost of added communication complexity and extra hyper parameters to tune.
- How much does the mis-specification of GMM affect the performance of FedTERM. Especially while dealing with clients with fewer samples, the updates for GMM can be high variance prone. How is the performance of FedTERM affected in such scenarios? Please consider adding additional experiments to consider aforementioned scenarios. Finally, misspecification of the prior at each step can lead to the algorithm underperforming severely.
summary_of_the_review: This paper handles the distributional shift observed in data in clients in a federated learning production setting. In particular, the distributional shift is modeled as a mixture of distributions. The innovations include learning a GMM in a federated fashion and the development of the algorithm FedTerm. Experiments show the performance gain with the proposed algorithm. However, there are some gaps in the paper which require additional clarification and experiments. Upon addition of the clarifications and experiments, the paper will be a strong paper to be accepted.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper is to solve the distribution shift between daytime modes and nighttime modes in federated learning with a mixture of distributions. The proposed method could be viewed as a two-group clustered federated learning method. In particular, the client clustering is based on the model parameters of prediction layers.
main_review: Strengths:

1. The problem setting is interesting and new. 
2. It is interesting studying the periodic distribution shift problem in FL. The authors model the distribution shift with the prior daytime and nighttime modes in daily life.
3. The authors propose a novel federated expectation-maximization algorithm (FedTEM) to learn a branched neural network model. Experiments on simulated environments show the proposed method achieves better performance when distribution shifts in the federation.

 
Weaknesses:

1. The periodic distribution shift problem in FL is very interesting. The distribution shift problem usually expects uncertain changes in every shift step. However, the targeting problem in this paper is a shift between two fixed distributions in a periodic manner. This setting is much less challenging than other distribution shift problems.  
2. The experiment cannot well support the mentioned application scenarios on “Diurnal or Nocturnal”. It simulates two distributions in FL benchmark datasets that are not designed to distinguish daytime and nighttime.
3. The baseline selection and experimental comparison is weak. 


Questions:

1. From an application perspective, there are many straightforward solutions to solve the proposed problem in training two models for “Diurnal or Nocturnal” respectively. For example, we can train two global models in a totally separate mode. I am not convinced that the proposed solution is necessary to solve the targeting problem. It would be much help to give more detailed examples to enhance the motivation.
2. Could you please provide more details about the difference between a daytime distribution and nighttime distribution? For example, how the distribution changes with respect to p(y), p(x) or p(y|x) ?
3. The proposed FedTEM estimates the mode a client belongs to and later chooses the corresponding branch of a neural network model for it. However, a similar strategy has been reported in the clustered FL framework as mentioned in A.4 - Clustering and mixture model, and other related methods, such as the below papers. Please pick up some paper as baseline methods to compare with.

https://arxiv.org/abs/2004.11791 

https://arxiv.org/abs/2108.08647

https://arxiv.org/abs/1705.10467 

https://arxiv.org/abs/2103.00697 

https://arxiv.org/abs/2108.09749

https://link.springer.com/chapter/10.1007/978-3-030-73194-6_3 

https://link.springer.com/chapter/10.1007/978-3-030-60548-3_13



summary_of_the_review: In this paper, the authors propose a novel setting for federated learning, and then design a solution to handle the periodic distribution shift problem in FL. However, the claim is not well supported by the contents.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates the problem of periodic client distribution shift in cross-device FL, as previously investiged by Eichner et al (semi-cyclic SGD). The authors propose to split the NN model in a shared part, with different heads for each component. A GMM model is added on top of the output of the shared part to allocate each client to a given head during training, with a temporal prior for the shift between classes. This GMM is fit via a federated EM scheme, in addition to the federation of the NN weights themselves. Experiments are conducted on EMNIST, CIFAR10/100 and StackOverflow datasets, for 2 components. The results demonstrate the superiority of the proposed method with respect to baseline approaches, including the addition of the temporal prior.
main_review: # Strengths

- Well written and clear paper, with a good motivation (fig 1)
- The proposed method is novel to the best of my knowledge, and provides an empirical solution to the problem initially tackled by Eichner et al.
- Although the authors focus on the case of 2 components in the experiments, the method is quite generic and could be applied in different settings
- The experiments are run on different datasets, with multiple relevant ablation baselines, and the results demonstrate the impact of the proposed method as well as the resilience to errors in modelisation (e.g. cosine instead of linear)

# Weaknesses

- In the experiments, the baselines are not that natural insofar as they also have multiple heads. I think it would also be relevant to compare to a network with a single branch.
- Fig 2 is difficult to read due to the large number of curves in each plot: I suggest adding different markers to make it more colorblind-friendly.
- The temporal prior is not learnt but fixed. Do you think it would be possible to fit it as well from data?
- Only 3 seeds are used: in addition to the means and stds, it would be great to provide the individual points, or, alternatively, to use more seeds.
summary_of_the_review: Given the technical contributions as well as the well-conducted experiments and its overall quality, I am in favour of accepting this paper.
review_title: null

Meta-review: The paper considers FL with periodically shifting distributions, which is a very relevant and timely research question in the area of federated learning, and learning under distributions shifts. The paper proposed an interesting unsupervised way to learn grouping clients into different branches during training, using a federated version of the EM algorithm. Overall the paper contains several solid contributions in some novel combinations, but remained borderline in terms over overall scores. While reviewers were generally positive about the approach, technical soundness and the importance of the question, still some concerns remained.

Concerns included on the level of novelty relative to several recent similar related FL works, and them being included as baselines. Several of these are now discussed in the rebuttal and revisions, but not all in sufficient depth. While the datasets used seem to offer sufficiently hard task from the split between the 'day' and 'night' distribution, several questions were raised if the treatment of priors is realistic enough. This includes the question of fair hyperparameter tuning with respect to the temporal priors, as well as potential misspecification of the same, towards a more principled treatment of the priors. The authors have answered several of the concerns in the revision, and have added more baseline comparisons, raising the paper narrowly above the acceptance bar in my assessment.

Time-wise, the very related paper Marfoq et al 2021 "Federated Multi-Task Learning under a Mixture of Distributions" seems to have been available 6 weeks before ICLR deadline. We thank the authors for having included it in discussion and experiments, but the discussion of related contributions needs to be expanded (main difference seems to be supervised vs unsupervised group assignment).

My impression is these points can be addressed in a camera ready version, and I hope the detailed feedback here by all reviewers below will be incorporated.
Decision: Accept (Poster)