Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes FedSR, a framework for learning (s)imple (r)epresentation in the (fed)erated setting so that the model can be applied to unseen domains/clients (in the sense of domain generalization). The representation is assumed to be one of the hidden layers in the neural network and is learned by minimizing the sum of (a) task prediction loss (2) conditional mutual information (CMI) and (3) L2 regularization (L2R) on the representation (see Eq.(9)). By theoretically justifying CMI and L2R terms, this work shows that together the objective function can lead to good generalization performance as demonstrated in common domain generalization benchmarks.
strengths_and_weaknesses: Strengths
- The problem is important yet under-explored in the literature
- The writing and presentation are clear and easy to follow
- Good empirical performance

Weaknesses
- The experiment setup is less typical for federated learning
questions: The proposed FedSR provides a solution for domain generalization in the federated learning setting, where each client can only see data on their own data thus comparing data across clients is infeasible. The paper is well-organized and the presentation is clear. Each term in the objective function is well-motivated and their connections to common domain generalization techniques are also interesting. There are a few comments/questions

1. It may be better to explicitly mention that z is assumed to be conditional independent of y given x before Eq.(4)

2. For the experiment, are the hidden dimensions for z independent with separate variances? Or a covariance matrix is learned?

3. The experiments are only using a handful of clients (since each of them corresponds to one domain). However, it is common in federated learning to have tens to hundreds of clients, which the current experiments fail to demonstrate. In this sense, it would be more convincing if the clients are indeed sampled from an underlying distribution as depicted in Eq.(2) and (3). Then (2) and (3) will be important metrics for the baselines and FedSR.

4. It will be interesting to see what the representations of FedSR look like. Currently, the experiments only show the performance advantage in terms of accuracy, but we do not know why we have such an advantage. Maybe looking at the quality of representations can help, such as the CMI and L2R on the hold-out domain, or visualization of the representations.

Minor points

- LHS of (23) misses a p in the log
- L269 the variational distribution can match (what?)
limitations: The authors have sufficiently addressed the limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Motivated by the need for domain generalization in a federated learning setup (where each client has a different data distribution or domain), this paper proposes two simple regularizers based on the most common L2 norm squared regularizer and (an upper bound of) conditional mutual information (CMI), i.e., $I(x,z|y)$.
The L2 norm regularizer is shown to be related to minimizing the KL divergence between a fixed Gaussian distribution and the prior distributions at each client.
This can be seen as pushing all the prior distributions to the same *fixed* distribution.
Thus, this encourages marginal alignment of the latent distributions similar to AlignFlow.
The CMI regularizer is approximated via a variational approximation similar to the ELBO objective.
Because the model is shared, this encourages all client-specific conditional distributions $p_i(z|y)$ to match the global $r(z|y)$---thus making a connection to conditional alignment.
Finally, the paper gives empirical DG results on multiple datasets showing that both regularizers are helpful.

strengths_and_weaknesses: Strengths:
- Proposes a simple algorithm that merely uses two locally-computable regularizers for domain generalization in a federated learning setting.
- Demonstrates insightful connections of these regularizers to marginal and conditional distribution alignment.
- Demonstrates the empirical effectiveness of these simple FL objectives for domain generalization on multiple datasets.
- Very easy to read paper that answers natural questions throughout the paper (e.g., Line 249-255, which I may suggest putting a little earlier or mentioning it a little earlier).

Weaknesses:
- It would be nice to evaluate these methods on real-world distribution shifts as in the WILDS benchmark dataset rather than only simulated or pseudo-realistic distribution shifts.

- An exploration of the number of optimization steps before synchronization (i.e., steps per round) would improve the paper by showing how this key FL parameter affects the results.  Currently, the synchronization is set at E = 5 (i.e., I think this means sync every 5 minibatches), which seems quite often unless this means 5 epochs. What is the performance if the models are synchronized every epoch, every 2 epochs or every half epoch?  Or what if E=10, 50, 100, 500?

[WILDS] Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., ... & Liang, P. (2021, July). Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (pp. 5637-5664). PMLR.

questions: - How is (11) related to the ELBO?  I expect this to be very close in terms of derivation using an auxiliary variational distribution r. Is the main difference that you are trying to match by conditioning on diferent things (i.e., $p$ conditions on $x$ but $r$ conditions on $y$)?

- Why discuss CMI first?  I would suggest putting L2R first.

- How does this relate to AlignFlow which aligns distributions by pushing them both towards a fixed standard normal distribution?

[AlignFlow] Grover, A., Chute, C., Shu, R., Cao, Z., & Ermon, S. (2020, April). Alignflow: Cycle consistent learning from multiple domains via normalizing flows. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 4028-4035).


limitations: - The paper mentions several limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper tackles the problem of domain generalization in the federated learning setting. Additional to optimizing the average of the local objective functions, this paper tries to learn the feature representation that is invariant with data distribution by optimizing a conditional mutual information $\ell^{CMI}$. An $\ell_2$-norm regularize is added to the loss function for the feature representation to regulate the feature complexity. The proposed algorithm is evaluated on several datasets to show its strength in dealing with data from unseen domains. The algorithm is compared with FedAvg and FedDGand a few centralized methods and has comparable performance as centralized algorithms.
strengths_and_weaknesses: Originality: This paper fails to include several important prior works in federated domain generalization, including [1,1], and [1,2], which are closely related to the proposed method in this paper. In specific, the idea of learning the feature representation with norm regularize has been used in [1.2]. The domain alignment (domain invariant representation) idea has also been used in [1.1]. Therefore, this paper seems to be incremental to the prior works.

Quality: The paper is well organized. It has theoretical justifications for the regularizers and made a clear connection to domain invariant representation learning. However, the analysis is still limited as no generalization or performance bound of the proposed loss is provided. 

Clarity: The presentation of the paper is clear. The notations and concepts have been clearly defined and explained. Theoretical results are fully discussed and the numerical results are thoroughly explained.

Significance: This paper tackles an important issue in federated learning of the domain shift between the training and deployment phase. The numerical results show a promising improvement in different datasets.

[1.1] Peng, X., Huang, Z., Zhu, Y., & Saenko, K. (2019, September). Federated Adversarial Domain Adaptation. In International Conference on Learning Representations.

[1.2] Mohri, M., Sivek, G., & Suresh, A. T. (2019, May). Agnostic federated learning. In International Conference on Machine Learning (pp. 4615-4625). PMLR.


questions: Q1: Could the authors include more discussions on existing works in domain generalization for federated learning and compare the proposed algorithm with the existing methods (e.g., [1.1], [1.2])

Q2: Can the author include an algorithm block demonstrating how $\ell^{CMI}_i$ is updated.

Q3: It is unclear how the data are separated as training/testing data. It is using one domain as testing data and the other domains as training data, or using 90% of the data from all domains as training and 10% of the data from all domains as testing data?
limitations: This paper has no negative social impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In the submission, a feature representation learning method was proposed to tackle the problem of federated learning based on the out-of-distribution classification setting. Specifically, two regularization terms were introduced, i.e., L2-norm and conditional mutual information regularization term. Experimental results are performed on 4 different benchmark datasets in the community of domain generalization, and some desired performances are reported. 
strengths_and_weaknesses: Strength:

- The problem of federated learning based on OOD setting is interesting and sounding. 
- The proposed method is simple and effective. 

Weakness:

- While the proposed method is simple, IMHO, it does not shed light on federated learning. Noted that feature representation regularization is actually not novel in the community of DG (e.g., [A]),  these methods can also be applied to the individual domain followed by FedAvg. 

[A] Adversarial Mutual Information-Guided Single Domain Generalization Network for Intelligent Fault Diagnosis


- FL based on heterogeneous environment (non iid) has been largely explored. It's interesting to see how these methods perform based on the setting DG. 

- I suggest considering more realistic datasets (e.g., medical imaging), which require privacy guarantee for FL setting, for evaluation.
questions: Followed by my first concern, since the main contribution is feature representation learning, therefore, it is necessary to discuss and compare with the DG baseline methods  which does not require domain alignment across domain, and can also be applied based on the FL setting.  
limitations: The limitation of the proposed method has been discussed.
review_title: null

Decision: Accept