Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper offers a beautiful and simple method for federated learning. Strong empirical results. 

Important area. 
                                                                                                                                                                                                                                                                                                                                                                                                .
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: Edit: Thanks for the thorough and responsive rebuttal! I'm particularly happy to see the additional background on BBP-MAP and the baselines you've added for handling data bias. You've comprehensively addressed my questions and I think this paper should be accepted.

Original review:

The authors extend the recently proposed Probabilistic Federated Neural Matching (PFNM) algorithm of Yurochkin et al. ( 2019) to more kinds of neural networks, show that it isn't as effective for larger models as it is for LeNet-sized ones, and propose enhancements that lead to a state-of-the-art approach they call FedMA. I'm convinced that this represents a meaningful advance in federated learning, although the paper could use some tightening up, and the experiments are somewhat limited.

Some feedback:

- I'd like to see a little bit more description of BBP-MAP, as even though it's not one of the components of the algorithm you directly modify it's still the underlying mathematical primitive. How far is it from having the same effect that the "best possible" permutation would? How is it able to allow the number of neurons in the federated model to grow relative to the size of the client models?

- Can you include the "entire data" baseline in more of the figures/plots (especially Figure 2)?

- The models and datasets covered in the experiments are adequate to demonstrate that the presented technique is worth exploring, but probably not for someone considering applying it in the context of a deployed federated learning application. Since federated learning is a problem domain motivated more by applied concerns (privacy, edge vs. cloud compute, on-device ML) than other areas of machine learning theory, it would be particularly valuable to see experiments at larger scale (in particular, on larger or more realistic datasets).

- The section that demonstrates how your model addresses skewed data domains is fascinating! That's one area in which your experiments are directly relevant to federated learning in practice, and it's a rapidly growing area of research in itself (e.g. in its relationship to causal learning that Leon Bottou has recently been exploring). Exploring this further could make for a whole separate paper. In the mean time, though, is there some kind of equivalent of the "entire data" baseline that would represent e.g. the best known technique for taking into account skewed domains outside the federated context?
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: Post Rebuttal Summary
---------------------------------
I have nudged my score up to an "Accept", based on my comments to the rebuttal below. I hope the authors continue to improve the readability of Sec. 2.1

Review Summary
--------------
Overall I think this is almost above the bar to be accepted, and I could be persuaded with a strong rebuttal.  The strengths here are the extensive experiments and the easy-to-implement method. The primary weakness of this paper is that it is a "straightfoward" way to extend the BBP-MAP method to CNNs and RNNs, so the methodological novelty is weak relative to the BBP-MAP past work (Yurochkin et al. ICML 2019). Other technical weaknesses limit the ability to use this method on clients with diverse class distributions, which will be common in real deployments.

Paper Summary
-------------
This paper addresses the problem of federated learning, where J separate "clients" with disjoint datasets each train a neural network model for a supervised problem, and then try to aggregate all J individual client models into one "global model" in a coherent way. The natural problem is that due to hidden units being permutable within one network, naively taking parameter averages across two client models will lead to bad accuracy without first coming up with a consistent ordering of the units in each layer. 

Previous work (Yurochkin et al. ICML 2019) has developed a Bayesian nonparametric model based on the Beta-Bernoulli Process (BBP) for the case of federated learning of multi-layer perceptrons. However, the extension to convolutional layers or recurrent layers has yet to be solved, which is the focus of this paper. 

This paper's algorithm (Federated Matched Averaging (FedMA), see Alg 1), proceeds by iteratively stepping thru the CNN or RNN layer by layer greedily from input to output. At each layer, we first solve a BBP-MAP optimization (bipartite matching using a BBP maximum a-posteriori objective as cost function, a subprocedure taken direclty from Yurochkin et al.). This obtains a consistent low-cost permutation for each client model. Then, the global model weights for that layer is the average of the aligned client weights. After the current layer update, each client keeps training, keeping all layers up to the current frozen but revising later layers. This layer-by-layer training can be applied to both CNNs and RNNs.

The proposed approach is compared to FedAvg and FedProx on MNIST and CIFAR image classification tasks with CNNs, and Shakespeare text classification tasks with RNNs. Later experiments explore the effect of communication efficiency (MB transfered between client and master), effect of local training epochs, handling biased class distributions, and interpretabilty.




Novelty & Significance
-----------------------
Solving federated learning problems is of increasing practical importance, and certainly trying to do so for CNNs and RNNs (more than just large MLPs) is important. So I like where the paper is going.

Although the method is "new", it is more or less a straightforward extension of work by Yurochkin et al. (ICML 2019) to CNNs and RNNs. If you read the last few sentences of Yurochkin et al., you'll see "Finally, it is of interest to extend our model-ing framework to other architectures such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). The permutation invariance necessitating matching inference also arises in CNNs since any permutation of the filters results in the same output, however additional bookkeeping is needed due to the pooling operations." I view this paper as a well-executed implementation of this "bookkeeping". Certainly not trivial, but to some readers perhaps not clearly "above the bar" for a top conference like ICLR.


Technical Concerns
------------------

## Concern 1: Client models will not always be alignable after permutation

My first concern is that there will not always be a one-to-one permutation of the neurons learned by two client models with different class distributions. Given fixed capacity at each layer, some clients may learn a filter for "horse hooves" (esp. if horse images are common to that client), while other clients may learn a filter for "snake skin" (if snakes are more common to that client). I wonder if we can quantify how well the aligned filters match in practice, and if there is any benefit to revising the alignment to allow some client-specific customizations (e.g. by having the global model can learn more units than the client model). 

## Concern 2: Use of the BBP-MAP subprocedure poorly motivated

The paper prioritizes a clean and easy-to-implement algorithm to resolve practical alignment issues between client CNN and RNN models. However, I was a bit underwhelmed that the BBP-MAP solution used by Yurochkin et al. was treated as a black-box subprocedure without much justification. I could see 2 preferable alternatives to the current use of BBP-MAP. Either a simpler approach using Eq. 2 with a squared error cost and the Munkres algorithm to solve bipartitite matching to obtain the permutation (which seems more in spirit of the rest of the paper). Or, a more sophisticated probabilistic approach (taking a Bayesian hierarchical model from Yurochkin et al. seriously and forming the estimated global weights from a weighted sums that includes both the clients (weighted by dataset size) and the assumed prior). As it is, I feel the BBP-MAP subprocedure in the current Algorithm 1 is poorly motivated for the task at hand.



Experimental Evaluation
-----------------------

Overall the experiments were extensive and demonstrated several apparent advantages (reduced need to transfer large memory during communication, etc.). 


Minor Presentation Concerns
---------------------
Before Eq. 2, you should introduce the "\theta" notation

I'm a bit confused about how "FedMA" differs from "FedMA with communication", even after reading Sec. 2.3. How exactly are communicate costs kept down? What are you sending from master to client at beginning of every "round" if not the full global model (all weights of the CNN)?
review_title: Official Blind Review #1

Meta-review: The authors presented a Federate Learning algorithm which constructs the global model layer-wise by matching and averaging hidden representations. They empirically demonstrate their method outperforms existing federated learning algorithms

This paper has received largely positive reviews. Unfortunately one reviewer wrote a very short review but was generally appreciative of the work. Fortunately, R1 wrote a detailed review with very specific questions and suggestions. The authors have addresses most of the concerns of the reviewers and I have no hesitation in recommending that this paper should be accepted. I request the authors to incorporate all suggestions made by the reviewers. 
Decision: Accept (Talk)