Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This manuscript proposes two strategies to improve both the robustness and accuracy of local agents under the setting of federated learning. Specifically, online reinforcement learning is used to perform adaptive hyperparameter search in order to maximize the utility of local models. This is quite an interesting idea since traditional hyperparameter tuning techniques, including random search or Bayesian optimization, usually requires access to monolithic dataset, which is clearly impractical under federated learning. The second contribution is an idea on using local distribution matching in order to synchronize the learning trajectories of different local models. This again is a novel and interesting idea. Overall the paper is well-written and clear to follow, which is a plus. Detailed comments and questions follow:

-   I understand that change of hyperparameter in local model affects the global model during in model aggregation stage. However, under the federated learning setting, if the number of local models is huge, then the influence of a single local model should be small. Hence instead of using online reinforcement learning for hyperparameter search, which is notoriously data-inefficient, why not framing the problem as a pure online learning problem? This helps to increase the data efficiency and also increases the stability of learning. 

-   In the design of matching network, is there any intuition why performance significantly improves if the activations of
one layer are instead derived from the activations of the next layer of interest above it? Furthermore, why we need to have additional model with parameters $\theta_i$ for alignment? Intuitively if we want the new model $M$ to be close to the original one $M^F$, shouldn't we just use some distance measure, e.g., $\ell_2$ norm, to measure the distance of feature activations in the corresponding layers directly? 

-   The experiments are illustrative, but might be too toyish under the federated learning setting. I appreciate the ablation studies the authors performed to show the relative impact of different strategies, which makes the relative contributions more clear. However, even in the case of MNIST and CIFAR, the improvement over baseline FA is not very significant. In this case, it would be better if the authors could also report the computational overhead in terms of running time.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors propose a novel representation matching scheme to reduce the divergence of local models in federated learning. In addition, the authors propose an online hyper-parameter tuning scheme. The paper is well-written. The empirical results show good performance. In overall, I think the authors propose an interesting alternative of weight regularization (also called weight divergence loss in this paper).

Detailed comments:

1. (Very minor, does not affect the score) The baseline FA+WD is actually the same as FedProx proposed in [1].

2. (Major concern) In most experiments, there is a huge gap in the performance between FA+WD and FA+RM. However, it is unclear such improvement is caused by the better robustness of RM, or simply simply caused by bad hyperparameters of FA+WD. Since FA, FA+WD, and FA+RM have different loss functions, it is unreasonable and unfair to use the same hyperparameters for them. The authors should report results with fine-tuned hyperparameters, so that we can confirm that RM really works. Otherwise, the results of the experiments are questionable. 

3. It seems that AH is irrelevant to federated learning. Even if we use fully synchronous SGD to train the model, we can still use AH to tune the hyperparameter on the server side. Ah does have some contribution, but seemingly it doesn't really contribute to the federated learning algorithm.



----------------
Reference

[1]  Li, Tian et al. “Federated Optimization for Heterogeneous Networks.” (2018).
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: - Good paper. However, the theoretical novelty is quite limited. There is no guarantee whatsoever whether the good empirical results achieved on the three experimented would persist with other datasets. Similarly, there is no analysis of the conditions required so that such empirical superiority would hold. 

- The flow of the ideas in the paper is clear and unequivocal. However, writing has a lot of room for improvement in terms of typos and grammatical mistakes. 
Examples
 -- p1: "are become"
 -- p5: "several fundamental difference"

- I think an analysis and comparisons based on computational run-time would be necessary to check how the RL-based methodology would fare w.r.t. FA and the other FL frameworks in comparison. 

- What about comparing to other federated learning frameworks, i.e. other than versions and extensions of FA? 

- In the Introduction, it was promised that the proposed method would stop catastrophic training failures; has that been actually described in the experiments?

- Similarly so for the robustness issue (which is also the first word in the paper title), where is the empirical demonstration of the robustness of the proposed method?

- Regarding the last paragraph in Section 2.2 and the first fundamental difference between the proposed modelling choice and RL, is the former still different from non-stationary RL? An example of recent works on non-stationary RL is "Reinforcement learning in non-stationary environments" by Padakandla et al. 2019. 

- This is not strictly necessary, but might be a done in a future work or so: Did you consider comparing to methods which are based on learning invariant representations (which is quite similar to the methodology pursued in the paper) adopted in different ML paradigms like domain adaptation?

review_title: Official Blind Review #3

Meta-review: This manuscript proposes strategies to improve both the robustness and accuracy of federated learning. Two proposals are online reinforcement learning for adaptive hyperparameter search, and local distribution matching to synchronize the learning trajectories of different local models. 

The reviewers and AC agree that the problem studied is timely and interesting, as it addresses known issues with federated learning. However, this manuscript also received quite divergent reviews, resulting from differences in opinion about the novelty and clarity of the conceptual and empirical results. Taken together, the AC's opinion is that the paper may not be ready for publication.
Decision: Reject