Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Could you elaborate a bit more on how the SGD is implemented on 4 GPUs? And, why not compare to other distributed optimization methods such as Downpour SGD and Elastic Averaging SGD?

Isn't the 4 GPU too small scale to evaluate a distributed algorithm?
review_title: Questions

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors present a partially asynchronous variant of the K-FAC method. The authors adapt/modify the K-FAC method in order to make it computationally tractable for optimizing deep neural networks. The method distributes the computation of the gradients and the other quantities required by the K-FAC method (2nd order statistics and Fisher Block inversion). The gradients are computed in synchronous manner by the ‘gradient workers’ and the quantities required by the K-FAC method are computed asynchronously by the ‘stats workers’ and ‘additional workers’. The method can be viewed as an augmented distributed Synchronous SGD method with additional computational nodes that update the approximate Fisher matrix and computes its inverse. The authors illustrate the performance of the method on the CIFAR-10 and ImageNet datasets using several models and compare with synchronous SGD.

The main contributions of the paper are:
1) Distributed variant of K-FAC that is efficient for optimizing deep neural networks. The authors mitigate the computational bottlenecks of the method (second order statistic computation and Fisher Block inverses) by asynchronous updating.
2) The authors propose a “doubly-factored” Kronecker approximation for layers whose inputs are too large to be handled by the standard Kronecker-factored approximation. They also present (Appendix A) a cheaper Kronecker factored approximation for convolutional layers.
3) Empirically illustrate the performance of the method, and show:
- Asynchronous Fisher Block inversions do not adversely affect the performance of the method (CIFAR-10)
- K-FAC is faster than Synchronous SGD (with and without BN, and with momentum) (ImageNet)
- Doubly-factored K-FAC method does not deteriorate the performance of the method (ImageNet and ResNet)
- Favorable scaling properties of K-FAC with mini-batch size

Pros:
- Paper presents interesting ideas on how to make computationally demanding aspects of K-FAC tractable. 
- Experiments are well thought out and highlight the key advantages of the method over Synchronous SGD (with and without BN).

Cons: 
- “…it should be possible to scale our implementation to a larger distributed system with hundreds of workers.” The authors mention that this should be possible, but fail to mention the potential issues with respect to communication, load balancing and node (worker) failure. That being said, as a proof-of-concept, the method seems to perform well and this is a good starting point.
- Mini-batch size scaling experiments: the authors do not provide validation curves, which may be interesting for such an experiment. Keskar et. al. 2016 (On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima) provide empirical evidence that large-batch methods do not generalize as well as small batch methods. As a result, even if the method has favorable scaling properties (in terms of mini-batch sizes), this may not be effective.

The paper is clearly written and easy to read, and the authors do a good job of communicating the motivation and main ideas of the method. There are a few minor typos and grammatical errors. 

Typos:
- “updates that accounts for” — “updates that account for”
- “Kronecker product of their inverse” — “Kronecker product of their inverses”
- “where P is distribution over” — “where P is the distribution over”
- “back-propagated loss derivativesas” — “back-propagated loss derivatives as”
- “inverse of the Fisher” — “inverse of the Fisher Information matrix”
- “which amounts of several matrix” — “which amounts to several matrix”
- “The diagram illustrate the distributed” — “The diagram illustrates the distributed”
- “Gradient workers computes” — “Gradient workers compute” 
- “Stat workers computes” — “Stat workers compute” 
- “occasionally and uses stale values” — “occasionally and using stale values” 
- “The factors of rank-1 approximations” — “The factors of the rank-1 approximations”
- “be the first singular value and its left and right singular vectors” — “be the first singular value and the left and right singular vectors … , respectively.”
- “\Psi is captures” — “\Psi captures”
- “multiplying the inverses of the each smaller matrices” — “multiplying the inverses of each of the smaller matrices”
- “which is a nested applications of the reshape” — “which is a nested application of the reshape”
- “provides a computational feasible alternative” — “provides a computationally feasible alternative”
- “according the geometric mean” — “according to the geometric mean”
- “analogous to shrink” — “analogous to shrinking”
- “applied to existing model-specification code” — “applied to the existing model-specification code”
- “: that the alternative parametrization” — “: the alternative parameterization”

Minor Issues:
- In paragraph 2 (Introduction) the authors mention several methods that approximate the curvature matrix. However, several methods that have been developed are not mentioned. For example:
1) (AdaGrad) Adaptive Subgradient Methods for Online Learning and Stochastic Optimization (http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf)
2) Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization (https://arxiv.org/abs/1607.01231)
3) adaQN: An Adaptive Quasi-Newton Algorithm for Training RNNs (http://link.springer.com/chapter/10.1007/978-3-319-46128-1_1)
4) A Self-Correcting Variable-Metric Algorithm for Stochastic Optimization (http://jmlr.org/proceedings/papers/v48/curtis16.html)
5) L-SR1: A Second Order Optimization Method for Deep Learning (https://openreview.net/pdf?id=By1snw5gl)
- Page 2, equation s = WA, is there a dimension issue in this expression?
- x-axis for top plots in Figures 3,4,5,7 (Updates x XXX) appear to be a headings for the lower plots.
- “James Martens. Deep Learning via Hessian-Free Optimization” appears twice in References section.
review_title: Review - Distributed K-FAC

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes an asynchronous distributed K-FAC method for efficient optimization of 
deep networks. The authors introduce interesting ideas that many computationally demanding 
parts of the original K-FAC algorithm can be efficiently implemented in distributed fashion. The
gradients and the second-order statistics are computed by distributed workers separately and 
aggregated at the parameter server along with the inversion of the approximate Fisher matrix 
computed by a separate CPU machine. The experiments are performed in CIFAR-10 and ImageNet
classification problems using models such as AlexNet, ResNet, and GoogleReNet.

The paper includes many interesting ideas and techniques to derive an asynchronous distributed 
version from the original K-FAC. And the experiments also show good results on a few 
interesting cases. However, I think the empirical results are not thorough and convincing 
enough yet. Particularly, experiments on various and large number of GPU workers (in the same machine, 
or across multiple workers) are desired. For example, as pointed by the authors in the answer of a comment,
Chen et.al. (Revisiting Distributed Synchronous SGD, 2015) used 100 workers to test their distributed deep 
learning algorithm. Even considering that the authors have a limitation in computing resource under the 
academic research setting, the maximum number of 4 or 8 GPUs seems too limited as the only test case of 
demonstrating the efficiency of a distributed learning algorithm.  
review_title: Official Review

Meta-review: All reviewers agreed that this is an interesting contribution, but all agreed that the testing environment was somewhat small-scale and that significant difficulties could arise is scaling it up. However, the overall sentiment was still sufficiently positive.
Decision: Accept (Poster)