Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Although I am not an expert on this area, but this paper clearly explains their contribution and provides enough evidences to prove their results.
Online distillation technique is introduced to accelerate traditional algorithms for large-scale distributed NN training.
Could the authors add more results on the CNN ?
review_title: Clear analysis, good motivation and sufficient verification

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes an online distillation method, called co-distillation, where the two different models are trained to match the predictions of other model in addition to minimizing its own loss. The proposed method is applied to two large-scale datasets and showed to perform better than other baselines such as label smoothing, and the standard ensemble. 

The paper is clearly written and was easy to understand. My major concern is the significance and originality of the proposed method. As written by the authors, the main contribution of the paper is to apply the codistillation method, which is pretty similar to Zhang et. al (2017), at scale. But, because from Zhang's method, I don't see any significant difficulty in applying to large-scale problems, I'm not sure that this can be a significant contribution. Rather, I think, it would have been better for the authors to apply the proposed methods to a smaller scale problems as well in order to explore more various aspects of the proposed methods including the effects of number of different models. In this sense, it is also a limitation that the authors showing experiments where only two models are codistillated. Usually, ensemble becomes stronger as the number of model increases.

review_title: Incremental algorithmic novelty and limited experiments

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper provides a very original & promising method to scale distributed training beyond the current limits of mini-batch stochastic gradient descent. As authors point out, scaling distributed stochastic gradient descent to more workers typically requires larger batch sizes in order to fully utilize computational resource, and increasing the batch size has a diminishing return. This is clearly a very important problem, as it is a major blocker for current machine learning models to scale beyond the size of models and datasets we currently use. Authors propose to use distillation as a mechanism of communication between workers, which is attractive because prediction scores are more compact than model parameters, model-agnostic, and can be considered to be more robust to out-of-sync differences. This is a simple and sensible idea, and empirical experiments convincingly demonstrate the advantage of the method in large scale distributed training.

I would encourage authors to experiment in broader settings, in order to demonstrate that the general applicability of the proposed method, and also to help readers better understand its limitations. Authors only provide a single positive data point; that co-distillation was useful in scaling up from 128 GPUs to 258 GPUs, for the particular language modeling problem (commoncrawl) which others have not previously studied. In order for other researchers who work on different problems and different system infrastructure to judge whether this method will be useful for them, however, they need to understand better when codistillation succeeds and when it fails. It will be more useful to provide experiments with smaller and (if possible) larger number of GPUs (16, 32, 64, and 512?, 1024?), so that we can more clearly understand how useful this method is under the regime mini-batch stochastic gradient continues to scale. Also, more diversity of models would also help understanding robustness of this method to the model. Why not consider ImageNet? Goyal et al reports that it took an hour for them to train ResNet on ImageNet with 256 GPUs, and authors may demonstrate it can be trained faster.

Furthermore, authors briefly mention that staleness of parameters up to tens of thousands of updates did not have any adverse effect, but it would good to know how the learning curve behaves as a function of this delay. Knowing how much delay we can tolerate will motivate us to design different methods of communication between teacher and student models.
review_title: Promising and interesting direction to scale distributed training

Meta-review: meta score: 7

The paper introduces an online distillation technique to parallelise large scale training.  Although the basic idea is not novel, the presented experimentation indicates that the authors' have made the technique work.  Thus this paper should be of interest to practitioners.

Pros:
 - clearly written, the approach is well-explained
 - good experimentation on large-scale common crawl data with 128-256 GPUs
 - strong experimental results

Cons:
 - the idea itself is not novel
 - the range of experimentation could be wider (e.g. different numbers of GPUs) but this is expensive!

Overall the novelty is in making this approach work well in practice, and demonstrating it experimentally.
Decision: Accept (Poster)