Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper was easy to read, the main idea was presented very clearly.

The main points of the paper (and my concerns are below) can be summarized as follows:
1. synchronous algoriths suffer from some struggeling nodes, for which the algorithm has to wait. From my own experience, this has never happend for me on e.g. Amazon EC2 cloud, however, it happens on our own cluster at my university, if the cluster is shared and some users make some nodes very busy. So maybe if the nodes would be dedicated to just user's job, it wouldn't be such a big concer (I am not sure what kind of cluster was used to produce Figure 3 and 4). Also how many experiments have you run? In my own experience, most of the time I get the gradient on time from all nodes equality fast, but maybe just in less than 0.1% of iterations I observe that it took maybe twice as long for some node. Also the increasing shape of the curve is somehow implying some weird implementation of communication. Isn't it only because you are somehow serialize the communication? And it would be maybe much faster if a "MPI_Reduce" would be used (even if we wait for the slowest guy)?
2. asynchronous algorithms are cutting the waiting time, however, the convergence speed may be slower. Moreover, those algorithms can be divergence it special care is not given to stale gradients. Also they have a nice guarantees for convex functions, but the non-convex DNN may cause pain.
3.they propose to take gradient from the first "N" workers out of "N+b" 
workers available. My concern here is that they focused only on the 
workers, but what if the "parameter server" will became to slow? What 
if the parameter server would be the bottleneck? How would you address 
this situation? But still if the number of nodes (N) is not large, and 
the deep DNN is used, I can imagine that the communciation will not 
take more than 30% of the run-time.


My largest concern is with the experiments. Different batch size 
implies that different learning rate should be chosen, right? How did 
you tune the learning rates and other parameters for e.g. Figure 5 you 
provide some formulas in (A2) but clearly this can bias your Figures, 
right? meaning, that if you tune "\gamma, \beta" for each N, it could 
be somehow more representative? also it would be nicer if you run the 
experiment many times and then report average, best and worst case 
behaviour. because now it can be just coinsidence, right? 

review_title: Interesting Idea, but can be improved and I have a few concerns about experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper claim that, when supported by a number of backup workers, synchronized-SGD 
actually works better than async-SGD. The paper first analyze the problem of staled updates
in async-SGDs, and proposed the sync-SGD with backup workers. In the experiments, the 
authors shows the effectiveness of the proposed method in applications to Inception Net
and PixelCNN.

The idea is very simple, but in practice it can be quite useful in industry settings where 
adding some backup workders is not a big problem in cost. Nevertheless, I think the 
proposed solution is quite straightforward to come up with when we assume that 
each worker contains the full dataset and we have budge to add more workers. So, 
under this setting, it seems quite natural to have a better performance with the additional 
backup workers that avoid the staggering worker problem. And, with this assumtion I'm not 
sure if the proposed solution is solving difficult enough problem with novel enough idea. 

In the experiments, for fair comparison, I think the Async-SGD should also have a mechanism 
to cut off updates of too much staledness just as the proposed method ignores all the remaining 
updates after having N updates. For example, one can measure the average time spent to 
obtain N updates in sync-SGD setting and use that time as the cut-off threashold in Async-SGD 
so that Async-SGD does not perform so poorly.
review_title: Official Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed a synchronous parallel SGD by employing several backup machines. The parameter server does not have to wait for the return from all machines to perform the update on the model, which reduce the synchronization overhead. It sounds like a reasonable and straightforward idea. 

My main concern is that this approach is only suitable for some very specific scenario, that is, most learners (except a small number of learners) are at the same pace to return the results. If the efficiency of learners does not follow such distribution, I do not think that the proposed algorithm will work. So I suggest two revisions:

- provide more experiments to show the performance with different efficiency distributions of learners.
- assuming that all learners follow the same distribution of efficiency and show the expected idle time is minor by using the proposed algorithm.
review_title: no title

final_score: rating: null
review_content: Do you assume that the number of slow learners are less than b?


review_title: assumption

final_score: rating: null
review_content: Do you assume that each of the workers have the same full replica of the dataset?
review_title: question

Meta-review: Counter to the current wisdom, this work proposes that synchronous training may be advantageous over asynchronous training (provided that "backup workers" are available). This is shown empirical and without theoretical results. The contribution is somewhat straightforward and designed for a specific large-scale hardware scenario, but this is often an important bottleneck in the learning process. However, there is some concern about the long-term impact of this work, due to its dependence on the hardware.
Decision: Reject