Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Paper refers to Dean & al and Chen & al as closest related works, yet doesn't appear to provide experimental results against them. Would it be possible to include these baseline figures?
review_title: Baselines

final_score: rating: null
review_content: Just to make the annoying message go away.
review_title: No questions.

final_score: rating: 3: Clear rejection
review_content: The authors present methods to speed-up gradient descent by leveraging asynchronicity in a layer-wise manner.

While they obtain up-to 1.7x speedup compared to synchronous training, their baseline is weak. More importantly, they dismiss parameter-server based methods, which are becoming standard, and so effectively just do not compare to the current state-of-the-art. They also do not present wall-time measurements. With these flaws, the paper is not ready for ICLR acceptance.
review_title: Lacks Strong Baselines and Wall-Time Results

final_score: rating: 3: Clear rejection
review_content: This paper is relatively difficult to parse. Much of the exposition of the proposed algorithm could be better presented using pseudo-code describing the compute flow, or a diagram describing exactly how the updates take place. As it stands, I'm not sure I understand everything. I would also have liked to see exactly described what the various labels in Fig 1 correspond to ("SGD task-wise, 1 comm"? Did you mean layer-wise?).
There are a couple of major issues with the evaluation: first, no comparison is reported against baseline async methods such as using a parameter server. Second, using AlexNet as a benchmark is not informative at all. AlexNet looks very different from any SOTA image recognition model, and in particular it has many fewer layers, which is especially relevant to the discussion in 6.3. It also uses lots of fully-connected layers which affect the compute/communication ratios in ways that are not relevant to most interesting architectures today.

review_title: Difficult to read paper. Lack of strong async baseline a major flaw.

final_score: rating: null
review_content: The overlap of computation and communication appears to be already handled by existing tools, e.g. TensorFlow(Dean et.al) and MXNet(Chen et.al.) Would be nice to have a discussion and comparison with these approaches
review_title: Overlapping techniques already in existing tools

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper describe an implementation of delayed synchronize SGD method for multi-GPU deep ne training.
Comments
1) The described manual implementation of delayed synchronization and state protection is helpful. However, such dependency been implemented by a dependency scheduler, without doing threading manually.
2) The overlap of computation and communication is a known technique implemented in existing solutions such as TensorFlow(as described in Chen et.al) and MXNet. The claimed contribution of this point is somewhat limited.
3) The convergence accuracy is only reported for the beginning iterations and only on AlexNet. It would be more helpful to include convergence curve till the end for all compared networks.

In summary, this is paper implements a variant of delayed SyncSGD approach. I find the novelty of the system somewhat limited (due to comment (2)). The experiments should have been improved to demonstrate the advantage of proposed approach.





review_title: review for Leveraging Asynchronicity in Gradient Descent for Scalable Deep Learning

Meta-review: A summary of strengths and weaknesses brought up in the reviews:
 
 Strengths
 -Paper presents a novel way to evaluate representations on generalizability to out-of-domain data (R2)
 -Experimental results are encouraging (R2)
 -Writing is clear (R1, R2)
 
 Weaknesses
 -More careful controls are needed to ascertain generalization (R2)
 -Experimental analysis is preliminary and lack of detailed analysis (R1, R2, R3)
 -Novelty and discussion of past related work (R3)
 
 The reviewers are in consensus that the idea is exciting and at least of moderate novelty, however the paper is just too preliminary for acceptance as-is. The authors did not provide a response. This is surprising because specific feedback was given to improve the paper and it seems that the paper was just under the bar. Therefore I have decided to align with the 3 reviewers in consensus and encourage the authors to revise the paper to respond to the fairly consistent suggestions for improvement and re-submit.
Decision: Reject