Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes new direction for asynchronous training. While many synchronous and asynchronous approaches for data parallelism have been proposed and implemented in the past, the space of asynchronous model parallelism hasn't really been explored before. This paper discusses an implementation of this approach and compares the results on dynamic neural networks as compared to existing parallel approaches.

Pros:
- Paper seems to cover and contrast well with the existing approaches and is able to clarify where it differs from existing papers.
- The new approach seems to show positive results on certain dynamic neural network problems.

Cons:
- Data parallelism is a very commonly used technique for scaling. While the paper mentions support for it, the results are only showed on a toy problem, and it is unclear that it will work well for real problems. It will be great to see more results that use multiple replicas.
- As the authors mention the messages also encapsulate meta-data or "state" as the authors refer to it. This does seem to make their compiler more complex. This doesn't seem to be a requirement for their design and proposal, and it will be good to see explorations to improve on this in the future.
- Comparisons with Dynet (somewhat hidden away) that offers auto-batching in a dynamic mode aren't very positive.

Questions: 
- It appears that only a single copy of the parameters is kept, thus it is possible that some of the gradients may be computed with newer values than what the forward computation used. Is this true? Does this cause convergence issues?

Overall it seems like a valuable area for exploration, especially given the growing interest in dynamic neural networks.

[Update] Lowered rating based on other feedback and revisiting empirical results. The ideas are still interesting, but the empirical results are less convincing.
review_title: New approach to asynchrony

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper describes a model-parallel training framework/algorithm that is specialized for new devises including FPGA. Because of the small memory of those devices, model-parallel training is necessary. Most current other frameworks are for model parallelism, so in this sense, the framework proposed by the authors is different and original. The framework includes a few interesting ideas including using intermediate representation (IR) to express static computation graph and execute it as dynamic control flow, combining pipeline model parallelism and data parallelism by splitting or replicating certain layers, and enabling asynchronous training, etc.  

Some concerns/questions are 
1) The framework is targeted at devices like FPGA, but the implementation is a multicore CPU SMP. It makes the computational result less convincing. Also, does the implementation use threading or message passing?
2) Pipeline model parallelism seems need a lot of load balance tuning. The reported speedup results confirm this conjecture. Can the limitation of pipeline model parallelism be improved?

Page 4, in the "min_update_interval" paragraph, why "Small min update interval may increase gradient staleness."? I would think it decreases staleness. 

The paper is clearly written and easy to follow. 


review_title: It is an interesting paper about model parallelism for dynamic neural networks. The proposed methodology is novel, but the implementation is not very complete.    

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents AMPNet, that addresses parallel training for dynamic networks. This is accomplished by building a static graph like IR that can serve as a target for compilation for high-level libraries such as tensor flow. In the IR each node of the computation graph is a parallel worker, and synchronization occurs when a sufficient number of gradients have been accumulated. The IR uses constructs such as concat, split, broadcast,.. allowing dynamic, instance dependent control flow decisions. The primary improvement in training performance is from reducing synchronization costs.

Comments for the author:

The paper proposes a solution to an important problem of model parallel training especially over dynamic batching that is increasingly important as we see more complex models where batching is not straightforward. The proposed solution can be effective. However, this is not really evident from the evaluation. Furthermore, the paper can be a little dense read for the ICLR audience. I have the following additional concerns:

1) The paper stresses new hardware throughout the paper. The paper also alludes to “simulator" of a 1 TFLOPs FPGA in the conclusion. However, your entire evaluation is over CPU. The said simulator is a bunch of sleep() calls (unless some details are skipped). I would encourage the authors to remove these references since these new devices have very different hardware behavior. For example, on a real constrained device, you may not enjoy a large L2 cache which you are benefitting from by doing an entire evaluation over CPUs. Likewise, the vector instruction processing behavior is also very different since these devices have limited power budgets and may not be able to support AVX style instructions. Unless an actual simulator like GEM5 is used, a correct representation of what hardware environment is being used is necessary before making claims that this is ideal for emerging hardware.

2) To continue on the hardware front and the evaluation, I feel for this paper to be accepted or appreciated, a simulated hardware is not necessary. Personally, I found the evaluation with simulated sleep functions more confusing than helpful. An appropriate evaluation for this paper can be just benefits over CPU or GPUs, For example, you have a 7 TFLOPS device (e.g. a GPU or a CPU). Existing algorithms extract X TFLOPs of processing power and using your IR/system one gets Y effective TFLOPs and Y>X. This is all that is required. Currently, looking at your evaluation riddled with hypothetical hardware, it is unclear to me if this is helpful for existing hardware. For example, in Table 1, are Tensorflow numbers only provided over the 1 TFLOPs device (they correspond to the 1 TFLOPs column for all workloads except for MNIST)?  Do you use the parallelism at all in your Tensorflow baseline?  Please clarify.

3) How do you compare for dynamic batching with dynamic IR platforms like pytorch? Furthermore, more details about how dynamic batching is happening in benchmarks mentioned in Table 1 will be nice to have. Finally, an emphasis on the novel contributions of the paper will also be appreciated.

4) Finally, the evaluation appears to be sensitive to the two hyper-parameters introduced. Are they dataset specific? I feel tuning them would be rather cumbersome for every model given how sensitive they are (Figure 5).

review_title: AMPNet review

Meta-review: The authors propose a system for asynchronous, model-parallel training, suitable for dynamic neural networks.  To summarize the reviewers:

PROS:
1. Paper contrasts well with existing work.
2. Positive results on dynamic neural network problems.
3. Well written and clear

CONS:
1. Some concern about extrapolations/estimates to hardware other than that on CPU.
2. Comparisons with Dynet seem to suggest auto-batching results in a dynamic mode aren't very positive.

For 1) the AC notes the author's objections to reviewer 1's views on the value of estimation/extrapolation to non-CPU hardware.  However, reviewer 3 voiced  a similar concern and  both still feel that there is more to be done to be convincing in the experiments.
Decision: Reject