Title: No title found

Abstract: No abstract found

:



Figures and Tables:

Peer review (iclr2020):
# Readability Review I am a complete outsider to the field of theoretical analysis of model parallelism, although I*m well familiar with deep learning and backpropagation in general. Thus, I*m going to review this paper in terms of readability and writing flow for an outsider rather than provide domain-specific feedback. The intent of the paper is clear for an outsider. The proposed solution is not. At a high level, my understanding is that the work proposes to parallelize backpropagation by updating multiple blocks/layers of a model in parallel, which requires sometimes using *stale* gradients/activations from previous timesteps as the *real* ones have not been computed yet. This is communicated reasonably well in Section 1. However, the specific parallelization insight is never explained at a high level and lost in notation (especially for a reader unfamiliar with the recomputation technique). I appreciated Figure 3 that attempted to illustrate the process on an example with K=3, but it was not self-sufficient to communicate the intuition. Figure 3 makes use of the concepts defined in Section 3.3, and thus illustrates Algorithm 2 rather than explains its high-level ideas. Without a high-level intuition, by the time the reader gets to Section 3.3, they are already lost. Layer-wise Staleness is introduced in Section 3.1 but is only used in Theorems 1-2. It doesn*t seem to have an effect on the actual algorithm, only on its analysis. As such, it*s confusing when introduced. Instead, I*d recommend starting Section 3 with introducing timestamps and a notion of possible computation across timestamps (especially explaining how it differs from vanilla backpropagation). This understanding is crucial for Section 3.2, but the notation ?t? is not. Figure 1 needs a number of improvements to adequately present the state of the field for an outsider on a first reading: * K (number of blocks/layers) is not defined in Section 1, only later in Section 2, but Figure 1 uses it. * Column 1, *Method*, needs citations to link it to the literature without having to read the main text. * Not all of the methods mentioned in the main text appear in the figure. * It is unclear whether all the computation boundaries must be aligned. For example, in vanilla BP the *Forward* phase at K=0 must finish before the *Forward* phase K=1 starts. Is it also true when, say, in FR the *Forward* phase at K=1 ends right when the *Backward* phase at K=0 begins, or is it just an artifact of how the figure is drawn? More generally, the figure shows the time when each phase happens but doesn*t show the dependency graph between phases. Why does Algorithm 1 appear? It*s just vanilla BP. The colors in Figures 4-5 are difficult to distinguish, especially different shades of blue or red. I struggle to see why DSP would improve the test accuracy of a network aside from just making the training faster. The networks in Table 3 go through the same number of updates, correct? DSP uses noisier gradients for its updates, which might act as a regularizer, but this is just speculation. One could try to test it by adding random noise to the true gradients of BP (with magnitude similar to the expected magnitude of the BP-DSP divergence at a corresponding epoch). If it similarly improves the performance of that network, that might be evidence towards this hypothesis. In general, gradient noise does not necessarily improve the network performance unless tuned to the architecture/task. In addition, the numbers in Table 3 must be listed with their corresponding standard deviation across multiple runs. Otherwise it*s difficult to ascertain whether the DSP improvement is real or thanks to a lucky seed. Spelling: - Section 3.1: *...the data is forwarded...* In summary, I have to give this paper a Weak Reject score for readability. I*m happy to increase it if my concerns are addressed in a new PDF version during the discussion period.