Title: No title found

Abstract: No abstract found

:



Figures and Tables:

Peer review (iclr2020):
This paper proposes a solution to improve the efficiency of distributed CNN training, which is an important and practical problem in accelerated training using multiple devices. The paper presents Diversely Stale Parameters (DSP) that applies different staleness to the blocks of CNN layers, which this work refers to as Layer-wise Staleness. Each block saves the input data while using it to compute initial activations in the block, making the output data of the block that will be taken by the next block. The saved input data is used once again: when the next block returns the error gradient, the activations within the block are recomputed from the saved input data that matches the error gradient*s timestamp, and the error gradient for this block is computed with these recomputed activations and the next block*s error gradient. The recomputation uses the current set of parameters that has been updated since the initial computation; the recomputed activations may differ from the initial activations. This design breaks data dependencies between forward and backward computation and creates more opportunities to overlap computation and communication of different layers. The paper analyzes the convergence property of DSP, and compares the empirical performance of DSP with standard back propagation (BP), the Decoupled Neural Interface, the Features Replay, and GPipe. I enjoyed reading this paper since it is written well, but I am not certain whether this paper should be accepted because the paper currently does not answer the following questions that are relevant to the main contributions of this paper. 1. Would it be possible to explain why the proposed solution does not maintain a queue for parameters? Such a queue could allow computing H = f_k(h_k^{n_k-m_k}; x_k^{n_k-m_k}) instead of H = f_k(h_k^{n_k-m_k}; x_k) in Algorithm 2. This makes the recomputation of activations in the backward pass the same as those in the forward pass, which makes it simpler to reason about convergence. For CNNs, the parameters of layers are much smaller than the activation of layers, especially with large batches enabled by pipelining, so the memory size of a parameter queue may not be large compared to the total memory size of block-level queues. A similar idea was explored in prior work: PipeDream [SOSP*19]; https://arxiv.org/abs/1806.03377. 2. How good is time-to-accuracy with DSP on a large dataset such as ImageNet? On CIFAR-10, the test accuracy is often better when some amount of staleness exists (as shown in sparsification-based gradient compression papers), which makes it unsurprising to see better accuracy on DSP than on BP. However, this result is less common on ImageNet, indicating that the empirical performance gain could also be due to the choice of the dataset, not just due to the training algorithm. In fact, Figure 5 (after magnification) seems to show that DSP has worse accuracy and loss compared to BP*s on ImageNet. The paper does not state the precise numbers, and makes it difficult for me to compare the time-to-accuracy of DSP with BP and FR. In addition, DSP introduces some staleness to parameter updates while GPipe adds no staleness to parameter updates, so it would be also interesting to see whether the small speedup difference between DSP (x2.7) and GPipe (x2.2) still translates into a time-to-accuracy difference. 3. What would be the practical benefit of overlapping the mini-batch forward and the mini-batch recomputation? If such overlapping is possible, it implies that the mini-batch recomputation alone was underutilizing the device memory as the forward computation still requires some memory; in other words, when not overlapping these stages, it might be possible to use a larger batch size or a different layer block size, which may improve computational efficiency and save all-reduce traffic. Since this paper tries to make a conscious decision on optimizing the memory use of distributed training, it would make the paper stronger if it examines the potentially increased memory use while overlapping forward and recomputation stages and explains why overlapping these stages is better than using the maximum amount of memory for individual stages.