Task: Summarization
Input (English): Data-parallel neural network training is network-intensive, so gradient dropping was designed to exchange only large gradients.   However, gradient dropping has been shown to slow convergence.   We propose to improve convergence by having each node combine its locally computed gradient with the sparse global gradient exchanged over the network. We empirically confirm with machine translation tasks that gradient dropping with local gradients approaches convergence 48% faster than non-compressed multi-node training and 28% faster compared to vanilla gradient dropping. We also show that gradient dropping with a local gradient update does not reduce the model's final quality.
Output (English): This paper proposes a 3 modes for combining local and global gradients to better use more computing nodes