Task: Language Identification
Input (Spanish, English): CONTRIBUTIONS When training LSTMs, many of the intermediate gradients are close to zero due to the flat shape of the tanh and sigmoid nonlinearities far from the origin. This paper shows that rounding these small gradients to zero results in matrices with up to 80% sparsity during training, and that training character-level LSTM language models with this sparsification does not significantly change the final performance of the model. The authors argue that this sparsity could be exploited with specialized hardware to improve the energy efficiency and speed of recurrent network training.  NOVELTY Thresholding gradients to induce sparsity and improve efficiency in RNN training is a novel result to my knowledge.  MISSING CITATIONS Prior work has explored low-precision arithmetic for recurrent neural network language models:  Hubara et al, “Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations”,
Output (English): en