Drop learning_rate to 1e-6 to avoid exploding weights,
caused by summing, not averaging, recurrent gradient updates.