Repository: dmlc/gluon-nlp

Issue:
clip_grad_global_norm doc needs notice on usage (was: clip_grad_global_norm produces problematic results)
I tried to use mulitple gpus to train the language model (e.g., AWD-LSTM), but the behaviour is not expected. I paste the training logs in the first 2 epochs as follows. I set hyper-parameters `alpha` and `beta` to zeros.

The logs with gradient clipping:

4 GPUS:
```
[Epoch 0 Batch 200/372] current loss 8.72, ppl 6128.99, throughput 660.63 samples/s, lr 29.57
[Epoch 0] throughput 45928.01 samples/s
[Epoch 0] time cost 52.74s, valid loss 8.34, valid ppl 4199.37，lr 30.00
[Epoch 0] test loss 8.31, test ppl 4053.50
[Epoch 1 Batch 200/372] current loss 8.47, ppl 4790.62, throughput 701.91 samples/s, lr 15.00
[Epoch 1] throughput 47520.37 samples/s
[Epoch 1] time cost 51.10s, valid loss 8.82, valid ppl 6737.68，lr 30.00
```

1 GPU:
```
[Epoch 0 Batch 200/372] current loss 7.70, ppl 2205.38, throughput 295.53 samples/s, lr 29.57
[Epoch 0] throughput 19927.64 samples/s
[Epoch 0] time cost 112.08s, valid loss 6.81, valid ppl 907.20，lr 30.00
[Epoch 0] test loss 6.74, test ppl 849.29
[Epoch 1 Batch 200/372] current loss 7.02, ppl 1116.47, throughput 302.28 samples/s, lr 15.00
[Epoch 1] throughput 20606.80 samples/s
[Epoch 1] time cost 108.55s, valid loss 6.51, valid ppl 671.14，lr 30.00
```

The logs without gradient clipping:

4 GPUS:
```
[Epoch 0 Batch 200/372] current loss 7.67, ppl 2153.44, throughput 775.13 samples/s, lr 29.57
[Epoch 0] throughput 53775.66 samples/s
[Epoch 0] time cost 46.28s, valid loss 6.78, valid ppl 881.91，lr 30.00
[Epoch 0] test loss 6.71, test ppl 821.79
[Epoch 1 Batch 200/372] current loss 7.00, ppl 1099.21, throughput 831.20 samples/s, lr 15.00
[Epoch 1] throughput 56021.61 samples/s
[Epoch 1] time cost 44.62s, valid loss 6.48, valid ppl 650.45，lr 30.00
```

1 GPU:
```
[Epoch 0 Batch 200/372] current loss 7.69, ppl 2182.02, throughput 309.02 samples/s, lr 29.57
[Epoch 0] throughput 20760.28 samples/s
[Epoch 0] time cost 107.76s, valid loss 6.76, valid ppl 865.22，lr 30.00
[Epoch 0] test loss 6.70, test ppl 809.79
[Epoch 1 Batch 200/372] current loss 7.01, ppl 1110.89, throughput 307.27 samples/s, lr 15.00
[Epoch 1] throughput 20919.05 samples/s
[Epoch 1] time cost 106.92s, valid loss 6.51, valid ppl 673.24，lr 30.00
```


Discussion:
I'm not sure if the log indicates a problem in the clip function as it depends on how you set the clipping value. Can you try narrow down the issue and see where the problem is? For example, you can try setting the clip value to a very large number, so that it won't actually be clipping, and see if the behavior approaches that without clipping.
@szha. I found that the error is due to that I use `trainer.step` instead of `trainer.update`, so the gradients are reduced twice. Perhaps, we can add such notice to the comment? As most of material about mxnet only mention `trainer.step`.
Indeed, we could have made a notice in the API doc about the `trainer.update` usage besides only showing in the examples. https://github.com/dmlc/gluon-nlp/blob/master/src/gluonnlp/utils/parameter.py#L34-L36

Resolving patch:
diff --git a/src/gluonnlp/utils/parameter.py b/src/gluonnlp/utils/parameter.py
--- a/src/gluonnlp/utils/parameter.py
+++ b/src/gluonnlp/utils/parameter.py
@@ -34,6 +34,9 @@ def clip_grad_global_norm(parameters, max_norm, check_isfinite=True):
     .. note::
 
         This function is only for use when `update_on_kvstore` is set to False in trainer.
+        In cases where training happens on multiple contexts, this method should be used in
+        conjunction with ``trainer.allreduce_grads()`` and ``trainer.update()``.
+        (**not** ``trainer.step()``)
 
     Example::
 


Test patch: