Repository: dmlc/gluon-nlp

Issue:
Log average loss metric of training GNMT
## Description
Current `log_avg_loss` in the `train_gnmt.py` script is incorrect.
```python
with mx.autograd.record():
        out, _ = model(src_seq, tgt_seq[:, :-1], src_valid_length, tgt_valid_length - 1)
        loss = loss_function(out, tgt_seq[:, 1:], tgt_valid_length - 1).mean()
        loss = loss * (tgt_seq.shape[1] - 1) / (tgt_valid_length - 1).mean()
        loss.backward()
step_loss = loss.asscalar()
log_avg_loss += step_loss
log_avg_gnorm += gnorm
```

`log_avg_loss` sums up averaged losses from different batches. It is divided by `args.log_interval` to compute the loss during the interval. It is inconsistent with the loss metric used in `train_transformer` script. In the latter script, losses and valid lengths during the interval are summed up separately. The average loss is computed from `interval_loss/interval_valid_length`.

Although these two computations are similar, I think it will be better to unify them in training scripts.

## Log Message
I print out the log message here:
```
2020-01-08 07:00:24,259 - root - [Epoch 0 Batch 100/1043] loss=6.2948, ppl=541.7368, gnorm=0.7030, throughput=36.26K wps, wc=606.57K
2020-01-08 07:00:37,441 - root - [Epoch 0 Batch 200/1043] loss=5.7232, ppl=305.8789, gnorm=0.3394, throughput=44.32K wps, wc=584.18K
2020-01-08 07:00:50,411 - root - [Epoch 0 Batch 300/1043] loss=5.2613, ppl=192.7247, gnorm=0.3442, throughput=45.71K wps, wc=592.78K
2020-01-08 07:01:03,364 - root - [Epoch 0 Batch 400/1043] loss=4.9213, ppl=137.1854, gnorm=0.3193, throughput=44.10K wps, wc=571.26K
2020-01-08 07:01:16,174 - root - [Epoch 0 Batch 500/1043] loss=4.6531, ppl=104.9109, gnorm=0.3304, throughput=43.29K wps, wc=554.48K
2020-01-08 07:01:28,367 - root - [Epoch 0 Batch 600/1043] loss=4.4185, ppl=82.9729, gnorm=0.3167, throughput=44.80K wps, wc=546.26K
2020-01-08 07:01:41,019 - root - [Epoch 0 Batch 700/1043] loss=4.3236, ppl=75.4571, gnorm=0.3128, throughput=44.82K wps, wc=566.96K
2020-01-08 07:01:53,287 - root - [Epoch 0 Batch 800/1043] loss=4.2010, ppl=66.7544, gnorm=0.3120, throughput=44.95K wps, wc=551.40K
2020-01-08 07:02:06,260 - root - [Epoch 0 Batch 900/1043] loss=4.0903, ppl=59.7581, gnorm=0.3177, throughput=44.53K wps, wc=577.61K
2020-01-08 07:02:19,849 - root - [Epoch 0 Batch 1000/1043] loss=4.0345, ppl=56.5169, gnorm=0.3055, throughput=43.28K wps, wc=588.16K
```


Discussion:

Resolving patch:
diff --git a/scripts/machine_translation/train_gnmt.py b/scripts/machine_translation/train_gnmt.py
--- a/scripts/machine_translation/train_gnmt.py
+++ b/scripts/machine_translation/train_gnmt.py
@@ -170,10 +170,10 @@ def evaluate(data_loader):
         tgt_valid_length = tgt_valid_length.as_in_context(ctx)
         # Calculating Loss
         out, _ = model(src_seq, tgt_seq[:, :-1], src_valid_length, tgt_valid_length - 1)
-        loss = loss_function(out, tgt_seq[:, 1:], tgt_valid_length - 1).mean().asscalar()
+        loss = loss_function(out, tgt_seq[:, 1:], tgt_valid_length - 1).sum().asscalar()
         all_inst_ids.extend(inst_ids.asnumpy().astype(np.int32).tolist())
         avg_loss += loss * (tgt_seq.shape[1] - 1)
-        avg_loss_denom += (tgt_seq.shape[1] - 1)
+        avg_loss_denom += (tgt_valid_length - 1).sum().asscalar()
         # Translate
         samples, _, sample_valid_length = translator.translate(
             src_seq=src_seq, src_valid_length=src_valid_length)
@@ -199,7 +199,8 @@ def train():
 
     best_valid_bleu = 0.0
     for epoch_id in range(args.epochs):
-        log_avg_loss = 0
+        log_loss = 0
+        log_denom = 0
         log_avg_gnorm = 0
         log_wc = 0
         log_start_time = time.time()
@@ -213,15 +214,18 @@ def train():
             with mx.autograd.record():
                 out, _ = model(src_seq, tgt_seq[:, :-1], src_valid_length, tgt_valid_length - 1)
                 loss = loss_function(out, tgt_seq[:, 1:], tgt_valid_length - 1).mean()
-                loss = loss * (tgt_seq.shape[1] - 1) / (tgt_valid_length - 1).mean()
+                loss = loss * (tgt_seq.shape[1] - 1)
+                log_loss += loss * tgt_seq.shape[0]
+                log_denom += (tgt_valid_length - 1).sum()
+                loss = loss / (tgt_valid_length - 1).mean()
                 loss.backward()
             grads = [p.grad(ctx) for p in model.collect_params().values()]
             gnorm = gluon.utils.clip_global_norm(grads, args.clip)
             trainer.step(1)
             src_wc = src_valid_length.sum().asscalar()
             tgt_wc = (tgt_valid_length - 1).sum().asscalar()
-            step_loss = loss.asscalar()
-            log_avg_loss += step_loss
+            log_loss = log_loss.asscalar()
+            log_denom = log_denom.asscalar()
             log_avg_gnorm += gnorm
             log_wc += src_wc + tgt_wc
             if (batch_id + 1) % args.log_interval == 0:
@@ -229,12 +233,13 @@ def train():
                 logging.info('[Epoch {} Batch {}/{}] loss={:.4f}, ppl={:.4f}, gnorm={:.4f}, '
                              'throughput={:.2f}K wps, wc={:.2f}K'
                              .format(epoch_id, batch_id + 1, len(train_data_loader),
-                                     log_avg_loss / args.log_interval,
-                                     np.exp(log_avg_loss / args.log_interval),
+                                     log_loss / log_denom,
+                                     np.exp(log_loss / log_denom),
                                      log_avg_gnorm / args.log_interval,
                                      wps / 1000, log_wc / 1000))
                 log_start_time = time.time()
-                log_avg_loss = 0
+                log_loss = 0
+                log_denom = 0
                 log_avg_gnorm = 0
                 log_wc = 0
         valid_loss, valid_translation_out = evaluate(val_data_loader)


Test patch: