Repository: facebookresearch/ParlAI

Issue:
num_updates reported at validation time
## Bug description
I apologize in advance, I have not looked deep into the code. This is a quick bug report based on what I saw.

Bug: It is observed in logs that the model log "num_of_update" that is not zero. The bug makes me very hesitate to report the accuracy I am getting in my paper. For example, this is a line from testing log:

"222s elapsed: {'exs': 7300, '%done': '98.01%', 'time_left': '4s', 'accuracy': 0.4058, 'f1': 0.4307, 'hits@1': 0.406, 'hits@5': 0.719, 'hits@10': 0.88, 'hits@100': 1.0, 'bleu': 0.2901, 'lr': 5e-10, **'num_updates': 92797**, 'examples': 7300, 'loss': 949.2, 'mean_loss': 0.13, 'mean_rank': 4.308, 'mrr': 0.5503}"

The value should be zero, according to the definition of the testing, otherwise it would indicate during testing, the model has been updated. 

See line [here](https://github.com/facebookresearch/ParlAI/blob/a0fe44646be2e8064b476b347e09cc1b20121e4b/parlai/core/torch_agent.py#L1002) in report function.

`metrics['num_updates'] = self._number_training_updates`

## Reproduction steps
Performed fine-turn on my own dataset. The bug is observed during testing.

Notice that the bug is also expected, but not supposed to be, in the BERT Light model. See [here](https://github.com/facebookresearch/ParlAI/blob/56513cade9d9a0cffddfc9b731d2e898ba10af7f/parlai/zoo/model_list.py#L251) and [here](https://parl.ai/docs/zoo.html?highlight=num_updates#light-bert-biranker-dialogue-model) in docs.

## Expected behavior
In testing, the num_updates should = 0.

## Additional context
Apologize again. I have not look into the code much deeper. Sorry can not be much help, u got to dig a bit.

In my opinion. Two possible situations are muturally exclusive, and may happened in this case:
* The model is indeed updating during testing. Which is very bad. The log is printing correctly as training update. See line [here](https://github.com/facebookresearch/ParlAI/blob/a0fe44646be2e8064b476b347e09cc1b20121e4b/parlai/core/torch_agent.py#L1002).
* The model is not updating, but somehow logs the num_of_update. Which in this case should not log that at all.

Thanks in advance!



Discussion:
tl;dr: everything is okay.

I appreciate the filing. The metric is misleading. In this case, `num_updates` is the TOTAL NUMBER of completed SGD steps we have completed. It is monotonically increasing, but should never increase during validation/test time.

As a point of comparison, run `eval_model.py -t yourtask -mf yourmodelfile -ltim 2`. You'll notice that `num_updates` is printed on every line, but never changes.

I find it useful to log because it gives me a point of comparison when I'm running sweeps to see if two models have had similar number of updates.
I'll discuss with my colleagues and decide if we want to keep the reporting.
Cool thanks for your reply and the quick update! :laughing:
To remove ambiguity, we'll rename it to `total_train_updates`.

Resolving patch:
diff --git a/parlai/core/torch_agent.py b/parlai/core/torch_agent.py
--- a/parlai/core/torch_agent.py
+++ b/parlai/core/torch_agent.py
@@ -39,7 +39,6 @@
     warn_once,
     round_sigfigs,
 )
-from parlai.core.distributed_utils import is_primary_worker
 
 try:
     import torch
@@ -999,7 +998,7 @@ def report(self):
         if hasattr(self, 'scheduler') and self.scheduler is not None:
             current_lr = round_sigfigs(self.optimizer.param_groups[0]['lr'], 4)
             metrics['lr'] = round_sigfigs(current_lr, 4)
-        metrics['num_updates'] = self._number_training_updates
+        metrics['total_train_updates'] = self._number_training_updates
 
         steps = self.metrics['updates']
         if steps > 0 and self.opt.get('gradient_clip', -1) > 0:
@@ -1013,7 +1012,7 @@ def report(self):
 
     def _gpu_usage(self):
         """
-        Computes GPU memory usage.
+        Compute GPU memory usage.
 
         Includes both allocated and cached memory; this should be close to the
         output of nvidia-smi, but not reflect of how much is currently demanded
diff --git a/parlai/zoo/model_list.py b/parlai/zoo/model_list.py
--- a/parlai/zoo/model_list.py
+++ b/parlai/zoo/model_list.py
@@ -248,7 +248,7 @@
             "-mf zoo:light/biranker_dialogue/model"
         ),
         "result": "{'exs': 6623, 'accuracy': 0.7586, 'f1': 0.7802, 'hits@1': 0.759, 'hits@5': 0.965,"  # noqa: E501
-        "'hits@10': 0.994, 'hits@100': 1.0, 'bleu': 0.7255, 'lr': 5e-05, 'num_updates': 15050,"  # noqa: E501
+        "'hits@10': 0.994, 'hits@100': 1.0, 'bleu': 0.7255, 'lr': 5e-05, 'total_train_updates': 15050,"  # noqa: E501
         "'examples': 6623, 'loss': 5307.0, 'mean_loss': 0.8013, 'mean_rank': 1.599, 'train_accuracy': 0}",  # noqa: E501
     },
     {
@@ -318,7 +318,7 @@
         ),
         "result2": (
             "[ Finished evaluating tasks ['convai2'] using datatype valid ]\n"
-            "{'exs': 7801, 'accuracy': 0.8942, 'f1': 0.9065, 'hits@1': 0.894, 'hits@5': 0.99, 'hits@10': 0.997, 'hits@100': 1.0, 'bleu': 0.8941, 'lr': 5e-09, 'num_updates': 0, 'examples': 7801, 'loss': 3004.0, 'mean_loss': 0.385, 'mean_rank': 1.234, 'mrr': 0.9359}"
+            "{'exs': 7801, 'accuracy': 0.8942, 'f1': 0.9065, 'hits@1': 0.894, 'hits@5': 0.99, 'hits@10': 0.997, 'hits@100': 1.0, 'bleu': 0.8941, 'lr': 5e-09, 'total_train_updates': 0, 'examples': 7801, 'loss': 3004.0, 'mean_loss': 0.385, 'mean_rank': 1.234, 'mrr': 0.9359}"
         ),
     },
     {
@@ -348,7 +348,7 @@
         ),
         "result2": (
             "[ Finished evaluating tasks ['convai2'] using datatype valid ]\n"
-            "{'exs': 7801, 'accuracy': 0.8686, 'f1': 0.8833, 'hits@1': 0.869, 'hits@5': 0.987, 'hits@10': 0.996, 'hits@100': 1.0, 'bleu': 0.8685, 'lr': 5e-09, 'num_updates': 0, 'examples': 7801, 'loss': 28.77, 'mean_loss': 0.003688, 'mean_rank': 1.301, 'mrr': 0.9197}"
+            "{'exs': 7801, 'accuracy': 0.8686, 'f1': 0.8833, 'hits@1': 0.869, 'hits@5': 0.987, 'hits@10': 0.996, 'hits@100': 1.0, 'bleu': 0.8685, 'lr': 5e-09, 'total_train_updates': 0, 'examples': 7801, 'loss': 28.77, 'mean_loss': 0.003688, 'mean_rank': 1.301, 'mrr': 0.9197}"
         ),
     },
     {
@@ -423,7 +423,7 @@
             "--round 3 -dt test -mf zoo:dialogue_safety/single_turn/model -bs 40"
         ),
         "result": (
-            "{'exs': 3000, 'accuracy': 0.9627, 'f1': 0.9627, 'bleu': 9.627e-10, 'lr': 5e-09, 'num_updates': 0, 'examples': 3000, 'mean_loss': 0.005441, 'class___notok___recall': 0.7833, 'class___notok___prec': 0.8333, 'class___notok___f1': 0.8076, 'class___ok___recall': 0.9826, 'class___ok___prec': 0.9761, 'class___ok___f1': 0.9793, 'weighted_f1': 0.9621}"
+            "{'exs': 3000, 'accuracy': 0.9627, 'f1': 0.9627, 'bleu': 9.627e-10, 'lr': 5e-09, 'total_train_updates': 0, 'examples': 3000, 'mean_loss': 0.005441, 'class___notok___recall': 0.7833, 'class___notok___prec': 0.8333, 'class___notok___f1': 0.8076, 'class___ok___recall': 0.9826, 'class___ok___prec': 0.9761, 'class___ok___f1': 0.9793, 'weighted_f1': 0.9621}"
         ),
     },
     {
@@ -440,7 +440,7 @@
             "python examples/eval_model.py -t dialogue_safety:multiturn -dt test -mf zoo:dialogue_safety/multi_turn/model --split-lines True -bs 40"
         ),
         "result": (
-            "{'exs': 3000, 'accuracy': 0.9317, 'f1': 0.9317, 'bleu': 9.317e-10, 'lr': 5e-09, 'num_updates': 0, 'examples': 3000, 'mean_loss': 0.008921, 'class___notok___recall': 0.7067, 'class___notok___prec': 0.6444, 'class___notok___f1': 0.6741, 'class___ok___recall': 0.9567, 'class___ok___prec': 0.9671, 'class___ok___f1': 0.9618, 'weighted_f1': 0.9331}"
+            "{'exs': 3000, 'accuracy': 0.9317, 'f1': 0.9317, 'bleu': 9.317e-10, 'lr': 5e-09, 'total_train_updates': 0, 'examples': 3000, 'mean_loss': 0.008921, 'class___notok___recall': 0.7067, 'class___notok___prec': 0.6444, 'class___notok___f1': 0.6741, 'class___ok___recall': 0.9567, 'class___ok___prec': 0.9671, 'class___ok___f1': 0.9618, 'weighted_f1': 0.9331}"
         ),
     },
 ]


Test patch:
diff --git a/tests/test_transformers.py b/tests/test_transformers.py
--- a/tests/test_transformers.py
+++ b/tests/test_transformers.py
@@ -80,8 +80,8 @@ def test_resuming(self):
             )
             # make sure the number of updates is being tracked correctly
             self.assertGreater(
-                valid2['num_updates'],
-                valid1['num_updates'],
+                valid2['total_train_updates'],
+                valid1['total_train_updates'],
                 'Number of updates is not increasing',
             )
             # make sure the learning rate is decreasing
@@ -476,8 +476,8 @@ def test_learning_rate_resuming(self, args):
         )
         # make sure the number of updates is being tracked correctly
         self.assertGreater(
-            valid2['num_updates'],
-            valid1['num_updates'],
+            valid2['total_train_updates'],
+            valid1['total_train_updates'],
             '({}) Number of updates is not increasing'.format(mdl),
         )
         # make sure the learning rate is decreasing
@@ -497,9 +497,10 @@ def test_learning_rate_resuming(self, args):
             )
         )
         self.assertEqual(
-            valid3['num_updates'],
-            valid1['num_updates'],
-            '({}) Finetuning LR scheduler reset failed ' '(num_updates).'.format(mdl),
+            valid3['total_train_updates'],
+            valid1['total_train_updates'],
+            '({}) Finetuning LR scheduler reset failed '
+            '(total_train_updates).'.format(mdl),
         )
         self.assertEqual(
             valid3['lr'],
@@ -516,9 +517,9 @@ def test_learning_rate_resuming(self, args):
             )
         )
         self.assertEqual(
-            valid4['num_updates'],
-            valid1['num_updates'],
-            '({}) LR scheduler change reset failed (num_updates).'
+            valid4['total_train_updates'],
+            valid1['total_train_updates'],
+            '({}) LR scheduler change reset failed (total_train_updates).'
             '\n{}'.format(mdl, stdout4),
         )
         self.assertEqual(