[trainer] port metrics logging and saving methods to all example scripts
Author: stas00
Pull request: no
Labels: name: Good First Issue

name: trainer

In an effort to make the examples easier to read, in https://github.com/huggingface/transformers/pull/10266 we added new trainer methods:

* `trainer.log_metrics` - to perform consistent formatting for logged metrics
* `trainer.save_metrics` - to save the metrics into a corresponding json file.

and deployed them in `run_seq2seq.py`.

The next task is do the same for all the other `examples/*/run_*.py` scripts.

Steps:
1. Study the diff for `run_seq2seq.py`. https://github.com/huggingface/transformers/pull/10266/files#diff-82bfb61a8b91894c2c2101734a6ab7b415be4ace5cd1e01b4c37663020d924ae
2. pick a script, e.g. `examples/multiple-choice/run_swag.py`
3. apply the same changes as in step 1 removing the explicit metrics printing lines and replacing them with the 2 new methods
4. test the modified script  (usually `README.md` for that folder should have the instructions to do so) and see that your change works - train/eval/test metrics are printed using the new way and that `(train|eval|test|all)_results.json` are generated.
You can use a very short datasample 5 records is enough, by just adding: `--max_train_samples 5 --max_val_samples 5 --max_test_samples 5`

repeat for other scripts.

Thank you very much!

The metrics log should be similar to this, with the exception of using different scoring metrics:

```


02/16/2021 17:06:39 - INFO - __main__ -   ***** train metrics *****
02/16/2021 17:06:39 - INFO - __main__ -     epoch                      =    1.0
02/16/2021 17:06:39 - INFO - __main__ -     init_mem_cpu_alloc_delta   =    2MB
02/16/2021 17:06:39 - INFO - __main__ -     init_mem_cpu_peaked_delta  =    0MB
02/16/2021 17:06:39 - INFO - __main__ -     init_mem_gpu_alloc_delta   =  230MB
02/16/2021 17:06:39 - INFO - __main__ -     init_mem_gpu_peaked_delta  =    0MB
02/16/2021 17:06:39 - INFO - __main__ -     total_flos                 = 2128GF
02/16/2021 17:06:39 - INFO - __main__ -     train_mem_cpu_alloc_delta  =   55MB
02/16/2021 17:06:39 - INFO - __main__ -     train_mem_cpu_peaked_delta =    0MB
02/16/2021 17:06:39 - INFO - __main__ -     train_mem_gpu_alloc_delta  =  692MB
02/16/2021 17:06:39 - INFO - __main__ -     train_mem_gpu_peaked_delta =  661MB
02/16/2021 17:06:39 - INFO - __main__ -     train_runtime              = 2.3114
02/16/2021 17:06:39 - INFO - __main__ -     train_samples              =    100
02/16/2021 17:06:39 - INFO - __main__ -     train_samples_per_second   =  3.028

02/16/2021 17:06:43 - INFO - __main__ -   ***** val metrics *****
02/16/2021 17:13:05 - INFO - __main__ -     epoch                     =     1.0
02/16/2021 17:13:05 - INFO - __main__ -     eval_bleu                 = 24.6502
02/16/2021 17:13:05 - INFO - __main__ -     eval_gen_len              =    32.9
02/16/2021 17:13:05 - INFO - __main__ -     eval_loss                 =  3.7533
02/16/2021 17:13:05 - INFO - __main__ -     eval_mem_cpu_alloc_delta  =     0MB
02/16/2021 17:13:05 - INFO - __main__ -     eval_mem_cpu_peaked_delta =     0MB
02/16/2021 17:13:05 - INFO - __main__ -     eval_mem_gpu_alloc_delta  =     0MB
02/16/2021 17:13:05 - INFO - __main__ -     eval_mem_gpu_peaked_delta =   510MB
02/16/2021 17:13:05 - INFO - __main__ -     eval_runtime              =  3.9266
02/16/2021 17:13:05 - INFO - __main__ -     eval_samples              =     100
02/16/2021 17:13:05 - INFO - __main__ -     eval_samples_per_second   =  25.467

02/16/2021 17:06:48 - INFO - __main__ -     ***** test metrics *****
02/16/2021 17:06:48 - INFO - __main__ -     test_bleu                 = 27.146
02/16/2021 17:06:48 - INFO - __main__ -     test_gen_len              =  41.37
02/16/2021 17:06:48 - INFO - __main__ -     test_loss                 = 3.6682
02/16/2021 17:06:48 - INFO - __main__ -     test_mem_cpu_alloc_delta  =    0MB
02/16/2021 17:06:48 - INFO - __main__ -     test_mem_cpu_peaked_delta =    0MB
02/16/2021 17:06:48 - INFO - __main__ -     test_mem_gpu_alloc_delta  =    0MB
02/16/2021 17:06:48 - INFO - __main__ -     test_mem_gpu_peaked_delta =  645MB
02/16/2021 17:06:48 - INFO - __main__ -     test_runtime              = 5.1136
02/16/2021 17:06:48 - INFO - __main__ -     test_samples              =    100
02/16/2021 17:06:48 - INFO - __main__ -     test_samples_per_second   = 19.556

```

Comments:
Can I work on this issue?

Yes, please and thank you!

Hi @stas00,
Sometimes it is saving it as txt file instead of a JSON file like the below code
```python
output_train_file = os.path.join(training_args.output_dir, "train_results.txt")
if trainer.is_world_process_zero():
    with open(output_train_file, "w") as writer:
        logger.info("***** Train results *****")
        for key, value in sorted(train_result.metrics.items()):
            logger.info(f"  {key} = {value}")
            writer.write(f"{key} = {value}\n")
```
Should we keep it in JSON format only or write code for saving it as a txt file?

I have seen such behavior in [run_qa.py](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_qa.py#L483), [run_mlm.py](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py#L416), run_clm.py, run_plm.py, and many other run_**.py files

`.json` format everywhere please, as this method writes out is a json data that it writes.

You don't need to keep the code that did .txt files writing.

These are examples and by definition they have no API as such to maintain, other than ensuring we don't drop functionality if someone uses these examples for something. And this effort will make things consistent on the metrics logging/saving front.

Thank you.

Hi @stas00 
In [run_ner.py](https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner.py#L432) we are saving two things `test_result` and `test_predictions` shall we add another function in a trainer for saving results apart from metrics? or just add an extra Argument for saving it as `test_predictions.json` file

```python
output_test_results_file = os.path.join(training_args.output_dir, "test_results.txt")
if trainer.is_world_process_zero():
    with open(output_test_results_file, "w") as writer:
        for key, value in sorted(metrics.items()):
            logger.info(f"  {key} = {value}")
            writer.write(f"{key} = {value}\n")

# Save predictions
output_test_predictions_file = os.path.join(training_args.output_dir, "test_predictions.txt")
if trainer.is_world_process_zero():
    with open(output_test_predictions_file, "w") as writer:
        for prediction in true_predictions:
            writer.write(" ".join(prediction) + "\n")
```

Yes, `test_predictions.txt` is a different feature that was unintentionally dropped in some of the scripts and ideally should be restored as well. 

Here is a request to restore it: https://github.com/huggingface/transformers/issues/10381

So if you'd like to tackle it as well together inside this one PR or as a separate PR that would be fantastic!

I'd say, it probably could be `trainer.save_predictions("test", predictions)`. And in which case please put it in all example scripts where it's relevant.

It probably should remain `test_predictions.txt`, as there is no data structure to it.

Note, that the secondary helper Trainer methods have been just moved in master (so rebase your branch), e.g. `save_metrics`:

```
src/transformers/trainer.py:    from .trainer_pt_utils import _get_learning_rate, log_metrics, metrics_format, save_metrics
src/transformers/trainer_pt_utils.py:def save_metrics(self, split, metrics):
```


Sure @stas00,
I would love to work on that as well if possible!
I would create separate PR because it will help me to make fewer mistakes.

Is there any way to test these four files faster: `run_tf_multiple_choice.py`, `run_xnli.py`, `run_tf_glue.py`, `run_tf_text_classification.py`.  For Other files testing script, I figure out from [test_examples.py](https://github.com/huggingface/transformers/blob/master/examples/test_examples.py) 
 

Great! thank you, @bhadreshpsavani!

> Is there any way to test these four files faster: `run_tf_multiple_choice.py`, `run_xnli.py`, `run_tf_glue.py`, `run_tf_text_classification.py`. For Other files testing script, I figure out from [test_examples.py](https://github.com/huggingface/transformers/blob/master/examples/test_examples.py)

If you'd like this could be your next challenge after this task. Ideally all examples should be tested, so missing tests, even one or two would be very welcome. We can create a separate issue and discuss the specifics if that appeals to you. If not, then please do not worry about it.

But otherwise, if you're testing manually, just use very short  `--max_train_samples 5 --max_val_samples 5 --max_test_samples 5`

Ya @stas00, we can add the missing tests after this task.
I have noticed that `--max_train_samples 5 --max_val_samples 5 --max_test_samples 5` is mostly not working for scripts other than `run_seq2seq.py`. 
It mostly giving this error
`ValueError: Some specified arguments are not used by the HfArgumentParser: ['--max_train_samples', '5', '--max_val_samples', '5', '--max_test_samples', '5']`

As you can tell I only ever use seq2seq for testing. You're absolutely correct that other examples don't have those.

I think it'd be greatly appreciated and very useful if other examples had a way to do the same. Let me check if others agree with that.

Oh and as you are doing an amazingly useful work syncing all examples to look and feel similar, there is one very crucial thing to sync and it's `templates/adding_a_new_example_script/` on which all new examples will be based, so we better have a good template to start with. I forgot to mention that earlier. Thank you!


> As you can tell I only ever use seq2seq for testing. You're absolutely correct that other examples don't have those.
> 
> I think it'd be greatly appreciated and very useful if other examples had a way to do the same. Let me check if others agree with that.

Created a dedicated issue for that now, should you be interested, @bhadreshpsavani 
https://github.com/huggingface/transformers/issues/10423
Thank you!

Sure @stas00,
I will be happy to work on it!