updated metrics saving and logging
Author: bhadreshpsavani
Pull request: yes

# What does this PR do?

I have updated redundant code for saving and logging metrics in the example scripts

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #10337 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@stas00  @sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
I want to mention one thing here while testing the file I found that,
For files `run_clm.py, run_mlm.py, run_plm.py, run_ner.py, run glue.py`
The logs are as expected like this
```
02/26/2021 20:31:22 - INFO - __main__ -   ***** eval metrics *****
02/26/2021 20:31:22 - INFO - __main__ -     HasAns_exact = 0.0
02/26/2021 20:31:22 - INFO - __main__ -     HasAns_f1 = 0.0
02/26/2021 20:31:22 - INFO - __main__ -     HasAns_total = 8
02/26/2021 20:31:22 - INFO - __main__ -     NoAns_exact = 100.0
02/26/2021 20:31:22 - INFO - __main__ -     NoAns_f1 = 100.0
02/26/2021 20:31:22 - INFO - __main__ -     NoAns_total = 6
02/26/2021 20:31:22 - INFO - __main__ -     best_exact = 42.857142857142854
02/26/2021 20:31:22 - INFO - __main__ -     best_exact_thresh = 0.0
02/26/2021 20:31:22 - INFO - __main__ -     best_f1 = 42.857142857142854
02/26/2021 20:31:22 - INFO - __main__ -     best_f1_thresh = 0.0
02/26/2021 20:31:22 - INFO - __main__ -     epoch = 1.43
02/26/2021 20:31:22 - INFO - __main__ -     exact = 42.857142857142854
02/26/2021 20:31:22 - INFO - __main__ -     f1 = 42.857142857142854
02/26/2021 20:31:22 - INFO - __main__ -     total = 14
```
But for other files like `run_qa.py, run_qa_beam_search.py, run_swags.py`
The logs were like below,
```
***** eval metrics *****
  HasAns_exact      =     0.0
  HasAns_f1         =     0.0
  HasAns_total      =       8
  NoAns_exact       =   100.0
  NoAns_f1          =   100.0
  NoAns_total       =       6
  best_exact        = 42.8571
  best_exact_thresh =     0.0
  best_f1           = 42.8571
  best_f1_thresh    =     0.0
  epoch             =    1.43
  exact             = 42.8571
  f1                = 42.8571
  total             =      14
```
Without a timestamp, log level, and function name. 

When I run the command `python -m unittest discover -s examples -t examples -v` it was giving proper logs



And also for consistency let's add this bit that is currently in `run_seq2seq.py`:

https://github.com/huggingface/transformers/blob/98569d4ba237d84714f6c15e2c301fd22d42d2b1/examples/seq2seq/run_seq2seq.py#L643-L644

this allows the user to load all metrics in one call.

It can be part of this PR, or a separate one if you'd like to make this completed faster. And I can make a separate issue to add it.

That is if @sgugger you're in agreement with that syncing proposal.

I'm fine with it.
One thing that's striking me is that all those calls are inside an `if trainer.is_world_process_zero()`. Shouldn't we refactor that bit in the `log_metrics`/`save_metrics` method?

Sure @sgugger and @stas00 
I will make changes in this PR to get the changes available it faster

It won't make much of a difference at the moment since there is other code that is running under `if trainer.is_world_process_zero()` - if we make that code refactored too then absolutely yes - it would make that part of the scripts so much simpler.

we could probably do something about:
```
            trainer.save_metrics("eval", metrics)
            all_metrics.update(metrics)
```

so that it's not done separately, by 
1. either having `save_metrics` always update `all_results.json` with every call
2. or simply have trainer store the metrics internally and then just have one call to flush it to the disk at the end of the run

suggestion 1 will require read+write but the cool thing is that it's totally automated and requires no extra calls later.

@bhadreshpsavani, I suggest we 

1. finish this PR w/o introducing new changes we are discussing, since they aren't yet thought out well/agreed upon yet.
2. Then we tweak `run_seq2seq.py` to do things better, have it as a model, 
3. and then sync to other scripts? 

how does that sound?

To be clear, perhaps leave out this suggestion for now https://github.com/huggingface/transformers/pull/10421#issuecomment-786806023 if we are going to refactor it anyway - unless you already did it, then please keep it in.

Okay @stas00,
I have added [this](https://github.com/huggingface/transformers/pull/10421#pullrequestreview-599815674) changes about logger and it is working perfectly. I will commit it in this PR for that three files.

OK, so the only missing step is to update the template `templates/adding_a_new_example_script`

@bhadreshpsavani, this is now merged: https://github.com/huggingface/transformers/pull/10427 and can be replicated to other scripts (and one template)

Please feel free to add it to this PR or open a new one - whatever works the best for you.

Thank you! 

@stas00, I will first try to make changes to this PR

in the `run_glue.py` we have code like this for saving test result,
```python
output_test_file = os.path.join(training_args.output_dir, f"test_results_{task}.txt")
if trainer.is_world_process_zero():
    with open(output_test_file, "w") as writer:
        logger.info(f"***** Test results {task} *****")
        writer.write("index\tprediction\n")
        for index, item in enumerate(predictions):
            if is_regression:
                writer.write(f"{index}\t{item:3.3f}\n")
            else:
                item = label_list[item]
                writer.write(f"{index}\t{item}\n")
```
I didn't change it because it was different than our general save_metrics() methods. 
Is there any way we can generalize it?


Hi @stas00 and @sgugger,

I was trying to make changes in the same PR and after doing rebase from master I needed to merge the new commits to my branch to push my changes. Let me know if this PR is not fine or I need to make another PR with all these changes and delete this one.

This is the first time I used `git rebase upstream/master` so I might have done it incorrectly.

Yes the rebase has made new files appear in the diff that are irrelevant to your work, so it would be great if you could close this PR and open a new one (no need to do anything else than that like creating a new branch, it's just git being annoying here).

For your earlier question, leave the part in `run_glue` that doesn't refactor nicely as it is I would say.

As Sylvain said you can make a new PR branch, but you can also fix this PR by rolling back to the last good commit before the failed rebase:

```
git reset --soft 4e529f1
git commit
git push -f
```

and then rebase

BTW, If you want to use an automated rebase process please consider this little script: https://github.com/stas00/git-tools/tree/master/git-rebase


Sure @stas00,
I will use an automatic rebase script next time.
For simplicity, I have created another [PR](https://github.com/huggingface/transformers/pull/10436) with all the changes and tested the changes.
I closing this PR.