feat: add logging through Weights & Biases
Author: borisdayma
Pull request: yes

Hi,

I've been using hugging-face for a while and I've been loving it. I think it could become a central reference for NLP problems and the new `Trainer` class made it so much easier to add new functionality.

I would like to contribute in solving a few limitations I observed:
* some models don't have clear details on how they have been trained (hyper-parameters) and for which task, making it hard to reproduce their results (and even ensure those results were actually reached) -> ideally we would track & log their training process as well as config parameters used
* some new models keep on being requested/proposed without seeing their actual efficiency -> ideally we would be able to compare them to baselines
* I've seen issues where "better" models are offered to replace previous ones but there is no way to prove they are actually better -> maybe we can add a metrics table (linked to actual runs) to the model card
* it is hard to keep track of models and datasets
* overall I'd like to bring more traceability and transparency between users

This is why I propose to use [Weights & Biases](https://docs.wandb.com/) which is free for open source and will help solve those issues. Also it is already part of Pytorch-Lightning so it will be easy to integrate it with those scripts, if extra setup is required.

This PR brings :

* logging through Weights & Biases when `wandb` is installed
* logging of evaluation metrics at the end of training

See example run on distilbert-base-cased  -> [W&B run](https://app.wandb.ai/borisd13/transformers-examples/runs/3iuvnwam?workspace=user-borisd13)

![image](https://user-images.githubusercontent.com/715491/80064053-d3e75680-84fc-11ea-954a-9590185c65df.png)

If you navigate around, you will see that config parameters, metrics, gradients, computer resources, actual code (including git repo & command line used) and stdout have all been uploaded for full traceability and reproducibility. I can also easily upload trained model but maybe we could add an arg to make it optional in case users have limited connection (as you prefer)?

I can also easily compare bert-base-uncased and distilbert-base-uncased from my [project page](https://app.wandb.ai/borisd13/transformers-examples).
*Note: I didn't try any fine-tuning and just used same parameters on both in this example.*

![image](https://user-images.githubusercontent.com/715491/80064311-62f46e80-84fd-11ea-8e9a-d4a61af437b3.png)

This makes it easy to quickly see if a new model is actually interesting.

Finally, this integration let us create [cool reports](https://app.wandb.ai/stacey/keras_finetune/reports/Curriculum-Learning-in-Nature--Vmlldzo1MjcxNw) where graphs directly pool data and are interactive.

If you like it, then I'd love to add other features in later PR's:

* log trained model -> I'll just need to have access to path of relevant files so that I don't upload entire output directory
* add [W&B init args](https://docs.wandb.com/library/init) as parameters based on what can be useful for hugging-face
* log all command line args -> I need to access the parser so that I can also log parameters such as `model_path_or_name` and `max_seq_length` (for run_glue script) which will be convenient to create custom summary tables and compare models (or finetune)
* maybe log called script (unless `finetuning_taks` is enough?)
* in addition to metrics, track model size & inference speed which can help users choose the model they want
* use artifacts (W&B version control system) to track models, tokenizers & datasets -> central repository that will let us see easily that runs are based on same dataset or same pre-trained model + see all trained tasks associated to one pre-trained model
* log prediction samples (based on the task)
* create a central repo on W&B (entity: hugging-face) to have access to other's people runs
* integrate model card issue template so that we always add a link to a run as well as performance metrics
* add [sweeps](https://docs.wandb.com/sweeps) -> make it easy to optimize hyper-parameters
* we could add a hook to run automatically on all tasks any new model added (and link the run to the model cards)

Please let me know your thoughts and feel free to contact me or Weights & Biases directly to discuss this integration.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3916?src=pr&el=h1) Report
> Merging [#3916](https://codecov.io/gh/huggingface/transformers/pull/3916?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/cb3c2212c79d7ff0a4a4e84c3db48371ecc1c15d&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `38.46%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3916/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3916?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3916      +/-   ##
==========================================
- Coverage   78.45%   78.43%   -0.02%     
==========================================
  Files         111      111              
  Lines       18521    18533      +12     
==========================================
+ Hits        14531    14537       +6     
- Misses       3990     3996       +6     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3916?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/3916/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `43.52% <38.46%> (-0.08%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3916/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.92% <0.00%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3916?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3916?src=pr&el=footer). Last update [cb3c221...1330605](https://codecov.io/gh/huggingface/transformers/pull/3916?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi,
I'm just checking if there's anything you would like me to change on this PR.
Ideally I would also like to be able to pass a few extra optional kwargs (saving model, project name…) but the integration can go as deep as you want based on the project philosophy.

Also feel free if you want me to test other use cases for this integration. I want to write a post with the W&B guys after that and compare models, so I'd be glad to do it on something you find most useful!

The idea is mainly to give more traceability to models and a convenient way to compare and optimize models.

Will take a look very soon @borisdayma! (can i ping you on the W&B Slack or is there a better channel?)

Perfect, you can ping me on the W&B slack!

That's great @julien-c !
The next big piece that could be useful is to log the trained model weights on W&B.
I am not sure it's a good idea to do it automatically. Would there be an easy way to add an arg for it `save_model`?
Otherwise if you think it makes more sense (like for `watch` method), people can call `wandb.save` at the end of their script separately.

I'll suggest to add it in wandb docs but do you think there would be a good place to document how to use `wandb` within hugging-face docs?

I'll add some doc about wandb to our examples' README.md

Awesome, let me know if I can be of any further help!