NER - pl example
Author: shubhamagarwal92
Pull request: yes

1. Solves most of the issues raised in #3159 
2. Streamlines shell script pipeline
3. pl logs related changes
4. added in Readme

Comments:
This looks great to me. Thanks @shubhamagarwal92. 

You need to just run the black command, I believe to is `make style` to fix formatting. Otherwise it looks good.

@srush I ran the `make style` which changed a lot of files, however, the `check_code_quality` is still failing! 
Do you want me to revert the last two commits and manually change the `'logs'` to `"logs"`

BTW, `pip install -e ".[dev]"` is failing on both mac and linux for `tensorflow` and on `sentencepiece` for mac. I had to manually install the `["black", "isort", "flake8"]` packages. Python=3.8.2 in conda env. 

> BTW, pip install -e ".[dev]" is failing on both mac and linux for tensorflow and on sentencepiece for mac.

That would be because [TensorFlow only supports python 3.5-3.7](https://www.tensorflow.org/install/pip?lang=python3#system-requirements), unfortunately.

> > BTW, pip install -e ".[dev]" is failing on both mac and linux for tensorflow and on sentencepiece for mac.
> 
> That would be because [TensorFlow only supports python 3.5-3.7](https://www.tensorflow.org/install/pip?lang=python3#system-requirements), unfortunately.

@LysandreJik Thanks. Installs on ubuntu with python 3.6. 

However, on mac: 
```
conda create -n transformers_dev python=3.6 -y
conda activate transformers_dev
pip install -e ".[dev]"


Failed to build tokenizers
ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly
```

Mac specs:
```
Python 3.6.10 | packaged by conda-forge | (default, Mar  5 2020, 09:56:10) 
[GCC Clang 9.0.1 ] on darwin
```

Hi @shubhamagarwal92 ,

thanks for that PR and fixing the issues :+1: 

I just ran the `run_pl.sh` script, training and final testing run without errors now. However, during training precision, recall and F1 are always 0 -> final output on the test set shows:

```bash
TEST RESULTS
{'val_loss': tensor(7.0679), 'precision': 0.0, 'recall': 0.0, 'f1': 0}
----------------------------------------------------------------------------------------------------
Testing: 200it [00:07, 27.98it/s]
```

Last lines of the prediction output:

```bash
der I-OTHderiv
Bibliothek I-OTHderiv
berufen I-OTHderiv
wurde I-OTHderiv
, I-OTHderiv
verließ I-OTHderiv
Gardthausen I-OTHderiv
den I-OTHderiv
Bibliotheksdienst I-OTHderiv
. I-OTHderiv
```


> I just ran the `run_pl.sh` script, training and final testing run without errors now. However, during training precision, recall and F1 are always 0 -> final output on the test set shows:
> 
> ```shell
> TEST RESULTS
> {'val_loss': tensor(7.0679), 'precision': 0.0, 'recall': 0.0, 'f1': 0}
> ----------------------------------------------------------------------------------------------------
> Testing: 200it [00:07, 27.98it/s]
> ```

Thanks for reporting this. Could you please verify the version of `pytorch-lightning`. For me it is
`pytorch-lightning==0.7.1`, `transformers==2.5.1` and the results as reported in the [README](https://github.com/huggingface/transformers/pull/3180/commits/ed39624dd0d0f3bce55352a8c4c9a8f515793e29#diff-eb7fd389de7be266012669eab7db207bR119).


Also could you please check if the results in `${OUTPUT_DIR}/test_results.txt` mentioned [here](https://github.com/huggingface/transformers/pull/3180/commits/84ee92d47ee6d659edaf6a61d09b393ebeea4d5b#diff-5a6311e9856e7b0057d9c1b85cd85fadR27)
also correspond to 0. 

It works for me as:

![Screen Shot 2020-03-09 at 3 39 26 PM](https://user-images.githubusercontent.com/7984532/76230915-556c7900-621c-11ea-82bc-1289b1933613.png)


Hi,

I'm using the same versions of both `pytorch-lightning` and `transformers` 😂

Output of `test_results` is:

```bash
$ cat germeval-model/test_results.txt 
f1 = 0
precision = 0.0
recall = 0.0
val_loss = tensor(9.4173)
```

But I'm going to test it on another machine :)


> But I'm going to test it on another machine :)

I am also attaching my environment file via `pip freeze > requirements.txt`:
[requirements.txt](https://github.com/huggingface/transformers/files/4307693/requirements.txt)

Please let me know if this doesn't work. 

@shubhamagarwal92 I think somehow you have the wrong version of our style checks installed. 

Can you try running under this command?
```
sudo pip install git+git://github.com/timothycrosley/isort.git@e63ae06ec7d70b06df9e528357650281a3d3ec22#egg=isort
sudo pip install .[tf,torch,quality]
```

@LysandreJik  we have to fix this, it is really confusing...

@stefan-it would love to see your log as well. Could you also try `rm cached*` I think maybe your feature cache got messed up?





# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3180?src=pr&el=h1) Report
> Merging [#3180](https://codecov.io/gh/huggingface/transformers/pull/3180?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e03129ad447ad7670fcc6206e5eb27a5435d4d86?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3180/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3180?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3180      +/-   ##
==========================================
+ Coverage   78.15%   78.16%   +0.01%     
==========================================
  Files          98       98              
  Lines       16641    16641              
==========================================
+ Hits        13006    13008       +2     
+ Misses       3635     3633       -2
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3180?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3180/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `94.72% <0%> (+0.31%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3180?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3180?src=pr&el=footer). Last update [e03129a...9f949d3](https://codecov.io/gh/huggingface/transformers/pull/3180?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


> ```
> sudo pip install git+git://github.com/timothycrosley/isort.git@e63ae06ec7d70b06df9e528357650281a3d3ec22#egg=isort
> sudo pip install .[tf,torch,quality]
> ```

@srush I reverted the last 3 style related commits, force-pushed and added a small commit to pass all the checks. Please merge if everything is fine. 

Also, for `isort`, I guess you meant using `git+https://` 
```
pip install git+https://github.com/timothycrosley/isort.git@e63ae06ec7d70b06df9e528357650281a3d3ec22#egg=isort
```
This link is also wrong at [contributing.md](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests). Could you please also state the python version in the md. 

This command `pip install .[tf,torch,quality]` is still failing on Mac as mentioned in my previous comment.

```
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly
```

Thanks @shubhamagarwal92. Sorry for the annoyance.

@LysandreJik this lgtm. 

@srush Happy to help! :) 

Thanks for approving the PR!

@shashwath94 I think I've found the reason for the bad evaluation results: I'm using apex and the `--fp16` parameter in the `run_pl.sh` script!

Do you have any idea, why it is not working using half precision 🤔

Ah, I will check with pytorch-lightning. It is plausible we are not integrating correctly with them 

@srush While you are it, could you please check the status of my PR in pl as well. 

https://github.com/PyTorchLightning/pytorch-lightning/pull/1094

Basically, I was observing memory leak on GPU0 if other GPU id (eg. [1]) was provided when running the NER example.

AFAIK, the solution is `torch.cuda.set_device()`