GPT-2 (medium size model, special_tokens, fine-tuning, attention) + repo code coverage metric 
Author: thomwolf
Pull request: yes

Superseded #560.

Improvements to GPT-2:
- add special tokens
- tested fine-tuning
- add medium size model

Improvements to GPT/GPT-2:
- option to extract attention weights.

Add code coverage

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/597?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@f9cde97`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `81%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/597/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/597?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##             master     #597   +/-   ##
=========================================
  Coverage          ?   67.04%           
=========================================
  Files             ?       18           
  Lines             ?     3835           
  Branches          ?        0           
=========================================
  Hits              ?     2571           
  Misses            ?     1264           
  Partials          ?        0
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/597?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_pretrained\_bert/tokenization\_openai.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/597/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvdG9rZW5pemF0aW9uX29wZW5haS5weQ==) | `81.34% <0%> (ø)` | |
| [pytorch\_pretrained\_bert/tokenization\_gpt2.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/597/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvdG9rZW5pemF0aW9uX2dwdDIucHk=) | `82.44% <20%> (ø)` | |
| [pytorch\_pretrained\_bert/modeling\_openai.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/597/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvbW9kZWxpbmdfb3BlbmFpLnB5) | `78.3% <68.51%> (ø)` | |
| [pytorch\_pretrained\_bert/modeling\_gpt2.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/597/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvbW9kZWxpbmdfZ3B0Mi5weQ==) | `79.04% <73.11%> (ø)` | |
| [pytorch\_pretrained\_bert/modeling.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/597/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvbW9kZWxpbmcucHk=) | `88.57% <98.09%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/597?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/597?src=pr&el=footer). Last update [f9cde97...35e6baa](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/597?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi, is the script run_openai.py updating weights of all layers or just the last few? Thanks in advance!