Pruning changes so that deleted heads are kept on save/load
Author: LysandreJik
Pull request: yes

The models saved with pruned heads will now be loaded correctly with a correct state dict and a correct configuration file. The changes in head structure are available in the config file via the property `config.pruned_heads`.

Pruned heads can be loaded from the config file:

```
config = GPT2Config(n_layer=4, n_head=4, pruned_heads={0: [1], 1: [2, 3]})
model = GPT2Model(config=config)

print([h.attn.n_head for h in model.h])

# [3, 2, 4, 4]
```

They are kept upon save:

```
model.save_pretrained("checkpoint")
model = GPT2Model.from_pretrained("checkpoint")

print([h.attn.n_head for h in model.h], model.config.pruned_heads)

# [3, 2, 4, 4] {0: [1], 1: [2, 3]}
```

And heads can be additionaly pruned, raising a warning if a head has already been pruned:

```
model.prune_heads({1: [1, 2], 3: [2]})

print([h.attn.n_head for h in model.h])

# Tried to remove head 2 of layer 1 but it was already removed. The current removed heads are {1: [1, 2], 3: [2]}
# [3, 1, 4, 3]
```

It is implemented for GPT, GPT-2, BERT, RoBERTa as well as XLM.



Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077?src=pr&el=h1) Report
> Merging [#1077](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/d7a4c3252ed5e630b7fb6e4b4616daddfe574fc5?src=pr&el=desc) will **increase** coverage by `0.46%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1077      +/-   ##
==========================================
+ Coverage   80.38%   80.84%   +0.46%     
==========================================
  Files          46       46              
  Lines        7749     7859     +110     
==========================================
+ Hits         6229     6354     +125     
+ Misses       1520     1505      -15
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/tests/modeling\_common\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvbW9kZWxpbmdfY29tbW9uX3Rlc3QucHk=) | `78.02% <100%> (+4.94%)` | :arrow_up: |
| [pytorch\_transformers/modeling\_distilbert.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfZGlzdGlsYmVydC5weQ==) | `96.77% <100%> (+0.03%)` | :arrow_up: |
| [pytorch\_transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfeGxuZXQucHk=) | `78.83% <100%> (-0.08%)` | :arrow_down: |
| [pytorch\_transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfdXRpbHMucHk=) | `90.02% <100%> (+3.98%)` | :arrow_up: |
| [pytorch\_transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfYmVydC5weQ==) | `88.03% <100%> (+0.04%)` | :arrow_up: |
| [pytorch\_transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfcm9iZXJ0YS5weQ==) | `75.89% <100%> (ø)` | :arrow_up: |
| [pytorch\_transformers/modeling\_transfo\_xl.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfdHJhbnNmb194bC5weQ==) | `57% <100%> (-0.12%)` | :arrow_down: |
| [pytorch\_transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfb3BlbmFpLnB5) | `81.95% <100%> (+0.11%)` | :arrow_up: |
| [pytorch\_transformers/modeling\_gpt2.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfZ3B0Mi5weQ==) | `84.03% <100%> (+0.19%)` | :arrow_up: |
| [pytorch\_transformers/modeling\_xlm.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfeGxtLnB5) | `87.08% <100%> (+0.34%)` | :arrow_up: |
| ... and [1 more](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077?src=pr&el=footer). Last update [d7a4c32...11600ed](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1077?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Let's have a talk about this one before merging (see my comment above)

Ok great, I think this is also ready to merge, now. Let's merge.