DistilBERT
Author: VictorSanh
Pull request: yes

Preparing the release for DistilBERT (smaller, faster, lighter, cheaper version of BERT)

Comments:
@VictorSanh Thanks for adding this :heart: (I'm currently adding this model to Flair)

One question: the BERT model configuration has a key `hidden_size`. For DilBERT it is now `dim`. Is this change intended 🤔

# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127?src=pr&el=h1) Report
> Merging [#1127](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/d06c5a2a0acd8525d969a8f8f5b968ec0ec110b4?src=pr&el=desc) will **increase** coverage by `1.1%`.
> The diff coverage is `96.79%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master    #1127     +/-   ##
=========================================
+ Coverage   79.61%   80.71%   +1.1%     
=========================================
  Files          42       46      +4     
  Lines        6898     7391    +493     
=========================================
+ Hits         5492     5966    +474     
- Misses       1406     1425     +19
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/modeling\_transfo\_xl.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfdHJhbnNmb194bC5weQ==) | `57.12% <ø> (-0.42%)` | :arrow_down: |
| [pytorch\_transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfeGxuZXQucHk=) | `79.01% <ø> (ø)` | :arrow_up: |
| [pytorch\_transformers/modeling\_xlm.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfeGxtLnB5) | `86.66% <ø> (ø)` | :arrow_up: |
| [pytorch\_transformers/tokenization\_distilbert.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX2Rpc3RpbGJlcnQucHk=) | `100% <100%> (ø)` | |
| [pytorch\_transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfYmVydC5weQ==) | `87.98% <100%> (ø)` | :arrow_up: |
| [...torch\_transformers/tests/tokenization\_bert\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX2JlcnRfdGVzdC5weQ==) | `98.66% <100%> (ø)` | :arrow_up: |
| [pytorch\_transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfYXV0by5weQ==) | `56.36% <71.42%> (+0.36%)` | :arrow_up: |
| [pytorch\_transformers/tests/modeling\_common\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvbW9kZWxpbmdfY29tbW9uX3Rlc3QucHk=) | `94.73% <80%> (-0.21%)` | :arrow_down: |
| [...ch\_transformers/tests/tokenization\_dilbert\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX2RpbGJlcnRfdGVzdC5weQ==) | `95.23% <95.23%> (ø)` | |
| [pytorch\_transformers/modeling\_distilbert.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfZGlzdGlsYmVydC5weQ==) | `96.73% <96.73%> (ø)` | |
| ... and [7 more](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127?src=pr&el=footer). Last update [d06c5a2...e7706f5](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1127?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).