Generative finetuning
Author: LysandreJik
Pull request: yes

Example script for fine-tuning generative models such as GPT-2 using causal language modeling (CLM). Will eventually cover masked language modeling (MLM) for BERT and RoBERTa as well.

Edit (thom): Added `max_len_single_sentence` and `max_len_sentences_pair` properties to the tokenizer to easily access the max length taking into account the special tokens.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987?src=pr&el=h1) Report
> Merging [#987](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/3566d2791905269b75014e8ea9db322c86f980b2?src=pr&el=desc) will **decrease** coverage by `0.12%`.
> The diff coverage is `77.66%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #987      +/-   ##
==========================================
- Coverage   79.22%   79.09%   -0.13%     
==========================================
  Files          38       42       +4     
  Lines        6406     6812     +406     
==========================================
+ Hits         5075     5388     +313     
- Misses       1331     1424      +93
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/tokenization\_xlnet.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3hsbmV0LnB5) | `84.25% <28.57%> (-3.86%)` | :arrow_down: |
| [pytorch\_transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX2JlcnQucHk=) | `93.39% <33.33%> (-1.75%)` | :arrow_down: |
| [pytorch\_transformers/tokenization\_xlm.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3hsbS5weQ==) | `79.67% <33.33%> (-2.38%)` | :arrow_down: |
| [pytorch\_transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3V0aWxzLnB5) | `83.03% <42.3%> (-3.28%)` | :arrow_down: |
| [pytorch\_transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfcm9iZXJ0YS5weQ==) | `73.52% <73.52%> (ø)` | |
| [...ytorch\_transformers/tests/modeling\_roberta\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvbW9kZWxpbmdfcm9iZXJ0YV90ZXN0LnB5) | `78.81% <78.81%> (ø)` | |
| [...ch\_transformers/tests/tokenization\_roberta\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX3JvYmVydGFfdGVzdC5weQ==) | `90.24% <90.24%> (ø)` | |
| [pytorch\_transformers/tokenization\_roberta.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3JvYmVydGEucHk=) | `92.45% <92.45%> (ø)` | |
| [pytorch\_transformers/file\_utils.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvZmlsZV91dGlscy5weQ==) | `71.22% <0%> (-2.88%)` | :arrow_down: |
| [pytorch\_transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfdXRpbHMucHk=) | `83.24% <0%> (ø)` | :arrow_up: |
| ... and [5 more](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987?src=pr&el=footer). Last update [3566d27...a448941](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987?src=pr&el=h1) Report
> Merging [#987](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/e00b4ff1de0591d5093407b16e665e5c86028f04?src=pr&el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #987      +/-   ##
==========================================
+ Coverage   79.61%   79.66%   +0.04%     
==========================================
  Files          42       42              
  Lines        6898     6914      +16     
==========================================
+ Hits         5492     5508      +16     
  Misses       1406     1406
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/tokenization\_xlnet.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3hsbmV0LnB5) | `89.18% <100%> (+0.19%)` | :arrow_up: |
| [pytorch\_transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX2JlcnQucHk=) | `94.88% <100%> (+0.04%)` | :arrow_up: |
| [pytorch\_transformers/tokenization\_transfo\_xl.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3RyYW5zZm9feGwucHk=) | `33.98% <100%> (+0.37%)` | :arrow_up: |
| [pytorch\_transformers/tokenization\_openai.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX29wZW5haS5weQ==) | `81.81% <100%> (+0.3%)` | :arrow_up: |
| [pytorch\_transformers/tokenization\_gpt2.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX2dwdDIucHk=) | `96.66% <100%> (+0.05%)` | :arrow_up: |
| [pytorch\_transformers/tokenization\_xlm.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3hsbS5weQ==) | `83.33% <100%> (+0.26%)` | :arrow_up: |
| [pytorch\_transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3V0aWxzLnB5) | `86.31% <100%> (+0.08%)` | :arrow_up: |
| [pytorch\_transformers/tokenization\_roberta.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3JvYmVydGEucHk=) | `96.29% <100%> (+0.06%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987?src=pr&el=footer). Last update [e00b4ff...06510cc](https://codecov.io/gh/huggingface/pytorch-transformers/pull/987?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@LysandreJik I think we are good to merge this new script for fine-tuning Bert/RoBERTa/GPT and GPT-2, right?

As a heads-up, when testing this using `--model-type=gpt2` on WikiText-103, I get this:

```
08/23/2019 17:57:44 - WARNING - pytorch_transformers.tokenization_utils -   Token indices sequence length is longer than the specified maximum sequence length for this model (119073253 > 1024). Running this sequence through the model will result in indexing errors
Traceback (most recent call last):
  File "examples/run_lm_finetuning.py", line 501, in <module>
    main()
  File "examples/run_lm_finetuning.py", line 450, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "examples/run_lm_finetuning.py", line 99, in load_and_cache_examples
    dataset = TextDataset(tokenizer, file_path=args.eval_data_file if evaluate else args.train_data_file, block_size=args.block_size)
  File "examples/run_lm_finetuning.py", line 75, in __init__
    tokenized_text = tokenizer.add_special_tokens_single_sentence(tokenized_text)
  File "~/.local/lib/python3.6/site-packages/pytorch_transformers/tokenization_utils.py", line 593, in add_special_tokens_single_sentence
    raise NotImplementedError
NotImplementedError
```

I think it is happening because `GPT2Tokenizer` doesn't implement `add_special_tokens_single_sentence`, which is used directly by `run_lm_finetuning.py` in `TextDataset.__init__`.

Ok, this looks good to me!