Added data collator for XLNet language modeling and related calls
Author: shngt
Pull request: yes
Labels: name: model card

Added `DataCollatorForXLNetLanguageModeling` in `data/data_collator.py` to return necessary inputs (applies masking and generates revelant tensors i.e. input_ids, perm_mask, target_mask and labels as per https://github.com/zihangdai/xlnet/blob/master/data_utils.py) for language modeling training with XLNetLMHeadModel. Also added related arguments, logic and calls in `examples/language-modeling/run_language_modeling.py`.

Resolves: #4739, #2008 (partially)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5354?src=pr&el=h1) Report
> Merging [#5354](https://codecov.io/gh/huggingface/transformers/pull/5354?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/28a690a80e6c8dbcb50b5628ef853146e1940125&el=desc) will **increase** coverage by `1.15%`.
> The diff coverage is `82.51%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5354/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5354?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5354      +/-   ##
==========================================
+ Coverage   76.18%   77.33%   +1.15%     
==========================================
  Files         138      141       +3     
  Lines       24292    24660     +368     
==========================================
+ Hits        18506    19071     +565     
+ Misses       5786     5589     -197     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5354?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/5354/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `28.21% <0.00%> (-0.45%)` | :arrow_down: |
| [src/transformers/testing\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5354/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90ZXN0aW5nX3V0aWxzLnB5) | `80.85% <ø> (ø)` | |
| [src/transformers/trainer\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/5354/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyX3RmLnB5) | `16.53% <4.34%> (-2.16%)` | :arrow_down: |
| [src/transformers/data/data\_collator.py](https://codecov.io/gh/huggingface/transformers/pull/5354/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFfY29sbGF0b3IucHk=) | `61.06% <17.30%> (-37.33%)` | :arrow_down: |
| [src/transformers/data/processors/glue.py](https://codecov.io/gh/huggingface/transformers/pull/5354/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvZ2x1ZS5weQ==) | `49.09% <20.00%> (-0.36%)` | :arrow_down: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/5354/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `37.96% <30.00%> (-1.43%)` | :arrow_down: |
| [src/transformers/training\_args\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/5354/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmluZ19hcmdzX3RmLnB5) | `47.45% <44.44%> (-3.71%)` | :arrow_down: |
| [src/transformers/tokenization\_marian.py](https://codecov.io/gh/huggingface/transformers/pull/5354/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbWFyaWFuLnB5) | `93.80% <66.66%> (+25.94%)` | :arrow_up: |
| [src/transformers/modeling\_reformer.py](https://codecov.io/gh/huggingface/transformers/pull/5354/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yZWZvcm1lci5weQ==) | `88.37% <77.96%> (-0.86%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5354/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `85.67% <85.67%> (ø)` | |
| ... and [39 more](https://codecov.io/gh/huggingface/transformers/pull/5354/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5354?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5354?src=pr&el=footer). Last update [28a690a...3397fb4](https://codecov.io/gh/huggingface/transformers/pull/5354?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks a lot for the PR @shngt - this looks really cool :-) 

It's won't actually be that easy to make this work with the trainer since we will have to pass `mems` to the `model(...)` and retrieve it from the outputs. This won't be solved in a nice clean way IMO. 

We will have to adapt this function here: https://github.com/huggingface/transformers/blob/331d8d2936e7a140225cf60301ba6469930fd216/src/transformers/trainer.py#L572

Because we don't have `namedtuples` (another case why we should have `namedtuples` :D @thomwolf @LysandreJik) 
I would suggest to add the following lines after this one: https://github.com/huggingface/transformers/blob/331d8d2936e7a140225cf60301ba6469930fd216/src/transformers/trainer.py#L580 for the moment (not very pretty to this :-/):

```python 
        outputs = model(**inputs)
        loss = outputs[0]  # model outputs are always tuple in transformers (see doc)
        
        ...
        if model.config_class.__name__ in ["XLNetConfig", "TransfoXLConfig"]:
            mems = outputs[1]
        else:
            mems = None
       return loss, mems
```

I don't see a better solution at the moment. What do you think @LysandreJik @julien-c ?

Would it be ok for you to add a couple of "not-so-pretty" if-statements to the trainer? 

@shngt - Let's first decide on how to adapt the `Trainer` before continuing.

Wait for #5399 to be merged :-) 

Sorry for the mess, I didn't know rebasing would do this. Repeating the commit message for clarity:
Changed the name of `DataCollatorForXLNetLanguageModeling` to the more general `DataCollatorForPermutationLanguageModelling`.
Removed the `--mlm` flag requirement for the new collator and defined a separate `--plm_probability` flag for its use.
CTRL uses a CLM loss just like GPT and GPT-2, so should work out of the box with this script (provided `past` is taken care of
similar to `mems` for XLNet). Added a few words in the comments to reflect this.
Changed calls and imports appropriately.


Yeah rebasing can be dangerous, happens to us all the time :D. 
#5399 is merged so it should be possible to add the XLNet data collator.

I would here to copy the only file that is changed from the branch `src/transformers/data/data_collator.py` (I tihnk) and then open a new PR where you paste in this file. Then this PR should be clean. We can close this PR then since it seems to be borked.