Add preprocessing step for transfo-xl tokenization to avoid tokenizing words followed by punction to <unk>
Author: patrickvonplaten
Pull request: yes

The problem is well shown in Issue #2000 .
This PR adds a preprocessing step to transfo-xl tokenization to better deal with the problem.


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2987?src=pr&el=h1) Report
> Merging [#2987](https://codecov.io/gh/huggingface/transformers/pull/2987?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/129f0604acb9e8b9cebd2897437324198fa37a0a?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `91.66%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2987/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2987?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2987      +/-   ##
==========================================
+ Coverage   77.17%   77.18%   +0.01%     
==========================================
  Files          98       98              
  Lines       15997    16009      +12     
==========================================
+ Hits        12345    12356      +11     
- Misses       3652     3653       +1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2987?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_transfo\_xl.py](https://codecov.io/gh/huggingface/transformers/pull/2987/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdHJhbnNmb194bC5weQ==) | `39.8% <91.66%> (+1.57%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2987?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2987?src=pr&el=footer). Last update [129f060...e0a9fd2](https://codecov.io/gh/huggingface/transformers/pull/2987?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Good to be merged then, I think!

Using this new preprocessing step, what is the proper way to format a paragraph of text so that it will work as a prompt for XLNet? Just a few lines of code as an example would make it clear to me, I think. 

The "padding text" in the example contains sentences with punctuation not separated by spaces. At the end of the text is "\<eod\> \</s\> \<eos\>" Shouldn't I include a \<eos\> after every sentence, by hand? If not and it can be done automatically, how do you avoid things like Mr. or Dr. that end in periods but don't end the sentence? 

Thanks.


Hi @summerstay, this preprocessing step is only necessary for Transfo-XL Net. For XLNet, you don't need to separate the text. Try out this code to see what I mean:

```
def show_tokenization(text, model_name):
    tok = AutoTokenizer.from_pretrained(model_name)
    print(tok.decode(tok.encode(text, add_special_tokens=False)))


show_tokenization('This is an example. See what happens with, and. ?', 'transfo-xl-wt103')
show_tokenization('This is an example. See what happens with, and. ?', 'xlnet-base-cased')

# prints:
# You might want to consider setting `add_space_before_punct_symbol=True` as an argument to the `tokenizer.encode()` to avoid tokenizing words with punctuation symbols to the `<unk>` token
# This is an <unk> See what happens <unk> <unk>?
# This is an example. See what happens with, and.?
```