Consider do_lower_case in PreTrainedTokenizer
Author: watkinsm
Pull request: yes

As pointed out in #1545, when using an uncased model, and adding a new uncased token, the tokenizer does not correctly identify this in the case that the input text contains the token in a cased format.

For instance, if we load bert-base-uncased into BertTokenizer, and then use .add_tokens() to add "cool-token", we get the expected result for .tokenize('this is a cool-token'). However, we get a possibly unexpected result for .tokenize('this is a cOOl-Token'), which in fact mirrors the result for the former from before the new token was added.

This PR adds
- functionality to PreTrainedTokenizer to handle this situation in case a tokenizer (currently Bert, DistilBert, and XLNet) has the do_lower_case=True kwarg by:
    1) lowercasing tokens added with .add_tokens()
    2) lowercasing text at the beginning of .tokenize()
- new common test case for tokenizers

XLMTokenizer's `do_lowercase_and_remove_accent` is a bit more complicated and is not included in this PR.

Comments:
this lgtm but let's wait for @thomwolf and @LysandreJik to chime in

I'd also like to improve the test cases. I'll try to find some time for that this weekend

Nice improvement, it would be even better with tests for DistilBERT and XLNet as both those models make use of the `do_lower_case` argument. TransfoXL also uses the `lower_case` argument and XLM the `do_lowercase_and_remove_accent` argument so it might be a good idea to test that those models have the correct behavior.

Putting tests in the `tokenization_tests_common` would probably be cleaner than in each model's test file, if we test all models rather than a single one.

@LysandreJik good points.

~Now that I'm thinking about it, it seems like it would make more sense to do the lowercasing/accent removal directly in the subclasses (`BertTokenizer`, `XLMtokenizer`, etc.) by overriding the `tokenize()` method from `PreTrainedTokenizer`, performing the normalization there, then calling the super `tokenize()` with the now-normalized text.~

Never mind, this would result in some silly code duplication.

Alright that looks good to me!

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1592?src=pr&el=h1) Report
> Merging [#1592](https://codecov.io/gh/huggingface/transformers/pull/1592?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/de2696f68e20019fef3a5e1b54de10351abb4145?src=pr&el=desc) will **decrease** coverage by `1.22%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1592/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1592?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1592      +/-   ##
==========================================
- Coverage   84.26%   83.03%   -1.23%     
==========================================
  Files         104      104              
  Lines       15431    15456      +25     
==========================================
- Hits        13003    12834     -169     
- Misses       2428     2622     +194
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1592?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tests/tokenization\_tests\_commons.py](https://codecov.io/gh/huggingface/transformers/pull/1592/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL3Rva2VuaXphdGlvbl90ZXN0c19jb21tb25zLnB5) | `100% <100%> (ø)` | :arrow_up: |
| [transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1592/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl91dGlscy5weQ==) | `92.21% <100%> (+0.07%)` | :arrow_up: |
| [transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1592/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX3B5dG9yY2hfdXRpbHMucHk=) | `9.85% <0%> (-83.1%)` | :arrow_down: |
| [transformers/tests/modeling\_tf\_common\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1592/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX3RmX2NvbW1vbl90ZXN0LnB5) | `81.55% <0%> (-15.54%)` | :arrow_down: |
| [transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/1592/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3JvYmVydGEucHk=) | `59.41% <0%> (-12.36%)` | :arrow_down: |
| [transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/1592/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3hsbmV0LnB5) | `71.18% <0%> (-2.44%)` | :arrow_down: |
| [transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/1592/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2N0cmwucHk=) | `94.24% <0%> (-2.22%)` | :arrow_down: |
| [transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/1592/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX29wZW5haS5weQ==) | `80.66% <0%> (-1.34%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1592?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1592?src=pr&el=footer). Last update [de2696f...21637d4](https://codecov.io/gh/huggingface/transformers/pull/1592?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Ok merging!