Fix bug which lowercases special tokens
Author: watkinsm
Pull request: yes

A previous PR (#1592), which lowercases input and added tokens if `do_lower_case` is set to `True` for a given tokenizer, introduced a bug which lowercases text without considering whether parts of the input are special tokens. The result is that special tokens may not be tokenized properly, e.g. "[CLS]" becomes 4 separate tokens when using the BERT tokenizer: "[", "cl", "##s", "]".

This change fixes that by only applying lowercasing to non-special tokens. The do_lower_case test case has also been expanded to use some special token based on the subclass.

Closes #2047 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2051?src=pr&el=h1) Report
> Merging [#2051](https://codecov.io/gh/huggingface/transformers/pull/2051?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/5bfcd0485ece086ebcbed2d008813037968a9e58?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2051/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2051?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2051      +/-   ##
==========================================
+ Coverage   83.58%   83.58%   +<.01%     
==========================================
  Files         105      105              
  Lines       15568    15574       +6     
==========================================
+ Hits        13012    13018       +6     
  Misses       2556     2556
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2051?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tests/tokenization\_tests\_commons.py](https://codecov.io/gh/huggingface/transformers/pull/2051/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL3Rva2VuaXphdGlvbl90ZXN0c19jb21tb25zLnB5) | `100% <100%> (ø)` | :arrow_up: |
| [transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2051/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl91dGlscy5weQ==) | `91.96% <100%> (+0.09%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2051?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2051?src=pr&el=footer). Last update [5bfcd04...0025a20](https://codecov.io/gh/huggingface/transformers/pull/2051?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@LysandreJik @thomwolf my last PR seems to have introduced a nasty bug, which got into the most recent release. Could one of you (or someone else at 🤗) review this PR, which should fix it? Sorry for the regression and inconvenience :disappointed: 

Indeed, this is an issue! No worries, bugs happen.

Using a regex may be a bit slow but we'll merge this as to fix the bug, and think of optimization afterward.

>Using a regex may be a bit slow but we'll merge this as to fix the bug, and think of optimization afterward.

@LysandreJik true! It just seemed like the fastest way to get to some fix for now. I can help improve it later if necessary :)