Split on punc should receive never_split list
Author: eisenjulian
Pull request: yes

When tokenizing when a never_split token that contains any punctuation, such as [ or ] they currently get split when they shouldn't be.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1928?src=pr&el=h1) Report
> Merging [#1928](https://codecov.io/gh/huggingface/transformers/pull/1928?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/176cd1ce1b337134425b426207fbe155099c18b4?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1928/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1928?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1928   +/-   ##
=======================================
  Coverage   84.04%   84.04%           
=======================================
  Files          97       97           
  Lines       14333    14333           
=======================================
  Hits        12046    12046           
  Misses       2287     2287
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1928?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/1928/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl9iZXJ0LnB5) | `95.92% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1928?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1928?src=pr&el=footer). Last update [176cd1c...35b06fa](https://codecov.io/gh/huggingface/transformers/pull/1928?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi, thanks for opening a PR! Could you provide an example of where this is an issue?

Hi @LysandreJik, thanks for the quick answer. The issue arises if you have `never_split` list that contains strings with punctuation, for example, square brackets. That means that you cannot easily append or use tokens in the vocabulary that have square brackets around them.

Two ways I can think of doing that is trying to reuse the [unusedN] tokens that come with BertTokenizer or adding new ones to the vocabulary. Something like the following:

    BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True, never_split=['[unused1]']).tokenize('hi how are you [unused1]')
    > ['hi', 'how', 'are', 'you', '[', 'unused', '##1', ']']

It's also telling that the method receives a never_split list that is never used, so it seems like it was originally meant to be used in that way.

Hi the `never_split` option is deprecated now (and kept for backward compatibility purposes only).

To avoid splitting a token, you should add it to the vocabulary using `tokenizer.add_tokens(['[unused1]'])`.