Repository: huggingface/transformers

Issue:
tokenizer of bert-base-uncased gives an incorrect split
I found the following code gives an incorrect split

`tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')`
`text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"`
`tokenized_text = tokenizer.tokenize(text)`
`print(' '.join(tokenized_text))`

> [ cl ##s ] who was jim henson ? [ sep ] jim henson was a puppet ##eer [ sep ]

The correct one, based on the Quick tour example, seems to be

> [CLS] who was jim henson ? [SEP] jim heson was a puppet ##eer [SEP]


Discussion:
As you can see, I'm not able to reproduce your bug, therefore **in my environment it works as expected without bugs**. Here the source code I've used (the same as you):
```
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import transformers
>>> from transformers import BertTokenizer
>>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
>>> text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
>>> tokenized_text = tokenizer.tokenize(text)
>>> tokenized_text
['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[SEP]', 'jim', 'henson', 'was', 'a', 'puppet', '##eer', '[SEP]']
>>> ' '.join(tokenized_text)
'[CLS] who was jim henson ? [SEP] jim henson was a puppet ##eer [SEP]' 
```

My environment specifications are the following:
- Python: **3.6.9**
- OS: **Ubuntu 16.04**
- Transformers: **2.2.2** (installed with `pip install --upgrade transformers`)
- PyTorch: **1.3.1**
- TensorFlow: **2.0**

Please specify your environment in order to understand why in your case it doesn't work.

UPDATE: I've just tested also with Transformers **2.1.1** and **2.0.0** and it works! Are you using `pytorch-transformers` or even `pytorch-pretrained-bert`?

> I found the following code gives an incorrect split
> 
> `tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')`
> `text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"`
> `tokenized_text = tokenizer.tokenize(text)`
> `print(' '.join(tokenized_text))`
> 
> > [ cl ##s ] who was jim henson ? [ sep ] jim henson was a puppet ##eer [ sep ]
> 
> The correct one, based on the Quick tour example, seems to be
> 
> > [CLS] who was jim henson ? [SEP] jim heson was a puppet ##eer [SEP]
This is an issue that is fixed in 2.2.2. It was present in earlier 2.2.1. If you update to the latest versions, it should be fixed. See https://github.com/huggingface/transformers/issues/2155
I am actually seeing that the behavior is now non-deterministic:
```
$ python -c 'import transformers; print(len(transformers.BertTokenizer.from_pretrained("bert-base-uncased").tokenize("A, [MASK] AllenNLP sentence.")))'
8
$ python -c 'import transformers; print(len(transformers.BertTokenizer.from_pretrained("bert-base-uncased").tokenize("A, [MASK] AllenNLP sentence.")))'
8
$ python -c 'import transformers; print(len(transformers.BertTokenizer.from_pretrained("bert-base-uncased").tokenize("A, [MASK] AllenNLP sentence.")))'
10
$ pip freeze | fgrep transformer
transformers==2.2.2
```

Resolving patch:
diff --git a/transformers/tokenization_utils.py b/transformers/tokenization_utils.py
--- a/transformers/tokenization_utils.py
+++ b/transformers/tokenization_utils.py
@@ -642,7 +642,7 @@ def tokenize(self, text, **kwargs):
         def lowercase_text(t):
             # convert non-special tokens to lowercase
             escaped_special_toks = [re.escape(s_tok) for s_tok in all_special_tokens]
-            pattern = r'(^' + r'|'.join(escaped_special_toks) + r')|' + \
+            pattern = r'(' + r'|'.join(escaped_special_toks) + r')|' + \
                       r'(.+?)'
             return re.sub(
                 pattern,


Test patch: