CTRLTokenizer not consistent with the fastBPE tokenizer used in Salesforce/CTRL
Author: orenmelamud
Pull request: no

## 🐛 Bug

cc @keskarnitish

<!-- Important information -->

I am using the transformers CTRL re-implementation to fine-tune the original pre-trained model released by Salesforce https://github.com/salesforce/ctrl.

When the input text consists of newline characters, the tokenization of the Transformers tokenizer differs from the one used by Salesforce CTRL.

## To Reproduce
Salesforce tokenization:
```
import fastBPE
import re
bpe = fastBPE.fastBPE('codes', 'vocab')
line = 'This is one sentence.\nAnd this is another sentence!\n'
tokenized_line = bpe.apply([line])[0]
tokenized_line = re.findall(r'\S+|\n', tokenized_line)
toks = list(filter(lambda x: x != u'@@', tokenized_line))
print(toks)

['This', 'is', 'one', 'sentenc@@', 'e.@@', '\n', 'And', 'this', 'is', 'another', 'sent@@', 'ence@@', '!@@', '\n']
```

Transformers tokenization:
```
from transformers import CTRLTokenizer
tokenizer = CTRLTokenizer.from_pretrained('ctrl', do_lower_case=False)
toks = tokenizer.tokenize(line)
print(toks)

['This', 'is', 'one', 'sentenc@@', 'e.@@', '\n@@', 'And', 'this', 'is', 'another', 'sent@@', 'ence!']
```

Also, I get this issue with double space in the input text:
```
line = 'And also a problem with more than one consecutive  space'
tokenized_line = tokenizer.tokenize(line)

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-37-747954838180> in <module>
      1 line = 'And also a problem with more than one consecutive  space'
----> 2 tokenized_line = tokenizer.tokenize(line)

~/anaconda3/envs/hugging/lib/python3.7/site-packages/transformers/tokenization_utils.py in tokenize(self, text, **kwargs)
    647 
    648         added_tokens = list(self.added_tokens_encoder.keys()) + self.all_special_tokens
--> 649         tokenized_text = split_on_tokens(added_tokens, text)
    650         return tokenized_text
    651 

~/anaconda3/envs/hugging/lib/python3.7/site-packages/transformers/tokenization_utils.py in split_on_tokens(tok_list, text)
    644             return sum((self._tokenize(token, **kwargs) if token not \
    645                     in self.added_tokens_encoder and token not in self.all_special_tokens \
--> 646                     else [token] for token in tokenized_text), [])
    647 
    648         added_tokens = list(self.added_tokens_encoder.keys()) + self.all_special_tokens

~/anaconda3/envs/hugging/lib/python3.7/site-packages/transformers/tokenization_utils.py in <genexpr>(.0)
    644             return sum((self._tokenize(token, **kwargs) if token not \
    645                     in self.added_tokens_encoder and token not in self.all_special_tokens \
--> 646                     else [token] for token in tokenized_text), [])
    647 
    648         added_tokens = list(self.added_tokens_encoder.keys()) + self.all_special_tokens

~/anaconda3/envs/hugging/lib/python3.7/site-packages/transformers/tokenization_ctrl.py in _tokenize(self, text)
    141 
    142         for token in text:
--> 143             split_tokens.extend([t for t in self.bpe(token).split(' ')])
    144         return split_tokens
    145 

~/anaconda3/envs/hugging/lib/python3.7/site-packages/transformers/tokenization_ctrl.py in bpe(self, token)
     94             return self.cache[token]
     95         word = tuple(token)
---> 96         word = tuple(list(word[:-1]) + [word[-1]+'</w>'])
     97         pairs = get_pairs(word)
     98 

## Expected behavior

I expect the tokenizers to output identical tokenizations so that fine-tuning is consistent with pre-training.

I expect the tokenizer to handle double spaces.

## Environment

* OS: Linux
* Python version: 3.7.4
* PyTorch version: 1.3.0
* PyTorch Transformers version (or branch): 2.1.1
* Using GPU ?
* Distributed of parallel setup ?
* Any other relevant information:




Comments:
Indeed, thanks a lot for the bug report.
This is fixed on master (by using regex to split instead of white spaces).