GPT2Tokenizer remove the ' ' (space) if it is at the end of text?
Author: carter54
Pull request: no
Labels: name: Core: Tokenization

# ❓ Questions & Help
GPT2Tokenizer remove the ' ' (space) if it is at the end of text?
## Details
<!-- Description of your issue -->
Hello~ I'm using transformers v2.11.0 to run some GPT2 model. 
When I tested the code, I found that the token id list generated by GPT2Tokenizer had confusing result when I input a text string end with a space.

To reproduce:
```
from transformers import GPT2Tokenizer
tokenizer_file = 'path_to_your_tokenzier_files'
tokenizer = GPT2Tokenizer.from_pretrained(tokenizer_file)

# case 1: without a space at the end input text string 
text_1 = 'import tensorflow as'
encoded_1 = tokenizer.encode(text_1, add_special_tokens=False)
print(encoded_1)

# case 2: with a space at the end input text string 
text_2 = 'import tensorflow as '   # noted there is a space at the end
encoded_2 = tokenizer.encode(text_2, add_special_tokens=False)
print(encoded_2)
```
the output is confusing
```
# case 1:
 [618, 1969, 573]

# case 2:
 [618, 1969, 573]
```
both these two case output the same token id list, with my tokenzier is [618, 1969, 573]. Which means the space at the end of case2 is ignored in the tokenization process.

I debug the code and found that the space is ignored at this step
https://github.com/huggingface/transformers/blob/b42586ea560a20dcadb78472a6b4596f579e9043/src/transformers/tokenization_utils.py#L1288

Is this step with some propose?

I also tried the ByteLevelBPETokenizer in tokenizers project
```
from tokenizers.implementations import ByteLevelBPETokenizer

tokenizer_file = 'path_to_your_tokenzier_files'
tokenizer = ByteLevelBPETokenizer(tokenizer_file)

# case 1: without a space at the end input text string 
text_1 = 'import tensorflow as'
encoded_1 = tokenizer.encode(text_1)
print(len(encoded_1))
print(encoded_1.ids)
print(encoded_1.tokens)

# case 2: with a space at the end input text string 
text_2 = 'import tensorflow as '
encoded_2 = tokenizer.encode(text_2)
print(len(encoded_2))
print(encoded_2.ids)
print(encoded_2.tokens)
```
the output are as expected:
```
# case 1
3
[618, 5015, 573]
['import', 'Ġtensorflow', 'Ġas']

# case 2
4
[618, 5015, 573, 231]
['import', 'Ġtensorflow', 'Ġas', 'Ġ']
```



Comments:
Hi @carter54,

Thanks for your issue! I think you're correct. This was a bug in a previous version.
Note that for v3.0.0 we have done a major refactoring that corrected this error as far as I can see. 

If you take a look at this line in current master in this line: https://github.com/huggingface/transformers/blob/9d9b872b66f9ab9b7b7c73f2c00985dd92c4121b/src/transformers/tokenization_utils.py#L310

You can see that the whitespace is now only stripped when `all_special_tokens_extended.get(tok, None).lstrip)` is explicitly set to `True` which is not the default case for `gpt2`. So using these versions:

```
tokenizers.__version__: 0.8.0.rc3
```
and 
```
transformers.__version__: 3.0.1 (master)
```

I cannot reproduce the error. 

Could you update transformers via:

```
pip install transformers --upgrade
```

and check if the error persists? I think v3.0.0 should have fixed it :-) 

@patrickvonplaten Yes, I can see it is fixed in V3.0.0.
Thanks~