GPT2Tokenizer 

Author: latkins
Pull request: no
Labels: name: Discussion

I am confused as to how the GPT2Tokenizer is intended to be used. It looks like `GPT2Tokenizer.encode` doesn't always take the byte pair encoding into account  -- is this intentional?

```
from pytorch_pretrained_bert import GPT2Tokenizer

tok = GPT2Tokenizer.from_pretrained("gpt2")

print(tok.encode("
"))
print(tok.encoder["
"])
```

Running the above gives:
```
[27, 91, 437, 1659, 5239, 91, 29]
50256
```
I would have (perhaps naively) expected that `tok.encode` gives `[50256]`. 

Comments:
`"
"` is a *special token* that is not intended to be feed through the tokenizer but added to the indices list after the tokenization process (see for example the way it is used in the short example [`run_gpt2.py`](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_gpt2.py#L98)).

It's common practice to distinguish the way we write *special tokens* for the constrains/behavior of the tokenizers as they (i) become unnecessary complex when we add exception rules for special tokens and (ii) are often designed externally (ex when you use SpaCy's tokenizer). The simplest way to do that is just to add the special tokens after the encoding process like it's done for GPT-2.

That makes sense! Is this in the documentation anywhere? I didn't come across it. Regardless, thanks for the clarification.

@thomwolf
I think after the new repo migration, you guys made '
' as first class citizen of GPT2 tokenizer. So now we can just have base inputs with this token and it'd work fine with tokenizer and model, right?

Note that in the new version of Transformer, the behavior of the GPT2 tokenizer changed. 

```python
text = "
 machine learning using PyTorch and TensorFlow"
tokenizer_folder = "gpt2"
tokenizer = GPT2Tokenizer.from_pretrained(tokenizer_folder)
encoded_ids = tokenizer.encode(text)
print(encoded_ids)
print("################### when tokenizing the whole sentence ######################")
print(encoded_ids)
for id in encoded_ids:
    print(
        f"""id: {id:<7}, original_token: {"'" + tokenizer.decoder[id] + "'":<10}, ignore_special_token: '{tokenizer.decode(id)}'""")
# endfor
```

output: 
```text
################### when tokenizing the whole sentence ######################
[50256, 4572, 4673, 1262, 9485, 15884, 354, 290, 309, 22854, 37535]
id: 50256  , original_token: '
', ignore_special_token: '
'
id: 4572   , original_token: 'Ġmachine', ignore_special_token: ' machine'
id: 4673   , original_token: 'Ġlearning', ignore_special_token: ' learning'
id: 1262   , original_token: 'Ġusing'  , ignore_special_token: ' using'
id: 9485   , original_token: 'ĠPy'     , ignore_special_token: ' Py'
id: 15884  , original_token: 'Tor'     , ignore_special_token: 'Tor'
id: 354    , original_token: 'ch'      , ignore_special_token: 'ch'
id: 290    , original_token: 'Ġand'    , ignore_special_token: ' and'
id: 309    , original_token: 'ĠT'      , ignore_special_token: ' T'
id: 22854  , original_token: 'ensor'   , ignore_special_token: 'ensor'
id: 37535  , original_token: 'Flow'    , ignore_special_token: 'Flow'
```


Please I need your help with the special token 

Hello to everyone,
 I would like someone to clarify/disaprove the following. I have found a pretrained gpt2 model trained on the Greek language from huggingface named nikokons/gpt2-greek and I want to fine tune it on my custom dataset. My dataset consists of samples of mathematical definitions with related questions written in the Greek language. Let me give some translated examples

Definition: Two angles are called complementary angles when they sum up to 180 degrees.
Question: What are complementary angles?

Definition: Two angles are called complementary angles when they sum up to 180 degrees.
Question: How do we call two angles which sum up to 180 degrees?

Definition: A triangle is called isosceles when it has two sides of equally length.
Question: What is an isosceles triangle?

Definition: A triangle is called isosceles when it has two sides of equally length.
Question: What do we call a triangle which has two equally in length sides?

Notice that for a Definition I might have multiple questions on my dataset. I want to fine tune the model in order to learn to answer the user’s question by answering to the user with the entire Definition related to the user’s question.

What are the steps I should follow?
First fine tune the model to the raw dataset ( I mean the dataset without special tokens) in order to learn the new terminology and then preprocess the dataset in order to add in the beginning and at the ending of each sample the
|endoftext| token and finetune the model again on the new preprocessed dataset?

the processed dataset would be like following?

|endoftext| A triangle is called isosceles when it has two sides of equally length. What is an isosceles triangle? |endoftext|
Two angles are called complementary angles when they sum up to 180 degrees.
How do we call two angles which sum up to 180 degrees?|endoftext|

Also should I use padding=right when tokenizing the samples?



Hi @redrocket8, thanks for participating on this issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

Ok I will delete it. Thanks