Error while adding new tokens to GPT2 tokenizer
Author: ZHAOTING
Pull request: no

A **NoneType Error** is encountered when I call `add_tokens()` to add new tokens to **GPT2 tokenizer** and the error is as following:

~~~~
File ".../pytorch_transformers/tokenization_utils.py", line 311, in add_tokens
    if self.convert_tokens_to_ids(token) == self.convert_tokens_to_ids(self.unk_token):
File ".../pytorch_transformers/tokenization_utils.py", line 381, in convert_tokens_to_ids
    for token in tokens:
TypeError: 'NoneType' object is not iterable
~~~~

The error comes from that it checks if the word id of a new `token` equals to that of the `unk_token` (as in tokenization_utils.py, line 311) but a **GPT2 tokenizer**'s `unk_token` is `None`. Therefore, the error happens when it tries iterating over `unk_token` in `convert_tokens_to_ids()` (as in tokenization_utils.py, line 381).

I think we can solve it by either 
1) changing the if-condition in line 311 and not checking the new token's equality to `unk_token` (I also don't quite understand the logic behind checking the equality here) 
or 
2) dealing with `None` input in `convert_tokens_to_ids(self, tokens)`so it returns `[]` if `tokens is None`.

Comments:
Added:

I also found that `_convert_token_to_id()` in tokenization_gpt2.py (line 182) uses `unk_token`, which is initially `None` in GPT2 tokenizer. This line of code can also lead to bugs.
~~~~
def _convert_token_to_id(self, token):
    """ Converts a token (str/unicode) in an id using the vocab. """
    if token in self.encoder:
        return self.encoder.get(token)
    return self.encoder.get(self.unk_token)
~~~~

Indeed, GPT-2 doesn't have a `unk_token` since it's supposed to be able to encode any string but this does have some unintended consequences since we also use the fact that a tokenizer returns the `unk_token` to check whether a token is in the current vocabulary or not.

I'll see how we can update this in the most coherent way. Probably mapping the special `
` token as `unk_token` and using the same logic as the other models (returning the `unk_token` when the token is not in the vocabulary) is the simplest way to fix it.

What will be the quick fix for this? When running `run_generation.py` in examples, I could resolve the `None' error by adding special tokens to the tokenizer like below:

` special_tokens = {"cls_token":"[CLS]", "unk_token":"[UNK]"} `
` tokenier = tokenizer_class.from_pretrained("gpt2", cls_token="[CLS]", unk_token="[UNK]")`
` tokenizer.add_special_tokens(special_tokens) `

But then, I got a CUDA error (probably) due to the different embedding size of the model and tokenizer. So, I resized the model's token embedding size from 50257 to 50259 like below:

` model.resize_token_embeddings(len(tokenizer)) `

Then, it tokenizes the tokens correctly with the additional token encoders that have `tokenizer.added_tokens_encoder.keys()` with [CLS] and [UNK]. But, regardless of input, the gpt2 output seems to be wrong: a sequence [CLS] [CLS] ...

![image](https://user-images.githubusercontent.com/2631683/61899403-5e0c7b00-aee9-11e9-9f5e-c578ffd80d78.png)
![image](https://user-images.githubusercontent.com/2631683/61899547-b17ec900-aee9-11e9-93b7-8befc6233875.png)







@dykang 
A quick fix would be cleaning the generated sentence by replacing unwanted tokens as "".
However, it is reasonable that the GPT2 outputs weird sentences given inputs including "[CLS]" as in your example because the new word embeddings are not trained. 

Thanks, @ZHAOTING. However, GH-910 seems to only add unk token though. @thomwolf, would the PR be generalizable to add any special tokens as my earlier comment above? When special tokens are added, how do existing pre-trained gpt2 models work properly? 

Adding some context.  Is it possible to add [CLS] and [SEP] tokens to gpt2-medium in a non destructive way.  After finetuning a bit following the structure indicated here: 
```
    # (Default, BERT/XLM pattern): [CLS] + A + [SEP] + B + [SEP]
    # (XLNet/GPT pattern): A + [SEP] + B + [SEP] + [CLS]
```
it is clear that GPT2 will attempt to predict the BPE tokens for [CLS] as ` [C LS]`, but adding the [SEP] and [CLS] tokens produce only an output of [CLS] / [SEP] tokens despite any top_k, top_p, or temperature settings.




I had a similar problem.

I tried to finetune a gpt2 model using the simpletransformers library. 

The error seems to originate from
"/usr/local/lib/python3.6/site-packages/transformers/optimization.py"
This is the file where the optimizer AdamW is implemented.

In my case, I made a backup of the 
"scheduler.pt" and "optimizer.pt" 
files in my saved checkpoint.

In file 
"/usr/local/lib/python3.6/site-packages/simpletransformers/language_modeling/language_modeling_model.py"
the optimizer and scheduler were also loaded from the checkpoint but this broke the code.
If these two files cannot be found in the path then your code will proceed with the rest of it.

This however remains a bug as your new optimizer and scheduler will start from scratch ignoring any "knowledge" they already contained about your previous optimization.