Hugging Face GPT-2 Tokenizer
Author: h56cho
Pull request: no

Hello,

I know that if I choose to add any new "special token" onto the pre-made GPT-2 tokenizer, and if I want to use the pre-trained GPT-2 model for my analysis, I will need to re-train the pre-trained GPT-2 to make the model learn that new special token.

But what if I just add an extra non-special token? for example, a word "paradox" is not included in the existing GPT-2 tokenizer, so say I add the word "paradox" to the existing set of GPT-2 vocabulary, like below:

```python
# load the pre-trained GPT2-tokenizer
gpt2_tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

# adding a new word (not special token) to the existing vocabulary,
# but I am not making any changes to the pre-assigned special tokens
gpt2_tokenizer.add_tokens("paradox")

# get the pre-trained HuggingFace GPT2DoubleHeadsModel 
model_gpt2DoubleHeadsModel = GPT2DoubleHeadsModel.from_pretrained('gpt2', output_hidden_states = True)

# resize the token embeddings
# (not sure what this function does)
model_gpt2DoubleHeadsModel.resize_token_embeddings(len(gpt2_tokenizer))
```
Given that I didn't make any changes to the special tokens in the GPT-2-tokenizer, do I still need to train the already pre-trained `GPT2DoubleHeadsModel` before I start using it,  just because I added a new word to the set of vocabulary?

Thank you,

Comments:
You would need to fine-tune your GPT-2 model on a dataset containing the word, yes. The reason being that your model needs to understand in which context is the word used, what it means, etc.

Hello,

Thank you for your reply.
So I have a set of multiple choice questions, and when I use the `add_tokens` function to add whichever the tokens from the dataset that are not originally included in the GPT-2 tokenizer, the length of my GPT-2 tokenizer jumps up by ~3000 (so my dataset contains 3000 new tokens)

even if I fine tune the pre-trained GPT-2 model on a portion of my dataset (say), I won't be able to train the pre-trained model on all of the new tokens. So I am not sure how I will go about this.

What I am trying to do is though, I want the pre-trained `GPT2DoubleHeadsModel` to solve a set of multiple-choice questions, and I want to compare the error rates generated by the hidden outputs of each of the 12 layers when they are fed directly into the multiple-choice head of the model. That is, my goal is not to minimize the overall error rate of the GPT-2 model, my goal is to simply compare the error rates generated by the different layers of the model. Given this information, do I still need to fine-tune my GPT-2 model on all of the new tokens that I am adding?

Thank you,

If you don't fine-tune your model on the new tokens you're adding, then when the model sees it at inference it will be a completely unknown token, and the model probably won't handle it correctly.

If you don't have any training data to fine-tune the model, why don't you keep the tokens as they are? The GPT-2 tokenizer should be able to correctly tokenize them, as it's a byte level BPE. 

Hello,

Thank you again for your reply.
I think I want to add the word as a new token mainly because I do not want the word to be treated as the mere `<unk>` token.

I am not sure what byte level BPE means, but if I do not add the new words as extra tokens, would it really work just because the tokenizer is a byte level BPE?

Thank you :s

Byte level BPEs should be able to tokenize everything. The GPT-2 tokenizer has no unknown token for that reason.

You should try to tokenize your tokens to see if some come back as unknown, but it shouldn't with GPT-2 (and RoBERTa for that matter)!

Thank you! I was able to confirm that what you mentioned in your previous post also works for my case.