Expanding vocab size for GTP2 pre-trained model.
Author: adigoryl
Pull request: no
Labels: name: wontfix

About the aim: 
I am trying to fine-tune a model on an English lyrics dataset in order to capture a style of a specific genre. To do this, at the fine-tuning input step, I wrap the lyrics with a "special token", e.g. <genre_type_tag> Lyrics text <genre_type_tag>. This means that I have to expand the vocab size by the number of special tokens.

Issue:
Using GTP2 tokenizer, I find that I can easily expand the vocab by specifying the special tokens:
`tokenizer = GPT2Tokenizer.from_pretrained(args.model_name, special_tokens=special_tokens)`.

However, the problem arises when I try to run the input through the model and get the following error:
> return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range at /Users/soumith/mc3build/conda-bld/pytorch_1549593514549/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:191

Which I believe says that the vocab id of the special token that I am using is out of bound since the model has been pre-trained without the them.

On the other hands, using OpenAIGTP model, I can see that this problem is solved by an additional parameter at the initialisation which tells the model to expect a number of special tags:
`model = OpenAIGPTDoubleHeadsModel.from_pretrained(args.model_name, num_special_tokens=len(special_tokens))`

I was wondering whether and how I can achieve a similar effect using GTP2 since it doesn't have such a parameter option.
To work around this issue I tried to alter the config file created using:
`config = GPT2Config.from_json_file(output_config_file)`, however, this gave me more issues and I am not sure whether that is the correct way to do it. 

Kind regards.


Comments:
@thomwolf Could you or someone from your team point me in the right direction to get the gtp2 model running with a small number of newly defined special tokens?
Any help very appreciated as I really need to move on with my research project.

Hi @adigoryl, I'm adding this feature with PR #560

You can have a look.

It should be merged soon I guess.

Hi @thomwolf, first of all, I would like to thank you for the quick response and solution. I have had a look at the added lines and have replaced the 'modelling_gpt2.py' file in my pytorch_pretrained_bert lib. Running the code: `model = GPT2LMHeadModel.from_pretrained(args.model_name, num_special_tokens=len(special_tokens))` gives me:

> model = cls(config, *inputs, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'num_special_tokens'

I am not sure whether this happens because of the way I have updated my lib or there still is something missing.
What is the best way to update my lib with the freshly made changes?

--------------------------UPDATE-------------------------------
Copy and paste seemed to work. The problem was that I needed to add a new line after the updated code paste since python is space sensitive. Having to fix it the num_special_tokens works in an anticipated way as I can see in the debugger that it sets the n_special field and updates total_tokens_embeddings. However, having this all fixed I still end up with the same issue I started with:

> Traceback (most recent call last):
  File "/Users/aw678/PycharmProjects/BERT/gtp2_train_lyrics_LM_copy.py", line 202, in <module>
    main()
  File "/Users/aw678/PycharmProjects/BERT/gtp2_train_lyrics_LM_copy.py", line 178, in main
    losses, past = model(input_ids, lm_labels, past=past)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling_gpt2.py", line 661, in forward
    hidden_states, presents = self.transformer(input_ids, position_ids, token_type_ids, past)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling_gpt2.py", line 587, in forward
    position_embeds = self.wpe(position_ids)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 118, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/functional.py", line 1454, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range at /Users/soumith/mc3build/conda-bld/pytorch_1549593514549/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:191

Not sure why it complains about "position_ids" since they are not compulsory. I believe this may not be an issue with my code (just in case someone wants to have a look):
[gtp2_train_lyrics_LM_copy.pdf](https://github.com/huggingface/pytorch-pretrained-BERT/files/3131933/gtp2_train_lyrics_LM_copy.pdf)

If you could provide a simplified working example of running GPT2 with new tokens then this should resolve my issue.


To replicate the error use the simplified version:
[gpt2_simplified.py.zip](https://github.com/huggingface/pytorch-pretrained-BERT/files/3132183/gpt2_simplified.py.zip)


You have to install the repo from source from the PR branch (see the instructions in the readme to install from source and after cloning git checkout to the PR branch before installing).

If it looks too complicated maybe the best is to wait for the PR to be merged.

I have managed to update the library on my machine but it seems that there is an incompatibility in the lib code. If you could provide a working toy example on how to fine-tune GPT2 with special symbols then I am sure the community would appreciate it and my issue would be resolved. I have attached such toy example above in the zip file, however, it has an issue which I believe is caused by the lib.

I am sorry to bother you so much but I just want to get on with my work. 
Regards, Adrian.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.