GPT2 lm_labels masking using (-1) throws an index out of range
Author: adigoryl
Pull request: no
Labels: name: wontfix

I am fine-tuning GPT2 model using the LMHead with a small number of special tokens. 
GPT2 underlying transformer takes the whole input at once, thus, it's important to pad inputs of varying lengths to a fixed length. The GPT2 model library offers -1 to be used as the padding value:

> lm_labels: optional language modeling labels: torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [-1, 0, ..., vocab_size]. All labels set to -1 are ignored (masked), the loss is only computed for the labels set in [0, ..., vocab_size].

**When I pad the lm_labels using -1's, the library throws an error.** On the other hand, using any other positive value for the masking works but the value acts like a vocabulary piece and thus makes the input incorrect. 

The other issue with **GPT2 is that the position_ids are compulsory**, whereas, the docs say they are optional.

Having an encoded and padded dataset like (not working case):

```
input_ids = torch.tensor([
    [[50257, 1212, 318, 43086, 2420, 2420, 2420, 50257]],
    [[50257, 1212, 318, 43086, 2420, 50257, 0, 0]]
])

position_ids = torch.tensor([
    [[1, 1, 1, 1, 1, 1, 1, 1]],
    [[1, 1, 1, 1, 1, 1, 0, 0]]
])

lm_labels = torch.tensor([
    [[1212, 318, 43086, 2420, 2420, 2420, 50257, 50257]],
    [[1212, 318, 43086, 2420, 2420, 2420, -1, -1]]
])

# Changing -1 padding makes the code work but also makes the input incorrect.
lm_labels = torch.tensor([
    [[1212, 318, 43086, 2420, 2420, 2420, 50257, 50257]],
    [[1212, 318, 43086, 2420, 50257, 50257, 0 , 0]]
])
```

@tholor Could you or someone from the team please fix the masking issue.

The full errors trace:

>   File "/Users/aw678/PycharmProjects/BERT/gpt2_simplified.py", line 181, in main
    losses, past = model(input_ids, position_ids, lm_labels, past=past)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling_gpt2.py", line 661, in forward
    hidden_states, presents = self.transformer(input_ids, position_ids, token_type_ids, past)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling_gpt2.py", line 590, in forward
    token_type_embeds = self.wte(token_type_ids)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 118, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/Users/aw678/anaconda3/envs/pytorch_conda_env/lib/python3.6/site-packages/torch/nn/functional.py", line 1455, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range at /Users/soumith/mc3build/conda-bld/pytorch_1549593514549/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:191


Comments:
Maybe you get the error because of the position_ids that are most likely wrong. 

I believe positional ids are not needed - you can use this:
predictions, past = model(tokens_tensor,position_ids=None token_type_ids=None, lm_labels=None, past=None)

and the use the parameters you want at the place you want or leave None if you do not want to use anything.



With zeros, it is not padded, zeros are "!" not "[PAD]"

tokenizer.convert_tokens_to_ids('[PAD]') this results in 0. I am confused why it happens

Really? When I use decode tokenizer.decode(0) results in !.

tokenizer.encode(x) = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(x)) [or should be], but when i try to use tokenizer.tokenize or tokenizer.convert_tokens_to_ids I am not sure which one, I get that its nto defined even tokenizer.encode works properly.

Does the 0 index actually stand for unknown tokens as well?
`tokenizer.convert_tokens_to_ids(["qrz"]) ` where "qrz" is supposed to be an unknown word. This will give [0]
But `tokenizer.convert_ids_to_tokens([0])` gives ["!"].


That strange - we together  get that ! == [PAD]

There was some issue fine-tuning GPT-2 with the master branch. This should now be fixed with the merge of #560 (which also add access to GPT-2 medium model).

But to actually answer the discussion on this issue (sorry I was reading to quickly), there is no padding token in GPT-2 vocabulary.
So either you manage to not pad (which is how the model is pretrained) or you need to add a new token to the vocabulary using the special_token functions. This method explained for instance in our blog post on fine-tuning GPT/GPT-2 here: https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313

Thanks. So we need to add the special token and then fine tune it?




This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.