Pad token for GPT2 and OpenAIGPT models
Author: dakshvar22
Pull request: no

## ❓ Questions & Help

I noticed that out of all the models `pad_token` is not set for only `OpenAIGPTModel` and `GPT-2Model`. 
I get a warning: `Using pad_token, but it is not set yet.` and `pad_token_id` is `None`

Is there any specific reason why is that so?

If not, what is the appropriate padding token to be used for these models?

Thanks


Comments:
Padding tokens were not used during the pre-training of GPT and GPT-2, therefore they have none. It shouldn't matter as when doing padding, you should specify an [attention mask](https://huggingface.co/transformers/glossary.html#attention-mask) to your model so that it doesn't attend to padded indices, therefore ignoring the value of the token.

i got the same issues any advice?

Also look at issue #3021 

What do you need the padding for? What is the use case? 
For both models using an attention mask over all tokens that you be padded should help (as explained above). 

Yes, using the attention mask over all tokens should help. Thanks

> Padding tokens were not used during the pre-training of GPT and GPT-2, therefore they have none. It shouldn't matter as when doing padding, you should specify an [attention mask](https://huggingface.co/transformers/glossary.html#attention-mask) to your model so that it doesn't attend to padded indices, therefore ignoring the value of the token.

I thought the same as your reply but my experiments shows this Attention mask does not work. 
See my recent [issue](https://github.com/huggingface/transformers/issues/3167), where i provided reproducible code to see my point. 


should the attention mask cover the labels as well?

for example i want to train "some passage <break> some content <pad> <pad>". 

so my input would be "some passage <break>", and my label would be "some passage <break> some content <pad> <pad>", in which the padding is necessary for batch processing.

In such a case how do I mask out the paddings in the labels?


Because GPT2 and GPT are causal LM you don't need to pad shorter sentences in batches. It is important though that the loss on these "unnecessary" tokens is not calculated. You should set all lables corresponding to "PADDED" tokens to `-100`. In the code snippet you can see in the `map_to_encoder_decoder_inputs` function how the `labels` are set to -100 for `attention_mask = 0`:
https://huggingface.co/patrickvonplaten/bert2gpt2-cnn_dailymail-fp16#training-script

> Because GPT2 and GPT are causal LM you don't need to pad shorter sentences in batches. 
    Why? The pad is to make up the length of the batch. Does this have anything to do with GPT2's causal model? 

> ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.

Just do `tokenizer.pad_token = tokenizer.eos_token`, and also set `tokenizer.padding_side = 'left'`.
It should work fine with batches. No need of `add_special_tokens`, otherwise the model embedding layer should be resized accordingly.



@ecolss  Why do we set 
```bash
tokenizer.padding_size = 'left'
```
. What is the problem if it stays as 'right' which is by default.
Thank you.

Specially when looking at these remarks: https://huggingface.co/docs/transformers/v4.30.0/en/model_doc/gpt2#transformers.GPT2Config

```
Tips:

- GPT-2 is a model with absolute position embeddings so it’s usually advised to pad the inputs on the right rather than the left.
```

> > ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.
> 
> Just do `tokenizer.pad_token = tokenizer.eos_token`, and also set `tokenizer.padding_side = 'left'`. It should work fine with batches. No need of `add_special_tokens`, otherwise the model embedding layer should be resized accordingly.

This does not seem consistent with the document  ? As mentioned by @roberth-plutoflume 

I am also facing this issue. Any update on it?
```
ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.
```

This happens when following t[he official example](https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-recognition/README.md#whisper-model) using GPT2 as a decoder (Warm Start of Speech Encoder Decoder Models)