Fine-tuning GPT: problems with padding
Author: ioana-blue
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.4.0
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.5.1+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help

 @LysandreJik 
 tokenizers: @mfuntowicz
## Information

Model I am using openai-gpt:

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The scripts are my own scripts inspired by the glue examples.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

Simple binary text classification, nothing fancy, inspired by the glue example files. 

## To reproduce

Steps to reproduce the behavior:

As reported in other issues, padding is not done for GPT* models. One workaround for this issue is to set the padding token to the eos token. This seems to work fine for the GPT2 models (I tried GPT2 and DistilGPT2), but creates some issues for the GPT model. Comparing the outputs of the two models, it looks like the config file for the GPT2 models contains ids for bos and eos tokens, while these are missing from the GPT config file (not sure this is the real problem). 
Some other interesting bits from the outputs:

```
ftfy or spacy is not installed using BERT BasicTokenizer instead of SpaCy & ftfy.
Using eos_token, but it is not set yet.
```

Bottom line, it crashes with ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})` - despite the fact that I have `tokenizer.pad_token = tokenizer.eos_token` in the code.

I'm expecting some issue with the tokenizer/missing ids for the special tokens. Wondering if there is something missing in the config file for the model. 

## Expected behavior
No error? :) I don't see any of these issues after setting the padding token to the eos token for the GPT2 model. As I briefly mentioned above, the only difference that I see in the config file is the ids for the eos/bos tokens, which seem to be missing from the GPT model config. 

Thanks for your help!


Comments:
Another suspicious flag in the config for GPT is `predict_special_tokens` which is set to `True` (no such thing for GPT2 config). I did a grep on this flag and it seems to be present only in the config class and not used anywhere else. Somewhat strange. 

I might have found a problem in my scripts/code. I've been using BERT-based models so far and when examples are converted to features, the batch encoder is initialized with padding set to max_length; I'm trying to initialize it to do_not_pad. In theory, this should fix it. In practice... we shall see :) 

Indeed, the root of the issue seems to be that you're asking your tokenizer to pad the sequences, but it does not have a padding token, and therefore cannot do so.

If setting the tokenizer's pad token to the eos token doesn't work, you can try adding a new token to the tokenizer with the `add_special_tokens()` method, and then resize the model embedding layer.

Seeing as you should use the attention mask when padding, these tokens should have close to zero influence on your training.

See the docs about the aforementioned methods [here](https://huggingface.co/transformers/internal/tokenization_utils.html?highlight=resize_token_embeddings#transformers.tokenization_utils_base.SpecialTokensMixin.add_special_tokens)

Isn't it more straightforward to ask the tokenizer to not pad the sequence (for gpt* models)?
 
The confusion came from the fact that setting the padding token to eos works for GPT2* models (because eos is defined in the config of the pretrained model), but doesn't for GPT (because eos is not defined)

So no padding seems to work looking at a few samples (but no batching possible). I'll start a few training jobs, I'll know tomorrow if it really trained properly (large dataset). 

Yes, it is more straightforward, but as you've said, no batching can be made. This is quite limiting and would tremendously slow down the training; if your training is small enough then that might still be enough!

@LysandreJik It's slow indeed, but I think I can live with it. I can't recall what the problem was, even fro gpt2 where I could assigned pad = eos, I got an error when I tried to batch. 

Ah, this is weird. If you ever stumble upon this issue again, please let us know so that we may see what's wrong. Thanks!

To get GPT2 to work, you'll also need to update the config's pad token to be the eos token:
`config.pad_token_id = config.eos_token_id`

For example, in `examples/lightning_base.py`, I've added the below lines right after loading the tokenizer in BaseTransformer().\_\_init\_\_():
```py
        if self.tokenizer.pad_token is None:
            self.tokenizer.pad_token = self.tokenizer.eos_token
            self.config.pad_token_id = self.config.eos_token_id
```

@ethanjperez thanks for the tip, I'll give it a try!


I think this works. I managed to train gpt and gpt2. I have an issue during evaluation with gpt2, but I don't think it's related. Closing this one, thanks @ethanjperez 

```
        if self.tokenizer.pad_token is None:
            self.tokenizer.pad_token = self.tokenizer.eos_token
            self.config.pad_token_id = self.config.eos_token_id
```

Using this can train, but save will meet so many problems!

https://github.com/huggingface/transformers/issues/5571