GPT2 MNLI training using run_glue.py
Author: nlp-student
Pull request: no

Running this on Google Colab,

```
!python run_glue.py \
  --model_name_or_path gpt2 \
  --task_name mnli \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_gpu_train_batch_size 10 \
  --gradient_accumulation_steps 32\
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir models/gpt2/mnli/
```

I get the following error,

```
 "Asking to pad but the tokenizer does not have a padding token. "
ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.
``` 
@LysandreJik : does the trainer need to be modified? Or am I supposed to be using additional commands for training using GPT2? (I have been following [this example](https://huggingface.co/transformers/v2.0.0/examples.html) using BERT)


Comments:
As explained in the documentation: "`run_glue.py`: This script can fine-tune the following models: BERT, XLM, XLNet and RoBERTa."

=> GPT-2 is a Transformer decoder, which can learn to generate text in an autoregressive way. It is not aimed at GLUE tasks, which are sequence classification tasks. 

Hi! Actually we've recently added `GPT2ForSequenceClassification` to enable support for sequence classification tasks (like GLUE). The support was added to enable some models such as EDIT: linked wrong model. Updated: [DialogRPT](https://huggingface.co/microsoft/DialogRPT-updown)!

However, as you have seen @nlp-student, the GPT-2 model isn't trainable out of the box with batch size > 1, as it has no padding token defined. Furthermore, MNLI is a three-way classification so you would need to set the number of labels appropriately.

I invite you to run the following script to create a model that you can then use with the `run_glue.py` script, initialized from the GPT-2 weights:
```py
from transformers import GPT2ForSequenceClassification, GPT2Tokenizer

model = GPT2ForSequenceClassification.from_pretrained("gpt2", num_labels=3)
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

# Define a padding token
tokenizer.pad_token = tokenizer.eos_token
model.config.pad_token_id = tokenizer.pad_token_id

# Save the model and tokenizer in a directory
model.save_pretrained("directory")
tokenizer.save_pretrained("directory")
```

Then you can launch the `run_glue.py` script by specifying that checkpoint.

However, as @NielsRogge pointed out, GPT-2 will not obtain as good results as a bi-directional encoders such as BERT.

Thank you so much for your responses. I will try that out.

Is GPT-2 generally expected to perform worse than BERT on sequence classification ? 

I've seen a lot of examples of BERT finetuned for classification, but not really for GPT-2. Is it not common (or that useful) to finetune GPT-2 for classification?

Thank you for any help or direction!

> Is GPT-2 generally expected to perform worse than BERT on sequence classification ?

Normally, yes. The reason for this is that GPT-2 is not really designed for such a task, whereas BERT is, as it has a special [CLS] token for classification tasks. GPT-2 is designed to process text autoregressively (i.e. left to right) in order to generate new text, whereas BERT is designed to process all tokens at once (hence creating a bidirectional representation of all input tokens), which is useful for tasks like sequence classification or extractive question answering for example. 

That said, you can still use GPT-2 to perform sequence classification. You can simply let GPT-2 process a sentence word by word from left to right, and then train it to predict the class of the sentence by placing a linear layer on top of the hidden representation of the final token of the sentence, which is done by looking at the [code](https://github.com/huggingface/transformers/blob/285c6262a84490270d2f1a1c06ee9ccfc1b60e8f/src/transformers/models/gpt2/modeling_gpt2.py#L1233) of `GPT2ForSequenceClassification`.

 So maybe an interesting thing to do is compare `BERTForSequenceClassification` and `GPT2ForSequenceClassification` on the same dataset, and see which one performs best. 

Many thanks for your detailed response! It was very helpful for my understanding.

You're welcome, I'm closing this issue! Feel free to reopen if you have other issues down the road.

> Hi! Actually we've recently added `GPT2ForSequenceClassification` to enable support for sequence classification tasks (like GLUE). The support was added to enable some models such as EDIT: linked wrong model. Updated: [DialogRPT](https://huggingface.co/microsoft/DialogRPT-updown)!
> 
> However, as you have seen @nlp-student, the GPT-2 model isn't trainable out of the box with batch size > 1, as it has no padding token defined. Furthermore, MNLI is a three-way classification so you would need to set the number of labels appropriately.
> 
> I invite you to run the following script to create a model that you can then use with the `run_glue.py` script, initialized from the GPT-2 weights:
> 
> ```python
> from transformers import GPT2ForSequenceClassification, GPT2Tokenizer
> 
> model = GPT2ForSequenceClassification.from_pretrained("gpt2", num_labels=3)
> tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
> 
> # Define a padding token
> tokenizer.pad_token = tokenizer.eos_token
> model.config.pad_token_id = tokenizer.pad_token_id
> 
> # Save the model and tokenizer in a directory
> model.save_pretrained("directory")
> tokenizer.save_pretrained("directory")
> ```
> 
> Then you can launch the `run_glue.py` script by specifying that checkpoint.
> 
> However, as @NielsRogge pointed out, GPT-2 will not obtain as good results as a bi-directional encoders such as BERT.

Hi,  met the same error as the original post although but I've set the per_device_train_batch_size=1. 
I was running run_swag.py with GPT2 (I added a class GPT2ForMultipleChoice referring to BertForMultipleChoice), could you offer some help with this problem?　Thanks a lot.

I follow the suggestion but got this error:

RuntimeError: Error(s) in loading state\_dict for GPT2ForSequenceClassification: size mismatch for score.weight: copying a param with shape torch.Size(\[100, 768\]) from checkpoint, the shape in current model is torch.Size(\[2, 768\]). You may consider adding \`ignore\_mismatched\_sizes=True\` in the model \`from\_pretrained\` method.

I then tried to flag `ignore_mismatched_sizes` as True like below but still, get the same error. 

```python
from transformers import GPT2ForSequenceClassification, GPT2Tokenizer

model = GPT2ForSequenceClassification.from_pretrained("gpt2", num_labels=100, ignore_mismatched_sizes=True)
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

# Define a padding token
tokenizer.pad_token = tokenizer.eos_token
model.config.pad_token_id = tokenizer.pad_token_id

# Save the model and tokenizer in a directory
model.save_pretrained("directory")
tokenizer.save_pretrained("directory")
```