GPT2: how to construct batch for Language Modeling
Author: cbaziotis
Pull request: no
Labels: name: wontfix

I am a little confused about how to prepare input bathces for GPT2LMHeadModel. I want to use GPT2 as an LM. For instance, I want to generate probability distributions over the vocabulary at each timestep, as well as computing the perplexities of sentences. It is important to note that I am working with sentences and not documents, so I will have to pad the inputs in the batch.

```python
from transformers import GPT2Tokenizer, GPT2LMHeadModel

# Prepare model
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()
model.to('cuda')

# input sentences
batch = ['this is a sentence.',
        'this is another sentence.',
        'this is another even longer sentence.']
```

## Question 1: Special tokens
a) Do I have to add a bos token id on my own or is it handled internally by GPT2Tokenizer? Same for the eos token.
```
# tokenize
tokens = [tokenizer.encode(x) for x in batch]

# add BOS and EOS
tokens = [[tokenizer.bos_token_id] + x + [tokenizer.eos_token_id]  for x in tokens]
```
```
[[50256, 428, 318, 257, 6827, 13, 50256],
 [50256, 428, 318, 1194, 6827, 13, 50256],
 [50256, 428, 318, 1194, 772, 2392, 6827, 13, 50256]]
```
b) `tokenizer.encode(x)` gives me a warning "This tokenizer does not make use of special tokens. Input is returned with no modification." I replaced it with `tokenizer.convert_tokens_to_ids(tokenizer.tokenize(x, add_prefix_space=True))` and the warning went away, but I am not sure what is the difference. Which tokenization should I use?

c) By looking at the properties of an instance of GPT2Tokenizer, I see that `bos_token` and `eos_token` are the same. Is this correct?


## Question 2: Padding
I want to pad based on the longest sentence in the batch. This is how I usually do it.
```
batch = pad_sequence([torch.LongTensor(x) for x in tokens], batch_first=True).to('cuda')
```
```
tensor([[50256,   428,   318,   257,  6827,    13, 50256,     0,     0],
        [50256,   428,   318,  1194,  6827,    13, 50256,     0,     0],
        [50256,   428,   318,  1194,   772,  2392,  6827,    13, 50256]])
```
a) What id does the model expect for the padded tokens? Do I have to pass the token id as an argument to the model or the tokenizer or you have a predifined one?


## Question 3: Model input
How is GPT2 made aware of the padded steps? For instance for an RNN I would do something like this:
```
lengths = (batch != 0).sum(-1) # tensor([7, 7, 9])
packed = pack_padded_sequence(x, lengths, batch_first=True)
out_packed, hn = rnn(packed)
```
but for GPT2 I havent found an example. The only ones i found are with batch size 1. So something like this wont work as expected:
```
outputs = model(x, labels=x)  # labels are shifted inside the model, right?
loss, logits = outputs[:2]
```
---
# Update
So there are still things unclear, but from reading other issues this is my current understanding:
 - GPT2 has no padding token, as it was trained on documents and not sentences.
 - In order to use GPT2 with variable length inputs, we can apply padding with an arbitrary token and ensure that those tokens are not used by the model with an `attention_mask`.
 - As for the labels, we should replace **only** on the `labels` variable the padded token ids with `-1`. 
So based on that, here is my current toy implementation:
```python
inputs = [
    'this is a sentence.',
    'this is another sentence.',
    'this is another even longer sentence.', ]

# tokenize
# tokens = [tokenizer.encode(x) for x in batch]
tokens = [tokenizer.convert_tokens_to_ids(
    tokenizer.tokenize(x, add_prefix_space=True))
    for x in inputs]

# add BOS and EOS
tokens = [[tokenizer.bos_token_id] + x + [tokenizer.eos_token_id]
          for x in tokens]

# padding_value can be whatever...
inputs = pad_sequence([torch.LongTensor(x) for x in tokens], batch_first=True, padding_value=0).to('cuda')
# 1 for real tokens and 0 for padded tokens
mask = (inputs != 0).float()
# replace the ids of the padded tokens (where token_id==padded_id) with `-1`
labels = inputs.masked_fill(inputs == 0, -1)

outputs = model(inputs, attention_mask=mask, labels=labels)
loss, logits = outputs[:2]
```
Is this correct??

### Bug: Padded tokens are not excluded from the loss 
However, I computed the loss on my own and found your implementation does not take into account the padded tokens when averaging, unless I am missing something
```python
_logits = logits.view(-1, logits.size(-1))  # flatten logits
_labels = torch.cat([inputs[:, 1:], inputs[:, :1] * 0], dim=1).view(-1) # shift inputs one position to the left and flatten
loss_real_avg = F.cross_entropy(_logits, _labels,  ignore_index=0, reduction='sum') / mask.sum() # ignore padded timesteps
loss_naive_avg = F.cross_entropy(_logits, _labels,  ignore_index=0, reduction='mean')

print("GPT2 loss:", loss.item())
print("loss_naive_avg:", loss_naive_avg.item())
print("loss_real_avg:", loss_real_avg.item())
```
```
GPT2 loss: 4.664564609527588
loss_naive_avg: 4.664564609527588
loss_real_avg: 4.056143283843994
```

Comments:
> I am a little confused about how to prepare input bathces for GPT2LMHeadModel. I want to use GPT2 as an LM. For instance, I want to generate probability distributions over the vocabulary at each timestep, as well as computing the perplexities of sentences. It is important to note that I am working with sentences and not documents, so I will have to pad the inputs in the batch.
> 
> ```python
> from transformers import GPT2Tokenizer, GPT2LMHeadModel
> 
> # Prepare model
> tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
> model = GPT2LMHeadModel.from_pretrained('gpt2')
> model.eval()
> model.to('cuda')
> 
> # input sentences
> batch = ['this is a sentence.',
>         'this is another sentence.',
>         'this is another even longer sentence.']
> ```
> 
> ## Question 1: Special tokens
> a) Do I have to add a bos token id on my own or is it handled internally by GPT2Tokenizer? Same for the eos token.
> 
> ```
> # tokenize
> tokens = [tokenizer.encode(x) for x in batch]
> 
> # add BOS and EOS
> tokens = [[tokenizer.bos_token_id] + x + [tokenizer.eos_token_id]  for x in tokens]
> ```
> 
> ```
> [[50256, 428, 318, 257, 6827, 13, 50256],
>  [50256, 428, 318, 1194, 6827, 13, 50256],
>  [50256, 428, 318, 1194, 772, 2392, 6827, 13, 50256]]
> ```
> 
> b) `tokenizer.encode(x)` gives me a warning "This tokenizer does not make use of special tokens. Input is returned with no modification." I replaced it with `tokenizer.convert_tokens_to_ids(tokenizer.tokenize(x, add_prefix_space=True))` and the warning went away, but I am not sure what is the difference. Which tokenization should I use?
> 
> c) By looking at the properties of an instance of GPT2Tokenizer, I see that `bos_token` and `eos_token` are the same. Is this correct?
> 
> ## Question 2: Padding
> I want to pad based on the longest sentence in the batch. This is how I usually do it.
> 
> ```
> batch = pad_sequence([torch.LongTensor(x) for x in tokens], batch_first=True).to('cuda')
> ```
> 
> ```
> tensor([[50256,   428,   318,   257,  6827,    13, 50256,     0,     0],
>         [50256,   428,   318,  1194,  6827,    13, 50256,     0,     0],
>         [50256,   428,   318,  1194,   772,  2392,  6827,    13, 50256]])
> ```
> 
> a) What id does the model expect for the padded tokens? Do I have to pass the token id as an argument to the model or the tokenizer or you have a predifined one?
> 
> ## Question 3: Model input
> How is GPT2 made aware of the padded steps? For instance for an RNN I would do something like this:
> 
> ```
> lengths = (batch != 0).sum(-1) # tensor([7, 7, 9])
> packed = pack_padded_sequence(x, lengths, batch_first=True)
> out_packed, hn = rnn(packed)
> ```
> 
> but for GPT2 I havent found an example. The only ones i found are with batch size 1. So something like this wont work as expected:
> 
> ```
> outputs = model(x, labels=x)  # labels are shifted inside the model, right?
> loss, logits = outputs[:2]
> ```
> 
> # Update
> So there are still things unclear, but from reading other issues this is my current understanding:
> 
> * GPT2 has no padding token, as it was trained on documents and not sentences.
> * In order to use GPT2 with variable length inputs, we can apply padding with an arbitrary token and ensure that those tokens are not used by the model with an `attention_mask`.
> * As for the labels, we should replace **only** on the `labels` variable the padded token ids with `-1`.
>   So based on that, here is my current toy implementation:
> 
> ```python
> inputs = [
>     'this is a sentence.',
>     'this is another sentence.',
>     'this is another even longer sentence.', ]
> 
> # tokenize
> # tokens = [tokenizer.encode(x) for x in batch]
> tokens = [tokenizer.convert_tokens_to_ids(
>     tokenizer.tokenize(x, add_prefix_space=True))
>     for x in inputs]
> 
> # add BOS and EOS
> tokens = [[tokenizer.bos_token_id] + x + [tokenizer.eos_token_id]
>           for x in tokens]
> 
> # padding_value can be whatever...
> inputs = pad_sequence([torch.LongTensor(x) for x in tokens], batch_first=True, padding_value=0).to('cuda')
> # 1 for real tokens and 0 for padded tokens
> mask = (inputs != 0).float()
> # replace the ids of the padded tokens (where token_id==padded_id) with `-1`
> labels = inputs.masked_fill(inputs == 0, -1)
> 
> outputs = model(inputs, attention_mask=mask, labels=labels)
> loss, logits = outputs[:2]
> ```
> 
> Is this correct??
> 
> ### Bug: Padded tokens are not excluded from the loss
> However, I computed the loss on my own and found your implementation does not take into account the padded tokens when averaging, unless I am missing something
> 
> ```python
> _logits = logits.view(-1, logits.size(-1))  # flatten logits
> _labels = torch.cat([inputs[:, 1:], inputs[:, :1] * 0], dim=1).view(-1) # shift inputs one position to the left and flatten
> loss_real_avg = F.cross_entropy(_logits, _labels,  ignore_index=0, reduction='sum') / mask.sum() # ignore padded timesteps
> loss_naive_avg = F.cross_entropy(_logits, _labels,  ignore_index=0, reduction='mean')
> 
> print("GPT2 loss:", loss.item())
> print("loss_naive_avg:", loss_naive_avg.item())
> print("loss_real_avg:", loss_real_avg.item())
> ```
> 
> ```
> GPT2 loss: 4.664564609527588
> loss_naive_avg: 4.664564609527588
> loss_real_avg: 4.056143283843994
> ```

What you proposed seems a valid walk-around to me. Also, look at #1464, which talked about adding `pad_token` to `tokenizer` and `embedding`. Perhaps that will help as well.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Check out [#3311](https://github.com/huggingface/transformers/issues/3311#issuecomment-601264426). GPT2 doesn't add BOS or EOS token, you must do it manually or use a tokenizer to do so. 

> Bug: Padded tokens are not excluded from the loss
> 
> However, I computed the loss on my own and found your implementation does not take into account the padded tokens when averaging, unless I am missing something
> ```py
> _logits = logits.view(-1, logits.size(-1))  # flatten logits
> _labels = torch.cat([inputs[:, 1:], inputs[:, :1] * 0], dim=1).view(-1) # shift inputs one position to the left and flatten
> loss_real_avg = F.cross_entropy(_logits, _labels,  ignore_index=0, reduction='sum') / mask.sum() # ignore padded timesteps
> loss_naive_avg = F.cross_entropy(_logits, _labels,  ignore_index=0, reduction='mean')
> print("GPT2 loss:", loss.item())
> print("loss_naive_avg:", loss_naive_avg.item())
> print("loss_real_avg:", loss_real_avg.item())
> GPT2 loss: 4.664564609527588
> loss_naive_avg: 4.664564609527588
> loss_real_avg: 4.056143283843994

For this bug, you may need to set `ignore_index` to -1 instead of 0 in `F.cross_entropy` according to this line:
> ```py
> # replace the ids of the padded tokens (where token_id==padded_id) with `-1`
> labels = inputs.masked_fill(inputs == 0, -1)
> ```