BioGPT causal language model with unexpected error
Author: junoriosity
Pull request: no

### System Info

transformers==4.28.0

### Who can help?

@ArthurZucker @younesbelkada @gante 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
input_sequence = "Hello, I'm a language model,"

inputs = torch.as_tensor(tokenizer.encode(input_sequence)).unsqueeze(0).to(device)
past_key_values = None

count = 0
complete_token = []
with torch.no_grad():
    while count<10:
        count += 1
        print("Iteration no.: " + str(count))
        if count > 1:
            inputs = input_token

        model_out = model(input_ids=inputs.to(device), past_key_values=past_key_values)
        logits = model_out.logits[:, -1, :]
        past_key_values = model_out.past_key_values

        topk_values, topk_indices = torch.topk(logits, 5)

        log_probs = F.softmax(topk_values, dim=-1)
        inputs_in_topk = torch.multinomial(log_probs, num_samples=1, replacement=True)
        input_token = torch.gather(topk_indices, 1, inputs_in_topk)
        complete_token.append(input_token)
```

### Expected behavior

I am trying to use a Causal Language Model from BioGPT. However, I got a strange error.

Here are my steps:

First, I installed `transformers` and `sacremoses`:

```
!pip install transformers sacremoses -q
```

Then I executed the code from above.

And here is the error I got:

```
Iteration no.: 1
Iteration no.: 2
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_18990/2689790310.py in <cell line: 8>()
     13             inputs = input_token
     14 
---> 15         model_out = model(input_ids=inputs.to(device), past_key_values=past_key_values)
     16         logits = model_out.logits[:, -1, :]
     17         past_key_values = model_out.past_key_values

~/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1192         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194             return forward_call(*input, **kwargs)
   1195         # Do not call functions when jit is used
   1196         full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/transformers/models/biogpt/modeling_biogpt.py in forward(self, input_ids, attention_mask, head_mask, inputs_embeds, past_key_values, labels, use_cache, output_attentions, output_hidden_states, return_dict)
    677         return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    678 
--> 679         outputs = self.biogpt(
    680             input_ids,
    681             attention_mask=attention_mask,

~/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1192         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194             return forward_call(*input, **kwargs)
   1195         # Do not call functions when jit is used
   1196         full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/transformers/models/biogpt/modeling_biogpt.py in forward(self, input_ids, attention_mask, head_mask, inputs_embeds, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
    589                 )
    590             else:
--> 591                 layer_outputs = decoder_layer(
    592                     hidden_states,
    593                     attention_mask=attention_mask,

~/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1192         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194             return forward_call(*input, **kwargs)
   1195         # Do not call functions when jit is used
   1196         full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/transformers/models/biogpt/modeling_biogpt.py in forward(self, hidden_states, attention_mask, layer_head_mask, past_key_value, output_attentions, use_cache)
    313         self_attn_past_key_value = past_key_value[:2] if past_key_value is not None else None
    314         # add present self-attn cache to positions 1,2 of present_key_value tuple
--> 315         hidden_states, self_attn_weights, present_key_value = self.self_attn(
    316             hidden_states=hidden_states,
    317             past_key_value=self_attn_past_key_value,

~/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1192         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194             return forward_call(*input, **kwargs)
   1195         # Do not call functions when jit is used
   1196         full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/transformers/models/biogpt/modeling_biogpt.py in forward(self, hidden_states, key_value_states, past_key_value, attention_mask, layer_head_mask, output_attentions)
    211         if attention_mask is not None:
    212             if attention_mask.size() != (bsz, 1, tgt_len, src_len):
--> 213                 raise ValueError(
    214                     f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.size()}"
    215                 )

ValueError: Attention mask should be of size (1, 1, 0, 12), but is torch.Size([1, 1, 1, 1])
```

So apparently, everything went fine in the first execution, but the in the second model call this error came up.

Do you know how to fix this? 🙂

Comments:
Hey @junoriosity 👋 

Two notes:
1. It seems like you are trying to generate text using BioGPT. Have you seen our `.generate()` function? ([guide](https://huggingface.co/docs/transformers/generation_strategies), [blog post](https://huggingface.co/blog/how-to-generate)); If you still want to do it manually, you need to configure the attention mask, which is why you see the exception. The attention mask is expected to have the shape `[batch_size, seq_len]`, where `seq_len` is the number of all input tokens so far (including the ones in `past_key_values`). 
2. When you share a script in an open-source repository for the contributors to help and/or debug, ensure it is self-contained (including imports and, if needed, all the data). We have many requests for help, which we can only attend at a decent pace if you help us too 🤗 See the script below for an example of a complete stand-alone reproducer:

```python
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForCausalLM

device = "cuda"
tokenizer = AutoTokenizer.from_pretrained("microsoft/biogpt")
model = AutoModelForCausalLM.from_pretrained("microsoft/biogpt").to(device)

input_sequence = "Hello, I'm a language model,"

inputs = torch.as_tensor(tokenizer.encode(input_sequence)).unsqueeze(0).to(device)
past_key_values = None

count = 0
complete_token = []
with torch.no_grad():
    while count < 10:
        count += 1
        print("Iteration no.: " + str(count))
        if count > 1:
            inputs = input_token

        model_out = model(input_ids=inputs.to(device), past_key_values=past_key_values)
        logits = model_out.logits[:, -1, :]
        past_key_values = model_out.past_key_values

        topk_values, topk_indices = torch.topk(logits, 5)

        log_probs = F.softmax(topk_values, dim=-1)
        inputs_in_topk = torch.multinomial(log_probs, num_samples=1, replacement=True)
        input_token = torch.gather(topk_indices, 1, inputs_in_topk)
        complete_token.append(input_token)
```

@gante Sorry, I have used a Jupyter notebook and so the initial loading of libraries etc. is something I usually do in the first paragraphs - and I have overlooked it. My bad ... sorry. 🙂

Regarding the attention mask, this is very new to me. For instance, when I make use of the OPT models, I just have to enter the the `past_key_values` like before (essentially `model_out.past_key_values`) and everything is fine. 

I have no clue right now, where I can get the `attention_mask` from resp. how I can create such an object from scratch.
If you could help me here, that would be awesome. 🙂

@junoriosity The attention mask is simply an integer tensor with the same shape as the inputs, with `1` on real tokens and `0` on padding. In particular for the attention mask update at generation time, see [this line](https://github.com/huggingface/transformers/blob/006da469dd5a465f4551f4245f780e3b1e92b76c/src/transformers/generation/utils.py#L766). However, it assumes that a starting attention mask exists, which you can obtain from `tokenizer(input_sequence).attention_mask`. 

BTW, I would highly recommend using `.generate()` unless you are experimenting with new decoding strategies. There are many corner cases handled in there :)

@gante Many thanks for your suggestion. 🙂 Here is what I did now :

```
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForCausalLM

device = "cuda"
tokenizer = AutoTokenizer.from_pretrained("microsoft/biogpt")
model = AutoModelForCausalLM.from_pretrained("microsoft/biogpt").to(device)

input_sequence = "Hello, I'm a language model,"

inputs = torch.as_tensor(tokenizer.encode(input_sequence)).unsqueeze(0).to(device)
attention_mask = torch.as_tensor(tokenizer(input_sequence).attention_mask).unsqueeze(0).to(device)
past_key_values = None

count = 0
complete_token = []
with torch.no_grad():
    while count < 10:
        count += 1
        print("Iteration no.: " + str(count))
        if count > 1:
            inputs = input_token
        
        print(inputs.to(device))
        print(attention_mask)
        
        model_out = model(input_ids=inputs.to(device), attention_mask=attention_mask, past_key_values=past_key_values)
        logits = model_out.logits[:, -1, :]
        past_key_values = model_out.past_key_values

        topk_values, topk_indices = torch.topk(logits, 5)

        log_probs = F.softmax(topk_values, dim=-1)
        inputs_in_topk = torch.multinomial(log_probs, num_samples=1, replacement=True)
        input_token = torch.gather(topk_indices, 1, inputs_in_topk)
        attention_mask = torch.as_tensor([1]).unsqueeze(0).to(device)
        complete_token.append(input_token)
```

and the output is 

```
Iteration no.: 1
tensor([[   2,  313, 3666,  399,    7,  174, 4617,  659,   14, 2545,  144,    7]],
       device='cuda:0')
tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], device='cuda:0')
Iteration no.: 2
tensor([[8]], device='cuda:0')
tensor([[1]], device='cuda:0')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_11426/272587798.py in <cell line: 3>()
     11         print(attention_mask)
     12 
---> 13         model_out = model(input_ids=inputs.to(device), attention_mask=attention_mask, past_key_values=past_key_values)
     14         logits = model_out.logits[:, -1, :]
     15         past_key_values = model_out.past_key_values

~/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1192         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194             return forward_call(*input, **kwargs)
   1195         # Do not call functions when jit is used
   1196         full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/transformers/models/biogpt/modeling_biogpt.py in forward(self, input_ids, attention_mask, head_mask, inputs_embeds, past_key_values, labels, use_cache, output_attentions, output_hidden_states, return_dict)
    677         return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    678 
--> 679         outputs = self.biogpt(
    680             input_ids,
    681             attention_mask=attention_mask,

~/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1192         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194             return forward_call(*input, **kwargs)
   1195         # Do not call functions when jit is used
   1196         full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/transformers/models/biogpt/modeling_biogpt.py in forward(self, input_ids, attention_mask, head_mask, inputs_embeds, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
    589                 )
    590             else:
--> 591                 layer_outputs = decoder_layer(
    592                     hidden_states,
    593                     attention_mask=attention_mask,

~/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1192         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194             return forward_call(*input, **kwargs)
   1195         # Do not call functions when jit is used
   1196         full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/transformers/models/biogpt/modeling_biogpt.py in forward(self, hidden_states, attention_mask, layer_head_mask, past_key_value, output_attentions, use_cache)
    313         self_attn_past_key_value = past_key_value[:2] if past_key_value is not None else None
    314         # add present self-attn cache to positions 1,2 of present_key_value tuple
--> 315         hidden_states, self_attn_weights, present_key_value = self.self_attn(
    316             hidden_states=hidden_states,
    317             past_key_value=self_attn_past_key_value,

~/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1192         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194             return forward_call(*input, **kwargs)
   1195         # Do not call functions when jit is used
   1196         full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/envs/pytorch_p39/lib/python3.9/site-packages/transformers/models/biogpt/modeling_biogpt.py in forward(self, hidden_states, key_value_states, past_key_value, attention_mask, layer_head_mask, output_attentions)
    211         if attention_mask is not None:
    212             if attention_mask.size() != (bsz, 1, tgt_len, src_len):
--> 213                 raise ValueError(
    214                     f"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.size()}"
    215                 )

ValueError: Attention mask should be of size (1, 1, 0, 12), but is torch.Size([1, 1, 1, 1])
```

Do you know, how would I have to set the `attention_mask` in the iteration step?


@junoriosity 

We reserve these issues for bugs, as we don't have the capacity to provide hands-on support. I'm afraid you'll have to dig deeper in the code I shared, the answer is there :)

@gante I understand you well. However, from my understanding the `attention_mask` in the second run should not reflect anything related to the 12 tokens and instead just have length 1, if I just enter one token.

That is not correct -- you are passing 1 new token, and ~11~ N past tokens (cached in `past_key_values`) :)

@gante But the initial input is already 12 tokens
<img width="1103" alt="Bildschirm­foto 2023-05-10 um 16 20 06" src="https://github.com/huggingface/transformers/assets/5286536/42e9537b-dc28-47e3-9cee-551f23dbdbe6">
Hence, we would be at 13 tokens then ... or am I confusing something?

Updated the message above. You are right, not 11 tokens, but the exact number of tokens is not the important part here.

One issue remains, though: the exception is not correct in the presence of `past_key_values` and is very misleading 👀 The attention mask must be of shape `[batch_size, new+cached tokens]`, so the answer consists of concatenating `[[1]]` at the end of each iteration. I'll open a PR to fix the exception message.

In the end, there was indeed a bug, so here's a working solution. And pardon me for my pushback -- it's the only way we can keep replying to blocking issues at a good pace :)

```py
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForCausalLM

device = "cuda"
tokenizer = AutoTokenizer.from_pretrained("microsoft/biogpt")
model = AutoModelForCausalLM.from_pretrained("microsoft/biogpt").to(device)

input_sequence = "Hello, I'm a language model,"

inputs = torch.as_tensor(tokenizer.encode(input_sequence)).unsqueeze(0).to(device)
attention_mask = torch.as_tensor(tokenizer(input_sequence).attention_mask).unsqueeze(0).to(device)
past_key_values = None

count = 0
complete_token = []
with torch.no_grad():
    while count < 10:
        count += 1
        print("Iteration no.: " + str(count))
        if count > 1:
            inputs = input_token

        print(inputs.to(device))
        print(attention_mask)
        print(past_key_values[0][0].shape if past_key_values else None)

        model_out = model(input_ids=inputs.to(device), attention_mask=attention_mask, past_key_values=past_key_values)
        logits = model_out.logits[:, -1, :]
        past_key_values = model_out.past_key_values

        topk_values, topk_indices = torch.topk(logits, 5)

        log_probs = F.softmax(topk_values, dim=-1)
        inputs_in_topk = torch.multinomial(log_probs, num_samples=1, replacement=True)
        input_token = torch.gather(topk_indices, 1, inputs_in_topk)
        attention_mask = torch.concat((attention_mask, torch.tensor([[1]]).to(attention_mask.device)), dim=1)
        complete_token.append(input_token)
```

@gante Many thanks for all your effort! 🤗

What is quite interesting, is that my initial approach works with 

```
from transformers import BloomTokenizerFast, BloomForCausalLM
from transformers.models.opt import OPTForCausalLM
from transformers import AutoTokenizer
```

i.e., if I use

```
tokenizer = AutoTokenizer.from_pretrained('facebook/opt-13b')
model = OPTForCausalLM.from_pretrained("facebook/opt-13b").to(device)
```

resp. 

```
tokenizer = BloomTokenizerFast.from_pretrained("bigscience/bloom-560m")
model = BloomForCausalLM.from_pretrained("bigscience/bloom-560m").to(device)
```

Perhaps it might be sensible to standardize it - but, of course, there might be some reasons that are a hindrance, that I am not aware of. 🙂

Some models (like OPT) have "better" default behavior in the absence of attention masks. We will probably move in that direction in the future.

However, the easier defaults come at a price -- at best, they require creating it from scratch at every forward pass (and at worst, you may get incorrect results). I'd recommend creating and manually manipulating the attention mask whenever possible, to avoid nasty surprises :) 

@gante In any case, many thanks for all your kind support. 🤗

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.