`modeling_opt.py` if `previous_key_values` given and `attention_mask==None` the model throws an error.
Author: GabrielKP
Pull request: no

### System Info

- `transformers` version: 4.26.1
- Platform: Linux-4.18.0-147.el8.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.16
- Huggingface_hub version: 0.12.1
- PyTorch version (GPU?): 1.13.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

## Code
1. Load opt/tokenizer
```py
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "facebook/opt-125m"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer  = AutoTokenizer.from_pretrained(model_name)
```
2. Precompute `past_key_values`
```py
text1 = "let's find a"
tokenized1 = tokenizer(text1, return_tensors='pt')
past_key_values = model(**tokenized1, use_cache=True)["past_key_values"]
```
4. Compute another set of values without `attention_mask`
```py
text2 = "bug"
tokenized2 = tokenizer(text2, return_tensors='pt')
model(input_ids=tokenized2["input_ids"], past_key_values=past_key_values)
# error! The mistakenly created an attention_mask that is too small.
```
(try `distilgpt2` and it will work)

## stack trace

```
Traceback (most recent call last):
  File "/home/gkressi1/opt/ldet/rate_in-context.py", line 334, in <module>
    main()
  File "/home/gkressi1/opt/ldet/rate_in-context.py", line 325, in main
    output_config = compute_surprisals(config=config, model_object=model_object)
  File "/home/gkressi1/opt/ldet/rate_in-context.py", line 219, in compute_surprisals
    output_rating = model_object.incontext(config, prompt_list)
  File "/home/gkressi1/opt/ldet/src/model_objects/model_hf_causal_lm_big.py", line 85, in incontext
    output = self.get_model_output(rest_prompt, use_cache=True)
  File "/home/gkressi1/opt/ldet/src/model_objects/model_hf_causal_lm_big.py", line 63, in get_model_output
    output = self.model(
  File "/home/gkressi1/.conda/envs/llm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/gkressi1/.conda/envs/llm/lib/python3.9/site-packages/accelerate/hooks.py", line 158, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/gkressi1/.conda/envs/llm/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 932, in forward
    outputs = self.model.decoder(
  File "/home/gkressi1/.conda/envs/llm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/gkressi1/.conda/envs/llm/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 639, in forward
    attention_mask = self._prepare_decoder_attention_mask(
  File "/home/gkressi1/.conda/envs/llm/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 546, in _prepare_decoder_attention_mask
    expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask
RuntimeError: The size of tensor a (93) must match the size of tensor b (1679) at non-singleton dimension 3
```

### Expected behavior

The model should create the attention mask by itself and not throw an error.

From the surface, this seems to be an easy fix:
1. Delete line [635](https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/models/opt/modeling_opt.py#L635) and [636](https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/models/opt/modeling_opt.py#L635)
2. Move line [639-642](https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/models/opt/modeling_opt.py#L639) of what is currently line [637](https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/models/opt/modeling_opt.py#L637)
3. Check TF/Flax models (?).

All the best!

Comments:
Hey! Thanks for submitting this issue! 
Passing attention  maks solves the problem, and usually we expect to pass attention masks when you are using the `past_key_values`(for example in generate). It is debatable whether the default behaviour should rely on the past_key_values. 
Do you have a specific usage in mind? 

The following works as expected: 
```python 
attn = torch.cat((tokenized1["attention_mask"], tokenized2["attention_mask"]), -1)
text2 = "bug"
tokenized2 = tokenizer(text2, return_tensors='pt')
model(input_ids=tokenized2["input_ids"], past_key_values=past_key_values,attention_mask =attn)
```
This way is the expected usage. When training or doing an inference, you should probably be in a for loop where the attention mask is defined based on the entire input. 


I agree that manually adding the attention_mask is an easy fix.

I am using a shared context as `past_key_values` and then computing different model outputs given the context. In that case I save the contexts `past_key_values` and use them later on. It is easy to recompute/save the contexts attention_mask and concat it for every output - but
* OPT model behavior is inconsistent to other model's I have been using (gpt-neo, bloom)
* it is [not documented](https://huggingface.co/docs/transformers/v4.26.1/en/model_doc/opt#transformers.OPTForCausalLM.forward.past_key_values) that the expected usage is passing the `attention_mask` when using `past_key_values`
* the thrown error is not descriptive of the issue

I do not understand what you mean with "default behaviour should rely on the past_key_values" - it seems to me that default behavior is not affected by changing this: line [636](https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/models/opt/modeling_opt.py#L636) seems to have exactly the same job that [639 - 642](https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/models/opt/modeling_opt.py#L639) has, just that it does not take into account `past_key_values` introducing the deviation of model behavior to other models.

I can understand if you say that passing `attention_mask` is expected behavior for using `past_key_values`, but maybe that could be mentioned somewhere?

Totally agree with you, will open a PR to adress this. I think this was also blocking us from adding the ONNX config for this model! 
Thanks for this 😉