Is there a data leakage in causal masking?
Author: Shaier
Pull request: no

### System Info

Following [this tutorial](https://huggingface.co/course/chapter7/6?fw=pt) on training a causal language model from scratch. I found the [source code](https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/models/gpt2/modeling_gpt2.py#L666) for the model they use (GPT2). On line 195 we define “causal_mask”. I tried commenting out this line and defining a new “causal_mask” with the same shape but either all True or all False entries (instead of the triangle masking). Though, the model still learned in both cases to generate natural language. This is unexpected as if all the inputs are masked all the time the model should not learn to generate coherent text. Am I missing something or is there data leakage?
![image](https://user-images.githubusercontent.com/43555163/223469858-b19308bc-1a67-43a3-9db3-9534c8287f34.png)

I don't know if the following is relevant to the issue, but I also find that on line 822 we have "attention_mask" which from the comments suppose to mask out as well:

> 
>             # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
>             # masked positions, this operation will create a tensor which is 0.0 for
>             # positions we want to attend and the dtype's smallest value for masked positions.
>             # Since we are adding it to the raw scores before the softmax, this is
>             # effectively the same as removing these entirely.

But I find that if I print
`print('attention_mask', torch.min(attention_mask))`
the result is always -0.0. So I assume this is not actually masking anything for some reason?

### Who can help?

@ArthurZucker, @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Steps to reproduce the behavior:

1. Comment line 195 which defines "causal_mask".
2. Instead, define causal_mask as: 
```
causal_mask = torch.rand(1,1,context_length,context_length)
causal_mask = causal_mask > 110.0 # all False
# or
causal_mask = causal_mask > 0.0 # all True
causal_mask = causal_mask.to(device)
```
3. Run script

Note that I'm using a small dataset with 10 short paragraphs.

### Expected behavior

Masking all the inputs all the time should not allow the model to learn to generate natural language. Instead. the model should generate random text. 

Comments:
Hey! 
1. The `attention_mask` can be assimilated to the `padding_mask` which tells the model where the pad tokens are. 
2. The `causal_mask` defined with :
```python 
            # if only "normal" attention layer implements causal mask
            query_length, key_length = query.size(-2), key.size(-2)
            causal_mask = self.bias[:, :, key_length - query_length : key_length, :key_length]
            mask_value = torch.finfo(attn_weights.dtype).min
            # Need to be a tensor, otherwise we get error: `RuntimeError: expected scalar type float but found double`.
            # Need to be on the same device, otherwise `RuntimeError: ..., x and y to be on the same device`
            mask_value = torch.full([], mask_value, dtype=attn_weights.dtype).to(attn_weights.device)
            attn_weights = torch.where(causal_mask, attn_weights.to(attn_weights.dtype), mask_value)
```
line 195 as you mention is the actual causal mask that is used in the SelfAttention, right before the softmax. 
When we create this attention mask, we make sure that the values that we want to mask have a `mask_value = torch.finfo(attn_weights.dtype).min`, this is a very big *negative* number. What you are using is a causal mask with values 0 and 1, which will no affect the attention scores. 

If you are using a pretrained model, its normal that this does not affect it. If you are not, it is also normal, but if you try to run inference, the model will perform worse than to a properly trained model.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.