attention_mask hold float values in [0,1] in T5
Author: pretidav
Pull request: no

### Feature request

Hello Everybody,

I was wondering whether attention_mask input for T5 could be used as a float in [0,1] instead of an integer as in the documentation
`attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked. `
Would you think passing something like (I write tokens for clarity, but they would be tokens_ids)
`tokens               = ['hello','how','are','you','pad','pad']
attention_mask       = [0.5,    0.9,  0.2, 1,     0,      0] `

to attribute a particular emphasis on several tokens respect to the others (still keeping components 0 for Pad tokens) ?
Clearly the model requires a finetuning, but I wonder if this different usage of attention_mask could harm it in some way…

As far as I see from [transformers/modeling_t5.py at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/modeling_t5.py)
It is not really clear to me how the attention_masks act in the attention blocks (see here for instance [transformers/modeling_t5.py at 8f46ac98498dd47701971064617e00d7e723a98e · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/8f46ac98498dd47701971064617e00d7e723a98e/src/transformers/models/t5/modeling_t5.py#L531)).
I was expecting some kind of “hard” attention, but as far as I see it’s a “soft” implementation shifting the position_bias. How this translates into the removal of ‘pad’ token contribution from the attention (is a shift of “1” as in the original attention_mask, enough to ensure a reasonable suppression of pads) ?

Any answer is very welcomed! Thank you!

### Motivation

This not-boolean usage of mask could offer important improvements over the model. 

### Your contribution

I am not an expert, so before doing any development I would like to hear about the feasibility (or if it is already supported) from developers. 

Comments:
cc @ydshieh that has given some thought to this in the past, I believe

Well, as far as I can recall, I didn't have this idea before. This looks like @pretidav want to provide manual (float) attention mask.

- with the introduction of `dtype.min` in #17306 (or even with `-1e4` currently) as the large negative value for masking value, `0` will be replaced by `dtype.min`, but how about `0.5`? Use something like `0.5 * dtype.min` doesn't really make sense: if there is also a `1` in the (original) attention mask which is replaced by `0`, then `0.5 * dtype.min` won't get any attention, just like `dtype.min`. Use more complex transformation won't be good (potentially) for performance reason.
- The number of places to be changed would be quite large (if we decide to do it)
- In order to make the decision, I think it would be great if we can see this usage is proposed in some papers, or in some real world examples, that show the improvements (if we want to do it correctly)
- Finally, I am not sure how a user would provide meaningful float attention mask values. Just using some heuristic to get hard-coded values in advance? I kinda feel that these soft values is the role of the attention probability through training.

cc @patrickvonplaten @patil-suraj @sgugger 



Thanks for the answer @ydshieh !
Float attention values could be provided from another model or a human annotator. For instance, a human annotator can try to "force" the model to pay more attention to some part of the input document, or some specific entities. It's just a way to introduce some external control over the model attention other then its own attention probability obtained during the training.

Is there a way currently such a feature could be used in T5? 
If I understand well, attn_mask is not the right feature to play with. 

@pretidav There is currently no such feature. As mentioned previously, attention mask is not good for this purpose, as the  range of the **final (processed)** attention mask is `[-inf, 0]`. The `-inf` means no attention at all (masked), and `0` means no mask (so attend to it). 

The attention mask is combined with the attention scores (computed using `query` and `keys`), whose range is **NOT** a fixed interval.

So from these facts, there is no clear & meaningful values to provide extra attention values (before `softmax`), as far as I can tell.

A **manual** change, if one want to do it, could be done after the following line of attention probability computation (which has values between 0 and 1).
https://github.com/huggingface/transformers/blob/8f46ac98498dd47701971064617e00d7e723a98e/src/transformers/models/t5/modeling_t5.py#L534-L536

However, it depends on how you would like to combine `attn_weights` with extra attention values.

Agree with what @ydshieh said!

This is an interesting question but to me it seems like a niche use-case and as @ydshieh said, this would require a lot of changes. IMO for this use-case one can just copy-paste the T5 model and tweak it to support this. The models in `Transformers` are designed in such a way that users could just take the modeling files and tweak it for their purpose rather than supporting everything in the framework.  Thanks :) 

Thank you very much for all the answers! I'll try to tweak the model.