Make use of attention_mask in Trainer's compute_metrics
Author: AlexBella365
Pull request: no

# 🚀 Feature request
In Trainer's training loop, the `compute_metrics` function takes a `EvalPrediction(predictions=preds, label_ids=label_ids)` object as input.
It should also be able to use `inputs['attention_mask']` to mask the irrelevant predictions (those for which attention_mask is 0).

## Motivation
I am working on a NER task and find myself having no way to filter out irrelevant predictions.
In the following example, the `raw_pred` leads to an accuracy of 43%, while the `masked_pred` gives me almost 77% (value 0 in attention_mask has been casted into `nan` before being applied to `raw_pred`)

**ground_truth**
```python
tensor([[0, 5, 5, 5, 0, 6, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
        [0, 1, 1, 1, 1, 2, 2, 0, 1, 2, 2, 2, 0, 0, 0, 0, 0, 0, 0],
        [0, 5, 6, 6, 6, 6, 6, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
        [0, 3, 3, 4, 0, 3, 3, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
        [0, 0, 0, 0, 0, 0, 0, 0, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0]])
```
**raw_pred**
```python
tensor([[0, 2, 5, 5, 6, 4, 1, 2, 3, 1, 3, 3, 2, 1, 2, 1, 2, 6, 3],
        [2, 1, 2, 2, 2, 2, 2, 0, 1, 2, 2, 2, 0, 0, 0, 0, 0, 0, 2],
        [4, 4, 4, 0, 0, 0, 0, 4, 0, 0, 0, 4, 4, 4, 4, 4, 4, 4, 0],
        [0, 6, 6, 4, 4, 5, 6, 6, 4, 0, 3, 3, 3, 2, 3, 3, 1, 2, 2],
        [0, 0, 0, 0, 5, 4, 5, 5, 5, 6, 4, 0, 2, 1, 2, 1, 1, 3, 3]])
```
**attention_mask**
```python
tensor([[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0]])
```
**masked_pred**
```python
tensor([[0., 2., 5., 5., 6., 4., 1., 2., nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,nan],
        [2., 1., 2., 2., 2., 2., 2., 0., 1., 2., 2., 2., 0., 0., 0., 0., 0., 0.,nan],
        [4., 4., 4., 0., 0., 0., 0., 4., 0., 0., 0., 4., 4., 4., 4., 4., 4., 4., 0.],
        [0., 6., 6., 4., 4., 5., 6., 6., 4., 0., nan, nan, nan, nan, nan, nan, nan, nan,nan],
        [0., 0., 0., 0., 5., 4., 5., 5., 5., 6., 4., 0., nan, nan, nan, nan, nan, nan,nan]])
```


Comments:
For this use case, it's best to subclass the `Trainer` and override the `evaluate` method. An example of this is given for question-answering [here](https://github.com/huggingface/transformers/blob/master/examples/question-answering/trainer_qa.py) where we need to post-process the predictions using the original dataset (a bit like your use case since the attention masks will be in the dataset). The predictions returned by the Trainer are in the same order as the elements of your dataset, so you're safe with that.

Thanks a lot Sylvain.
Actually I realised that the `predictions.label_ids` coming from the training loop were padded with the value `-100`. By using the same padding value in my preprocessing, I can recover `the attention_mask` by putting a threshold.
Cheers 

Oh even easier then! Can we close the issue?

> For this use case, it's best to subclass the `Trainer` and override the `evaluate` method. An example of this is given for question-answering [here](https://github.com/huggingface/transformers/blob/master/examples/question-answering/trainer_qa.py) where we need to post-process the predictions using the original dataset (a bit like your use case since the attention masks will be in the dataset). The predictions returned by the Trainer are in the same order as the elements of your dataset, so you're safe with that.

Hello, can I ask where I can find the proper example again? Because the example link you mentioned has an error with 404 now. @sgugger 

Hi @saekomdalkom, you can find it here: https://github.com/huggingface/transformers/blob/main/examples/pytorch/question-answering/trainer_qa.py

> Hi @saekomdalkom, you can find it here: https://github.com/huggingface/transformers/blob/main/examples/pytorch/question-answering/trainer_qa.py

Thanks a lot!!