Should I use `attention_mask`?
Author: ShaneTian
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
#### My Target
1. Transfer sentences to `ids`
2. Padding `ids` when 
3. Encode `ids` to vector(`last_hidden_states`)
4. Put this vector to my own downstream model.

#### My code
```py
original_text = "Hello world!"
ids = model.encode(original_text)
padding_mask = [1] * len(ids)  # Padding mask
while ids < max_length:
    ids.append(0)  # Padding
    padding_mask.append(0)  # Mask using 0

# === Use `attention_mask` ===
outputs = model(ids, attention_mask=padding_mask)
last_hidden_states = outputs[0]  # Get vector such as '<CLS>'
```

#### But...
In official demo code, **it does not use padding mask**:
```py
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
# === Not use `attention_mask` ===
outputs = model(input_ids)
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
```
So, why? Should I use `attention_mask`?
Thanks!

Comments:
Hi,

The attention mask is useful when there are padding indices on which you do not want to perform attention. You should use it when you're using padding, which should only happen when having a batch size superior to one with different sized sequences in those batches.

> Hi,
> 
> The attention mask is useful when there are padding indices on which you do not want to perform attention. You should use it when you're using padding, which should only happen when having a batch size superior to one with different sized sequences in those batches.

Thanks for your reply! @LysandreJik 
So, you mean I should use it when I use batch data and there are different sized sequences in batch, otherwise I shouldn’t. 
But why the official demo is ok? I guess it will implement PAD again no matter what I have already implemented PAD manually or not? Right?

> So, you mean I should use it when I use batch data and there are different sized sequences in batch, otherwise I shouldn’t.

Exactly. You can use them, but you don't need to. You probably shouldn't because of the performance cost.

> But why the official demo is ok? I guess it will implement PAD again no matter what I have already implemented PAD manually or not? Right?

It is ok because there is only one sentence.

> > So, you mean I should use it when I use batch data and there are different sized sequences in batch, otherwise I shouldn’t.
> 
> Exactly. You can use them, but you don't need to. You probably shouldn't because of the performance cost.
> 
> > But why the official demo is ok? I guess it will implement PAD again no matter what I have already implemented PAD manually or not? Right?
> 
> It is ok because there is only one sentence.

Thank you! @rlouf 
1. I think those tokens that have been padded should not be paid attention. So I don't know why I don't need to.
2. The most models have a fixed length(max_length), the sentences should pad to it before feed. I mean why the official demo doesn't pad.

> 1. I think those tokens that have been padded should not be paid attention. So I don't know why I don't need to.

Sorry, now I re-read my response I realize it was not very clear. I meant that if they are the same size you can still pad, but you'd hit the performance. If they are not, then you should absolutely pad and pass the appropriate attention mask.

> 2. The most models have a fixed length(max_length), the sentences should pad to it before feed. I mean why the official demo doesn't pad.

Yes but sentences don't all have the same length:

```
// No need to pad this
[[1, 2,  3],
[5, 6, 7]]
```

BUT

```
// Here you should pad
[[1, 2, 3, pad_token_id],
[5, 6, 7, 8]]
```

Thank you very much! I got it. @rlouf