transformers model doesn't output zeros for padded subtokens
Author: dayyass
Pull request: no
Labels: name: bug

### System Info

I have noticed that mean pooling over last_hidden_state returns different results depending on the batch size. When I started to go deeper to understand the reason of the problem I found that padded by tokenizer subtokens after passing it through the model are not zeros and actually this leads to the problem behavior. Then batch size = 1 there is no padded subtokens, so results are 100% correct, but batch size = 1 leads to much longer computations (obviously).

Is it correct behavior and is it possible to avoid this?

The snippet below uses `bert-base-uncased` model, but this behavior also related to other models.


### Who can help?

@LysandreJik @SaulLu

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python3
import torch
from transformers import AutoModel, AutoTokenizer

model_name = 'bert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

sentences = [
    "Hello, World!",
    "How are you doing today?",
]

tokenizer_kwargs = {
    'return_tensors': 'pt',
    'padding':        True,
    'truncation':     True,
    'max_length':     512,
}

tokens_1 = tokenizer(sentences, **tokenizer_kwargs)

tokens_2 = []

for sentence in sentences:
    tokens = tokenizer(sentence, **tokenizer_kwargs)
    tokens_2.append(tokens)

with torch.no_grad():
    tensor_1 = model(**tokens_1)['last_hidden_state']

tensor_2 = []

for tokens in tokens_2:
    with torch.no_grad():
        tensor = model(**tokens)['last_hidden_state']
    tensor_2.append(tensor)

print(torch.allclose(tensor_1[1], tensor_2[1].squeeze(0), atol=1e-05))  # True
print(torch.allclose(tensor_1[0][:6], tensor_2[0].squeeze(0), atol=1e-05))  # True

print(tensor_1[0][6:])  # padded tokens
```

### Expected behavior

I expect that embeddings corresponding to padded subtokens would be equal to zero.

Comments:
> This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
> 
> Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

ping

Hi @dayyass ,

Indeed, I don't think there is a condition on embeddings that would be at an masked position. But as you have shown in your two tests:
```python
print(torch.allclose(tensor_1[1], tensor_2[1].squeeze(0), atol=1e-05))  # True
print(torch.allclose(tensor_1[0][:6], tensor_2[0].squeeze(0), atol=1e-05))  # True
```

The most important thing is that the embeddings of the unmasked positions are identical, no matter how many pad tokens are added at the end of the sequence - which is verified on your tests.

Did I miss something in your question? Why do you expect that embeddings corresponding to padded subtokens would be equal to zero?

Hi, @SaulLu!
Thanks for answering!

No, I put all information in issue.

I assume that embeddings corresponding to padded subtokens should be equal to zero, because it is easier to find such embeddings and exclude those from averaging. But if we doesn't exclude those, it is better (most likely) to average embeddings when it is equal to zero.


Still actual issue.

> I assume that embeddings corresponding to padded subtokens should be equal to zero, because it is easier to find such embeddings and exclude those from averaging. But if we doesn't exclude those, it is better (most likely) to average embeddings when it is equal to zero.

Thanks for the details, in your case, I think the attention mask is exactly what you're looking for. If you ever want your output to have the value set to 0 for pad tokens, you can multiply the output by the attention mask. In your example:
```python
tensor_1 * tokens_1.attention_mask.unsqueeze(2)
```


> > I assume that embeddings corresponding to padded subtokens should be equal to zero, because it is easier to find such embeddings and exclude those from averaging. But if we doesn't exclude those, it is better (most likely) to average embeddings when it is equal to zero.
> 
> Thanks for the details, in your case, I think the attention mask is exactly what you're looking for. If you ever want your output to have the value set to 0 for pad tokens, you can multiply the output by the attention mask. In your example:
> 
> ```python
> tensor_1 * tokens_1.attention_mask.unsqueeze(2)
> ```

That's an excellent and elegant solution, thank you for that!
I guess the issue might be closed.

Glad it helped you! :hugs: