Padding changes model outputs (even with attention_mask)
Author: thashim
Pull request: no
Labels: name: wontfix

name: Core: Modeling

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Consider the following example code which gets a BERT embedding, and compares the BERT embedding on the same input with padding + masking.
```
bertModel=BertModel.from_pretrained('bert-base-uncased')
bertModel=bertModel.eval()
print('token id:'+str(id))
print('token types:'+str(typeout))

bert_out = torch.mean(bertModel(id, token_type_ids = typeout, attention_mask = mask)[0],1)

add_pad = lambda x: torch.cat((x, torch.zeros(1, 10, dtype=x.dtype)),1)
print('mask:'+str(mask))
print('padded id:'+str(add_pad(id)))
print('padded type:'+str(add_pad(typeout)))
print('padded mask:'+str(add_pad(mask)))
bert_out_2 = torch.mean(bertModel(add_pad(id), token_type_ids = add_pad(typeout), attention_mask = add_pad(mask))[0],1)

print(bert_out[0,0:10])
print(bert_out_2[0,0:10])
```
The output here is 
```
token id:tensor([[ 101, 5292, 3270,  102, 8638, 2060,  102]])
token types:tensor([[0, 0, 0, 0, 1, 1, 1]])
mask:tensor([[1., 1., 1., 1., 1., 1., 1.]])
padded id:tensor([[ 101, 5292, 3270,  102, 8638, 2060,  102,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0]])
padded type:tensor([[0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])
padded mask:tensor([[1., 1., 1., 1., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])
tensor([ 0.2600,  0.1568, -0.2576, -0.0110,  0.1021, -0.1336,  0.2546, -0.5071,
         0.0462, -0.2283], grad_fn=<SliceBackward>)
tensor([ 0.0996,  0.0061, -0.3331, -0.0237, -0.1110, -0.0050,  0.2755, -0.3335,
        -0.0565, -0.2542], grad_fn=<SliceBackward>)
```
The two tensors which were generated by the same input but have different padding have drastically different embeddings.


## Expected behavior

The last two lines of the output should be the same
## Environment info
     
- `transformers` version: 2.1.1.
- Platform: osx
- Python version: 3.7.3
- PyTorch version (GPU?): 1.2.0
- Tensorflow version (GPU?): N/A
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Comments:
I tried to replicate the input as close as I could given the output you gave:

```py
from transformers import BertModel
import torch

token_id = torch.tensor([[ 101, 5292, 3270,  102, 8638, 2060,  102]])
token_types = torch.tensor([[0, 0, 0, 0, 1, 1, 1]])
mask = torch.tensor([[1., 1., 1., 1., 1., 1., 1.]])

padded_id = torch.tensor([[ 101, 5292, 3270,  102, 8638, 2060,  102,    0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])
padded_type = torch.tensor([[0, 0, 0, 0, 1, 1, 1,                       0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])
padded_mask = torch.tensor([[1., 1., 1., 1., 1., 1., 1.,                0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])

bertModel = BertModel.from_pretrained('bert-base-uncased')
bertModel = bertModel.eval()

output = bertModel(token_id, token_type_ids=token_types, attention_mask=mask)
padded_output = bertModel(padded_id, token_type_ids=padded_type, attention_mask=padded_mask)
```

I then print the maximum difference between the output of the model with the non-padded input and the output with the padded input:

```py
print(torch.max(output[0] - padded_output[0][:, :7]))
print(torch.max(output[1] - padded_output[1]))
```

Which outputs the following (negligible) difference:

```py
tensor(3.6359e-06, grad_fn=<MaxBackward1>)
tensor(5.9605e-07, grad_fn=<MaxBackward1>)
```

Would it be possible for you to give a completely reproducible script so that I may see where the issue lies?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.