Post-padding affects the Bert embedding output
Author: XinnuoXu
Pull request: no

# 🐛 Bug

## Information

Model: BertModel
Language: English

The problem arises when using:

```
# Load model
from transformers import *
import torch
model_class = BertModel
tokenizer_class = BertTokenizer
pretrained_weights = 'bert-base-uncased'
tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
model = model_class.from_pretrained(pretrained_weights).to('cuda')

# First example
batch_id = [[101, 1996, 3035, 2038, 2741, 1037, 1056, 28394, 2102, 2000, 1996, 3035, 2012, 17836, 4186, 2000, 8439, 2014, 3938, 2705, 5798, 102]]
batch_id = torch.tensor(batch_id).to('cuda')
with torch.no_grad():
    last_hidden_states = model(batch_id)[0].cpu().numpy()
print (last_hidden_states[0][:10])

# Second example
batch_id = [[101, 1996, 3035, 2038, 2741, 1037, 1056, 28394, 2102, 2000, 1996, 3035, 2012, 17836, 4186, 2000, 8439, 2014, 3938, 2705, 5798, 102, 0, 0]]
batch_id = torch.tensor(batch_id).to('cuda')
with torch.no_grad():
    last_hidden_states = model(batch_id)[0].cpu().numpy()
print (last_hidden_states[0][:10])
```

Output for the first example
```
array([[ 0.00197573, -0.06912418,  0.24121636, ..., -0.13239928,
         0.13210389,  0.3860737 ],
       [ 0.18745837, -0.15252575,  0.16234997, ..., -0.34497464,
         1.0031146 ,  0.20545363],
       [ 0.40690556, -0.7345518 ,  1.1162403 , ..., -1.148023  ,
        -0.38943186, -0.6397534 ],
       ...,
       [ 1.3574413 , -0.87637144,  1.007168  , ..., -0.7466023 ,
        -0.5337318 , -0.02415964],
       [ 0.0907229 , -1.0051603 ,  0.7100666 , ..., -0.00599465,
        -0.37829682,  0.4773703 ],
       [-0.00619348, -0.34730428,  0.9920887 , ...,  0.28678447,
         0.2980772 ,  0.8005251 ]], dtype=float32)
```

Output for the second example
```
array([[-0.10877508,  0.0271297 ,  0.17947783, ..., -0.2650592 ,
         0.15821457,  0.35017303],
       [-0.1396759 , -0.25098413,  0.3990493 , ..., -0.52468735,
         0.8060062 ,  0.42330667],
       [ 0.18865047, -1.0035415 ,  1.3446846 , ..., -1.1652598 ,
        -0.60856164, -0.419513  ],
       ...,
       [ 1.3687737 , -0.9032434 ,  1.0184443 , ..., -0.7951573 ,
        -0.56618035, -0.00522863],
       [ 0.02363256, -0.962884  ,  0.68822455, ..., -0.03798304,
        -0.34567115,  0.5442954 ],
       [-0.00341167, -0.33559048,  1.0627198 , ...,  0.31898227,
         0.2941662 ,  0.7981017 ]], dtype=float32)
```

I also checked that the output for `tokenizer.convert_tokens_to_ids(tokenizer.pad_token)` is `0`

## Expected behavior

The embeddings for the padded sequence should be same with the ones without padding. 

## Environment info

- `transformers` version: transformers 2.4.1
- Platform: Linux
- Python version: Python 3.7.5
- PyTorch version (GPU?): PyTorch 1.4.0 (CUDA Version 10.1.243)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Comments:
Hi, please look into the documentation of the [attention mask](https://huggingface.co/transformers/glossary.html#attention-mask).

Actually, it was a valid question. The output will be numerically different for sure since there are extra positions to attend and even if those are paddings -> there is a difference among the float values. However, the real question is whether the (cosine/dot) similarity among the resulting vectors have changed at all.