Padding tokens affect MobileBert output
Author: johnmccain
Pull request: no

## Environment info

- `transformers` version: 4.2.2
- Platform: Windows-10-10.0.17134-SP0
- Python version: 3.6.8
- PyTorch version (GPU?): 1.7.1+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

## Information

Model I am using (Bert, XLNet ...): MobileBert

Adding padding tokens to the end of a sequence affects MobileBert output even when masked. I've tried this on a few other models (`bert-base-uncased`, `roberta-base`, `xlm-roberta-base`) and was only able to replicate this with `google/mobilebert-uncased`.

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_string = 'google/mobilebert-uncased'
tokenizer = AutoTokenizer.from_pretrained(model_string)
model = AutoModelForSequenceClassification.from_pretrained(model_string)

example_text = 'Hello, world!'

input_with_pad = tokenizer.encode_plus(
    example_text,
    padding='max_length',
    max_length=32,
    return_tensors='pt'
)
print(input_with_pad)
# {'input_ids': tensor([[ 101, 7592, 1010, 2088,  999,  102,    0,    0,    0,    0,    0,    0,
#             0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
#             0,    0,    0,    0,    0,    0,    0,    0]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
#          0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
#          0, 0, 0, 0, 0, 0, 0, 0]])}

input_without_pad = tokenizer.encode_plus(
    example_text,
    padding='longest',
    max_length=32,
    return_tensors='pt'
)

print(input_without_pad)
# {'input_ids': tensor([[ 101, 7592, 1010, 2088,  999,  102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1]])} 

with torch.no_grad():
    model.eval()

    out_with_pad = model(**input_with_pad)
    print(out_with_pad.logits)
    # tensor([[12693366., -5310913.]])

    out_without_pad = model(**input_without_pad)
    print(out_without_pad.logits)
    # tensor([[12741167., -5327575.]])
```

## Expected behavior

Padding tokens should not affect the output of the model as long as they are masked. As far as I can tell, this only occurs with mobilebert.

Comments:
Following up on this, is there anyone in particular that I should tag to take a look at this issue?

Hi @johnmccain, I'll take a look in the coming days.

Hi! I ran your example and added an additional relative difference computation: 
```py
r_tol = torch.max(torch.abs(out_with_pad - out_without_pad) / torch.abs(out_without_pad))
```

What I gather from this is that the difference between the two outputs varies from 0.3% to 0.4%.

Some things to note:

- The attention mask is useful to hide tokens, but isn't perfect: the attention mask essentially adds a very large negative value to the attentions of the tokens we don't want to attend to (-10000), but that is not (-inf) either, so it doesn't erase them from existence. Even if padding is correctly done with an attention mask, some differences of ~1e-4 or ~1e-5 can still happen.
- Unfortunately, there’s not much we can do, given that this is the way the original model was trained, using an adder (-10000). We have to keep as close as possible to the original implementation.
- While keeping in mind that these differences are usually very very small, and shouldn’t have an impact on your model, the way to get closer to the expected behavior if to have as few padding tokens as possible.

Now, MobileBERT is peculiar in that it has extremely high outputs compared to other models, but from what I'm seeing it's still within the 0.3%-0.4%. It is slightly higher than for other models, such as BERT which are in the ~0.0001% range. I didn't dive in enough to see exactly why this is so, but my guess is that all the tweaks to make it smaller (bottlenecks) might be responsible, as well as the very high outputs.

If you randomly initialize a MobileBERT and run through the same tests:

```py
config = AutoConfig.from_pretrained(model_string)
model = AutoModelForSequenceClassification.from_config(config)
```

You'll get results that are comparable to BERT:

```py
    print(out_with_pad_logits, out_without_pad_logits)
    print(out_without_pad_logits - out_with_pad_logits)
    print(r_tol)
```

yields 

```
tensor([[ 0.0255, -0.0048]]) tensor([[ 0.0255, -0.0048]])
tensor([[-2.9769e-05,  8.6003e-06]])
tensor(0.0018)
```

on my side (random each run as randomly initialized weights)

Investigated with @jplu 

Thanks for looking into this!
That makes sense that this phenomenon would only be immediately visible with MobileBert with its extremely large logits. I will say that this can affect downstream tasks in my experience--for a MobileBert model finetuned on a binary classification task, switching from `padding='max_length'` to `padding='longest'` changed a handful of logits on my test set enough to affect the predicted class. (~1 in 500-1000 examples were altered enough to flip from 0 to 1 or vice versa). I haven't experienced that same sort of impact when using other Huggingface models like RoBERTa or Bert.

I wonder if the effect of padding tokens is diminished when using an activation in the classifier head to avoid the extremely large logits as suggested in #8938. I will comment back with what I find.

Hey guys, I opened a similar issue a while ago https://github.com/huggingface/transformers/issues/7070. It was automatically closed due to inactivity, but we are still struggling with the issue every day and don't use batching when predicting.

When computing the relative difference, for the example input shown in the issue mentioned above (I compared `emb2` and `emb4` from my issue), I got a quite disturbing result:
- mean error: 4% 
- median error: 0.9%
- max error: 200% (5033.1235 VS 1674.1803)

I might have done some mistake, but I just used the formula written above:

> ```python
> r_tol = torch.max(torch.abs(out_with_pad - out_without_pad) / torch.abs(out_without_pad))
> ```

Hi @swecooo! Thanks for letting us know. There might be a deeper issue than what I've seen then, I'll take a deeper look as soon as I have time.

Could you specify how you computed these, for example with a code snippet so that I can investigate? Thanks!

Hey @LysandreJik, thank you very much for looking into this issue of ours. :slightly_smiling_face: This is the snippet that I used for computing the mean, median and max errors. I believe it should be identical to your formula mentioned above.

```python
import torch
from transformers import AutoModel, AutoTokenizer

model = 'google/mobilebert-uncased'
tokenizer = AutoTokenizer.from_pretrained(model)
model = AutoModel.from_pretrained(model)
text = 'Hey, how are you?'

i1 = tokenizer.batch_encode_plus([text], padding=True, return_tensors='pt')
emb1 = model(**i1).pooler_output[0]  # Only one in batch (not padded): [-2.5088e+07,  7.3279e+04,  1.7544e+05, ...]

i2 = tokenizer.batch_encode_plus([text, text + ' hey'], padding=True, return_tensors='pt')
emb2 = model(**i2).pooler_output[0]  # Not longest (padded): [-2.4871e+07,  8.1873e+04,  1.6693e+05, ...]

diff = torch.abs(emb2 - emb1) / torch.abs(emb1)
print("Mean", torch.mean(diff))  # 0.0432
print("Median", torch.median(diff))  # 0.0090
print("Max", torch.max(diff))  # 2.0063
top_10 = torch.argsort(diff, descending=True)[:10]
print(diff[top_10], emb1[top_10], emb2[top_10], sep="\n")  # [2.0063, 1.5129, 1.1135, 0.6702, ...]
                                                           # [1674.1803, 7940.5342, 2012.5089, -13467.0508, ...]
                                                           # [5033.1235, 19954.0098, 4253.4575, -4441.5933 ...]
```

From the outputs, it seems that, naturally, smaller output values have a larger error. Please do let me know if you can (or cannot) reproduce the issue in the same magnitude as it happens for me, or if I can provide any more details.

I used `torch==1.7.1` and `transformers==4.3.2` on Python 3.7.

Cool, thanks for providing this snippet! I'll need to take a few hours to deep dive into it and see what's happening, so you expect an answer by the end of the next week if that's alright.

Will also look into your previous issue https://github.com/huggingface/transformers/issues/7070 (Sorry that it felt through the cracks!)

> you expect an answer by the end of the next week if that's alright

Sure, thanks a lot for looking into this. About #7070, I believe it's basically the same issue as here. :slightly_smiling_face:

Hello! I've taken a look, and you are both right: padding tokens affect MobileBERT's output values. One thing that MobileBERT does differently to other models, is that it uses an embedding size of `128` which is different to the `hidden_size`. 

Before adding the word embeddings to the position embeddings and token type embeddings, these word embeddings are first passed through a 1D convolution with kernel size 3, effectively casting a tensor of size `(batch_size, sequence_length, 128)` to a tensor of size `(batch_size, sequence_length, 384)`. 

This happens here: https://github.com/huggingface/transformers/blob/a85eb616f73c3e7eedb22146972ea41921164671/src/transformers/models/mobilebert/modeling_mobilebert.py#L199-L214

Then, this value is passed through a linear layer of output size `512`, resulting in a final value of size `(batch_size, sequence_length, 512)`. 

This happens here: https://github.com/huggingface/transformers/blob/a85eb616f73c3e7eedb22146972ea41921164671/src/transformers/models/mobilebert/modeling_mobilebert.py#L215-L216

Due to these two transformations, if we have a single padding token, it now has an impact on the token that is right before it. One can easily test is with the following code:

```py
from transformers import MobileBertModel, MobileBertTokenizer
import torch

# Instantiate model and tokenizer
model = MobileBertModel.from_pretrained("google/mobilebert-uncased")
tokenizer = MobileBertTokenizer.from_pretrained("google/mobilebert-uncased")

# Create an array of just "1"
input_embeds = torch.ones([1, 10, 128])

# Fill the last token's embeddings with a very high value
input_embeds[:, -1, :] = 100000

resulting_embeddings = model.embeddings(inputs_embeds=input_embeds)
# Resulting embeddings of shape [1, 10, 512]

maximum_values_per_token_embedding = resulting_embeddings.squeeze().max(dim=1).values.round().tolist()
# [16.0, 17.0, 17.0, 17.0, 17.0, 17.0, 17.0, 17.0, 210926.0, 1566259.0]
```

As we can see, the last two tokens are affected by the very high value of the last token. This is due to the 1D convolution. Unfortunately, the attention mask can't really do anything about that now, as it's only aware of the last value, and only ignoring that one.

---

Steps from here: I'm contacting the author to see if we have an error in our implementation w.r.t padding tokens. In the meantime I'll think about how we can handle it from there.

Thank you for opening this issue, this is quite an error in the expected behavior vs actual behavior!

Hi all! There seems to have been an error with the weights conversion, as this issue stems from the padding token (0) embeddings seem to have values, where they should not.

Could you please confirm that adding the following line right after model instantiation solves your issues:

If the model is a `MobileBertModel` (for example with `AutoModel`)
```py
model.embeddings.word_embeddings.weight[0, :] = 0
```
@swecooo after adding the line mentioned above, running your code results in:
```out
Mean tensor(8.5514e-06, grad_fn=<MeanBackward0>)
Median tensor(6.2345e-07, grad_fn=<MedianBackward0>)
Max tensor(0.0003, grad_fn=<MaxBackward1>)
```

If the model is a mobilebert with a head, for example sequence classification:
```py
model.mobilebert.embeddings.word_embeddings.weight[0, :] = 0
```

@johnmccain after adding the line mentioned above, running your code results in:
```out
tensor([[-2765138.0000,  1868917.2500]])
tensor([[-2765139.7500,  1868916.5000]])
```

If you confirm this solves your issues, I will update the checkpoints on the hub.

Hey @LysandreJik, I can confirm that setting the pad token embeddings to zero solves the issue with my code. 
I went ahead and trained up the model on a classification task to check the real-world impact of zeroing the pad token embeddings, and I am no longer seeing discrepancies in classification output when using max_length vs longest padding 😃

Thank you!

This is great news!

Hi @LysandreJik, I can also confirm that zeroing the embedding solves the issue for me. Thanks a lot!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hey @LysandreJik, I just want to ping about the model checkpoint update, because it seems that the issue is still present in the model. I use the workaround for now, but if you found some time, it would be great to close this! :)

Thanks for the ping @sewco, I have just updated the weights. This can be closed now, feel free to reopen if you still feel something is missing.