Positional embeddings are not applied when input embeddings are passed in for Pytorch DistilBert model
Author: randimah
Pull request: no

## Environment info
- `transformers` version: 4.1.1
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.6.6
- PyTorch version (GPU?): 1.8.0+cpu (False)
- Tensorflow version (GPU?): 2.4.0 (False)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@julien-c 

Models:
- albert, bert, xlm: @LysandreJik

## Information

Model I am using (DistilBertModel):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load pre-trained models for DistilBertForSequenceClassification and DistilBertTokenizer
2. Encode input text
3. Find input embeddings by passing in input ids through pre-trained model's word emebedding layer
4. Forward pass model with encoded input ids
5. Forward pass the model with input embeddings found in step 3
6. Compare the logits of steps 4 and 5
```python
from transformers import DistilBertTokenizer, DistilBertForSequenceClassification

input_text = '''This is some sample text. But I would like a model prediction on this.'''

pre_trained_model = 'distilbert-base-uncased'

model = DistilBertForSequenceClassification.from_pretrained(pre_trained_model)
tokenizer = DistilBertTokenizer.from_pretrained(pre_trained_model)

encoded_tokens = tokenizer.encode_plus(input_text, add_special_tokens=True, return_token_type_ids=True,
                                       return_tensors='pt')
input_embeds = model.distilbert.embeddings.word_embeddings(encoded_tokens['input_ids'])

scores_for_input_ids = model(input_ids=encoded_tokens['input_ids'], attention_mask=encoded_tokens["attention_mask"])
scores_for_input_embeds = model(inputs_embeds=input_embeds, attention_mask=encoded_tokens["attention_mask"])

print('Logits for input ids', scores_for_input_ids.logits)
print('Logits for input embeds', scores_for_input_embeds.logits)

```
Output
Logits for input ids tensor([[-0.0721,  0.0499]], grad_fn=<AddmmBackward>)
Logits for input embeds tensor([[ 0.0675, -0.0452]], grad_fn=<AddmmBackward>)

## Expected behavior

The logits returned for steps 4 and 5 above should be the same. For other pytorch models such as Bert and Roberta as well as Tensorflow implementation of DistilBert (TFDistilBert) the logits returned for steps 4 and 5 are the same.
```python
from transformers import DistilBertTokenizer, BertForSequenceClassification

input_text = '''This is some sample text. But I would like a model prediction on this.'''

pre_trained_model = 'bert-base-uncased'

model = BertForSequenceClassification.from_pretrained(pre_trained_model)
tokenizer = DistilBertTokenizer.from_pretrained(pre_trained_model)

encoded_tokens = tokenizer.encode_plus(input_text, add_special_tokens=True, return_token_type_ids=True,
                                       return_tensors='pt')
input_embeds = model.bert.embeddings.word_embeddings(encoded_tokens['input_ids'])

scores_for_input_ids = model(input_ids=encoded_tokens['input_ids'], attention_mask=encoded_tokens["attention_mask"])
scores_for_input_embeds = model(inputs_embeds=input_embeds, attention_mask=encoded_tokens["attention_mask"])

print('Logits for input ids', scores_for_input_ids.logits)
print('Logits for input embeds', scores_for_input_embeds.logits)
```
Output
Logits for input ids tensor([[-0.1336,  0.1173]], grad_fn=<AddmmBackward>)
Logits for input embeds tensor([[-0.1336,  0.1173]], grad_fn=<AddmmBackward>)

I digged deeper into this in the transformers library and found out that positional embeddings are not applied when input embeddings are passed for the model forward pass particularly in the DistilBert model. On the other hand if the input ids are passed in, input embeddings are calculated from input ids and positional embeddings are applied on top of that before passing into the underlying transformer. In https://github.com/huggingface/transformers/blob/master/src/transformers/models/distilbert/modeling_distilbert.py lines 479-482.


Comments:
That's correct indeed, and seems like a bug to me. Would you like to open a PR to fix the issue?

Thanks for the confirmation. Yes, I should be able to do that.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.