Repository: huggingface/transformers

Issue:
Roberta Positional Embeddings Not Masked
## Bug

Hi, I think for the RoBERTa model, the positional embeddings are created slightly wrong. In both this library and in `fairseq` the positional embeddings are sequential integers starting from `padding_idx + 1`. However, in `fairseq.utils.make_positions` all indicies of the input that correspond to the padding index are masked as they go into the positional embedding layer.

## Example

```python
import torch
padding_idx = 0

tensor = torch.as_tensor([
    [1, 13, 54, 0, 0],
    [1, 55, 12, 24, 0]
])
```

### In Fairseq

```
from fairseq import utils
utils.make_positions(tensor, padding_idx = padding_idx)
>>> tensor([
        [1, 2, 3, 0, 0],
        [1, 2, 3, 4, 0]
        ])
```

The positional sequence begins at padding_idx + 1 but all occurrences of the padding index in the input tensor are unmanipulated. This means the position embeddings are only adjusted for tokens which are not padding.

### In Transformers

This code comes from [modeling_roberta.py#L65](https://github.com/huggingface/transformers/blob/master/transformers/modeling_roberta.py#L65)
```python
# Position numbers begin at padding_idx+1. Padding symbols are ignored.
# cf. fairseq's `utils.make_positions`
position_ids = torch.arange(self.padding_idx+1, seq_length+self.padding_idx+1, dtype=torch.long, device=input_ids.device)
position_ids = position_ids.unsqueeze(0).expand_as(input_shape)
```

The output tensor of these function calls do not retain the masked tokens:
```
>>> tensor([
        [1, 2, 3, 4, 5],
        [1, 2, 3, 4, 5]
        ])
```
## Complete Example

```python

from fairseq import utils

tensor = torch.as_tensor([
    [1, 13, 54, 0, 0],
    [1, 55, 12, 24, 0]
])

def fairseq_make_positions(tensor, padding_idx):
    mask = tensor.ne(padding_idx).int()
    return (
        torch.cumsum(mask, dim=1).type_as(mask) * mask
    ).long() + padding_idx

def transformers_make_positions(input_ids, padding_idx):
    input_shape = input_ids.size()
    seq_length = input_shape[1]
    position_ids = torch.arange(padding_idx+1, seq_length+padding_idx+1, dtype=torch.long, device=input_ids.device)
    position_ids = position_ids.unsqueeze(0).expand(input_shape)
    return position_ids


print(fairseq_make_positions(tensor, padding_idx = 0))
print(transformers_make_positions(tensor, padding_idx = 0))
```
Output:
```
# Fairseq:
tensor([[1, 2, 3, 0, 0],
        [1, 2, 3, 4, 0]])
# Transformers:
tensor([[1, 2, 3, 4, 5],
        [1, 2, 3, 4, 5]])
```

## Expected behavior
I expect all padded tokens in the input tensor to be padded as they go into the positional embeddings layer as well as the token embeddings layer.

## Additional context

Happy to make a PR if this is agreed to be a bug, but this will mean the positional embeddings train slightly differently to the existing `RobertaEmbeddings` module.



Discussion:
I think you are right. Awesome if you can fix this in a PR.

Resolving patch:
diff --git a/transformers/modeling_roberta.py b/transformers/modeling_roberta.py
--- a/transformers/modeling_roberta.py
+++ b/transformers/modeling_roberta.py
@@ -51,24 +51,44 @@ def __init__(self, config):
                                                 padding_idx=self.padding_idx)
 
     def forward(self, input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None):
-        if input_ids is not None:
-            input_shape = input_ids.size()
-        else:
-            input_shape = inputs_embeds.size()[:-1]
-
-        seq_length = input_shape[1]
-        device = input_ids.device if input_ids is not None else inputs_embeds.device
-
         if position_ids is None:
-            # Position numbers begin at padding_idx+1. Padding symbols are ignored.
-            # cf. fairseq's `utils.make_positions`
-            position_ids = torch.arange(self.padding_idx+1, seq_length+self.padding_idx+1, dtype=torch.long, device=device)
-            position_ids = position_ids.unsqueeze(0).expand(input_shape)
+            if input_ids is not None:
+                # Create the position ids from the input token ids. Any padded tokens remain padded.
+                position_ids = self.create_position_ids_from_input_ids(input_ids).to(input_ids.device)
+            else:
+                position_ids = self.create_position_ids_from_inputs_embeds(inputs_embeds)
+
         return super(RobertaEmbeddings, self).forward(input_ids,
                                                       token_type_ids=token_type_ids,
                                                       position_ids=position_ids,
                                                       inputs_embeds=inputs_embeds)
 
+    def create_position_ids_from_input_ids(self, x):
+        """ Replace non-padding symbols with their position numbers. Position numbers begin at
+        padding_idx+1. Padding symbols are ignored. This is modified from fairseq's
+        `utils.make_positions`.
+
+        :param torch.Tensor x:
+        :return torch.Tensor:
+        """
+        mask = x.ne(self.padding_idx).long()
+        incremental_indicies = torch.cumsum(mask, dim=1) * mask
+        return incremental_indicies + self.padding_idx
+
+    def create_position_ids_from_inputs_embeds(self, inputs_embeds):
+        """ We are provided embeddings directly. We cannot infer which are padded so just generate
+        sequential position ids.
+
+        :param torch.Tensor inputs_embeds:
+        :return torch.Tensor:
+        """
+        input_shape = inputs_embeds.size()[:-1]
+        sequence_length = input_shape[1]
+
+        position_ids = torch.arange(self.padding_idx+1, sequence_length+self.padding_idx+1, dtype=torch.long,
+                                    device=inputs_embeds.device)
+        return position_ids.unsqueeze(0).expand(input_shape)
+
 
 ROBERTA_START_DOCSTRING = r"""    The RoBERTa model was proposed in
     `RoBERTa: A Robustly Optimized BERT Pretraining Approach`_


Test patch:
diff --git a/transformers/tests/modeling_roberta_test.py b/transformers/tests/modeling_roberta_test.py
--- a/transformers/tests/modeling_roberta_test.py
+++ b/transformers/tests/modeling_roberta_test.py
@@ -25,6 +25,7 @@
     import torch
     from transformers import (RobertaConfig, RobertaModel, RobertaForMaskedLM,
                               RobertaForSequenceClassification, RobertaForTokenClassification)
+    from transformers.modeling_roberta import RobertaEmbeddings
     from transformers.modeling_roberta import ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP
 
 from .modeling_common_test import (CommonTestCases, ids_tensor)
@@ -205,6 +206,57 @@ def test_model_from_pretrained(self):
             shutil.rmtree(cache_dir)
             self.assertIsNotNone(model)
 
+    def test_create_position_ids_respects_padding_index(self):
+        """ Ensure that the default position ids only assign a sequential . This is a regression
+        test for https://github.com/huggingface/transformers/issues/1761
+
+        The position ids should be masked with the embedding object's padding index. Therefore, the
+        first available non-padding position index is RobertaEmbeddings.padding_idx + 1
+        """
+        config = self.model_tester.prepare_config_and_inputs()[0]
+        model = RobertaEmbeddings(config=config)
+
+        input_ids = torch.as_tensor([[12, 31, 13, model.padding_idx]])
+        expected_positions = torch.as_tensor([[
+            0 + model.padding_idx + 1,
+            1 + model.padding_idx + 1,
+            2 + model.padding_idx + 1,
+            model.padding_idx
+        ]])
+
+        position_ids = model.create_position_ids_from_input_ids(input_ids)
+        self.assertEqual(
+            position_ids.shape,
+            expected_positions.shape
+        )
+        self.assertTrue(torch.all(torch.eq(position_ids, expected_positions)))
+
+    def test_create_position_ids_from_inputs_embeds(self):
+        """ Ensure that the default position ids only assign a sequential . This is a regression
+        test for https://github.com/huggingface/transformers/issues/1761
+
+        The position ids should be masked with the embedding object's padding index. Therefore, the
+        first available non-padding position index is RobertaEmbeddings.padding_idx + 1
+        """
+        config = self.model_tester.prepare_config_and_inputs()[0]
+        embeddings = RobertaEmbeddings(config=config)
+
+        inputs_embeds = torch.Tensor(2, 4, 30)
+        expected_single_positions = [
+            0 + embeddings.padding_idx + 1,
+            1 + embeddings.padding_idx + 1,
+            2 + embeddings.padding_idx + 1,
+            3 + embeddings.padding_idx + 1,
+        ]
+        expected_positions = torch.as_tensor([expected_single_positions, expected_single_positions])
+        position_ids = embeddings.create_position_ids_from_inputs_embeds(inputs_embeds)
+        self.assertEqual(
+            position_ids.shape,
+            expected_positions.shape
+        )
+        self.assertTrue(
+            torch.all(torch.eq(position_ids, expected_positions))
+        )
 
 
 class RobertaModelIntegrationTest(unittest.TestCase):