The reasons for offseting the position embedding ids by 2 for OPT Model.
Author: Zcchill
Pull request: no

### System Info

transformers 4.20.1

### Who can help?

@sgugger 
@stevhliu 
@gante 
@ArthurZucker 
@younesbelkada

### Information

- [X] The official example scripts
- [x] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction
The source code in modeling_opt.py (HuggingFace)

```python
class OPTLearnedPositionalEmbedding(nn.Embedding):
    """
    This module learns positional embeddings up to a fixed maximum size.
    """

    def __init__(self, num_embeddings: int, embedding_dim: int):
        # OPT is set up so that if padding_idx is specified then offset the embedding ids by 2
        # and adjust num_embeddings appropriately. Other models don't have this hack
        self.offset = 2
        super().__init__(num_embeddings + self.offset, embedding_dim)

    def forward(self, attention_mask: torch.LongTensor, past_key_values_length: int = 0):
        """`input_ids_shape` is expected to be [bsz x seqlen]."""
        attention_mask = attention_mask.long()

        # create positions depending on attention_mask
        positions = (torch.cumsum(attention_mask, dim=1).type_as(attention_mask) * attention_mask).long() - 1

        # cut positions if `past_key_values_length` is > 0
        positions = positions[:, past_key_values_length:]

        return super().forward(positions + self.offset)
```

The source code in metaseq 
```python
class LearnedPositionalEmbedding(nn.Embedding):
    """
    This module learns positional embeddings up to a fixed maximum size.
    Padding ids are ignored by either offsetting based on padding_idx
    or by setting padding_idx to None and ensuring that the appropriate
    position ids are passed to the forward function.
    """

    def __init__(self, num_embeddings: int, embedding_dim: int, padding_idx: int):
        super().__init__(num_embeddings, embedding_dim, padding_idx)
        if self.padding_idx is not None:
            self.max_positions = self.num_embeddings - self.padding_idx - 1
        else:
            self.max_positions = self.num_embeddings

    def forward(
        self,
        input: Tensor,
        incremental_state: Optional[Dict[str, Dict[str, Optional[Tensor]]]] = None,
        positions: Optional[Tensor] = None,
    ):
        """Input is expected to be of size [bsz x seqlen]."""
        assert (positions is None) or (
            self.padding_idx is None
        ), "If positions is pre-computed then padding_idx should not be set."

        # we cannot use incremental state here because we must be aware of
        # padding.
        if positions is None and self.padding_idx is not None:
            positions = utils.make_positions(input, self.padding_idx)

        return F.embedding(
            positions,
            self.weight,
            self.padding_idx,
            self.max_norm,
            self.norm_type,
            self.scale_grad_by_freq,
            self.sparse,
        )
```

### Expected behavior

Why do we need to conduct offset for the positional embedding here and why it is 2? I haven't found the same setting in the source code of meta.

Comments:
Hey! That's an interesting question. 
The main reason behind this is that the original code uses `nn.Embedding(..., ..., padding_idx)` where the argument is passed to the `torch module`. The `padding_idx` can be found in the tokenizer and is `1`. 
Now in transformers we realised that using the `nn.Embedding`'s native `padding_idx` is actually not very good for training. There is a big thread where you can learn more about why here : #10200. 

Since we are not doing this, we need to manually update the indexes, so we have the padding index plus the `-1` that is already there. That sums up to a shift of `2`


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.