DeBERTa `p2p` attention type is not used
Author: guillaume-be
Pull request: no

### Who can help
DeBERTa: @LysandreJik

## Information

The current implementation of DeBERTa handles 3 types of attentions: `p2c`, `c2p` and `p2p` with complex branching handling the different types of attentions that may be provided by the user. It is my current understanding that the `p2p` attention currently does not impact the output of the model but leads to unnecessary  intermediate calculations and unused code.

The handling of the attention types is mainly done in the `disentangled_att_bias` method at https://github.com/huggingface/transformers/blob/73ec4340ec651ca1fe4f8ead9206297a4d4ed79c/src/transformers/models/deberta/modeling_deberta.py#L656

While the option of having `p2p` in the attention type is used on lines 673, 677 and 690, the scores are only updated if either `p2c` or `c2p` are in the attention types when these types are checked on lines 683 and 700. I am proposing to simplify the attention calculation to handle only the `c2p` and `p2c` attention types, as follows:

```python
    def disentangled_att_bias(self, query_layer, key_layer, relative_pos, rel_embeddings, scale_factor):
        if relative_pos is None:
            q = query_layer.size(-2)
            relative_pos = build_relative_position(q, key_layer.size(-2), query_layer.device)
        if relative_pos.dim() == 2:
            relative_pos = relative_pos.unsqueeze(0).unsqueeze(0)
        elif relative_pos.dim() == 3:
            relative_pos = relative_pos.unsqueeze(1)
        # bxhxqxk
        elif relative_pos.dim() != 4:
            raise ValueError(f"Relative position ids must be of dim 2 or 3 or 4. {relative_pos.dim()}")

        att_span = min(max(query_layer.size(-2), key_layer.size(-2)), self.max_relative_positions)
        relative_pos = relative_pos.long().to(query_layer.device)
        rel_embeddings = rel_embeddings[
            self.max_relative_positions - att_span : self.max_relative_positions + att_span, :
        ].unsqueeze(0)

        score = 0

        if "c2p" in self.pos_att_type:
            pos_key_layer = self.pos_proj(rel_embeddings)
            pos_key_layer = self.transpose_for_scores(pos_key_layer)
            c2p_att = torch.matmul(query_layer, pos_key_layer.transpose(-1, -2))
            c2p_pos = torch.clamp(relative_pos + att_span, 0, att_span * 2 - 1)
            c2p_att = torch.gather(c2p_att, dim=-1, index=c2p_dynamic_expand(c2p_pos, query_layer, relative_pos))
            score += c2p_att

        # position->content
        if "p2c" in self.pos_att_type:
            pos_query_layer = self.pos_q_proj(rel_embeddings)
            pos_query_layer = self.transpose_for_scores(pos_query_layer)
            pos_query_layer /= math.sqrt(pos_query_layer.size(-1) * scale_factor)
            if query_layer.size(-2) != key_layer.size(-2):
                r_pos = build_relative_position(key_layer.size(-2), key_layer.size(-2), query_layer.device)
            else:
                r_pos = relative_pos
            p2c_pos = torch.clamp(-r_pos + att_span, 0, att_span * 2 - 1)
            p2c_att = torch.matmul(key_layer, pos_query_layer.transpose(-1, -2))
            p2c_att = torch.gather(
                p2c_att, dim=-1, index=p2c_dynamic_expand(p2c_pos, query_layer, key_layer)
            ).transpose(-1, -2)

            if query_layer.size(-2) != key_layer.size(-2):
                pos_index = relative_pos[:, :, :, 0].unsqueeze(-1)
                p2c_att = torch.gather(p2c_att, dim=-2, index=pos_dynamic_expand(pos_index, p2c_att, key_layer))
            score += p2c_att
  
        return score
```

I have tested this implementation on a miminal example, and the results are identical, with the benefit of simpler attention calculation:
```python
from transformers import  DebertaForSequenceClassification, DebertaTokenizer,  DebertaConfig

if __name__ == '__main__':
    config = DebertaConfig.from_pretrained("microsoft/deberta-base-mnli")
    config.pos_att_type = ["c2p", "p2p"]
    model = DebertaForSequenceClassification.from_pretrained("microsoft/deberta-base-mnli")
    tokenizer = DebertaTokenizer.from_pretrained('microsoft/deberta-base-mnli')

    input_context = ["[CLS] I love you. [SEP] I like you. [SEP]"]

    input_ids = tokenizer(input_context, return_tensors="pt", padding=True)

    output = model(input_ids=input_ids.input_ids)
    print(output.logits.softmax(-1))
```

Simplifying this calculation also allows a marginally simpler instantiation of the `DisentangledSelfAttention` module
If you would be open to it, I'd be happy to submit these changes as part of a PR.

Thank you!


Comments:
Thanks for reporting! Pinging @BigBird01 for advice :)

Yes, P2P is not used actually. In our previous experiments, we run with P2P for ablation study and finally we didn't apply it due to the additional cost and marginal performance improvement. Please feel free to remove it.