DeBERTa-v2's build_relative_position method initializes tensor on cpu and costs much time
Author: qq775294390
Pull request: no

### System Info


- `transformers` version: 4.24.0
- Platform: Linux-4.14.105-1-tlinux3-0013-x86_64-with-glibc2.10
- Python version: 3.8.3
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.10.1+cu113 (True)
- Tensorflow version (GPU?): 2.10.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@LysandreJik 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

hello, I am using DeBERTa-v2 in my code, and it runs slowly. 

With `torch.profile`, I find `Self CPU time`(more than 1.7s) is much larger than `Self CUDA time`(about 30ms). And most CPU time is from function `build_relative_position`, where two tensors are initialized without specifying their `device`. So the two tensors and the following codes are running on cpu, and it costs much time.
https://github.com/huggingface/transformers/blob/main/src/transformers/models/deberta_v2/modeling_deberta_v2.py#592:~:text=q_ids%20%3D%20torch,%2C%20key_size)
![image](https://user-images.githubusercontent.com/30080441/203538636-a0698855-af21-45b0-b2fa-a344620d15d3.png)
![image](https://user-images.githubusercontent.com/30080441/203538516-eec6a4ce-9a95-49d2-9b41-debd3a993413.png)


My test script is a simple forward process with `torch.profile` to get time cost.
```
import torch
from torch.profiler import profile, record_function, ProfilerActivity
from transformers import AutoModelForMaskedLM, AutoTokenizer, FillMaskPipeline, BertTokenizer, AutoConfig, AutoModel, DebertaV2ForMaskedLM

model_path = 
data_path = 

tokenizer=AutoTokenizer.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path).to("cuda")
data = open(data_path).readlines()
max_length, batch_size = 256, 16

with torch.no_grad():
    for step in range(10):
        L = data[step * batch_size: step * batch_size + batch_size]
        batch_inputs = {i: torch.tensor(j).to('cuda') for (i, j) in
                        tokenizer(L, max_length=max_length, padding='max_length').items()}
        with profile(
                activities=[ProfilerActivity.CUDA, ProfilerActivity.CPU],
                with_stack=True,
        ) as prof:
            outputs = model(**batch_inputs)
        print(prof.key_averages(group_by_stack_n=15, group_by_input_shape=True).table(sort_by="self_cpu_time_total", row_limit=2))
```


### Expected behavior

 I hope the tensors in  function `build_relative_position` are on the proper device(cuda if accessible) when initialized, instead of making the results `to(xxx.device)` after some computation.

In my modified code where the two tensors are initialized on cuda, the cost of cpu time will be reduced to about 77ms from 1s on my machine.
![image](https://user-images.githubusercontent.com/30080441/203542086-18226d81-5b44-4135-af61-907bf8ea148a.png)
![image](https://user-images.githubusercontent.com/30080441/203542455-f3279dd9-289f-4d61-9635-f8ffe00bc682.png)

Comments:
We'd be happy to review a PR if you want to fix this :-)

> We'd be happy to review a PR if you want to fix this :-)

Thanks for replying, I've created a PR for this issue  :-)