Device mismatch VITS speaker embedding and speaker_id
Author: fakhirali
Pull request: no

### System Info

- `transformers` version: 4.33.1
- Platform: Linux-6.2.0-32-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): 2.13.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@sanchit-gandhi @hollance 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

To reproduce the error

```python
from transformers import VitsModel, AutoTokenizer
import torch
model = VitsModel.from_pretrained("Matthijs/vits-vctk")
tokenizer = AutoTokenizer.from_pretrained("Matthijs/vits-vctk")
model.to('cuda')

text = "Hey, it's Hugging Face on the phone"
inputs = tokenizer(text, return_tensors="pt")
inputs.to('cuda')
inputs['speaker_id'] = 3
with torch.no_grad():
    output = model(**inputs).waveform[0]
    
# or

text = "Hey, it's Hugging Face on the phone"
inputs = tokenizer(text, return_tensors="pt")
inputs['speaker_id'] = torch.tensor(3)
inputs.to('cuda')
with torch.no_grad():
    output = model(**inputs).waveform[0]
```



### Error

```terminal
Traceback (most recent call last):
  File "/home/fakhir/Code/VoiceChat/tts.py", line 27, in <module>
    mouth.say(text)
  File "/home/fakhir/Code/VoiceChat/tts.py", line 19, in say
    output = self.model(**inputs).waveform[0].to('cpu')
  File "/home/fakhir/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/fakhir/.local/lib/python3.10/site-packages/transformers/models/vits/modeling_vits.py", line 1438, in forward
    speaker_embeddings = self.embed_speaker(torch.tensor([speaker_id])).unsqueeze(-1)
  File "/home/fakhir/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/fakhir/.local/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 162, in forward
    return F.embedding(
  File "/home/fakhir/.local/lib/python3.10/site-packages/torch/nn/functional.py", line 2210, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA__index_select)
```

My temporary solution is to add this to [modeling_vits.py](https://github.com/huggingface/transformers/blob/18ee1fe76295239335bf1528c744fe1cfba21cc8/src/transformers/models/vits/modeling_vits.py#L1438)

```python
if self.config.num_speakers > 1 and speaker_id is not None:
    if not 0 <= speaker_id < self.config.num_speakers:
        raise ValueError(f"Set `speaker_id` in the range 0-{self.config.num_speakers - 1}.")
    speaker_id = torch.tensor([speaker_id]).to(self.embed_speaker.weight.device) #added this
    speaker_embeddings = self.embed_speaker(speaker_id).unsqueeze(-1)
else:
    speaker_embeddings = None
```


Comments:
Hey @fakhirali - could you try using the official checkpoint (i.e. the one on the [kakao-enterprise org](https://huggingface.co/kakao-enterprise/vits-vctk)) and the following modified script?
```python
from transformers import VitsModel, AutoTokenizer
import torch

model = VitsModel.from_pretrained("kakao-enterprise/vits-vctk")
tokenizer = AutoTokenizer.from_pretrained("kakao-enterprise/vits-vctk")
model.to("cuda:0")

text = "Hey, it's Hugging Face on the phone"
inputs = tokenizer(text, return_tensors="pt")
inputs["speaker_id"] = torch.tensor(3)
inputs = inputs.to("cuda:0")

with torch.no_grad():
    output = model(**inputs).waveform[0]
```

=> I've tested on MPS and this script worked for me. The trick is overriding your `inputs` with the version placed on your accelerator device.

I still get the same error. It is due to the following line in [modeling_vits.py](https://github.com/huggingface/transformers/blob/18ee1fe76295239335bf1528c744fe1cfba21cc8/src/transformers/models/vits/modeling_vits.py#L1438)

```python
speaker_embeddings = self.embed_speaker(torch.tensor([speaker_id])).unsqueeze(-1)
```
I just changed it to the following and now it works.
```python
speaker_embeddings = self.embed_speaker(torch.tensor([speaker_id], device=self.device)).unsqueeze(-1)
```
I have a [fork](https://github.com/fakhirali/transformers) with this fix should I submit a PR?

Yes please! Thanks @fakhirali