BarkModel is not working on apple silicon
Author: sagargulabani
Pull request: no
Labels: name: Audio

### System Info

transformers            4.39.1
python 3.10.14
Apple M3 Max Macbook pro.

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


I am trying to get the bark model running from here.

https://huggingface.co/docs/transformers/en/model_doc/bark




```
from transformers import AutoProcessor, BarkModel

# load in fp16 and use Flash Attention 2
import torch

mps_device = torch.device("mps")

processor = AutoProcessor.from_pretrained("suno/bark")
model = BarkModel.from_pretrained("suno/bark-small").to("mps")

# enable CPU offload
model.enable_cpu_offload()

voice_preset = "v2/en_speaker_6"
inputs = processor("Hello, my dog is cute", voice_preset=voice_preset)

audio_array = model.generate(**inputs)
audio_array = audio_array.cpu().numpy().squeeze()
````

I am running into the following error

AssertionError: Torch not compiled with CUDA enabled"



```
{
	"name": "AssertionError",
	"message": "Torch not compiled with CUDA enabled",
	"stack": "---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[21], line 4
      1 voice_preset = \"v2/en_speaker_6\"
      2 inputs = processor(\"Hello, my dog is cute\", voice_preset=voice_preset)
----> 4 audio_array = model.generate(**inputs)
      5 audio_array = audio_array.cpu().numpy().squeeze()

File /opt/anaconda3/envs/hf/lib/python3.10/site-packages/torch/utils/_contextlib.py:115, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    112 @functools.wraps(func)
    113 def decorate_context(*args, **kwargs):
    114     with ctx_factory():
--> 115         return func(*args, **kwargs)

File /opt/anaconda3/envs/hf/lib/python3.10/site-packages/transformers/models/bark/modeling_bark.py:1821, in BarkModel.generate(self, input_ids, history_prompt, return_output_lengths, **kwargs)
   1818             kwargs_fine[key] = value
   1820 # 1. Generate from the semantic model
-> 1821 semantic_output = self.semantic.generate(
   1822     input_ids,
   1823     history_prompt=history_prompt,
   1824     semantic_generation_config=semantic_generation_config,
   1825     **kwargs_semantic,
   1826 )
   1828 # 2. Generate from the coarse model
   1829 coarse_output = self.coarse_acoustics.generate(
   1830     semantic_output,
   1831     history_prompt=history_prompt,
   (...)
   1836     **kwargs_coarse,
   1837 )

File /opt/anaconda3/envs/hf/lib/python3.10/site-packages/transformers/models/bark/modeling_bark.py:981, in BarkSemanticModel.generate(self, input_ids, semantic_generation_config, history_prompt, attention_mask, **kwargs)
    973     semantic_history = torch.tensor(
    974         [semantic_generation_config.semantic_pad_token] * max_input_semantic_length, dtype=torch.int
    975     ).to(self.device)
    977 semantic_history = torch.repeat_interleave(semantic_history[None], batch_size, dim=0)
    979 infer_array = torch.tensor(
    980     [[semantic_generation_config.semantic_infer_token]] * batch_size, dtype=torch.int
--> 981 ).to(self.device)
    983 input_embeds = torch.cat(
    984     [
    985         self.input_embeds_layer(input_ids[:, :max_input_semantic_length])
   (...)
    989     dim=1,
    990 )
    992 tokens_to_suppress = list(
    993     range(semantic_generation_config.semantic_vocab_size, semantic_generation_config.semantic_pad_token)
    994 )

File /opt/anaconda3/envs/hf/lib/python3.10/site-packages/torch/cuda/__init__.py:293, in _lazy_init()
    288     raise RuntimeError(
    289         \"Cannot re-initialize CUDA in forked subprocess. To use CUDA with \"
    290         \"multiprocessing, you must use the 'spawn' start method\"
    291     )
    292 if not hasattr(torch._C, \"_cuda_getDeviceCount\"):
--> 293     raise AssertionError(\"Torch not compiled with CUDA enabled\")
    294 if _cudart is None:
    295     raise AssertionError(
    296         \"libcudart functions unavailable. It looks like you have a broken build?\"
    297     )

AssertionError: Torch not compiled with CUDA enabled"
}

```

How can I resolve this issue ?


### Expected behavior

I should be able to generate the audio. 

Comments:
I can indeed reproduce: 
```ptyhon
>>> model.device
device(type='cuda', index=0)
```
while I am on MPS.
For now you should remove the call to `model.enable_cpu_offload()` which is failing this. 
You will then get: 
```python

File ~/Work/transformers/src/transformers/generation/logits_process.py:1727, in SuppressTokensLogitsProcessor.__call__(self, input_ids, scores)
   1725 vocab_tensor = torch.arange(scores.shape[-1], device=scores.device)
   1726 suppress_tokens = torch.tensor(self.suppress_tokens, device=scores.device)
-> 1727 suppress_token_mask = torch.isin(vocab_tensor, suppress_tokens)
   1728 scores = torch.where(suppress_token_mask, -float("inf"), scores)
   1729 return scores

NotImplementedError: The operator 'aten::isin.Tensor_Tensor_out' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764. As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.
```

cc @ylacombe for the ` model.enable_cpu_offload()`

Looks like the current offloading code of Bark isn't compatible with MPS, I don't have a MPS device, any idea how I could still test a fix on MPS @ArthurZucker ?

Send me the PR, otherwise you can spin an instance I think they support `mps` now. Otherwise should I handle it? 

still needing help on this one? i have an m3 system.

Hey @bghira, it would be great!  Could you check the issue is still happening ?
Also note that we should cast the processor output to the device:
`inputs = processor("Hello, my dog is cute", voice_preset=voice_preset).to(mps_device)`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.