Add support for MacOS Apple Metal "mps" backend
Author: acostin1
Pull request: no
Labels: name: bug

### System Info

```shell
MacOS, M1 architecture, Python 3.10, Pytorch 1.12 nightly, Transformers latest (4.19.2)
```


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Try to set backend to newly released "mps" backend (Apple Metal) in Pytorch.

```
from transformers import pipeline

classifier = pipeline("sentiment-analysis")
classifier.device = "mps"

classifier("We are very sad to mps backend is not supporter in Transformers.")
```




### Expected behavior

Transformers should run on the GPU. 

Instead, an error is thrown.

```

File ~/miniforge3/envs/pytorch-nightly/lib/python3.10/site-packages/transformers/pipelines/base.py:826, in Pipeline.device_placement(self)
    824         yield
    825 else:
--> 826     if self.device.type == "cuda":
    827         torch.cuda.set_device(self.device)
    829     yield

AttributeError: 'str' object has no attribute 'type'
```


Comments:
```
>>> classifier = pipeline("sentiment-analysis")
>>> classifier.device = torch.device("mps")
>>> classifier.model.to("mps")
>>> classifier("We are very sad to mps backend is not supporter in Transformers.")
[{'label': 'NEGATIVE', 'score': 0.5000945329666138}]
```

Unfortunately, as evidenced in the output, the PyTorch MPS backend is still very much broken. Even a lot of basic operations do not work correctly. For example:

```
>>> torch.arange(10, device="mps")
tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0], device='mps:0')
>>> torch.ones(10, device="mps").type(torch.int32)
tensor([1065353216, 1065353216, 1065353216, 1065353216, 1065353216, 1065353216,
        1065353216, 1065353216, 1065353216, 1065353216], device='mps:0',
       dtype=torch.int32)
```

So, it's unlikely that you can use it yet, until the Torch maintainers shake out some bugs.

I was wondering if I am using it correctly. Thanks!

I think they know about these bugs, they are being reported on the Pytorch repo.

I believe @Narsil has been working on enabling better devices for the pipelines

Hi, you can now do:

```python
classifier = pipeline("sentiment-analysis", device=torch.device("mps"))
```

Not sure how/when TF is going to add support, but we'll figure out a way to enable this cross library too afterwards.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@Narsil 
Hi I am trying to run this 
`mps_device = torch.device('mps')`
`mnli_pipe = pipeline('zero-shot-classification', device = mps_device)`
`sent = 'The weather is awesome'`
`mnli_pipe(sent, labels = ['positive', 'negative'])`
But I am getting an error 
`RuntimeError: Placeholder storage has not been allocated on MPS device!`

My guess is the 'sent' is not on the mps device. How do i send it to the mps_device?

Hi @AsaKal .

I don't have acces to a `mps` device to try and debug whatś going on.
The code that does take care of placing all tensors on the correct before running the model is here : https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/base.py#L960

If you could run a debugger from that line you could see if the tensors are correctly placed or not ! 

Any fuller stacktrace would also help. 

Hi @Narsil 

I've also just came across this issue. 

here's a fuller stack-trace:
```
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.9/site-packages/transformers/pipelines/base.py", line 842, in device_placement
    yield
  File "/usr/local/lib/python3.9/site-packages/transformers/pipelines/base.py", line 959, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/usr/local/lib/python3.9/site-packages/transformers/pipelines/text_generation.py", line 215, in _forward
    generated_sequence = self.model.generate(input_ids=input_ids, **generate_kwargs)  # BS x SL
  File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/transformers/generation_utils.py", line 1320, in generate
    return self.sample(
  File "/usr/local/lib/python3.9/site-packages/transformers/generation_utils.py", line 1938, in sample
    outputs = self(
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/transformers/models/gptj/modeling_gptj.py", line 816, in forward
    transformer_outputs = self.transformer(
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/transformers/models/gptj/modeling_gptj.py", line 617, in forward
    inputs_embeds = self.wte(input_ids)
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/sparse.py", line 158, in forward
    return F.embedding(
  File "/usr/local/lib/python3.9/site-packages/torch/nn/functional.py", line 2199, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Placeholder storage has not been allocated on MPS device!
```

from the Debugger: 
- at the line you indicated:
self.device is "mps" (of Class TextGenerationPipeline)
but self.model.device is "cpu" 

- at the last line of the stack trace (functional.py):
input.device is "mps"
weight.device is "cpu"

it seems that the loaded model is not moved to the "mps" backend, if done manually after defining the pipeline

`
generator = pipeline('text-generation',model=model,tokenizer=tokenizer,device=torch.device("mps"))
generator.model.to("mps")
`
it works. 

I'm not familiar enough with the code to suggest where to fix this, but I hope this is enough for you to track it down. 
Thank you! 


Culprit code is here !

https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/base.py#L772

Basically we only handle `cuda` and I think it's because of multi gpu setup!

Proposed fix at #18494 

I got issue here:
 NotImplementedError: The operator 'aten::quantize_per_tensor.tensor_qparams' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764. 
thanks!