Intel macOS system with AMD 6900XT GPU, using MPS: cannot get any usable result back from any model
Author: TheBloke
Pull request: no

### System Info

- `transformers` version: 4.28.0.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.10.10
- Huggingface_hub version: 0.13.3
- Safetensors version: not installed
- PyTorch version (GPU?): 2.1.0.dev20230402 (False) says false but I am using PyTorch on GPU via `mps`
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

OS: macOS x64 Ventura 13.3
Hardware: Intel system with AMD 6900XT GPU

### Who can help?

@sgugger (possibly PyTorch related?)

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

My goal is to try models like Alpaca and GPT4ALL on my home Intel macOS system using my AMD 6900XT GPU.

I tried and failed to get tloen's Alpaca Lora UI and the LLaMa_MPS project running (details below).  In investigating this, I have run this simple test script I wrote to test using MPS to get a result from `t5-small`. As a base for the code I used https://github.com/huggingface/transformers/issues/22122#issuecomment-1475302212 as the discussions in that thread indicated it should work fine:

```
import torch
import transformers
from transformers import T5ForConditionalGeneration, AutoTokenizer

print("PyTorch version: ", torch.__version__)
print("transformers version: ", transformers.__version__)
print()

tokenizer = AutoTokenizer.from_pretrained('t5-small', model_max_length=512)
input_string = 'translate English to German: "The house is wonderful."'
print("Input string:", input_string)

## On CPU
print("Trying CPU")
model_cpu = T5ForConditionalGeneration.from_pretrained('t5-small', device_map='auto')
print("Running on: ", model_cpu.device)

inputs = tokenizer(input_string, return_tensors='pt').input_ids
outputs = model_cpu.generate(inputs, max_length=200)
print("Decoded Output: ", tokenizer.decode(outputs[0]))
print("raw output: ", outputs)

## On MPS
print()
print("Trying mps")
model_mps = T5ForConditionalGeneration.from_pretrained('t5-small')
model_mps = model_mps.to('mps')
print("Running on: ", model_mps.device)

inputs_mps = tokenizer(input_string, return_tensors='pt').input_ids
inputs_mps = inputs_mps.to('mps')
outputs = model_mps.generate(inputs_mps, max_length=200)
try:
    print("Decoded Output: ", tokenizer.decode(outputs[0]))
except Exception as e:
    print(e)
    print("raw output: ", outputs)
```

This produces the following result; CPU works fine, MPS produces a strange, repeating and very long result which throws an exception when being decoded:
```
PyTorch version:  2.1.0.dev20230402
transformers version:  4.28.0.dev0

Input string: translate English to German: "The house is wonderful."
Trying CPU
Running on:  cpu
Decoded Output:  <pad> "Das Haus ist wunderbar."

raw output:  tensor([[    0,    96, 17266,  4598,   229, 19250,   535,     1]])

Trying mps
Running on:  mps:0
out of range integral type conversion attempted
raw output:  tensor([[                   0, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808, -9223372036854775808,
         -9223372036854775808, -9223372036854775808]], device='mps:0')
```

Before trying this script, I first tried tloen's alpaca-lora GUI which has MPS support (https://huggingface.co/spaces/tloen/alpaca-lora/blob/main/app.py), and LLaMA_MPS (https://github.com/jankais3r/LLaMA_MPS)

Both of these exhibit the same or a very similar problem: the code appears to run fine, it uses my AMD 6900XT GPU (as detected via macOS Activity Manager), but I either get no output at all (alpaca-lora), or the output is corrupted, showing each token as a `??` symbol (LLaMa_MPS).

I am an AI newbie so I'm unsure how to try and debug this but I am pretty sure that all three of these examples are exhibiting the same problem.

I don't know if it's an issue in transformers or in PyTorch which provides the MPS backend, so I thought I'd start here.

Note: I'm running PyTorch 2.1 dev version because trying 2.0.0 with alpaca-lora gave me the error `RuntimeError: MPS does not support cumsum op with int64 input` - this was fixed by updating to 2.1-dev and Ventura 13.3.  I have tested LLaMa_MPS with PyTorch 2.0.0 with the same result, so I don't believe it's specific to 2.1-dev.

Thanks in advance for any help.

### Expected behavior

The mps code shown above should output the same result as the CPU code.

Comments:
I just did some mores searching and realised that -9223372036854775808 is "the smallest value that can be stored in a 64-bit signed integer" and that this issue looks the same as the Pytorch MPS issue reported here: https://github.com/pytorch/pytorch/issues/92311

In that thread, someone reported this workaround:
> Just replace argmax(...) with max(...).indices for instance replace output.argmax(dim=1) with output.max(dim=1).indices

I don't know if this helps me here as I'm not running any PyTorch code directly, but rather calling it through Transformers.

Anyway I guess this likely shows this isn't a transformers issue but is in PyTorch, and has already been reported. In which case, apologies for not noticing this before reporting this. 

yea it looks like this is something that needs to be fixed on the PyTorch side. We can't just change the code of `generate` on our side to accommodate MPS devices and this is clearly a bug in PyTorch. So we just have to wait a bit for them to fix it.

Understood, thanks for the quick reply.