`addmm_impl_cpu_` not implemented for 'Half'
Author: kanpuriyanawab
Pull request: no

### System Info

- `transformers` version: 4.32.1
- Platform: macOS-13.5-arm64-arm-64bit
- Python version: 3.11.3
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.3
- Accelerate version: 0.21.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): 2.13.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@gante @ArthurZucker

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Code:
```
chkpt = "meta-llama/Llama-2-7b-chat-hf"
tokenizer = AutoTokenizer.from_pretrained(chkpt)
model = AutoModelForCausalLM.from_pretrained(chkpt, torch_dtype=torch.float16)

inputs = tokenizer("Hello, I am", return_tensors="pt")
tokens = model.generate(**inputs)
```

StackTrace:

<details> <summary> Traceback </summary>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/flip/anaconda3/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/flip/anaconda3/lib/python3.11/site-packages/transformers/generation/utils.py", line 1642, in generate
    return self.sample(
           ^^^^^^^^^^^^
  File "/Users/flip/anaconda3/lib/python3.11/site-packages/transformers/generation/utils.py", line 2724, in sample
    outputs = self(
              ^^^^^
  File "/Users/flip/anaconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/flip/anaconda3/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 809, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/Users/flip/anaconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/flip/anaconda3/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 697, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/Users/flip/anaconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/flip/anaconda3/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 413, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "/Users/flip/anaconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/flip/anaconda3/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 310, in forward
    query_states = self.q_proj(hidden_states)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/flip/anaconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/flip/anaconda3/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'
</details>


### Expected behavior

Model should generate output.

Comments:
Hey! Make sure to put the model on `cuda`, the errors is here because you are trying to use `float16` on CPU which is not supported 😉 

@ArthurZucker  I have a Mac (😞 )
Does HF not support fp16 for MPS in mac?

It's not HF, it's `pytorch`. You can put the model on `mps` using `to('mps')` should work 

and then will I be able to use fp16 inference?

@ArthurZucker Sadly it doesn't work

<details><summary>Stacktrace</summary>
<p>

>>> tokens = model.generate(**inputs)
/Users/flip/anaconda3/lib/python3.11/site-packages/transformers/generation/utils.py:736: UserWarning: MPS: no support for int64 repeats mask, casting it to int32 (Triggered internally at /private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_1aidzjezue/croot/pytorch_1687856425340/work/aten/src/ATen/native/mps/operations/Repeat.mm:236.)
  input_ids = input_ids.repeat_interleave(expand_size, dim=0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/flip/anaconda3/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/flip/anaconda3/lib/python3.11/site-packages/transformers/generation/utils.py", line 1642, in generate
    return self.sample(
           ^^^^^^^^^^^^
  File "/Users/flip/anaconda3/lib/python3.11/site-packages/transformers/generation/utils.py", line 2721, in sample
    model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/flip/anaconda3/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 864, in prepare_inputs_for_generation
    position_ids = attention_mask.long().cumsum(-1) - 1
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: MPS does not support cumsum op with int64 input

</p>
</details> 

You could try to pass the position ids yourself, computing them outside mps by calling `prepare_inputs_for_generation`! Also make sure you have the latest version of `torch`


You should try the following, which usually better handles the devices. I think we can do something about this, but the problem is more with `torch` not supporting cumsum for `long()`. Will see if it's something we want to change or not
```python 
>>> from transformers import pipeline 
>>> pipeline = pipeline("text-generation", model = chkpt, device = 'mps', torch_dtype = torch.float16))
>>> print(pipeline("Where should I eat?")
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.