RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'
Author: jerin-metrum-ai
Pull request: no

### System Info

Transformers: 4.35.2
Torch: 2.1.1-cpu

### Who can help?

@Narsil 
I attempted text generation with the Llama 2 model using the transformer text generation pipeline on a CPU. The model precision was configured to float16 using the torch_dtype argument, but encountered the following issues. While I successfully executed FP32 and BF16 models on the CPU, it seems that transformers does not support FP16 on CPU.  Is FP16 supported by transformers on CPU ? The detailed error is provided below:
```
Traceback (most recent call last):
  File "/home/user/sagar/wrkld-llm-hf-cpu/performance-testing/llm/test.py", line 21, in <module>
    output = text_generator(
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/transformers/pipelines/text_generation.py", line 208, in __call__
    return super().__call__(text_inputs, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1140, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1147, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1046, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/transformers/pipelines/text_generation.py", line 271, in _forward
    generated_sequence = self.model.generate(input_ids=input_ids, attention_mask=attention_mask, **generate_kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/transformers/generation/utils.py", line 1673, in generate
    return self.greedy_search(
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/transformers/generation/utils.py", line 2521, in greedy_search
    outputs = self(
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1009, in forward
    outputs = self.model(
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 897, in forward
    layer_outputs = decoder_layer(
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 626, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 244, in forward
    query_states = self.q_proj(hidden_states)
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.cache/pypoetry/virtualenvs/llama2-benchmark-qWVSPitQ-py3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'
```



### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
import torch

model_id = "meta-llama/Llama-2-7b-chat-hf"
device = "cpu"
torch_dtype = torch.float16
tokenizer = AutoTokenizer.from_pretrained(model_id)
input_text = "Discuss the history and evolution of artificial intelligence in 80 words"

text_generator = pipeline(
    "text-generation",
    model=model_id,
    tokenizer=tokenizer,
    return_tensors=True,
    device=device,
    torch_dtype = torch_dtype,
)


# Inference benchmarking
output = text_generator(
    input_text,
    max_new_tokens=100,
    temperature=0.1,
)
print(tokenizer.decode(output[0]["generated_token_ids"]))
num_tokens = len(output[0]["generated_token_ids"])
```

### Expected behavior

Is FP16 supported by transformers on CPU ?

Comments:
Hello, the issue does not lie with transformers but `torch` which does not support FP16 on CPU. 🤗 

@ArthurZucker  Thanks for the quick response

Does INT8 is supported on CPU ? I was trying out load_in_8bit argument but got below error:
```
Traceback (most recent call last):
  File "/home/ubuntu/jerin/infer.py", line 14, in <module>
    model = AutoModelForCausalLM.from_pretrained(model_id, load_in_8bit=True)
  File "/home/ubuntu/jerin/venv/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 566, in from_pretrained
    return model_class.from_pretrained(
  File "/home/ubuntu/jerin/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2714, in from_pretrained
    raise ImportError(
ImportError: Using `load_in_8bit=True` requires Accelerate: `pip install accelerate` and the latest version of bitsandbytes `pip install -i https://test.pypi.org/simple/ bitsandbytes` or pip install bitsandbytes` 
```

Note: I've already installed the latest version of accelerate and bitsandbytes.

Yep this is also gonna be fixed by #27764, you need a GPU for this package. Maybe some very specific libraries (like `candle`). with custom kernels (Maybe GPTQ?) support it on CPU but it's rare and def not torch!

Hi @jerin-scalers-ai, you can also check the following [repository](https://github.com/intel/intel-extension-for-transformers). The intel team recently added support for QLora on CPU. We might support it on transformers in a near future.

bfloat16 simply has no CPU support because no CPU supports bf16 instructions, just use `torch_dtype=torch.float16`. It might fail on some particular models, but should work for most.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.