LayerNormKernelImpl" not implemented for 'Half when using bloom-560m with int8, device cpu
Author: Nguyendat-bit
Pull request: no

### System Info

- `transformers` version: 4.31.0
- Platform: Linux-5.4.0-153-generic-x86_64-with-glibc2.31
- Python version: 3.10.0
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0.dev20230810+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

Hi @ArthurZucker, I'm encountering an error: 'LayerNormKernelImpl' not implemented for 'Half' when generating text with the Bloom-560m + int8 model, device=cpu. However, when I run it on CUDA, the error does not occur. Could you help me, please?
![image](https://github.com/huggingface/transformers/assets/34463182/e0331364-a9f0-4861-bf7a-6f05d1b4bfd8)

![image](https://github.com/huggingface/transformers/assets/34463182/2e1b1da7-eb24-48de-b72f-16d417108909)

Thanks you

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

.

### Expected behavior

.

Comments:
Output exceeds the [size limit](command:workbench.action.openSettings?%5B%22notebook.output.textLineLimit%22%5D). Open the full output data [in a text editor](command:workbench.action.openLargeOutput?feedb4df-81a7-4c54-ab34-8c946d0d421b)---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[12], line 1
----> 1 generate("1 + 1 ? ")

Cell In[9], line 7, in generate(text)
      2 with torch.no_grad():
      3     encode= tokenizer.batch_encode_plus(list(map(lambda x: x.format(qus= text[:256]), prompts)), 
      4                                padding= 'longest',
      5                                return_tensors= 'pt')
----> 7 output_sequences= model.generate(input_ids= encode['input_ids'], attention_mask= encode['attention_mask'], max_new_tokens= 20)
      8 return tokenizer.batch_decode(output_sequences, skip_special_tokens= True)

File [~/miniconda3/envs/cls-lm/lib/python3.10/site-packages/torch/utils/_contextlib.py:115](https://vscode-remote+ssh-002dremote-002b183-002e81-002e34-002e200.vscode-resource.vscode-cdn.net/home/tungnk/cls-lm/Doamin-news/experiment/~/miniconda3/envs/cls-lm/lib/python3.10/site-packages/torch/utils/_contextlib.py:115), in context_decorator..decorate_context(*args, **kwargs)
    112 @functools.wraps(func)
    113 def decorate_context(*args, **kwargs):
    114     with ctx_factory():
--> 115         return func(*args, **kwargs)

File [~/miniconda3/envs/cls-lm/lib/python3.10/site-packages/transformers/generation/utils.py:1538](https://vscode-remote+ssh-002dremote-002b183-002e81-002e34-002e200.vscode-resource.vscode-cdn.net/home/tungnk/cls-lm/Doamin-news/experiment/~/miniconda3/envs/cls-lm/lib/python3.10/site-packages/transformers/generation/utils.py:1538), in GenerationMixin.generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, **kwargs)
   1532         raise ValueError(
   1533             "num_return_sequences has to be 1 when doing greedy search, "
   1534             f"but is {generation_config.num_return_sequences}."
   1535         )
   1537     # 11. run greedy search
...
   2543         layer_norm, (input, weight, bias), input, normalized_shape, weight=weight, bias=bias, eps=eps
   2544     )
-> 2545 return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)

RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

The 8-bit quantization with `load_in_8bit=True` only works on the GPU. You can't use it (or in general flloat16) on the CPU.

> The 8-bit quantization with `load_in_8bit=True` only works on the GPU. You can't use it (or in general flloat16) on the CPU.

Thanks you