`device_map = "auto"` failed for LLaMA model on H800
Author: ruikangliu
Pull request: no

### System Info

When I use `device_map = "auto"`  for LLaMA model on more than 3 H800 GPUs, errors pop up during model inference. But when I use fewer than 3 H800 GPUs, everything is OK. It seems to be something wrong with data transfer across devices on H800 GPUs.

My transformers version is 4.36.0, cuda version is 11.8, torch version is 2.0.0. I also tried transformers 4.35.2, cuda 12.1, torch 2.1.1, which also failed.

```
Exception has occurred: RuntimeError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
  File "/home/liuruikang/anaconda3/envs/ptqlora/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 1193, in forward
    logits = logits.float()
  File "/home/liuruikang/anaconda3/envs/ptqlora/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/liuruikang/anaconda3/envs/ptqlora/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/liuruikang/anaconda3/envs/ptqlora/lib/python3.8/site-packages/transformers/generation/utils.py", line 2579, in greedy_search
    outputs = self(
  File "/home/liuruikang/anaconda3/envs/ptqlora/lib/python3.8/site-packages/transformers/generation/utils.py", line 1718, in generate
    return self.greedy_search(
  File "/home/liuruikang/anaconda3/envs/ptqlora/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/liuruikang/anaconda3/envs/ptqlora/lib/python3.8/site-packages/transformers/pipelines/text_generation.py", line 271, in _forward
    generated_sequence = self.model.generate(input_ids=input_ids, attention_mask=attention_mask, **generate_kwargs)
  File "/home/liuruikang/anaconda3/envs/ptqlora/lib/python3.8/site-packages/transformers/pipelines/base.py", line 1046, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/home/liuruikang/anaconda3/envs/ptqlora/lib/python3.8/site-packages/transformers/pipelines/base.py", line 1147, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
  File "/home/liuruikang/anaconda3/envs/ptqlora/lib/python3.8/site-packages/transformers/pipelines/base.py", line 1140, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/home/liuruikang/anaconda3/envs/ptqlora/lib/python3.8/site-packages/transformers/pipelines/text_generation.py", line 208, in __call__
    return super().__call__(text_inputs, **kwargs)
  File "/home/liuruikang/workspace/quant/ptq-lora/tmp.py", line 7, in <module>
    print(generator("More and more large language models are opensourced so Hugging Face has"))
  File "/home/liuruikang/anaconda3/envs/ptqlora/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/liuruikang/anaconda3/envs/ptqlora/lib/python3.8/runpy.py", line 194, in _run_module_as_main (Current frame)
    return _run_code(code, main_globals, None,
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```

Here is the sample code for reproducing the error (more than 3 H800 GPUs needed):

```
import torch
from transformers import pipeline

checkpoint = "./modelzoo/llama/llama-7b"    # path to llama ckpt
generator = pipeline("text-generation", model=checkpoint, device_map="auto", torch_dtype=torch.float16)

print(generator("More and more large language models are opensourced so Hugging Face has"))
```

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Here is the sample code for reproducing the error (more than 3 H800 GPUs needed):

```
import torch
from transformers import pipeline

checkpoint = "./modelzoo/llama/llama-7b"    # path to llama ckpt
generator = pipeline("text-generation", model=checkpoint, device_map="auto", torch_dtype=torch.float16)

print(generator("More and more large language models are opensourced so Hugging Face has"))
```

### Expected behavior

`device_map = "auto"` failed for LLaMA model on more than 3 H800 GPUs during model inference

Comments:
cc @younesbelkada 

This may arise from hardware issues... Thanks for your time and attention.

> This may arise from hardware issues... Thanks for your time and attention.

hi, i have encountered the same issue with a4500 and a6000 gpu. may i ask if you have identified the cause of this problem later?