RuntimeError: CUDA error: an illegal instruction was encountered
Author: dongxu
Pull request: no

### System Info

When I run “AutoModelForCausalLM.from_pretrained” or chat function, I sometimes get "RuntimeError: CUDA error: an illegal instruction was encountered", but sometimes it works. Any one has ideas?
The error message is below:
> RuntimeError                              Traceback (most recent call last)
> Cell In[14], line 1
> ----> 1 model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B-Chat", device_map="auto", trust_remote_code=True).eval()
> 
> File ~\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py:488, in _BaseAutoModelClass.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
>     486     else:
>     487         cls.register(config.__class__, model_class, exist_ok=True)
> --> 488     return model_class.from_pretrained(
>     489         pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
>     490     )
>     491 elif type(config) in cls._model_mapping.keys():
>     492     model_class = _get_model_class(config, cls._model_mapping)
> 
> File ~\AppData\Roaming\Python\Python310\site-packages\transformers\modeling_utils.py:2824, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, *model_args, **kwargs)
>    2819     logger.warn(
>    2820         "This model has some weights that should be kept in higher precision, you need to upgrade "
>    2821         "`accelerate` to properly deal with them (`pip install --upgrade accelerate`)."
>    2822     )
>    2823 if device_map != "sequential":
> -> 2824     max_memory = get_balanced_memory(
>    2825         model,
>    2826         dtype=target_dtype,
>    2827         low_zero=(device_map == "balanced_low_0"),
>    2828         max_memory=max_memory,
>    2829         **kwargs,
>    2830     )
>    2831 kwargs["max_memory"] = max_memory
>    2832 # Make sure tied weights are tied before creating the device map.
> 
> File ~\AppData\Roaming\Python\Python310\site-packages\accelerate\utils\modeling.py:731, in get_balanced_memory(model, max_memory, no_split_module_classes, dtype, special_dtypes, low_zero)
>     703 """
>     704 Compute a `max_memory` dictionary for [`infer_auto_device_map`] that will balance the use of each available GPU.
>     705 
>    (...)
>     728         Transformers generate function).
>     729 """
>     730 # Get default / clean up max_memory
> --> 731 max_memory = get_max_memory(max_memory)
>     733 if not (torch.cuda.is_available() or is_xpu_available()) or is_mps_available():
>     734     return max_memory
> 
> File ~\AppData\Roaming\Python\Python310\site-packages\accelerate\utils\modeling.py:624, in get_max_memory(max_memory)
>     622 if not is_xpu_available():
>     623     for i in range(torch.cuda.device_count()):
> --> 624         _ = torch.tensor([0], device=i)
>     625     max_memory = {i: torch.cuda.mem_get_info(i)[0] for i in range(torch.cuda.device_count())}
>     626 else:
> 
> RuntimeError: CUDA error: an illegal instruction was encountered
> CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
> For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
> Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

just run the code blow:
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import GenerationConfig

# Note: The default behavior now has injection attack prevention off.
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen-7B-Chat", trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B-Chat", device_map="auto", trust_remote_code=True).eval()

# Specify hyperparameters for generation
model.generation_config = GenerationConfig.from_pretrained("Qwen/Qwen-7B-Chat", trust_remote_code=True) # 可指定不同的生成长度、top_p等相关超参

# 第一轮对话 1st dialogue turn
response, history = model.chat(tokenizer, "你好", history=None)
print(response)

but not 100% happens. Sometimes work fine, sometimes not.

### Expected behavior

I just need to run smoothly and get response.I want to know the solution.

Comments:
That's an issue for this particular repo since they use code on the Hub, the model itself is not implemented in the Transformers library :-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.