RuntimeError: t == DeviceType::CUDA INTERNAL ASSERT FAILED at HIPGuardImplMasqueradingAsCUDA.h:60, please report a bug to PyTorch
Labels: module: rocm, triaged

### 🐛 Describe the bug

I get the following runtime error when trying to use the Koala LLM and Microsoft Guidance : 

```
Exception in thread Thread-8 (generate):
Traceback (most recent call last):

  File "/home/x00/anaconda3/envs/GPT/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/home/x00/anaconda3/envs/GPT/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/transformers/generation/utils.py", line 1522, in generate
    return self.greedy_search(
  File "/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/transformers/generation/utils.py", line 2339, in greedy_search
    outputs = self(
  File "/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 691, in forward
    outputs = self.model(
  File "/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 579, in forward
    layer_outputs = decoder_layer(
  File "/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 293, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 195, in forward
    query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
  File "/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/x00/Bureau/libs/gptq/quant.py", line 279, in forward
    quant_cuda.vecquant4matmul(x.float(), self.qweight, out, self.scales.float(), self.qzeros, self.g_idx)

RuntimeError: t == DeviceType::CUDA INTERNAL ASSERT FAILED at "/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/torch/include/ATen/hip/impl/HIPGuardImplMasqueradingAsCUDA.h":60, please report a bug to PyTorch. 
```

```
import os
import guidance
import transformers
import llama_inference

llama_inference.transformers = transformers
tokenizer = transformers.LlamaTokenizer.from_pretrained("TheBloke/koala-7B-GPTQ-4bit-128g")
model = llama_inference.load_quant("TheBloke/koala-7B-GPTQ-4bit-128g","koala-7B-4bit-128g.safetensors",4,128,0)

llm = guidance.llms.transformers.Koala(model=model, tokenizer=tokenizer)

valid_weapons = ["sword", "axe", "mace", "spear", "bow", "crossbow"]

character_maker = guidance("""The following is a character profile for an RPG game in JSON format.
```json
{
    "id": "{{id}}",
    "description": "{{description}}",
    "name": "{{gen 'name'}}",
    "age": {{gen 'age' pattern='[0-9]+' stop=','}},
    "armor": "{{#select 'armor'}}leather{{or}}chainmail{{or}}plate{{/select}}",
    "weapon": "{{select 'weapon' options=valid_weapons}}",
    "class": "{{gen 'class'}}",
    "mantra": "{{gen 'mantra' temperature=0.7}}",
    "strength": {{gen 'strength' pattern='[0-9]+' stop=','}},
    "items": [{{#geneach 'items' num_iterations=5 join=', '}}"{{gen 'this' temperature=0.7}}"{{/geneach}}]
}```""")

character_maker(
    id="e1f491f7-7ab8-4dac-8c20-c92b5e7d883d",
    description="A quick and nimble fighter.",
    valid_weapons=valid_weapons, llm=llm
)
```

Expected behavior : This code should return the json completed by the LLM

I use this model but get the same error with others : https://huggingface.co/TheBloke/koala-7B-GPTQ-4bit-128g/tree/main
The llama_inference library comes from GPTQ-for-LLaMa (Cuda branch) : https://github.com/qwopqwop200/GPTQ-for-LLaMa

### Versions

PyTorch version: 2.0.1+rocm5.4.2
Is debug build: False
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: 5.4.22803-474e8620

OS: Linux Mint 21.1 (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0
Clang version: Could not collect
CMake version: version 3.25.0
Libc version: glibc-2.35

Python version: 3.10.11 (main, May 16 2023, 00:28:57) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-75-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.5.119
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: AMD Radeon RX 6800 XT
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: 5.4.22803
MIOpen runtime version: 2.19.0
Is XNNPACK available: True

CPU:
Architecture :                          x86_64
Mode(s) opératoire(s) des processeurs : 32-bit, 64-bit
Address sizes:                          48 bits physical, 48 bits virtual
Boutisme :                              Little Endian
Processeur(s) :                         12
Liste de processeur(s) en ligne :       0-11
Identifiant constructeur :              AuthenticAMD
Nom de modèle :                         AMD Ryzen 5 5500
Famille de processeur :                 25
Modèle :                                80
Thread(s) par cœur :                    2
Cœur(s) par socket :                    6
Socket(s) :                             1
Révision :                              0
Frequency boost:                        disabled
Vitesse maximale du processeur en MHz : 3800,0000
Vitesse minimale du processeur en MHz : 1400,0000
BogoMIPS :                              7600.33
Drapaux :                               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm
Virtualisation :                        AMD-V
Cache L1d :                             192 KiB (6 instances)
Cache L1i :                             192 KiB (6 instances)
Cache L2 :                              3 MiB (6 instances)
Cache L3 :                              16 MiB (1 instance)
Nœud(s) NUMA :                          1
Nœud NUMA 0 de processeur(s) :          0-11
Vulnerability Itlb multihit:            Not affected
Vulnerability L1tf:                     Not affected
Vulnerability Mds:                      Not affected
Vulnerability Meltdown:                 Not affected
Vulnerability Mmio stale data:          Not affected
Vulnerability Retbleed:                 Not affected
Vulnerability Spec store bypass:        Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:               Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:               Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                    Not affected
Vulnerability Tsx async abort:          Not affected

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] pytorch-lightning==2.0.3
[pip3] pytorch-triton-rocm==2.0.1
[pip3] torch==2.0.1+rocm5.4.2
[pip3] torchaudio==2.0.2+rocm5.4.2
[pip3] torchmetrics==0.11.4
[pip3] torchvision==0.15.2+rocm5.4.2
[conda] numpy                     1.24.1                   pypi_0    pypi
[conda] pytorch-lightning         2.0.3                    pypi_0    pypi
[conda] pytorch-triton-rocm       2.0.1                    pypi_0    pypi
[conda] torch                     2.0.1+rocm5.4.2          pypi_0    pypi
[conda] torchaudio                2.0.2+rocm5.4.2          pypi_0    pypi
[conda] torchmetrics              0.11.4                   pypi_0    pypi
[conda] torchvision               0.15.2+rocm5.4.2          pypi_0    pypi

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang