internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: vllm-project/vllm

Issue:
Support both CUDA 11.8 and CUDA 12.1
Hi vLLM maintainers. I suggest maintaining compatibility with torch 2.0.1 and CUDA 11.8.0 for a few more versions. The way this would work is that you create two versions of the wheel:
- PyPi: torch 2.1.0 and CUDA 12.1.1 wheel
- GitHub release: additional torch 2.0.1 and CUDA 11.8.0 wheel

The idea is that people can still install vLLM from the GitHub release if they do not have the latest CUDA version yet. This can be achieved by the following:

```python
VLLM_VERSION = "0.2.2"
PYPI_BUILD = os.getenv("PYPI_BUILD", "0") == "1"

if not PYPI_BUILD:
    try:
        CUDA_VERSION = "".join(os.environ.get("CUDA_VERSION", torch.version.cuda).split("."))[:3]
        VLLM_VERSION += f"+cu{CUDA_VERSION}"
    except Exception as ex:
        raise RuntimeError("Your system must have an Nvidia GPU for installing vLLM")
```

In the GitHub workflow, add a conditional on which and check if the current CUDA version being used to build is the same as the one you want to release on PyPi.

```
if ( $env:CUDA_VERSION -eq $env:PYPI_CUDA_VERSION ){
    $env:PYPI_BUILD = 1
}
```

For reference, you can look at `setup.py` and `build.yaml` in AutoAWQ.
https://github.com/casper-hansen/AutoAWQ/blob/main/.github/workflows/build.yaml

_Originally posted by @casper-hansen in https://github.com/vllm-project/vllm/issues/1527#issuecomment-1799749890_
            


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -12,6 +12,8 @@
 
 ROOT_DIR = os.path.dirname(__file__)
 
+MAIN_CUDA_VERSION = "12.1"
+
 # Supported NVIDIA GPU architectures.
 SUPPORTED_ARCHS = {"7.0", "7.5", "8.0", "8.6", "8.9", "9.0"}
 
@@ -225,7 +227,7 @@ def get_path(*filepath) -> str:
     return os.path.join(ROOT_DIR, *filepath)
 
 
-def find_version(filepath: str):
+def find_version(filepath: str) -> str:
     """Extract version information from the given filepath.
 
     Adapted from https://github.com/ray-project/ray/blob/0b190ee1160eeca9796bc091e07eaebf4c85b511/python/setup.py
@@ -238,6 +240,15 @@ def find_version(filepath: str):
         raise RuntimeError("Unable to find version string.")
 
 
+def get_vllm_version() -> str:
+    version = find_version(get_path("vllm", "__init__.py"))
+    cuda_version = str(nvcc_cuda_version)
+    if cuda_version != MAIN_CUDA_VERSION:
+        cuda_version_str = cuda_version.replace(".", "")[:3]
+        version += f"+cu{cuda_version_str}"
+    return version
+
+
 def read_readme() -> str:
     """Read the README file if present."""
     p = get_path("README.md")
@@ -256,7 +267,7 @@ def get_requirements() -> List[str]:
 
 setuptools.setup(
     name="vllm",
-    version=find_version(get_path("vllm", "__init__.py")),
+    version=get_vllm_version(),
     author="vLLM Team",
     license="Apache 2.0",
     description=("A high-throughput and memory-efficient inference and "


Test patch:
Repository: vllm-project/vllm

Issue:
Error while creating inference server with EleutherAI/gpt-j-6b model.
 I am trying to run GPTj-6b on an instance with 8 L4 GPU getting the below error. Using version 0.2. but tried with latest as well.

 python -m vllm.entrypoints.api_server     --model EleutherAI/gpt-j-6b     --tensor-parallel-size 8  --dtype float16 --host 0.0.0.0 --port 8000 --gpu-memory-utilization 0.95
  File "/opt/conda/envs/myenv/lib/python3.8/site-packages/ray/_private/auto_init_hook.py", line 24, in auto_init_wrapper
    return fn(args, **kwargs)
  File "/opt/conda/envs/myenv/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(args, kwargs)
  File "/opt/conda/envs/myenv/lib/python3.8/site-packages/ray/_private/worker.py", line 2547, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(AssertionError): ray::RayWorker.execute_method() (pid=17501, ip=10.138.15.207, actor_id=6393f8b00b5b463275043b0b01000000, repr=<vllm.engine.ray_utils.RayWorker object at 0x7f1ac80b5550>)
  File "/opt/conda/envs/myenv/lib/python3.8/site-packages/vllm/engine/ray_utils.py", line 32, in execute_method
    return executor(*args, kwargs)
  File "/opt/conda/envs/myenv/lib/python3.8/site-packages/vllm/worker/worker.py", line 68, in init_model
    self.model = get_model(self.model_config)
  File "/opt/conda/envs/myenv/lib/python3.8/site-packages/vllm/model_executor/model_loader.py", line 101, in get_model
    model.load_weights(model_config.model, model_config.download_dir,
  File "/opt/conda/envs/myenv/lib/python3.8/site-packages/vllm/model_executor/models/gpt_j.py", line 251, in load_weights
    assert param_slice.shape == loaded_weight.shape
AssertionError


Discussion:
I also had the same issue. 
I think this error happened because `load_weights()` function of `gpt_j.py` didn't consider the size of TP.
same question when I load Chinese-llama-2-7b model with "awq". How can I fix it ? 

Resolving patch:
diff --git a/vllm/model_executor/models/gpt_j.py b/vllm/model_executor/models/gpt_j.py
--- a/vllm/model_executor/models/gpt_j.py
+++ b/vllm/model_executor/models/gpt_j.py
@@ -250,7 +250,7 @@ def load_weights(self,
                 if att_weight_name not in name:
                     continue
                 param = state_dict[name.replace(att_weight_name, "qkv_proj")]
-                shard_size = param.shape[1]
+                shard_size = param.shape[0] // 3
                 loaded_weight = loaded_weight[shard_size * tp_rank:shard_size *
                                               (tp_rank + 1)]
                 param_slice = param.data[shard_size * stride_id:shard_size *


Test patch:
Repository: vllm-project/vllm

Issue:
KeyError: 'decoder.layers.23.fc2.weight'  when test model "facebook/opt-6.7b"
<img width="1133" alt="image" src="https://github.com/vllm-project/vllm/assets/73372584/23ac48c4-2b1f-42e3-a55f-a252fee7901b">
It looks like the model structure and parameters don't match, how should I fix this 


Discussion:

Resolving patch:
diff --git a/vllm/model_executor/models/opt.py b/vllm/model_executor/models/opt.py
--- a/vllm/model_executor/models/opt.py
+++ b/vllm/model_executor/models/opt.py
@@ -331,6 +331,9 @@ def load_weights(self,
                 model_name_or_path, cache_dir, load_format, revision):
             if "lm_head.weight" in name:
                 continue
+            if name.startswith("decoder."):
+                name = "model." + name
+
             for (param_name, weight_name, shard_id) in stacked_params_mapping:
                 if weight_name not in name:
                     continue


Test patch:
Repository: vllm-project/vllm

Issue:
Fix num_gpus when TP > 1
 For now we should do:
```
num_gpus=self.cache_config.gpu_memory_utilization if self.parallel_config.tensor_parallel_size < 2 else 1
```

_Originally posted by @Yard1 in https://github.com/vllm-project/vllm/issues/1821#issuecomment-1833028091_
            


Discussion:

Resolving patch:
diff --git a/vllm/engine/async_llm_engine.py b/vllm/engine/async_llm_engine.py
--- a/vllm/engine/async_llm_engine.py
+++ b/vllm/engine/async_llm_engine.py
@@ -301,7 +301,16 @@ def _init_engine(self, *args,
         elif self.worker_use_ray:
             engine_class = ray.remote(num_cpus=0)(self._engine_class).remote
         else:
-            engine_class = ray.remote(num_gpus=1)(self._engine_class).remote
+            # FIXME(woosuk): This is a bit hacky. Be careful when changing the
+            # order of the arguments.
+            cache_config = args[1]
+            parallel_config = args[2]
+            if parallel_config.tensor_parallel_size == 1:
+                num_gpus = cache_config.gpu_memory_utilization
+            else:
+                num_gpus = 1
+            engine_class = ray.remote(num_gpus=num_gpus)(
+                self._engine_class).remote
         return engine_class(*args, **kwargs)
 
     async def engine_step(self) -> bool:
diff --git a/vllm/engine/llm_engine.py b/vllm/engine/llm_engine.py
--- a/vllm/engine/llm_engine.py
+++ b/vllm/engine/llm_engine.py
@@ -158,9 +158,13 @@ def _init_workers_ray(self, placement_group: "PlacementGroup",
         for bundle in placement_group.bundle_specs:
             if not bundle.get("GPU", 0):
                 continue
+            if self.parallel_config.tensor_parallel_size == 1:
+                num_gpus = self.cache_config.gpu_memory_utilization
+            else:
+                num_gpus = 1
             worker = ray.remote(
                 num_cpus=0,
-                num_gpus=self.cache_config.gpu_memory_utilization,
+                num_gpus=num_gpus,
                 scheduling_strategy=PlacementGroupSchedulingStrategy(
                     placement_group=placement_group,
                     placement_group_capture_child_tasks=True),


Test patch:
Repository: vllm-project/vllm

Issue:
benchmark_latency.py will hang  when --batchsize=1 and --n=2
benchmark_latency.py will hang if setting --batchsize=1 and --n=2or4or8.
should max_num_seqs been set as args.batch_size * args.n ？
https://github.com/vllm-project/vllm/blob/main/benchmarks/benchmark_latency.py#L23


Discussion:
Hmm I'm getting a different error, which model are you using?

```
(base) xmo@simon-dev-l4x2:~/vllm/benchmarks$ python benchmark_latency.py --batch-size=1 --n=2
Namespace(model='facebook/opt-125m', tokenizer=None, quantization=None, tensor_parallel_size=1, input_len=32, output_len=128, batch_size=1, n=2, use_beam_search=False, num_iters=3, trust_remote_code=False, dtype='auto')
Traceback (most recent call last):
  File "/home/xmo/vllm/benchmarks/benchmark_latency.py", line 101, in <module>
    main(args)
  File "/home/xmo/vllm/benchmarks/benchmark_latency.py", line 18, in main
    llm = LLM(
  File "/home/xmo/vllm/vllm/entrypoints/llm.py", line 93, in __init__
    self.llm_engine = LLMEngine.from_engine_args(engine_args)
  File "/home/xmo/vllm/vllm/engine/llm_engine.py", line 225, in from_engine_args
    engine_configs = engine_args.create_engine_configs()
  File "/home/xmo/vllm/vllm/engine/arg_utils.py", line 199, in create_engine_configs
    scheduler_config = SchedulerConfig(self.max_num_batched_tokens,
  File "/home/xmo/vllm/vllm/config.py", line 294, in __init__
    self._verify_args()
  File "/home/xmo/vllm/vllm/config.py", line 298, in _verify_args
    raise ValueError(
ValueError: max_num_batched_tokens (32) is smaller than max_model_len (2048). This effectively limits the maximum sequence length to max_num_batched_tokens and makes vLLM reject longer sequences. Please increase max_num_batched_tokens or decrease max_model_len.
```
 After this patch (https://github.com/vllm-project/vllm/pull/1198),  llm model requires max_num_batched_tokens >max_model_len , so if args.batch_size * args.input_len<max_model_len , you will get this error.  Commenting out this line will fix the problem(https://github.com/vllm-project/vllm/blob/main/benchmarks/benchmark_latency.py#L24).

Here is my test case：
python benchmark_latency.py --model falcon-7b --tokenizer falcon-7b --n 2 --batch-size 1 --trust-remote-code 
Ah you are right, I believe the calculation for `max_num_batched_tokens` need to be updated, as well as `max_model_len` for this benchmark script. However, I think `max_num_seqs` should not change because it is orthogonal to `n` as `n` is for sampling per sequence. 
is there any update for this question, when i using the qwen/Qwen-7B-Chat for the latency benchmark, it produce the same error.

Resolving patch:
diff --git a/benchmarks/benchmark_latency.py b/benchmarks/benchmark_latency.py
--- a/benchmarks/benchmark_latency.py
+++ b/benchmarks/benchmark_latency.py
@@ -19,7 +19,6 @@ def main(args: argparse.Namespace):
         tokenizer=args.tokenizer,
         quantization=args.quantization,
         tensor_parallel_size=args.tensor_parallel_size,
-        max_num_seqs=args.batch_size,
         trust_remote_code=args.trust_remote_code,
         dtype=args.dtype,
     )


Test patch:
Repository: vllm-project/vllm

Issue:
Specify `--load-format pt` for Mixtral
As explained [here](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1/discussions/3), HF pushed some changes to the official repo to make it compatible with `transformers==4.36.0.dev0`. Need to double check that this does not break vLLM integration (in particular, we don't want to load weights twice, and model name changed from `mistral` to `mixtral`).

https://github.com/vllm-project/vllm/blob/b5f882cc98e2c9c6dde7357dbac2ec0c2c57d8cd/requirements.txt#L10C20-L10C20


Discussion:
Adding `--load-format pt` seems do to the trick. I'll let you guys (@WoosukKwon @zhuohan123) advertise it (maybe an assert in the code)?
> Adding `--load-format pt` seems do to the trick. I'll let you guys (@WoosukKwon @zhuohan123) advertise it (maybe an assert in the code)?

I met this problem too. How to add this param ?

Resolving patch:
diff --git a/vllm/config.py b/vllm/config.py
--- a/vllm/config.py
+++ b/vllm/config.py
@@ -119,6 +119,16 @@ def _verify_load_format(self) -> None:
             # Force ROCm to load from pt weights if nothing specific is set
             if load_format == "auto":
                 load_format = "pt"
+
+        # FIXME(woosuk): This is a temporary hack. Support safetensor weights.
+        architectures = getattr(self.hf_config, "architectures", [])
+        if "MixtralForCausalLM" in architectures and load_format != "pt":
+            logger.info(
+                "Currently, only 'pt' format is supported for Mixtral. "
+                "Changing the format to 'pt'. This may re-download the "
+                "weights if you have downloaded the safetensor weights.")
+            load_format = "pt"
+
         self.load_format = load_format
 
     def _verify_tokenizer_mode(self) -> None:


Test patch:
Repository: vllm-project/vllm

Issue:
ROCm Docker build fails after GPTQ support
hey, i just noticed the recent most push fails on rocm
https://github.com/vllm-project/vllm/blob/b81a6a6bb342d7b9166a1c7a6b69507fb53ff33e/setup.py#L222 

can you add this line in the setup.py along with awq quantization like so
https://github.com/vllm-project/vllm/blob/b81a6a6bb342d7b9166a1c7a6b69507fb53ff33e/setup.py#L228

If this feature is suppose to support rocm can you also check as hipify is not able to import #include<hipblas.h>

Thank you

_Originally posted by @hex-plex in https://github.com/vllm-project/vllm/issues/916#issuecomment-1858585757_
            


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -219,13 +219,13 @@ def get_torch_arch_list() -> Set[str]:
     "csrc/activation_kernels.cu",
     "csrc/layernorm_kernels.cu",
     "csrc/quantization/squeezellm/quant_cuda_kernel.cu",
-    "csrc/quantization/gptq/q_gemm.cu",
     "csrc/cuda_utils_kernels.cu",
     "csrc/pybind.cpp",
 ]
 
 if _is_cuda():
     vllm_extension_sources.append("csrc/quantization/awq/gemm_kernels.cu")
+    vllm_extension_sources.append("csrc/quantization/gptq/q_gemm.cu")
 
 vllm_extension = CUDAExtension(
     name="vllm._C",
diff --git a/vllm/config.py b/vllm/config.py
--- a/vllm/config.py
+++ b/vllm/config.py
@@ -143,7 +143,7 @@ def _verify_tokenizer_mode(self) -> None:
 
     def _verify_quantization(self) -> None:
         supported_quantization = ["awq", "gptq", "squeezellm"]
-        rocm_not_supported_quantization = ["awq"]
+        rocm_not_supported_quantization = ["awq", "gptq"]
         if self.quantization is not None:
             self.quantization = self.quantization.lower()
 


Test patch:
Repository: vllm-project/vllm

Issue:
torch.distributed.all_reduce does not free memory
 I've visualized the memory usage:

* llama 7B, TP=1
<img width="3346" alt="Screenshot 2023-12-16 at 11 14 03 PM" src="https://github.com/vllm-project/vllm/assets/46394894/e6ed7069-2190-4823-8f25-8e27bd94fe35">

The activation memory is reused after every layer.

* llama-70B, TP=8
<img width="3247" alt="Screenshot 2023-12-16 at 11 20 10 PM" src="https://github.com/vllm-project/vllm/assets/46394894/b5f492bb-7262-4c06-a040-7796e0f7fc06">

**However, when using TP, the activation memory for all reduce is not reused**

_Originally posted by @WoosukKwon in https://github.com/vllm-project/vllm/pull/2031#discussion_r1429046645_
            


Discussion:

Resolving patch:
diff --git a/vllm/worker/worker.py b/vllm/worker/worker.py
--- a/vllm/worker/worker.py
+++ b/vllm/worker/worker.py
@@ -48,6 +48,14 @@ def __init__(
         self.gpu_cache = None
 
     def init_model(self, cupy_port: Optional[int] = None):
+        # torch.distributed.all_reduce does not free the input tensor until
+        # the synchronization point. This causes the memory usage to grow
+        # as the number of all_reduce calls increases. This env var disables
+        # this behavior.
+        # Related issue:
+        # https://discuss.pytorch.org/t/cuda-allocation-lifetime-for-inputs-to-distributed-all-reduce/191573
+        os.environ["TORCH_NCCL_AVOID_RECORD_STREAMS"] = "1"
+
         # This env var set by Ray causes exceptions with graph building.
         os.environ.pop("NCCL_ASYNC_ERROR_HANDLING", None)
         # Env vars will be set by Ray.


Test patch:
Repository: vllm-project/vllm

Issue:
GPTQ models don't support CUDA graph
Got the following error while running `python examples/llm_engine_example.py --model TheBloke/Mixtral-8x7B-v0.1-GPTQ --dtype half`:
```
  File "/home/wskwon/workspace/vllm/vllm/model_executor/layers/sampler.py", line 396, in _random_sample
    random_samples = torch.multinomial(probs,
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
```
The error didn't appear when the `--enforce-eager` flag was set.

*AWQ models did not raise errors.

I guess this is somehow related to exllama v2 kernels.


Discussion:
@WoosukKwon thx for your work!

You mentioned "*AWQ models did not raise errors.". Does that mean mixtral AWQ is working now or in general AWQ (not mixtral)?

I saw in https://github.com/vllm-project/vllm/issues/2074 that mixtral AWQ is not yet working, right?
Hi @r000bin, Mixtral AWQ (e.g., `ybelkada/Mixtral-8x7B-Instruct-v0.1-AWQ`) is supported.
> Hi @r000bin, Mixtral AWQ (e.g., ybelkada/Mixtral-8x7B-Instruct-v0.1-AWQ) is supported.

I am still facing the KeyError issue
![image](https://github.com/vllm-project/vllm/assets/32456947/2332d641-e93e-4895-938c-4c4cb5cce97d)

I did a few tests today and found that I have to get current cuda stream and launch kernels with `kernel<<<blocks, threads, 0, stream>>>` instead of using default `kernel<<<blocks, threads>>>` to make cuda graph stream capture to work.

Resolving patch:
diff --git a/vllm/config.py b/vllm/config.py
--- a/vllm/config.py
+++ b/vllm/config.py
@@ -181,12 +181,6 @@ def _verify_cuda_graph(self) -> None:
             self.max_context_len_to_capture = self.max_model_len
         self.max_context_len_to_capture = min(self.max_context_len_to_capture,
                                               self.max_model_len)
-        if (self.quantization in ["gptq", "squeezellm"]
-                and not self.enforce_eager):
-            # Related issue: https://github.com/vllm-project/vllm/issues/2147
-            logger.warning(f"{self.quantization} does not support CUDA graph "
-                           "yet. Disabling CUDA graph.")
-            self.enforce_eager = True
 
     def verify_with_parallel_config(
         self,


Test patch:
Repository: vllm-project/vllm

Issue:
[v0.2.7] Release Tracker
**ETA**: Jan 3rd - 4th

## Major changes

TBD

## PRs to be merged before the release

- [x] #2221 
- [ ] ~~#2293~~ (deferred)


Discussion:
Would it be possible to fix tensor parallel > 1 with AWQ and GPTQ before the new release? I’m seeing about 5 people reporting now that it’s broken (most seem to mention Mixtral)
Hi @casper-hansen, I think it was fixed by #2208 Do you still have the issue on the main branch?

Resolving patch:
diff --git a/vllm/__init__.py b/vllm/__init__.py
--- a/vllm/__init__.py
+++ b/vllm/__init__.py
@@ -8,7 +8,7 @@
 from vllm.outputs import CompletionOutput, RequestOutput
 from vllm.sampling_params import SamplingParams
 
-__version__ = "0.2.6"
+__version__ = "0.2.7"
 
 __all__ = [
     "LLM",


Test patch:
Repository: vllm-project/vllm

Issue:
prefix caching error with baichuan model
## Machine Info
- Ubuntu
- cuda driver 12.1
- T4 GPU x4

## Reproduce Step

Only change llm line in `examples/offline_inference_with_prefix.py` with
```
llm = LLM(model="baichuan-inc/Baichuan2-13B-Chat", tensor_parallel_size=4, enforce_eager=True, dtype="half", trust_remote_code=True)
``` 

## Error message:
```
triton.runtime.autotuner.OutOfResources: out of resource: shared memory, Required: 65538, Hardware limit: 65536. Reducing block sizes or `num_stages` may help.
```

When i use a common llama arch like `TInyllama` model, it's ok. Might related `Alibi`? 

@DouHappy @caoshiyi @zhuohan123  Please take a look!


Discussion:

Resolving patch:
diff --git a/vllm/model_executor/layers/triton_kernel/prefix_prefill.py b/vllm/model_executor/layers/triton_kernel/prefix_prefill.py
--- a/vllm/model_executor/layers/triton_kernel/prefix_prefill.py
+++ b/vllm/model_executor/layers/triton_kernel/prefix_prefill.py
@@ -618,7 +618,9 @@ def context_attention_fwd(q,
                               b_ctx_len,
                               max_input_len,
                               alibi_slopes=None):
-        BLOCK = 128
+
+        cap = torch.cuda.get_device_capability()
+        BLOCK = 128 if cap[0] >= 8 else 64
         # shape constraints
         Lq, Lk, Lv = q.shape[-1], k.shape[-1], v.shape[-1]
         assert Lq == Lk and Lk == Lv


Test patch:
Repository: vllm-project/vllm

Issue:
Unable to build docker image on machines without GPU
Hello all, I tried to build the ROCM docker image on a machine without GPU, but I met the problem:

> Traceback (most recent call last):
    File "/var/lib/jenkins/vllm/setup.py", line 207, in <module>
        amd_arch = get_amdgpu_offload_arch()
   File "/var/lib/jenkins/vllm/setup.py", line 59, in get_amdgpu_offload_arch
        raise RuntimeError(error_message) from e
   RuntimeError: Error: Command '['/opt/rocm/llvm/bin/amdgpu-offload-arch']' returned non-zero exit status 1.

It looks like it's unable to pip install vllm on a machine without GPU now, but I remember that vllm 0.2.1 on branch https://github.com/EmbeddedLLM/vllm-rocm/tree/v0.2.1.post1-rocm can be installed without GPUs. 

BTW, this problem also exists on CUDA platform. Hope you give some advice!



Discussion:
+1

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -282,11 +282,15 @@ def get_torch_arch_list() -> Set[str]:
                 },
             ))
 elif _is_hip():
-    amd_arch = get_amdgpu_offload_arch()
-    if amd_arch not in ROCM_SUPPORTED_ARCHS:
-        raise RuntimeError(
-            f"Only the following arch is supported: {ROCM_SUPPORTED_ARCHS}"
-            f"amdgpu_arch_found: {amd_arch}")
+    amd_archs = os.getenv("GPU_ARCHS")
+    if amd_archs is None:
+        amd_archs = get_amdgpu_offload_arch()
+    for arch in amd_archs.split(";"):
+        if arch not in ROCM_SUPPORTED_ARCHS:
+            raise RuntimeError(
+                f"Only the following arch is supported: {ROCM_SUPPORTED_ARCHS}"
+                f"amdgpu_arch_found: {arch}")
+        NVCC_FLAGS += [f"--offload-arch={arch}"]
 
 elif _is_neuron():
     neuronxcc_version = get_neuronxcc_version()


Test patch: