Repository: vllm-project/vllm
Pull request #17699: [Misc] Remove  qlora_adapter_name_or_path
Author: login: jeejeelee
Labels: name: documentation
description: Improvements or additions to documentation
name: ready
description: ONLY add when PR is ready to merge/full CI is needed
name: ci/build
Created: 2025-05-06T07:21:07Z | Merged: 2025-05-07T06:10:38Z

`qlora_adapter_name_or_path` is a redundant argument and is now deprecated


Diff:
diff --git a/examples/offline_inference/lora_with_quantization_inference.py b/examples/offline_inference/lora_with_quantization_inference.py
index ab235ddd7545..b6608ec6e958 100644
--- a/examples/offline_inference/lora_with_quantization_inference.py
+++ b/examples/offline_inference/lora_with_quantization_inference.py
@@ -75,43 +75,38 @@ def initialize_engine(model: str, quantization: str,
                       lora_repo: Optional[str]) -> LLMEngine:
     """Initialize the LLMEngine."""
 
-    if quantization == "bitsandbytes":
-        # QLoRA (https://arxiv.org/abs/2305.14314) is a quantization technique.
-        # It quantizes the model when loading, with some config info from the
-        # LoRA adapter repo. So need to set the parameter of load_format and
-        # qlora_adapter_name_or_path as below.
-        engine_args = EngineArgs(model=model,
-                                 quantization=quantization,
-                                 qlora_adapter_name_or_path=lora_repo,
-                                 enable_lora=True,
-                                 max_lora_rank=64)
-    else:
-        engine_args = EngineArgs(model=model,
-                                 quantization=quantization,
-                                 enable_lora=True,
-                                 max_loras=4)
+    engine_args = EngineArgs(model=model,
+                             quantization=quantization,
+                             enable_lora=True,
+                             max_lora_rank=64,
+                             max_loras=4)
     return LLMEngine.from_engine_args(engine_args)
 
 
 def main():
     """Main function that sets up and runs the prompt processing."""
 
-    test_configs = [{
-        "name": "qlora_inference_example",
-        'model': "huggyllama/llama-7b",
-        'quantization': "bitsandbytes",
-        'lora_repo': 'timdettmers/qlora-flan-7b'
-    }, {
-        "name": "AWQ_inference_with_lora_example",
-        'model': 'TheBloke/TinyLlama-1.1B-Chat-v0.3-AWQ',
-        'quantization': "awq",
-        'lora_repo': 'jashing/tinyllama-colorist-lora'
-    }, {
-        "name": "GPTQ_inference_with_lora_example",
-        'model': 'TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ',
-        'quantization': "gptq",
-        'lora_repo': 'jashing/tinyllama-colorist-lora'
-    }]
+    test_configs = [
+        # QLoRA (https://arxiv.org/abs/2305.14314)
+        {
+            "name": "qlora_inference_example",
+            'model': "huggyllama/llama-7b",
+            'quantization': "bitsandbytes",
+            'lora_repo': 'timdettmers/qlora-flan-7b'
+        },
+        {
+            "name": "AWQ_inference_with_lora_example",
+            'model': 'TheBloke/TinyLlama-1.1B-Chat-v0.3-AWQ',
+            'quantization': "awq",
+            'lora_repo': 'jashing/tinyllama-colorist-lora'
+        },
+        {
+            "name": "GPTQ_inference_with_lora_example",
+            'model': 'TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ',
+            'quantization': "gptq",
+            'lora_repo': 'jashing/tinyllama-colorist-lora'
+        }
+    ]
 
     for test_config in test_configs:
         print(
diff --git a/vllm/engine/arg_utils.py b/vllm/engine/arg_utils.py
index 876a70dfe432..5d8b5d1f618a 100644
--- a/vllm/engine/arg_utils.py
+++ b/vllm/engine/arg_utils.py
@@ -6,6 +6,7 @@
 import json
 import re
 import threading
+import warnings
 from dataclasses import MISSING, dataclass, fields
 from itertools import permutations
 from typing import (Any, Callable, Dict, List, Literal, Optional, Type,
@@ -394,7 +395,13 @@ def __post_init__(self):
         if isinstance(self.compilation_config, (int, dict)):
             self.compilation_config = CompilationConfig.from_cli(
                 str(self.compilation_config))
-
+        if self.qlora_adapter_name_or_path is not None:
+            warnings.warn(
+                "The `qlora_adapter_name_or_path` is deprecated "
+                "and will be removed in v0.10.0. ",
+                DeprecationWarning,
+                stacklevel=2,
+            )
         # Setup plugins
         from vllm.plugins import load_general_plugins
         load_general_plugins()
@@ -504,10 +511,14 @@ def add_cli_args(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
                                 **load_kwargs["ignore_patterns"])
         load_group.add_argument("--use-tqdm-on-load",
                                 **load_kwargs["use_tqdm_on_load"])
-        load_group.add_argument('--qlora-adapter-name-or-path',
-                                type=str,
-                                default=None,
-                                help='Name or path of the QLoRA adapter.')
+        load_group.add_argument(
+            "--qlora-adapter-name-or-path",
+            type=str,
+            default=None,
+            help="The `--qlora-adapter-name-or-path` has no effect, do not set"
+            " it, and it  will be removed in v0.10.0.",
+            deprecated=True,
+        )
         load_group.add_argument('--pt-load-map-location',
                                 **load_kwargs["pt_load_map_location"])
 
@@ -534,7 +545,7 @@ def add_cli_args(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
             deprecated=True,
             help="[DEPRECATED] The `--enable-reasoning` flag is deprecated as "
             "of v0.8.6. Use `--reasoning-parser` to specify the reasoning "
-            "parser backend insteadThis flag (`--enable-reasoning`) will be "
+            "parser backend instead. This flag (`--enable-reasoning`) will be "
             "removed in v0.10.0. When `--reasoning-parser` is specified, "
             "reasoning mode is automatically enabled.")
         guided_decoding_group.add_argument(
@@ -896,12 +907,6 @@ def create_model_config(self) -> ModelConfig:
 
     def create_load_config(self) -> LoadConfig:
 
-        if(self.qlora_adapter_name_or_path is not None) and \
-            self.quantization != "bitsandbytes":
-            raise ValueError(
-                "QLoRA adapter only support "
-                f"'bitsandbytes' quantization, but got {self.quantization}")
-
         if self.quantization == "bitsandbytes":
             self.load_format = "bitsandbytes"
 
@@ -1098,11 +1103,6 @@ def create_engine_config(
             max_cpu_loras=self.max_cpu_loras if self.max_cpu_loras
             and self.max_cpu_loras > 0 else None) if self.enable_lora else None
 
-        if self.qlora_adapter_name_or_path is not None and \
-            self.qlora_adapter_name_or_path != "":
-            self.model_loader_extra_config[
-                "qlora_adapter_name_or_path"] = self.qlora_adapter_name_or_path
-
         # bitsandbytes pre-quantized model need a specific model loader
         if model_config.quantization == "bitsandbytes":
             self.quantization = self.load_format = "bitsandbytes"
diff --git a/vllm/model_executor/model_loader/weight_utils.py b/vllm/model_executor/model_loader/weight_utils.py
index 10bc55ca5f7d..beff33414ad7 100644
--- a/vllm/model_executor/model_loader/weight_utils.py
+++ b/vllm/model_executor/model_loader/weight_utils.py
@@ -162,23 +162,15 @@ def get_quant_config(model_config: ModelConfig,
                                   None)
     if hf_quant_config is not None:
         return quant_cls.from_config(hf_quant_config)
-    # In case of bitsandbytes/QLoRA, get quant config from the adapter model.
+    # Inflight BNB quantization
     if model_config.quantization == "bitsandbytes":
-        if (not load_config.model_loader_extra_config
-                or "qlora_adapter_name_or_path"
-                not in load_config.model_loader_extra_config):
-            return quant_cls.from_config({"adapter_name_or_path": ""})
-        model_name_or_path = load_config.model_loader_extra_config[
-            "qlora_adapter_name_or_path"]
-
-    else:
-        model_name_or_path = model_config.model
-    is_local = os.path.isdir(model_name_or_path)
+        return quant_cls.from_config({})
+    is_local = os.path.isdir(model_config.model)
     if not is_local:
         # Download the config files.
-        with get_lock(model_name_or_path, load_config.download_dir):
+        with get_lock(model_config.model, load_config.download_dir):
             hf_folder = snapshot_download(
-                model_name_or_path,
+                model_config.model,
                 revision=model_config.revision,
                 allow_patterns="*.json",
                 cache_dir=load_config.download_dir,
@@ -186,7 +178,7 @@ def get_quant_config(model_config: ModelConfig,
                 tqdm_class=DisabledTqdm,
             )
     else:
-        hf_folder = model_name_or_path
+        hf_folder = model_config.model
 
     possible_config_filenames = quant_cls.get_config_filenames()
 
@@ -213,7 +205,7 @@ def get_quant_config(model_config: ModelConfig,
         config = json.load(f)
 
         if model_config.quantization == "bitsandbytes":
-            config["adapter_name_or_path"] = model_name_or_path
+            config["adapter_name_or_path"] = model_config.model
         elif model_config.quantization == "modelopt":
             if config["producer"]["name"] == "modelopt":
                 return quant_cls.from_config(config)


Reviews:
user: login: DarkLight1337
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T07:30:49Z
user: login: DarkLight1337
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T07:40:43Z
user: login: jeejeelee
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-06T07:54:48Z
user: login: DarkLight1337
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T07:57:18Z
user: login: jeejeelee
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-06T08:01:11Z
user: login: DarkLight1337
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T08:02:43Z
user: login: jeejeelee
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-06T08:12:30Z
user: login: DarkLight1337
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-06T08:14:17Z
user: login: hmellor
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T08:59:09Z
user: login: jeejeelee
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-06T09:58:02Z
user: login: hmellor
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T10:08:22Z
user: login: jeejeelee
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-06T10:39:31Z
user: login: jeejeelee
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-06T10:40:58Z
user: login: hmellor
body: LGTM
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-06T11:56:35Z

Review comments:
diff_hunk: @@ -394,7 +395,13 @@ def __post_init__(self):
         if isinstance(self.compilation_config, (int, dict)):
             self.compilation_config = CompilationConfig.from_cli(
                 str(self.compilation_config))
-
+        if self.qlora_adapter_name_or_path is not None:
+            warnings.warn(
+                "The `qlora_adapter_name_or_path` is deprecated "
+                "and will be removed in a future release. ",
path: vllm/engine/arg_utils.py
user: login: DarkLight1337
body: Can you explain here how can users update their code?
created_at: 2025-05-06T07:30:49Z
author_association: MEMBER
diff_hunk: @@ -504,10 +511,13 @@ def add_cli_args(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
                                 **load_kwargs["ignore_patterns"])
         load_group.add_argument("--use-tqdm-on-load",
                                 **load_kwargs["use_tqdm_on_load"])
-        load_group.add_argument('--qlora-adapter-name-or-path',
-                                type=str,
-                                default=None,
-                                help='Name or path of the QLoRA adapter.')
+        load_group.add_argument(
+            "--qlora-adapter-name-or-path",
+            type=str,
+            default=None,
+            help="The `--qlora-adapter-name-or-path` is "
+            "deprecated and will be removed in a future release.",
path: vllm/engine/arg_utils.py
user: login: DarkLight1337
body: Let's indicate that it will be removed in v0.10.0
created_at: 2025-05-06T07:40:42Z
author_association: MEMBER
diff_hunk: @@ -504,10 +511,13 @@ def add_cli_args(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
                                 **load_kwargs["ignore_patterns"])
         load_group.add_argument("--use-tqdm-on-load",
                                 **load_kwargs["use_tqdm_on_load"])
-        load_group.add_argument('--qlora-adapter-name-or-path',
-                                type=str,
-                                default=None,
-                                help='Name or path of the QLoRA adapter.')
+        load_group.add_argument(
+            "--qlora-adapter-name-or-path",
+            type=str,
+            default=None,
+            help="The `--qlora-adapter-name-or-path` is "
+            "deprecated and will be removed in a future release.",
path: vllm/engine/arg_utils.py
user: login: jeejeelee
body: Done
created_at: 2025-05-06T07:54:48Z
author_association: COLLABORATOR
diff_hunk: @@ -394,7 +395,13 @@ def __post_init__(self):
         if isinstance(self.compilation_config, (int, dict)):
             self.compilation_config = CompilationConfig.from_cli(
                 str(self.compilation_config))
-
+        if self.qlora_adapter_name_or_path is not None:
+            warnings.warn(
+                "The `qlora_adapter_name_or_path` is deprecated "
+                "and will be removed in v0.10.0. ",
path: vllm/engine/arg_utils.py
user: login: DarkLight1337
body: Is there any alternative for the user to use this feature? Or is this feature simply removed?
created_at: 2025-05-06T07:57:18Z
author_association: MEMBER
diff_hunk: @@ -394,7 +395,13 @@ def __post_init__(self):
         if isinstance(self.compilation_config, (int, dict)):
             self.compilation_config = CompilationConfig.from_cli(
                 str(self.compilation_config))
-
+        if self.qlora_adapter_name_or_path is not None:
+            warnings.warn(
+                "The `qlora_adapter_name_or_path` is deprecated "
+                "and will be removed in v0.10.0. ",
path: vllm/engine/arg_utils.py
user: login: jeejeelee
body: Just remove it
created_at: 2025-05-06T08:01:11Z
author_association: COLLABORATOR
diff_hunk: @@ -394,7 +395,13 @@ def __post_init__(self):
         if isinstance(self.compilation_config, (int, dict)):
             self.compilation_config = CompilationConfig.from_cli(
                 str(self.compilation_config))
-
+        if self.qlora_adapter_name_or_path is not None:
+            warnings.warn(
+                "The `qlora_adapter_name_or_path` is deprecated "
+                "and will be removed in v0.10.0. ",
path: vllm/engine/arg_utils.py
user: login: DarkLight1337
body: Is it removed because it's unnecessary anymore?
created_at: 2025-05-06T08:02:42Z
author_association: MEMBER
diff_hunk: @@ -394,7 +395,13 @@ def __post_init__(self):
         if isinstance(self.compilation_config, (int, dict)):
             self.compilation_config = CompilationConfig.from_cli(
                 str(self.compilation_config))
-
+        if self.qlora_adapter_name_or_path is not None:
+            warnings.warn(
+                "The `qlora_adapter_name_or_path` is deprecated "
+                "and will be removed in v0.10.0. ",
path: vllm/engine/arg_utils.py
user: login: jeejeelee
body: Yeah, so we can remove it 
created_at: 2025-05-06T08:12:30Z
author_association: COLLABORATOR
diff_hunk: @@ -504,10 +511,13 @@ def add_cli_args(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
                                 **load_kwargs["ignore_patterns"])
         load_group.add_argument("--use-tqdm-on-load",
                                 **load_kwargs["use_tqdm_on_load"])
-        load_group.add_argument('--qlora-adapter-name-or-path',
-                                type=str,
-                                default=None,
-                                help='Name or path of the QLoRA adapter.')
+        load_group.add_argument(
+            "--qlora-adapter-name-or-path",
+            type=str,
+            default=None,
+            help="The `--qlora-adapter-name-or-path` has no effect, do not set"
+            " it, and it  will be removed in v0.10.0.",
+        )
path: vllm/engine/arg_utils.py
user: login: hmellor
body: You could also add `deprecated=True` here (you'd probably need to merge from main though)
created_at: 2025-05-06T08:55:11Z
author_association: MEMBER
diff_hunk: @@ -162,17 +162,8 @@ def get_quant_config(model_config: ModelConfig,
                                   None)
     if hf_quant_config is not None:
         return quant_cls.from_config(hf_quant_config)
-    # In case of bitsandbytes/QLoRA, get quant config from the adapter model.
-    if model_config.quantization == "bitsandbytes":
-        if (not load_config.model_loader_extra_config
-                or "qlora_adapter_name_or_path"
-                not in load_config.model_loader_extra_config):
-            return quant_cls.from_config({"adapter_name_or_path": ""})
-        model_name_or_path = load_config.model_loader_extra_config[
-            "qlora_adapter_name_or_path"]
 
-    else:
-        model_name_or_path = model_config.model
+    model_name_or_path = model_config.model
path: vllm/model_executor/model_loader/weight_utils.py
user: login: hmellor
body: Is this variable necessary at all now? We access the other attributes from `model_config` directly everywhere else in this method.
created_at: 2025-05-06T08:56:34Z
author_association: MEMBER
diff_hunk: @@ -394,7 +395,13 @@ def __post_init__(self):
         if isinstance(self.compilation_config, (int, dict)):
             self.compilation_config = CompilationConfig.from_cli(
                 str(self.compilation_config))
-
+        if self.qlora_adapter_name_or_path is not None:
+            warnings.warn(
path: vllm/engine/arg_utils.py
user: login: hmellor
body: Should we be using `logger.warning` instead?
created_at: 2025-05-06T08:58:20Z
author_association: MEMBER
diff_hunk: @@ -394,7 +395,13 @@ def __post_init__(self):
         if isinstance(self.compilation_config, (int, dict)):
             self.compilation_config = CompilationConfig.from_cli(
                 str(self.compilation_config))
-
+        if self.qlora_adapter_name_or_path is not None:
+            warnings.warn(
path: vllm/engine/arg_utils.py
user: login: jeejeelee
body: I searched the codebase, and for the deprecated warnings, most use this approach
created_at: 2025-05-06T09:58:02Z
author_association: COLLABORATOR
diff_hunk: @@ -394,7 +395,13 @@ def __post_init__(self):
         if isinstance(self.compilation_config, (int, dict)):
             self.compilation_config = CompilationConfig.from_cli(
                 str(self.compilation_config))
-
+        if self.qlora_adapter_name_or_path is not None:
+            warnings.warn(
path: vllm/engine/arg_utils.py
user: login: hmellor
body: Ok, I wasn't sure
created_at: 2025-05-06T10:08:22Z
author_association: MEMBER
diff_hunk: @@ -162,17 +162,8 @@ def get_quant_config(model_config: ModelConfig,
                                   None)
     if hf_quant_config is not None:
         return quant_cls.from_config(hf_quant_config)
-    # In case of bitsandbytes/QLoRA, get quant config from the adapter model.
-    if model_config.quantization == "bitsandbytes":
-        if (not load_config.model_loader_extra_config
-                or "qlora_adapter_name_or_path"
-                not in load_config.model_loader_extra_config):
-            return quant_cls.from_config({"adapter_name_or_path": ""})
-        model_name_or_path = load_config.model_loader_extra_config[
-            "qlora_adapter_name_or_path"]
 
-    else:
-        model_name_or_path = model_config.model
+    model_name_or_path = model_config.model
path: vllm/model_executor/model_loader/weight_utils.py
user: login: jeejeelee
body: Done
created_at: 2025-05-06T10:39:31Z
author_association: COLLABORATOR
diff_hunk: @@ -504,10 +511,13 @@ def add_cli_args(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
                                 **load_kwargs["ignore_patterns"])
         load_group.add_argument("--use-tqdm-on-load",
                                 **load_kwargs["use_tqdm_on_load"])
-        load_group.add_argument('--qlora-adapter-name-or-path',
-                                type=str,
-                                default=None,
-                                help='Name or path of the QLoRA adapter.')
+        load_group.add_argument(
+            "--qlora-adapter-name-or-path",
+            type=str,
+            default=None,
+            help="The `--qlora-adapter-name-or-path` has no effect, do not set"
+            " it, and it  will be removed in v0.10.0.",
+        )
path: vllm/engine/arg_utils.py
user: login: jeejeelee
body: Done
created_at: 2025-05-06T10:40:57Z
author_association: COLLABORATOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-05-06T07:21:16Z
body: 👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run `fastcheck` CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your `fastcheck` build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping `simon-mo` or `khluu` to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add `ready` label to the PR or enable auto-merge.

🚀
author_association: NONE
user: login: DarkLight1337
created_at: 2025-05-06T08:14:29Z
body: LGTM then
author_association: MEMBER
user: login: jeejeelee
created_at: 2025-05-06T14:19:31Z
body: @hmellor  It looks like we can't add `deprecated=True`, see: https://buildkite.com/vllm/ci/builds/19389#0196a552-9af7-4bcc-9835-0d684e85cc46
author_association: COLLABORATOR
user: login: hmellor
created_at: 2025-05-06T14:30:48Z
body: Oh, sorry about that, I thought https://github.com/vllm-project/vllm/pull/17426 had merged
author_association: MEMBER
user: login: jeejeelee
created_at: 2025-05-06T14:38:15Z
body: It doesn't matter, let's wait for #17426
author_association: COLLABORATOR
user: login: hmellor
created_at: 2025-05-06T15:32:37Z
body: Ok it's merged
author_association: MEMBER
user: login: DarkLight1337
created_at: 2025-05-07T06:10:36Z
body: I'll just force merge this
author_association: MEMBER