Model: Qwen/Qwen2.5-7b-Instruct-GPTQ-Int4

Adapter config:
```
{
  "alpha_pattern": {},
  "auto_mapping": null,
  "base_model_name_or_path": "Qwen/Qwen2.5-14B-Instruct",
  "bias": "none",
  "fan_in_fan_out": false,
  "inference_mode": true,
  "init_lora_weights": true,
  "layer_replication": null,
  "layers_pattern": null,
  "layers_to_transform": null,
  "loftq_config": {},
  "lora_alpha": 16,
  "lora_dropout": 0.0,
  "megatron_config": null,
  "megatron_core": "megatron.core",
  "modules_to_save": null,
  "peft_type": "LORA",
  "r": 64,
  "rank_pattern": {},
  "revision": null,
  "target_modules": [
    "down_proj",
    "v_proj",
    "o_proj",
    "up_proj",
    "k_proj",
    "gate_proj",
    "q_proj"
  ],
  "task_type": "CAUSAL_LM",
  "use_dora": true,
  "use_rslora": true
}
```

But vllm is still crashing.
```
INFO 10-05 17:57:13 logger.py:36] Received request chat-840b486a221745679e3ca6a42c22e754: prompt: '
system\nYou are a helpful assistant who helps people in finding answers to complicated legal questions.Always think step by step.
\n
user\nPlease write a passage to answer the question\nTry to include as many key details as possible.\n\n\nhi\n\n\nPassage:"""\n
\n
assistant\n', params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.4, top_p=1.0, top_k=-1, min_p=0.0, seed=None, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=[], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=8132, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), prompt_token_ids: [151644, 8948, 198, 2610, 525, 264, 10950, 17847, 879, 8609, 1251, 304, 9271, 11253, 311, 16965, 5777, 4755, 9636, 2284, 1744, 3019, 553, 3019, 13, 151645, 198, 151644, 872, 198, 5501, 3270, 264, 21085, 311, 4226, 279, 3405, 198, 21453, 311, 2924, 438, 1657, 1376, 3565, 438, 3204, 4192, 6023, 1406, 12187, 424, 2974, 89262, 151645, 198, 151644, 77091, 198], lora_request: LoRARequest(lora_name='acts-lora', lora_int_id=1, lora_path='/home/azureuser/LLaMA-Factory/saves/Qwen2.5-14B-Instruct/lora/train_2024-09-26-19-25-51-unsloth-v7-bnb', lora_local_path=None, long_lora_max_len=None, base_model_name='Qwen/Qwen2.5-14b-Instruct-GPTQ-Int4'), prompt_adapter_request: None.
INFO 10-05 17:57:13 engine.py:288] Added request chat-840b486a221745679e3ca6a42c22e754.
INFO 10-05 17:57:13 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20241005-175713.pkl...
WARNING 10-05 17:57:13 model_runner_base.py:143] Failed to pickle inputs of failed execution: Can't pickle local object 'weak_bind.<locals>.weak_bound'
ERROR 10-05 17:57:13 engine.py:157] RuntimeError('Error in model execution: Loading lora /home/azureuser/LLaMA-Factory/saves/Qwen2.5-14B-Instruct/lora/train_2024-09-26-19-25-51-unsloth-v7-bnb failed')
ERROR 10-05 17:57:13 engine.py:157] Traceback (most recent call last):
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/lora/worker_manager.py", line 94, in _load_adapter
ERROR 10-05 17:57:13 engine.py:157]     lora = self._lora_model_cls.from_local_checkpoint(
ERROR 10-05 17:57:13 engine.py:157]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/lora/models.py", line 216, in from_local_checkpoint
ERROR 10-05 17:57:13 engine.py:157]     module_name, _ = parse_fine_tuned_lora_name(lora_module)
ERROR 10-05 17:57:13 engine.py:157]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/lora/utils.py", line 113, in parse_fine_tuned_lora_name
ERROR 10-05 17:57:13 engine.py:157]     raise ValueError(f"{name} is unsupported LoRA weight")
ERROR 10-05 17:57:13 engine.py:157] ValueError: base_model.model.model.layers.0.mlp.down_proj.lora_magnitude_vector is unsupported LoRA weight
ERROR 10-05 17:57:13 engine.py:157] 
ERROR 10-05 17:57:13 engine.py:157] The above exception was the direct cause of the following exception:
ERROR 10-05 17:57:13 engine.py:157] 
ERROR 10-05 17:57:13 engine.py:157] Traceback (most recent call last):
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/worker/model_runner_base.py", line 116, in _wrapper
ERROR 10-05 17:57:13 engine.py:157]     return func(*args, **kwargs)
ERROR 10-05 17:57:13 engine.py:157]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/worker/model_runner.py", line 1552, in execute_model
ERROR 10-05 17:57:13 engine.py:157]     self.set_active_loras(model_input.lora_requests,
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/worker/model_runner.py", line 1249, in set_active_loras
ERROR 10-05 17:57:13 engine.py:157]     self.lora_manager.set_active_adapters(lora_requests, lora_mapping)
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/lora/worker_manager.py", line 136, in set_active_adapters
ERROR 10-05 17:57:13 engine.py:157]     set_active_adapters_worker(requests, mapping, self._apply_adapters,
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/adapter_commons/utils.py", line 52, in set_active_adapters_worker
ERROR 10-05 17:57:13 engine.py:157]     apply_adapters_func(requests)
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/lora/worker_manager.py", line 195, in _apply_adapters
ERROR 10-05 17:57:13 engine.py:157]     self.add_adapter(lora)
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/lora/worker_manager.py", line 204, in add_adapter
ERROR 10-05 17:57:13 engine.py:157]     lora = self._load_adapter(lora_request)
ERROR 10-05 17:57:13 engine.py:157]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/lora/worker_manager.py", line 107, in _load_adapter
ERROR 10-05 17:57:13 engine.py:157]     raise RuntimeError(f"Loading lora {lora_path} failed") from e
ERROR 10-05 17:57:13 engine.py:157] RuntimeError: Loading lora /home/azureuser/LLaMA-Factory/saves/Qwen2.5-14B-Instruct/lora/train_2024-09-26-19-25-51-unsloth-v7-bnb failed
ERROR 10-05 17:57:13 engine.py:157] 
ERROR 10-05 17:57:13 engine.py:157] The above exception was the direct cause of the following exception:
ERROR 10-05 17:57:13 engine.py:157] 
ERROR 10-05 17:57:13 engine.py:157] Traceback (most recent call last):
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 155, in start
ERROR 10-05 17:57:13 engine.py:157]     self.run_engine_loop()
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 218, in run_engine_loop
ERROR 10-05 17:57:13 engine.py:157]     request_outputs = self.engine_step()
ERROR 10-05 17:57:13 engine.py:157]                       ^^^^^^^^^^^^^^^^^^
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 236, in engine_step
ERROR 10-05 17:57:13 engine.py:157]     raise e
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 227, in engine_step
ERROR 10-05 17:57:13 engine.py:157]     return self.engine.step()
ERROR 10-05 17:57:13 engine.py:157]            ^^^^^^^^^^^^^^^^^^
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 1264, in step
ERROR 10-05 17:57:13 engine.py:157]     outputs = self.model_executor.execute_model(
ERROR 10-05 17:57:13 engine.py:157]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/executor/gpu_executor.py", line 130, in execute_model
ERROR 10-05 17:57:13 engine.py:157]     output = self.driver_worker.execute_model(execute_model_req)
ERROR 10-05 17:57:13 engine.py:157]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/worker/worker_base.py", line 327, in execute_model
ERROR 10-05 17:57:13 engine.py:157]     output = self.model_runner.execute_model(
ERROR 10-05 17:57:13 engine.py:157]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 10-05 17:57:13 engine.py:157]     return func(*args, **kwargs)
ERROR 10-05 17:57:13 engine.py:157]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 10-05 17:57:13 engine.py:157]   File "/home/azureuser/miniconda3/envs/model/lib/python3.11/site-packages/vllm/worker/model_runner_base.py", line 146, in _wrapper
ERROR 10-05 17:57:13 engine.py:157]     raise type(err)(f"Error in model execution: "
ERROR 10-05 17:57:13 engine.py:157] RuntimeError: Error in model execution: Loading lora /home/azureuser/LLaMA-Factory/saves/Qwen2.5-14B-Instruct/lora/train_2024-09-26-19-25-51-unsloth-v7-bnb failed
CRITICAL 10-05 17:57:13 launcher.py:72] AsyncLLMEngine has failed, terminating server process
INFO:     127.0.0.1:35622 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [4014262]
```