Repository: vllm-project/vllm

Issue:
AWQ: vLLM cannot load AWQ models in Safetensors format
Hi guys

## Issue: vLLM cannot load AWQ models saved in Safetensors format by AutoAWQ.  

### The error
```
  File "/home/vllm/vllm/vllm/model_executor/models/llama.py", line 340, in load_weights
    loaded_weight = loaded_weight.T
AttributeError: 'builtins.PySafeSlice' object has no attribute 'T'
```

### Background

AutoAWQ recently gained the ability to save models in safetensors format. I requested this was added before I started mass AWQ production, because:
- safetensors is considered the best format for the future, as it's safer and quicker
- it is hoped that Transformers will soon add native AWQ support, and when they do they will definitely require safetensors support.

Safetensors was added to AutoAWQ in this PR: https://github.com/casper-hansen/AutoAWQ/pull/47

I would really like for there to be working AWQ support in vLLM before I start mass releasing AWQs, but I also want to be able to release models in Safetensors format, so that I don't have update them all later.

So it'd be great if we could get a fix for this, so I can start pushing out lots of models for people to try.

### Example Safetensors AWQ model for testing, and log of trying to load it in vLLM

Model: [TheBlokeAI/Test-AWQ-13B-128](https://huggingface.co/TheBlokeAI/Test-AWQ-13B-128/tree/main)

Log:
```
Launching vLLM with args: --model TheBlokeAI/Test-AWQ-13B-128 --quantization awq --host 0.0.0.0
INFO 09-17 14:21:24 llm_engine.py:72] Initializing an LLM engine with config: model='TheBlokeAI/Test-AWQ-13B-128', tokenizer='TheBlokeAI/Test-AWQ-13B-128', tokenizer_mode=auto, revision=None, trust_remote_code=False, dtype=torch.float16, download_dir=None, load_format=auto, tensor_parallel_size=1, quantization=awq, seed=0)
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/vllm/vllm/vllm/entrypoints/api_server.py", line 83, in <module>
    engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/vllm/vllm/vllm/engine/async_llm_engine.py", line 487, in from_engine_args
    engine = cls(engine_args.worker_use_ray,
  File "/home/vllm/vllm/vllm/engine/async_llm_engine.py", line 271, in __init__
    self.engine = self._init_engine(*args, **kwargs)
  File "/home/vllm/vllm/vllm/engine/async_llm_engine.py", line 307, in _init_engine
    return engine_class(*args, **kwargs)
  File "/home/vllm/vllm/vllm/engine/llm_engine.py", line 105, in __init__
    self._init_workers(distributed_init_method)
  File "/home/vllm/vllm/vllm/engine/llm_engine.py", line 137, in _init_workers
    self._run_workers(
  File "/home/vllm/vllm/vllm/engine/llm_engine.py", line 688, in _run_workers
    output = executor(*args, **kwargs)
  File "/home/vllm/vllm/vllm/worker/worker.py", line 67, in init_model
    self.model = get_model(self.model_config)
  File "/home/vllm/vllm/vllm/model_executor/model_loader.py", line 92, in get_model
    model.load_weights(model_config.model, model_config.download_dir,
  File "/home/vllm/vllm/vllm/model_executor/models/llama.py", line 340, in load_weights
    loaded_weight = loaded_weight.T
AttributeError: 'builtins.PySafeSlice' object has no attribute 'T'
```

### Example model that works, not saved in Safetensors (pytorch_model.bin)

Model: [TheBlokeAI/Test-AWQ-13B-128-No_Safetensors](https://huggingface.co/TheBlokeAI/Test-AWQ-13B-128-No_Safetensors/tree/main)

Log:
```
Launching vLLM with args: --model TheBlokeAI/Test-AWQ-13B-128-No_Safetensors --quantization awq --host 0.0.0.0
INFO 09-17 14:15:33 llm_engine.py:72] Initializing an LLM engine with config: model='TheBlokeAI/Test-AWQ-13B-128-No_Safetensors', tokenizer='TheBlokeAI/Test-AWQ-13B-128-No_Safetensors', tokenizer_mode=auto, revision=None, trust_remote_code=False, dtype=torch.float16, download_dir=None, load_format=auto, tensor_parallel_size=1, quantization=awq, seed=0)
INFO 09-17 14:17:03 llm_engine.py:202] # GPU blocks: 2600, # CPU blocks: 327
INFO:     Started server process [38]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
```

Both models are Llama-2-13B-Chat-HF, saved with AutoAWQ. Only difference is one is saved in Safetensors, other is not.


Discussion:

Resolving patch:
diff --git a/vllm/model_executor/models/llama.py b/vllm/model_executor/models/llama.py
--- a/vllm/model_executor/models/llama.py
+++ b/vllm/model_executor/models/llama.py
@@ -43,8 +43,8 @@
     VocabParallelEmbedding)
 from vllm.model_executor.quantization_utils import QuantizationConfig
 from vllm.model_executor.weight_utils import (
-    load_tensor_parallel_weights, load_padded_tensor_parallel_vocab,
-    hf_model_weights_iterator)
+    convert_pyslice_to_tensor, hf_model_weights_iterator,
+    load_tensor_parallel_weights, load_padded_tensor_parallel_vocab)
 from vllm.sequence import SamplerOutput
 
 KVCache = Tuple[torch.Tensor, torch.Tensor]
@@ -337,6 +337,7 @@ def load_weights(self,
                 is_packed = self.quant_config.is_packed(name)
                 is_transposed = self.quant_config.is_transposed(name)
             if is_transposed:
+                loaded_weight = convert_pyslice_to_tensor(loaded_weight)
                 loaded_weight = loaded_weight.T
 
             is_attention_weight = False


Test patch: