Update README.md
Browse files
README.md
CHANGED
|
@@ -11,6 +11,9 @@ This model is a int4 model with group_size 128 and symmetric quantization of [Qw
|
|
| 11 |
Please follow the license of the original model.
|
| 12 |
|
| 13 |
## How To Use
|
|
|
|
|
|
|
|
|
|
| 14 |
### INT4 Inference
|
| 15 |
```python
|
| 16 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
| 11 |
Please follow the license of the original model.
|
| 12 |
|
| 13 |
## How To Use
|
| 14 |
+
|
| 15 |
+
For vllm, this pr is required https://github.com/vllm-project/vllm/pull/24818
|
| 16 |
+
|
| 17 |
### INT4 Inference
|
| 18 |
```python
|
| 19 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|