|
--- |
|
license: mit |
|
language: |
|
- en |
|
base_model: |
|
- microsoft/Phi-3-mini-128k-instruct |
|
--- |
|
|
|
## GGUF quantized version of Phi-3 Model (128k-instruct mini) |
|
|
|
project original [source](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) (base model) |
|
|
|
Q_2_K (not nice) |
|
|
|
Q_3_K_M (acceptable) |
|
|
|
Q_4_K_M is recommanded (good for running with CPU as well) |
|
|
|
Q_5_K_M (good in general) |
|
|
|
Q_6_K is good also; if you want a better results; take this one instead of Q_5_K_M |
|
|
|
Q_8_0 which is very good; need a reasonable size of RAM otherwise you might expect a long wait |
|
|
|
16-bit and 32-bit are not provided here; since the size is similar to the original safetensors; once you have a GPU, go ahead with the saftetensors, pretty much the same |
|
|
|
### how to run it |
|
|
|
use any connector for interacting with gguf; i.e., [gguf-connector](https://pypi.org/project/gguf-connector/) |
