File size: 906 Bytes
2b0945e
 
 
 
 
 
522cdd9
2b0945e
 
017391b
40309c6
c1455ff
40309c6
28198cc
 
 
 
40309c6
28198cc
2b32e49
28198cc
b448787
28198cc
2b32e49
28198cc
6fb8bef
49670bb
8e44694
49670bb
d80b8ee
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
---
license: mit
language:
- en
base_model:
- microsoft/Phi-3-mini-128k-instruct
pipeline_tag: text-generation
---

## GGUF quantized version of Phi-3 Model (128k-instruct mini)

project original [source](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) (base model)

Q_2_K (not nice)

Q_3_K_M (acceptable)

Q_4_K_M is recommanded (good for running with CPU as well)

Q_5_K_M (good in general)

Q_6_K is good also; if you want a better result; take this one instead of Q_5_K_M

Q_8_0 which is very good; need a reasonable size of RAM otherwise you might expect a long wait

16-bit and 32-bit are not provided here; since the file size is similar to the original safetensors; once you have a GPU, go ahead with the safetensors, pretty much the same

### how to run it

use any connector for interacting with gguf; i.e., [gguf-connector](https://pypi.org/project/gguf-connector/)

welcome to ai era