Hugging Face's logo

754geg
/
kpok

Text Generation
Transformers
GGUF
Safetensors
quantized
2-bit
3-bit
4-bit precision
5-bit
6-bit
8-bit precision
GGUF
mistral
arxiv:2304.12244
arxiv:2306.08568
arxiv:2308.09583
Inference Endpoints
text-generation-inference
Model card Files Community
754geg commited on
Commit
be3f729
•
1 Parent(s): b9f4796

Upload 2 files

Browse files
Files changed (2) hide show
  1. README.md +74 -0
  2. gitattributes +97 -0
README.md ADDED
@@ -0,0 +1,74 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - quantized
4
+ - 2-bit
5
+ - 3-bit
6
+ - 4-bit
7
+ - 5-bit
8
+ - 6-bit
9
+ - 8-bit
10
+ - GGUF
11
+ - transformers
12
+ - safetensors
13
+ - mistral
14
+ - text-generation
15
+ - arxiv:2304.12244
16
+ - arxiv:2306.08568
17
+ - arxiv:2308.09583
18
+ - license:apache-2.0
19
+ - autotrain_compatible
20
+ - endpoints_compatible
21
+ - text-generation-inference
22
+ - region:us
23
+ - text-generation
24
+ model_name: WizardLM-2-8x22B-GGUF
25
+ base_model: microsoft/WizardLM-2-8x22B
26
+ inference: false
27
+ model_creator: microsoft
28
+ pipeline_tag: text-generation
29
+ quantized_by: MaziyarPanahi
30
+ ---
31
+ # [MaziyarPanahi/WizardLM-2-8x22B-GGUF](https://huggingface.co/MaziyarPanahi/WizardLM-2-8x22B-GGUF)
32
+ - Model creator: [microsoft](https://huggingface.co/microsoft)
33
+ - Original model: [microsoft/WizardLM-2-8x22B](https://huggingface.co/microsoft/WizardLM-2-8x22B)
34
+
35
+ ## Description
36
+ [MaziyarPanahi/WizardLM-2-8x22B-GGUF](https://huggingface.co/MaziyarPanahi/WizardLM-2-8x22B-GGUF) contains GGUF format model files for [microsoft/WizardLM-2-8x22B](https://huggingface.co/microsoft/WizardLM-2-8x22B).
37
+
38
+ ## How to download
39
+ You can download only the quants you need instead of cloning the entire repository as follows:
40
+
41
+
42
+ ```
43
+ huggingface-cli download MaziyarPanahi/WizardLM-2-8x22B-GGUF --local-dir . --include '*Q2_K*gguf'
44
+ ```
45
+
46
+ On Windows:
47
+
48
+ ```sh
49
+ huggingface-cli download MaziyarPanahi/WizardLM-2-8x22B-GGUF --local-dir . --include *Q4_K_S*gguf
50
+ ```
51
+
52
+ ## Load sharded model
53
+
54
+ `llama_load_model_from_file` will detect the number of files and will load additional tensors from the rest of files.
55
+
56
+ ```sh
57
+ llama.cpp/main -m WizardLM-2-8x22B.Q2_K-00001-of-00005.gguf -p "Building a website can be done in 10 simple steps:\nStep 1:" -n 1024 -e
58
+ ```
59
+
60
+ ## Prompt template
61
+
62
+ ```
63
+ {system_prompt}
64
+ USER: {prompt}
65
+ ASSISTANT: </s>
66
+ ```
67
+
68
+ or
69
+
70
+ ```
71
+ A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful,
72
+ detailed, and polite answers to the user's questions. USER: Hi ASSISTANT: Hello.</s>
73
+ USER: {prompt} ASSISTANT: </s>......
74
+ ```
gitattributes ADDED
@@ -0,0 +1,97 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ WizardLM-2-8x22B.Q2_K-00001-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
37
+ WizardLM-2-8x22B.Q2_K-00002-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
38
+ WizardLM-2-8x22B.Q2_K-00003-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
39
+ WizardLM-2-8x22B.Q2_K-00004-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
40
+ WizardLM-2-8x22B.Q2_K-00005-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
41
+ WizardLM-2-8x22B.IQ3_XS-00001-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
42
+ WizardLM-2-8x22B.IQ3_XS-00002-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
43
+ WizardLM-2-8x22B.IQ3_XS-00003-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
44
+ WizardLM-2-8x22B.IQ3_XS-00004-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
45
+ WizardLM-2-8x22B.IQ3_XS-00005-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
46
+ WizardLM-2-8x22B.Q3_K_S-00001-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
47
+ WizardLM-2-8x22B.Q3_K_S-00002-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
48
+ WizardLM-2-8x22B.Q3_K_S-00003-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
49
+ WizardLM-2-8x22B.Q3_K_S-00004-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
50
+ WizardLM-2-8x22B.Q3_K_S-00005-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
51
+ WizardLM-2-8x22B.Q3_K_L-00001-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
52
+ WizardLM-2-8x22B.Q3_K_L-00002-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
53
+ WizardLM-2-8x22B.Q3_K_L-00003-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
54
+ WizardLM-2-8x22B.Q3_K_L-00004-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
55
+ WizardLM-2-8x22B.Q3_K_L-00005-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
56
+ WizardLM-2-8x22B.Q3_K_M-00001-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
57
+ WizardLM-2-8x22B.Q3_K_M-00002-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
58
+ WizardLM-2-8x22B.Q3_K_M-00003-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
59
+ WizardLM-2-8x22B.Q3_K_M-00004-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
60
+ WizardLM-2-8x22B.Q3_K_M-00005-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
61
+ WizardLM-2-8x22B.Q4_K_S-00001-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
62
+ WizardLM-2-8x22B.Q4_K_S-00002-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
63
+ WizardLM-2-8x22B.Q4_K_S-00003-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
64
+ WizardLM-2-8x22B.Q4_K_S-00004-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
65
+ WizardLM-2-8x22B.Q4_K_S-00005-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
66
+ WizardLM-2-8x22B.IQ4_XS-00001-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
67
+ WizardLM-2-8x22B.IQ4_XS-00002-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
68
+ WizardLM-2-8x22B.IQ4_XS-00003-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
69
+ WizardLM-2-8x22B.IQ4_XS-00004-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
70
+ WizardLM-2-8x22B.IQ4_XS-00005-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
71
+ WizardLM-2-8x22B.Q4_K_M-00001-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
72
+ WizardLM-2-8x22B.Q4_K_M-00002-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
73
+ WizardLM-2-8x22B.Q4_K_M-00003-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
74
+ WizardLM-2-8x22B.Q4_K_M-00004-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
75
+ WizardLM-2-8x22B.Q4_K_M-00005-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
76
+ WizardLM-2-8x22B.Q5_K_M-00001-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
77
+ WizardLM-2-8x22B.Q5_K_M-00002-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
78
+ WizardLM-2-8x22B.Q5_K_M-00003-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
79
+ WizardLM-2-8x22B.Q5_K_M-00004-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
80
+ WizardLM-2-8x22B.Q5_K_M-00005-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
81
+ WizardLM-2-8x22B.Q5_K_S-00001-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
82
+ WizardLM-2-8x22B.Q5_K_S-00002-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
83
+ WizardLM-2-8x22B.Q5_K_S-00003-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
84
+ WizardLM-2-8x22B.Q5_K_S-00004-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
85
+ WizardLM-2-8x22B.Q5_K_S-00005-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
86
+ WizardLM-2-8x22B.Q6_K-00001-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
87
+ WizardLM-2-8x22B.Q6_K-00002-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
88
+ WizardLM-2-8x22B.Q6_K-00003-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
89
+ WizardLM-2-8x22B.Q6_K-00004-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
90
+ WizardLM-2-8x22B.Q6_K-00005-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
91
+ WizardLM-2-8x22B.Q8_0-00001-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
92
+ WizardLM-2-8x22B.Q8_0-00002-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
93
+ WizardLM-2-8x22B.Q8_0-00003-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
94
+ WizardLM-2-8x22B.Q8_0-00004-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
95
+ WizardLM-2-8x22B.Q8_0-00005-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
96
+ WizardLM-2-8x22B.IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text
97
+ WizardLM-2-8x22B.IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text