Jezzarax commited on
Commit
ba6bec3
1 Parent(s): 1631d92

Initial commit with draft readme and quantized models

Browse files
.gitattributes CHANGED
@@ -33,3 +33,13 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ yi-6b.Q4_K.gguf filter=lfs diff=lfs merge=lfs -text
37
+ yi-6b.Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
38
+ yi-6b.Q5_K.gguf filter=lfs diff=lfs merge=lfs -text
39
+ yi-6b.Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
40
+ yi-6b.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
41
+ yi-6b.Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
42
+ yi-6b.Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
43
+ yi-6b.Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
44
+ yi-6b.f16.gguf filter=lfs diff=lfs merge=lfs -text
45
+ yi-6b.Q3_K.gguf filter=lfs diff=lfs merge=lfs -text
config.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ {
2
+ "model_type": "yi"
3
+ }
readme.md ADDED
@@ -0,0 +1,328 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: 01-ai/Yi-6B
3
+ inference: false
4
+ license: other
5
+ license_link: LICENSE
6
+ license_name: yi-license
7
+ model_creator: 01-ai
8
+ model_name: Yi 6B
9
+ model_type: yi
10
+ prompt_template: 'Human: {prompt} Assistant:
11
+
12
+ '
13
+ quantized_by: jezzarax
14
+ ---
15
+ <!-- markdownlint-disable MD041 -->
16
+
17
+ # Yi 6B - GGUF
18
+ - Model creator: [01-ai](https://huggingface.co/01-ai)
19
+ - Original model: [Yi 34B](https://huggingface.co/01-ai/Yi-6B)
20
+ - Readme and repo format by [TheBloke](https://huggingface.co/TheBloke/) and his [Yi-34B-GGUF repo](https://huggingface.co/TheBloke/Yi-34B-GGUF)
21
+
22
+ <!-- description start -->
23
+ ## Description
24
+
25
+ This repo contains GGUF format model files for [01-ai's Yi 6B](https://huggingface.co/01-ai/Yi-6B).
26
+
27
+ These files were quantised using hardware kindly provided by [Massed Compute](https://massedcompute.com/).
28
+
29
+ <!-- description end -->
30
+ <!-- README_GGUF.md-about-gguf start -->
31
+ ### About GGUF
32
+
33
+ GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.
34
+
35
+ Here is an incomplete list of clients and libraries that are known to support GGUF:
36
+
37
+ * [llama.cpp](https://github.com/ggerganov/llama.cpp). The source project for GGUF. Offers a CLI and a server option.
38
+ * [text-generation-webui](https://github.com/oobabooga/text-generation-webui), the most widely used web UI, with many features and powerful extensions. Supports GPU acceleration.
39
+ * [KoboldCpp](https://github.com/LostRuins/koboldcpp), a fully featured web UI, with GPU accel across all platforms and GPU architectures. Especially good for story telling.
40
+ * [LM Studio](https://lmstudio.ai/), an easy-to-use and powerful local GUI for Windows and macOS (Silicon), with GPU acceleration.
41
+ * [LoLLMS Web UI](https://github.com/ParisNeo/lollms-webui), a great web UI with many interesting and unique features, including a full model library for easy model selection.
42
+ * [Faraday.dev](https://faraday.dev/), an attractive and easy to use character-based chat GUI for Windows and macOS (both Silicon and Intel), with GPU acceleration.
43
+ * [ctransformers](https://github.com/marella/ctransformers), a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server.
44
+ * [llama-cpp-python](https://github.com/abetlen/llama-cpp-python), a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
45
+ * [candle](https://github.com/huggingface/candle), a Rust ML framework with a focus on performance, including GPU support, and ease of use.
46
+
47
+ <!-- README_GGUF.md-about-gguf end -->
48
+ <!-- repositories-available start -->
49
+ ## Repositories available
50
+
51
+ * [2, 3, 4, 5, 6 and 8-bit GGUF models for CPU+GPU inference](https://huggingface.co/jezzarax/yi-6b-GGUF)
52
+ * [01-ai's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/01-ai/Yi-6B)
53
+ <!-- repositories-available end -->
54
+
55
+ <!-- prompt-template start -->
56
+ ## Prompt template: Yi
57
+
58
+ ```
59
+ Human: {prompt} Assistant:
60
+
61
+ ```
62
+
63
+ <!-- prompt-template end -->
64
+
65
+
66
+ <!-- compatibility_gguf start -->
67
+ ## Compatibility
68
+
69
+ These quantised GGUFv2 files are compatible with llama.cpp from August 27th onwards, as of commit [d0cee0d](https://github.com/ggerganov/llama.cpp/commit/d0cee0d36d5be95a0d9088b674dbb27354107221)
70
+
71
+ They are also compatible with many third party UIs and libraries - please see the list at the top of this README.
72
+
73
+ ## Explanation of quantisation methods
74
+
75
+ <details>
76
+ <summary>Click to see details</summary>
77
+
78
+ The new methods available are:
79
+
80
+ * GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
81
+ * GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
82
+ * GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
83
+ * GGML_TYPE_Q5_K - "type-1" 5-bit quantization. Same super-block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
84
+ * GGML_TYPE_Q6_K - "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw
85
+
86
+ Refer to the Provided Files table below to see what files use which methods, and how.
87
+ </details>
88
+ <!-- compatibility_gguf end -->
89
+
90
+ <!-- README_GGUF.md-provided-files start -->
91
+ ## Provided files
92
+
93
+ | Name | Quant method | Bits | Size | Max RAM required | Use case |
94
+ | ---- | ---- | ---- | ---- | ---- | ----- |
95
+ | [yi-6b.Q2_K.gguf](https://huggingface.co/jezzarax/yi-6b-GGUF/blob/main/yi-6b.Q2_K.gguf) | Q2_K | 2 | 14.56 GB| 17.06 GB | smallest, significant quality loss - not recommended for most purposes |
96
+ | [yi-6b.Q3_K_S.gguf](https://huggingface.co/jezzarax/yi-6b-GGUF/blob/main/yi-6b.Q3_K_S.gguf) | Q3_K_S | 3 | 14.96 GB| 17.46 GB | very small, high quality loss |
97
+ | [yi-6b.Q3_K_M.gguf](https://huggingface.co/jezzarax/yi-6b-GGUF/blob/main/yi-6b.Q3_K_M.gguf) | Q3_K_M | 3 | 16.64 GB| 19.14 GB | very small, high quality loss |
98
+ | [yi-6b.Q3_K_L.gguf](https://huggingface.co/jezzarax/yi-6b-GGUF/blob/main/yi-6b.Q3_K_L.gguf) | Q3_K_L | 3 | 18.14 GB| 20.64 GB | small, substantial quality loss |
99
+ | [yi-6b.Q4_0.gguf](https://huggingface.co/jezzarax/yi-6b-GGUF/blob/main/yi-6b.Q4_0.gguf) | Q4_0 | 4 | 19.47 GB| 21.97 GB | legacy; small, very high quality loss - prefer using Q3_K_M |
100
+ | [yi-6b.Q4_K_S.gguf](https://huggingface.co/jezzarax/yi-6b-GGUF/blob/main/yi-6b.Q4_K_S.gguf) | Q4_K_S | 4 | 19.54 GB| 22.04 GB | small, greater quality loss |
101
+ | [yi-6b.Q4_K_M.gguf](https://huggingface.co/jezzarax/yi-6b-GGUF/blob/main/yi-6b.Q4_K_M.gguf) | Q4_K_M | 4 | 20.66 GB| 23.16 GB | medium, balanced quality - recommended |
102
+ | [yi-6b.Q5_0.gguf](https://huggingface.co/jezzarax/yi-6b-GGUF/blob/main/yi-6b.Q5_0.gguf) | Q5_0 | 5 | 23.71 GB| 26.21 GB | legacy; medium, balanced quality - prefer using Q4_K_M |
103
+ | [yi-6b.Q5_K_S.gguf](https://huggingface.co/jezzarax/yi-6b-GGUF/blob/main/yi-6b.Q5_K_S.gguf) | Q5_K_S | 5 | 23.71 GB| 26.21 GB | large, low quality loss - recommended |
104
+ | [yi-6b.Q5_K_M.gguf](https://huggingface.co/jezzarax/yi-6b-GGUF/blob/main/yi-6b.Q5_K_M.gguf) | Q5_K_M | 5 | 24.32 GB| 26.82 GB | large, very low quality loss - recommended |
105
+ | [yi-6b.Q6_K.gguf](https://huggingface.co/jezzarax/yi-6b-GGUF/blob/main/yi-6b.Q6_K.gguf) | Q6_K | 6 | 28.21 GB| 30.71 GB | very large, extremely low quality loss |
106
+ | [yi-6b.Q8_0.gguf](https://huggingface.co/jezzarax/yi-6b-GGUF/blob/main/yi-6b.Q8_0.gguf) | Q8_0 | 8 | 36.54 GB| 39.04 GB | very large, extremely low quality loss - not recommended |
107
+
108
+ **Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
109
+
110
+
111
+
112
+ <!-- README_GGUF.md-provided-files end -->
113
+
114
+ <!-- README_GGUF.md-how-to-download start -->
115
+ ## How to download GGUF files
116
+
117
+ **Note for manual downloaders:** You almost never want to clone the entire repo! Multiple different quantisation formats are provided, and most users only want to pick and download a single file.
118
+
119
+ The following clients/libraries will automatically download models for you, providing a list of available models to choose from:
120
+
121
+ * LM Studio
122
+ * LoLLMS Web UI
123
+ * Faraday.dev
124
+
125
+ ### In `text-generation-webui`
126
+
127
+ Under Download Model, you can enter the model repo: jezzarax/yi-6b-GGUF and below it, a specific filename to download, such as: yi-6b.Q4_K_M.gguf.
128
+
129
+ Then click Download.
130
+
131
+ ### On the command line, including multiple files at once
132
+
133
+ I recommend using the `huggingface-hub` Python library:
134
+
135
+ ```shell
136
+ pip3 install huggingface-hub
137
+ ```
138
+
139
+ Then you can download any individual model file to the current directory, at high speed, with a command like this:
140
+
141
+ ```shell
142
+ huggingface-cli download jezzarax/yi-6b-GGUF yi-6b.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False
143
+ ```
144
+
145
+ <details>
146
+ <summary>More advanced huggingface-cli download usage</summary>
147
+
148
+ You can also download multiple files at once with a pattern:
149
+
150
+ ```shell
151
+ huggingface-cli download jezzarax/yi-6b-GGUF --local-dir . --local-dir-use-symlinks False --include='*Q4_K*gguf'
152
+ ```
153
+
154
+ For more documentation on downloading with `huggingface-cli`, please see: [HF -> Hub Python Library -> Download files -> Download from the CLI](https://huggingface.co/docs/huggingface_hub/guides/download#download-from-the-cli).
155
+
156
+ To accelerate downloads on fast connections (1Gbit/s or higher), install `hf_transfer`:
157
+
158
+ ```shell
159
+ pip3 install hf_transfer
160
+ ```
161
+
162
+ And set environment variable `HF_HUB_ENABLE_HF_TRANSFER` to `1`:
163
+
164
+ ```shell
165
+ HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download jezzarax/yi-6b-GGUF yi-6b.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False
166
+ ```
167
+
168
+ Windows Command Line users: You can set the environment variable by running `set HF_HUB_ENABLE_HF_TRANSFER=1` before the download command.
169
+ </details>
170
+ <!-- README_GGUF.md-how-to-download end -->
171
+
172
+ <!-- README_GGUF.md-how-to-run start -->
173
+ ## Example `llama.cpp` command
174
+
175
+ Make sure you are using `llama.cpp` from commit [d0cee0d](https://github.com/ggerganov/llama.cpp/commit/d0cee0d36d5be95a0d9088b674dbb27354107221) or later.
176
+
177
+ ```shell
178
+ ./main -ngl 32 -m yi-6b.Q4_K_M.gguf --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "Human: {prompt} Assistant:"
179
+ ```
180
+
181
+ Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
182
+
183
+ Change `-c 2048` to the desired sequence length. For extended sequence models - eg 8K, 16K, 32K - the necessary RoPE scaling parameters are read from the GGUF file and set by llama.cpp automatically.
184
+
185
+ If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`
186
+
187
+ For other parameters and how to use them, please refer to [the llama.cpp documentation](https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md)
188
+
189
+ ## How to run in `text-generation-webui`
190
+
191
+ Further instructions here: [text-generation-webui/docs/llama.cpp.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp.md).
192
+
193
+ ## How to run from Python code
194
+
195
+ You can use GGUF models from Python using the [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) or [ctransformers](https://github.com/marella/ctransformers) libraries.
196
+
197
+ ### How to load this model in Python code, using ctransformers
198
+
199
+ #### First install the package
200
+
201
+ Run one of the following commands, according to your system:
202
+
203
+ ```shell
204
+ # Base ctransformers with no GPU acceleration
205
+ pip install ctransformers
206
+ # Or with CUDA GPU acceleration
207
+ pip install ctransformers[cuda]
208
+ # Or with AMD ROCm GPU acceleration (Linux only)
209
+ CT_HIPBLAS=1 pip install ctransformers --no-binary ctransformers
210
+ # Or with Metal GPU acceleration for macOS systems only
211
+ CT_METAL=1 pip install ctransformers --no-binary ctransformers
212
+ ```
213
+
214
+ #### Simple ctransformers example code
215
+
216
+ ```python
217
+ from ctransformers import AutoModelForCausalLM
218
+
219
+ # Set gpu_layers to the number of layers to offload to GPU. Set to 0 if no GPU acceleration is available on your system.
220
+ llm = AutoModelForCausalLM.from_pretrained("jezzarax/yi-6b-GGUF", model_file="yi-6b.Q4_K_M.gguf", model_type="yi", gpu_layers=50)
221
+
222
+ print(llm("AI is going to"))
223
+ ```
224
+
225
+ ## How to use with LangChain
226
+
227
+ Here are guides on using llama-cpp-python and ctransformers with LangChain:
228
+
229
+ * [LangChain + llama-cpp-python](https://python.langchain.com/docs/integrations/llms/llamacpp)
230
+ * [LangChain + ctransformers](https://python.langchain.com/docs/integrations/providers/ctransformers)
231
+
232
+ <!-- README_GGUF.md-how-to-run end -->
233
+
234
+ <!-- original-model-card start -->
235
+ # Original model card: 01-ai's Yi 6B
236
+
237
+ <div align="center">
238
+
239
+ <img src="./Yi.svg" width="200px">
240
+
241
+ </div>
242
+
243
+ ## Introduction
244
+
245
+ The **Yi** series models are large language models trained from scratch by
246
+ developers at [01.AI](https://01.ai/). The first public release contains two
247
+ bilingual(English/Chinese) base models with the parameter sizes of 6B([`Yi-6B`](https://huggingface.co/01-ai/Yi-6B))
248
+ and 34B([`Yi-34B`](https://huggingface.co/01-ai/Yi-34B)). Both of them are trained
249
+ with 4K sequence length and can be extended to 32K during inference time.
250
+ The [`Yi-6B-200K`](https://huggingface.co/01-ai/Yi-6B-200K)
251
+ and [`Yi-34B-200K`](https://huggingface.co/01-ai/Yi-34B-200K) are base model with
252
+ 200K context length.
253
+
254
+ ## News
255
+
256
+ - 🎯 **2023/11/06**: The base model of [`Yi-6B-200K`](https://huggingface.co/01-ai/Yi-6B-200K)
257
+ and [`Yi-34B-200K`](https://huggingface.co/01-ai/Yi-34B-200K) with 200K context length.
258
+ - 🎯 **2023/11/02**: The base model of [`Yi-6B`](https://huggingface.co/01-ai/Yi-6B) and
259
+ [`Yi-34B`](https://huggingface.co/01-ai/Yi-34B).
260
+
261
+
262
+ ## Model Performance
263
+
264
+ | Model | MMLU | CMMLU | C-Eval | GAOKAO | BBH | Common-sense Reasoning | Reading Comprehension | Math & Code |
265
+ | :------------ | :------: | :------: | :------: | :------: | :------: | :--------------------: | :-------------------: | :---------: |
266
+ | | 5-shot | 5-shot | 5-shot | 0-shot | 3-shot@1 | - | - | - |
267
+ | LLaMA2-34B | 62.6 | - | - | - | 44.1 | 69.9 | 68.0 | 26.0 |
268
+ | LLaMA2-70B | 68.9 | 53.3 | - | 49.8 | 51.2 | 71.9 | 69.4 | 36.8 |
269
+ | Baichuan2-13B | 59.2 | 62.0 | 58.1 | 54.3 | 48.8 | 64.3 | 62.4 | 23.0 |
270
+ | Qwen-14B | 66.3 | 71.0 | 72.1 | 62.5 | 53.4 | 73.3 | 72.5 | **39.8** |
271
+ | Skywork-13B | 62.1 | 61.8 | 60.6 | 68.1 | 41.7 | 72.4 | 61.4 | 24.9 |
272
+ | InternLM-20B | 62.1 | 59.0 | 58.8 | 45.5 | 52.5 | 78.3 | - | 30.4 |
273
+ | Aquila-34B | 67.8 | 71.4 | 63.1 | - | - | - | - | - |
274
+ | Falcon-180B | 70.4 | 58.0 | 57.8 | 59.0 | 54.0 | 77.3 | 68.8 | 34.0 |
275
+ | Yi-6B | 63.2 | 75.5 | 72.0 | 72.2 | 42.8 | 72.3 | 68.7 | 19.8 |
276
+ | Yi-6B-200K | 64.0 | 75.3 | 73.5 | 73.9 | 42.0 | 72.0 | 69.1 | 19.0 |
277
+ | **Yi-34B** | **76.3** | **83.7** | 81.4 | 82.8 | **54.3** | **80.1** | 76.4 | 37.1 |
278
+ | Yi-34B-200K | 76.1 | 83.6 | **81.9** | **83.4** | 52.7 | 79.7 | **76.6** | 36.3 |
279
+
280
+ While benchmarking open-source models, we have observed a disparity between the
281
+ results generated by our pipeline and those reported in public sources (e.g.
282
+ OpenCompass). Upon conducting a more in-depth investigation of this difference,
283
+ we have discovered that various models may employ different prompts,
284
+ post-processing strategies, and sampling techniques, potentially resulting in
285
+ significant variations in the outcomes. Our prompt and post-processing strategy
286
+ remains consistent with the original benchmark, and greedy decoding is employed
287
+ during evaluation without any post-processing for the generated content. For
288
+ scores that were not reported by the original authors (including scores reported
289
+ with different settings), we try to get results with our pipeline.
290
+
291
+ To evaluate the model's capability extensively, we adopted the methodology
292
+ outlined in Llama2. Specifically, we included PIQA, SIQA, HellaSwag, WinoGrande,
293
+ ARC, OBQA, and CSQA to assess common sense reasoning. SquAD, QuAC, and BoolQ
294
+ were incorporated to evaluate reading comprehension. CSQA was exclusively tested
295
+ using a 7-shot setup, while all other tests were conducted with a 0-shot
296
+ configuration. Additionally, we introduced GSM8K (8-shot@1), MATH (4-shot@1),
297
+ HumanEval (0-shot@1), and MBPP (3-shot@1) under the category "Math & Code". Due
298
+ to technical constraints, we did not test Falcon-180 on QuAC and OBQA; the score
299
+ is derived by averaging the scores on the remaining tasks. Since the scores for
300
+ these two tasks are generally lower than the average, we believe that
301
+ Falcon-180B's performance was not underestimated.
302
+
303
+ ## Usage
304
+
305
+ Please visit our [github repository](https://github.com/01-ai/Yi) for general
306
+ guidance on how to use this model.
307
+
308
+ ## Disclaimer
309
+
310
+ Although we use data compliance checking algorithms during the training process
311
+ to ensure the compliance of the trained model to the best of our ability, due to
312
+ the complexity of the data and the diversity of language model usage scenarios,
313
+ we cannot guarantee that the model will generate correct and reasonable output
314
+ in all scenarios. Please be aware that there is still a risk of the model
315
+ producing problematic outputs. We will not be responsible for any risks and
316
+ issues resulting from misuse, misguidance, illegal usage, and related
317
+ misinformation, as well as any associated data security concerns.
318
+
319
+ ## License
320
+
321
+ The Yi series models are fully open for academic research and free commercial
322
+ usage with permission via applications. All usage must adhere to the [Model
323
+ License Agreement 2.0](https://huggingface.co/01-ai/Yi-34B/blob/main/LICENSE). To
324
+ apply for the official commercial license, please contact us
325
+ ([yi@01.ai](mailto:yi@01.ai)).
326
+
327
+ <!-- original-model-card end -->
328
+
yi-6b.Q2_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5b4d4c24aabbc67790aaca772d0f7477f36390f2aed006685043070cd88e8f0c
3
+ size 2621230304
yi-6b.Q3_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a0f46f2a68122a35e08de0429b0f0dd1d1b6941c0acc7b35aaeef84bb2b1532b
3
+ size 2992835808
yi-6b.Q3_K_L.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1155f53b918d76c202abc0e4a886251c73f2080c69a15190bcbfecb4c62eed45
3
+ size 3236891872
yi-6b.Q3_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7ea180a61c51274e85ff8c3e9aa03f2281950d7006c62ead1df12838ce71af5f
3
+ size 2709196000
yi-6b.Q4_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:789a7696442cd5b0fea0703161ceb3628b10c4f029862596ad4da736a5bf94ea
3
+ size 3673967840
yi-6b.Q4_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:610c40948d6bff7121c20055f1a194a472bac0f747eaa32b50e9515189226f9c
3
+ size 3502918880
yi-6b.Q5_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e7fcc7e7a590a04608e066f3d4097f37c9ea1494903b2adcd79f0aee5f120257
3
+ size 4304424160
yi-6b.Q5_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8526a05809c9ca29b10a15279f9dfe4f8d8eb9e32efec913ab61520972acd6d7
3
+ size 4204154080
yi-6b.Q6_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b3631b1336c2bc992562a3e1eeb42b1f55ad1dca5684c0ce9464984954d9a56e
3
+ size 4974284000
yi-6b.f16.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1712debba47099a8165a1bb030432fd9537690caa8d6d25089e271166be2ce62
3
+ size 12124097696