Hugging Face's logo

about0
/
llama-v2-chinese-ymcui-alpaca-GGML-13B

Text Generation
PyTorch
Chinese
English
llama
llama-2
alpaca
chinese
中文
Model card Files Community
- commited on
Commit
caf7a4a
•
1 Parent(s): 123d2cf

readme: add

Browse files
Files changed (1) hide show
  1. README.md +76 -0
README.md ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - zh
4
+ - en
5
+ tags:
6
+ - pytorch
7
+ - llama
8
+ - llama-2
9
+ - alpaca
10
+ - chinese
11
+ - 中文
12
+ model_name: Llama 2 Chinese Alpaca 13B
13
+ inference: false
14
+ model_type: llama-alpaca
15
+ pipeline_tag: text-generation
16
+ quantized_by: about0
17
+ ---
18
+
19
+ # LLaMA-v2-chinese-alpaca-13B-GGML (ymcui)
20
+
21
+ Here are the GGML converted and/or quantized models for [ymcui's Chinese LLaMA-v2 Alpaca 13B]([https://github.com/ymcui/Chinese-LLaMA-Alpaca-2]).
22
+
23
+ **!NOTE! The GGML filetype is outdated. Prefer GGUF format going forward.**
24
+
25
+ ## Explanation of quantisation methods
26
+ <details>
27
+ <summary>Click to see details</summary>
28
+
29
+ Methods:
30
+ * type-0 (Q4_0, Q5_0, Q8_0) - weights w are obtained from quants q using w = d * q, where d is the block scale.
31
+ * type-1 (Q4_1, Q5_1) - weights are given by w = d * q + m, where m is the block minimum
32
+
33
+ The new methods available are:
34
+ * GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
35
+ * GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
36
+ * GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
37
+ * GGML_TYPE_Q5_K - "type-1" 5-bit quantization. Same super-block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
38
+ * GGML_TYPE_Q6_K - "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw
39
+ * GGML_TYPE_Q8_K - "type-0" 8-bit quantization. Only used for quantizing intermediate results. The difference to the existing Q8_0 is that the block size is 256. All 2-6 bit dot products are implemented for this quantization type.
40
+
41
+ This is exposed via llama.cpp quantization types that define various "quantization mixes" as follows:
42
+
43
+ * LLAMA_FTYPE_MOSTLY_Q2_K - uses GGML_TYPE_Q4_K for the attention.vw and feed_forward.w2 tensors, GGML_TYPE_Q2_K for the other tensors.
44
+ * LLAMA_FTYPE_MOSTLY_Q3_K_S - uses GGML_TYPE_Q3_K for all tensors
45
+ * LLAMA_FTYPE_MOSTLY_Q3_K_M - uses GGML_TYPE_Q4_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K
46
+ * LLAMA_FTYPE_MOSTLY_Q3_K_L - uses GGML_TYPE_Q5_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K
47
+ * LLAMA_FTYPE_MOSTLY_Q4_K_S - uses GGML_TYPE_Q4_K for all tensors
48
+ * LLAMA_FTYPE_MOSTLY_Q4_K_M - uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q4_K
49
+ * LLAMA_FTYPE_MOSTLY_Q5_K_S - uses GGML_TYPE_Q5_K for all tensors
50
+ * LLAMA_FTYPE_MOSTLY_Q5_K_M - uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q5_K
51
+ * LLAMA_FTYPE_MOSTLY_Q6_K- uses 6-bit quantization (GGML_TYPE_Q8_K) for all tensors
52
+ </details>
53
+
54
+
55
+ ## Provided files
56
+
57
+ | Name | Quant method | Bits | Size | Max RAM required | Use case |
58
+ | ---- | ---- | ---- | ---- | ---- | ----- |
59
+ | [llama-v2-chinese-alpaca-13B-Q2_K.ggml](https://huggingface.co/about0/llama-v2-chinese-ymcui-alpaca-GGML-13B/blob/main/llama-v2-chinese-alpaca-13B-Q2_K.ggml) | Q2_K | 2 | 5.65 GB| 8.15 GB | smallest, significant quality-loss - not recommended for most purposes |
60
+ | [llama-v2-chinese-alpaca-13B-Q3_K_S.ggml](https://huggingface.co/about0/llama-v2-chinese-ymcui-alpaca-GGML-13B/blob/main/llama-v2-chinese-alpaca-13B-Q3_K_S.ggml) | Q3_K_S | 3 | 5.81 GB | 8.31 GB | very small, high quality-loss |
61
+ | [llama-v2-chinese-alpaca-13B-Q3_K_M.ggml](https://huggingface.co/about0/llama-v2-chinese-ymcui-alpaca-GGML-13B/blob/main/llama-v2-chinese-alpaca-13B-Q3_K_M.ggml) | Q3_K_M | 3 | 6.46 GB | 7.96 GB | very small, high quality-loss |
62
+ | [llama-v2-chinese-alpaca-13B-Q3_K_L.ggml](https://huggingface.co/about0/llama-v2-chinese-ymcui-alpaca-GGML-13B/blob/main/llama-v2-chinese-alpaca-13B-Q3_K_L.ggml) | Q3_K_L | 3 | 7.08 GB| 9.58 GB | small, substantial quality-loss |
63
+ | [llama-v2-chinese-alpaca-13B-Q4_0.ggml](https://huggingface.co/about0/llama-v2-chinese-ymcui-alpaca-GGML-13B/blob/main/llama-v2-chinese-alpaca-13B-Q4_0.ggml) | Q4_0 | 4 | 7.53 GB| 10.03 GB | legacy; small, very high quality-loss - prefer using Q3_K_M |
64
+ | [llama-v2-chinese-alpaca-13B-Q4_1.ggml](https://huggingface.co/about0/llama-v2-chinese-ymcui-alpaca-GGML-13B/blob/main/llama-v2-chinese-alpaca-13B-Q4_1.ggml) | Q4_1 | 4 | 8.34 GB| 10.84 GB | legacy; small, very high quality-loss - prefer using Q3_K_M |
65
+ | [llama-v2-chinese-alpaca-13B-Q4_K_S.ggml](https://huggingface.co/about0/llama-v2-chinese-ymcui-alpaca-GGML-13B/blob/main/llama-v2-chinese-alpaca-13B-Q4_K_S.ggml) | Q4_K_S | 4 | 7.53 GB| 10.03 GB | small, greater quality-loss |
66
+ | [llama-v2-chinese-alpaca-13B-Q4_K_M.ggml](https://huggingface.co/about0/llama-v2-chinese-ymcui-alpaca-GGML-13B/blob/main/llama-v2-chinese-alpaca-13B-Q4_K_M.ggml) | Q4_K_M | 4 | 8.03 GB| 10.53 GB | medium, balanced quality - recommended |
67
+ | [llama-v2-chinese-alpaca-13B-Q5_0.ggml](https://huggingface.co/about0/llama-v2-chinese-ymcui-alpaca-GGML-13B/blob/main/llama-v2-chinese-alpaca-13B-Q5_0.ggml) | Q5_0 | 5 | 9.15 GB| 11.65 GB | legacy; medium, balanced quality - prefer using Q4_K_M |
68
+ | [llama-v2-chinese-alpaca-13B-Q5_1.ggml](https://huggingface.co/about0/llama-v2-chinese-ymcui-alpaca-GGML-13B/blob/main/llama-v2-chinese-alpaca-13B-Q5_1.ggml) | Q5_1 | 5 | 9.96 GB| 12.46 GB | legacy; medium, balanced quality - prefer using Q4_K_M |
69
+ | [llama-v2-chinese-alpaca-13B-Q5_K_S.ggml](https://huggingface.co/about0/llama-v2-chinese-ymcui-alpaca-GGML-13B/blob/main/llama-v2-chinese-alpaca-13B-Q5_K_S.ggml) | Q5_K_S | 5 | 9.15 GB | 11.65 GB | large, low quality-loss - recommended |
70
+ | [llama-v2-chinese-alpaca-13B-Q5_K_M.ggml](https://huggingface.co/about0/llama-v2-chinese-ymcui-alpaca-GGML-13B/blob/main/llama-v2-chinese-alpaca-13B-Q5_K_M.ggml) | Q5_K_M | 5 | 9.41 GB | 11.91 GB | large, very low quality-loss - recommended |
71
+ | [llama-v2-chinese-alpaca-13B-Q6_K.ggml](https://huggingface.co/about0/llama-v2-chinese-ymcui-alpaca-GGML-13B/blob/main/llama-v2-chinese-alpaca-13B-Q6_K.ggml) | Q6_K | 6 | 10.9 GB| 13.4 GB | very large, extremely low quality-loss |
72
+ | [llama-v2-chinese-alpaca-13B-Q8_0.ggml](https://huggingface.co/about0/llama-v2-chinese-ymcui-alpaca-GGML-13B/blob/main/llama-v2-chinese-alpaca-13B-Q8_0.ggml) | Q8_0 | 8 | 14 GB| 16.5 GB | very large, extremely low quality-loss - not recommended |
73
+ | [llama-v2-chinese-alpaca-13B-f16.ggml](https://huggingface.co/about0/llama-v2-chinese-ymcui-alpaca-GGML-13B/blob/main/llama-v2-chinese-alpaca-13B-f16.ggml) | f16 | 16 | 26.5 GB| 29 GB | very large, almost no quality-loss - not recommended |
74
+
75
+ ### Model Sources
76
+ - **Repository:** [https://github.com/ymcui/Chinese-LLaMA-Alpaca-2]