Main branch
Browse files- README.md +47 -0
- measurement.json +0 -0
README.md
ADDED
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: other
|
3 |
+
license_name: deepseek
|
4 |
+
license_link: LICENSE
|
5 |
+
quantized_by: bartowski
|
6 |
+
pipeline_tag: text-generation
|
7 |
+
---
|
8 |
+
|
9 |
+
## Exllama v2 Quantizations of deepseek-coder-6.7b-instruct
|
10 |
+
|
11 |
+
Using <a href="https://github.com/turboderp/exllamav2/releases/tag/master">turboderp's ExLlamaV2 master</a> for quantization.
|
12 |
+
|
13 |
+
Each branch contains an individual bits per weight, with the main one containing only the meaurement.json for further conversions.
|
14 |
+
|
15 |
+
Conversion was done using Evol-Instruct-Code-80k-v1.parquet as calibration dataset.
|
16 |
+
|
17 |
+
Default arguments used except when the bits per weight is above 6.0, at that point the lm_head layer is quantized at 8 bits per weight instead of the default 6.
|
18 |
+
|
19 |
+
Original model: https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-instruct
|
20 |
+
|
21 |
+
## Download instructions
|
22 |
+
|
23 |
+
With git:
|
24 |
+
|
25 |
+
```shell
|
26 |
+
git clone --single-branch --branch 4_0 https://huggingface.co/bartowski/deepseek-coder-6.7b-instruct-exl2
|
27 |
+
```
|
28 |
+
|
29 |
+
With huggingface hub (credit to TheBloke for instructions):
|
30 |
+
|
31 |
+
```shell
|
32 |
+
pip3 install huggingface-hub
|
33 |
+
```
|
34 |
+
|
35 |
+
To download the `main` (only useful if you only care about measurement.json) branch to a folder called `deepseek-coder-6.7b-instruct-exl2`:
|
36 |
+
|
37 |
+
```shell
|
38 |
+
mkdir deepseek-coder-6.7b-instruct-exl2
|
39 |
+
huggingface-cli download bartowski/deepseek-coder-6.7b-instruct-exl2 --local-dir deepseek-coder-6.7b-instruct-exl2 --local-dir-use-symlinks False
|
40 |
+
```
|
41 |
+
|
42 |
+
To download from a different branch, add the `--revision` parameter:
|
43 |
+
|
44 |
+
```shell
|
45 |
+
mkdir deepseek-coder-6.7b-instruct-exl2
|
46 |
+
huggingface-cli download bartowski/deepseek-coder-6.7b-instruct-exl2 --revision 4_0 --local-dir deepseek-coder-6.7b-instruct-exl2 --local-dir-use-symlinks False
|
47 |
+
```
|
measurement.json
ADDED
The diff for this file is too large to render.
See raw diff
|
|