File size: 3,934 Bytes
13520c3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
137ff21
81389b9
 
96d6a6a
 
33d6741
 
137ff21
13520c3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
---
license: apache-2.0
tags:
- alignment-handbook
- generated_from_trainer
- juanako
- mistral
- UNA
datasets:
- HuggingFaceH4/ultrafeedback_binarized
model-index:
- name: juanako-7b-UNA
  results:
  - task:
      type: text-generation
      name: TruthfulQA (MC2)
    dataset:
      type: text-generation
      name: truthful_qa
      config: multiple_choice
      split: validation
    metrics:
      - type: accuracy
        value: 65.13
        verified: true
  - task:
      type: text-generation
      name: ARC-Challenge
    dataset:
      type: text-generation
      name: ai2_arc
      config: ARC-Challenge
      split: test
    metrics:
      - type: accuracy
        value: 68.17
        verified: true
  - task:
      type: text-generation
      name: HellaSwag
    dataset:
      type: text-generation
      name: Rowan/hellaswag
      split: test
    metrics:
      - type: accuracy
        value: 85.34
        verified: true
  - task:
      type: text-generation
      name: Winogrande
    dataset:
      type: text-generation
      name: winogrande
      config: winogrande_debiased
      split: test
    metrics:
      - type: accuracy
        value: 78.85
        verified: true
  - task:
      type: text-generation
      name: MMLU
    dataset:
      type: text-generation
      name: cais/mmlu
      config: all
      split: test
    metrics:
      - type: accuracy
        value: 62.47
        verified: true
  - task:
      type: text-generation
      name: PiQA
    dataset:
      type: text-generation
      name: piqa
      split: test
    metrics:
      - type: accuracy
        value: 83.57
  - task:
      type: text-generation
      name: DROP
    dataset:
      type: text-generation
      name: drop
      split: validation
    metrics:
      - type: accuracy
        value: 38.74
        verified: true
  - task:
      type: text-generation
      name: PubMedQA
    dataset:
      type: text-generation
      name: bigbio/pubmed_qa
      config: pubmed_qa_artificial_bigbio_qa
      split: validation
    metrics:
      - type: accuracy
        value: 76.0
quantized_by: bartowski
pipeline_tag: text-generation
---

## Exllama v2 Quantizations of juanako-7b-UNA

Using <a href="https://github.com/turboderp/exllamav2/releases/tag/v0.0.10">turboderp's ExLlamaV2 v0.0.10</a> for quantization.

Each branch contains an individual bits per weight, with the main one containing only the meaurement.json for further conversions.

Conversion was done using wikitext-103-raw-v1-test.parquet as calibration dataset.

Default arguments used except when the bits per weight is above 6.0, at that point the lm_head layer is quantized at 8 bits per weight instead of the default 6.

Original model: https://huggingface.co/fblgit/juanako-7b-UNA
 
<a href="https://huggingface.co/bartowski/juanako-7b-UNA-exl2/tree/4_0">4.0 bits per weight</a>
 
<a href="https://huggingface.co/bartowski/juanako-7b-UNA-exl2/tree/5_0">5.0 bits per weight</a>
 
<a href="https://huggingface.co/bartowski/juanako-7b-UNA-exl2/tree/6_0">6.0 bits per weight</a>
 
<a href="https://huggingface.co/bartowski/juanako-7b-UNA-exl2/tree/8_0">8.0 bits per weight</a>

## Download instructions

With git:

```shell
git clone --single-branch --branch 4_0 https://huggingface.co/bartowski/juanako-7b-UNA-exl2
```

With huggingface hub (credit to TheBloke for instructions):

```shell
pip3 install huggingface-hub
```

To download the `main` (only useful if you only care about measurement.json) branch to a folder called `juanako-7b-UNA-exl2`:

```shell
mkdir juanako-7b-UNA-exl2
huggingface-cli download bartowski/juanako-7b-UNA-exl2 --local-dir juanako-7b-UNA-exl2 --local-dir-use-symlinks False
```

To download from a different branch, add the `--revision` parameter:

```shell
mkdir juanako-7b-UNA-exl2
huggingface-cli download bartowski/juanako-7b-UNA-exl2 --revision 4_0 --local-dir juanako-7b-UNA-exl2 --local-dir-use-symlinks False
```