SpiridonSunRotator commited on
Commit
8a53461
1 Parent(s): 132c138

Added evaluation results

Browse files
Files changed (1) hide show
  1. README.md +6 -2
README.md CHANGED
@@ -11,10 +11,14 @@ tags:
11
  Official [AQLM](https://arxiv.org/abs/2401.06118) quantization of [meta-llama/Meta-Llama-3.1-8B
12
  ](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B) finetuned with [PV-Tuning](https://arxiv.org/abs/2405.14852).
13
 
14
- For this quantization, we used 1 codebook of 16 bits and groupsize of 16.
15
 
16
  Results:
17
  | Model | Quantization | MMLU (5-shot) | ArcC| ArcE| Hellaswag | PiQA | Winogrande | Model size, Gb |
18
  |------|------|-------|------|------|------|------|------|------|
19
  |meta-llama/Meta-Llama-3.1-8B| None |0.6521 | 0.5145 | 0.8144 | 0.5998 | 0.8014 | 0.7356 | 16.1 |
20
- | | 1x16g16 | 0.3057 | 0.2952 | 0.5981 | 0.4457 | 0.7138 | 0.5912 | 3.4 |
 
 
 
 
 
11
  Official [AQLM](https://arxiv.org/abs/2401.06118) quantization of [meta-llama/Meta-Llama-3.1-8B
12
  ](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B) finetuned with [PV-Tuning](https://arxiv.org/abs/2405.14852).
13
 
14
+ For this quantization, we used 1 codebook of 16 bits and groupsize of 8.
15
 
16
  Results:
17
  | Model | Quantization | MMLU (5-shot) | ArcC| ArcE| Hellaswag | PiQA | Winogrande | Model size, Gb |
18
  |------|------|-------|------|------|------|------|------|------|
19
  |meta-llama/Meta-Llama-3.1-8B| None |0.6521 | 0.5145 | 0.8144 | 0.5998 | 0.8014 | 0.7356 | 16.1 |
20
+ | | 1x16g8 | 0.5524 | 0.4437 | 0.7765 | 0.5603 | 0.7797 | 0.6851 | 4.1 |
21
+
22
+ **Note**
23
+
24
+ We used `lm-eval=0.4.0` for evaluation.