NicoNico commited on
Commit
818df15
1 Parent(s): 3b54e17

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +19 -0
README.md CHANGED
@@ -6,3 +6,22 @@ license: apache-2.0
6
  This is GreenBitAI's pretrained **low-bit** LLMs with extreme compression yet still strong performance.
7
 
8
  Please refer to our [Github page](https://github.com/GreenBitAI/green-bit-llm) for the code to run the model and more information.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
  This is GreenBitAI's pretrained **low-bit** LLMs with extreme compression yet still strong performance.
7
 
8
  Please refer to our [Github page](https://github.com/GreenBitAI/green-bit-llm) for the code to run the model and more information.
9
+
10
+ ### zero-shot evaluation
11
+ | **Repository (Phi Family)** | **Avg Acc.** | **OpenBQ** | **ARC-E** | **Winogr.** | **HellaS.** | **ARC-C** | **PIQA** | **BoolQ** | **RACE** | **ANLI-R1** | **ANLI-R2** | **ANLI-R3** | **WiC** |
12
+ |:---------------------------------------------|:------------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:---------:|:--------:|:-----------:|:-----------:|:-----------:|:-------:|
13
+ | `Phi-3-mini-128k-instruct-layer-mix-bpw-2.2` | 0.510 | 0.270 | 0.706 | 0.648 | 0.479 | 0.411 | 0.736 | 0.783 | 0.381 | 0.393 | 0.38 | 0.399 | 0.536 |
14
+ | `Phi-3-mini-128k-instruct-layer-mix-bpw-2.5` | 0.514 | 0.290 | 0.719 | 0.656 | 0.488 | 0.401 | 0.750 | 0.778 | 0.401 | 0.392 | 0.410 | 0.407 | 0.493 |
15
+ | `Phi-3-mini-128k-instruct-layer-mix-bpw-3.0` | 0.548 | 0.318 | 0.761 | 0.663 | 0.519 | 0.453 | 0.777 | 0.798 | 0.393 | 0.473 | 0.404 | 0.442 | 0.579 |
16
+ | `Phi-3-mini-128k-instruct-layer-mix-bpw-4.0` | 0.582 | 0.346 | 0.779 | 0.708 | 0.582 | 0.495 | 0.787 | 0.840 | 0.412 | 0.529 | 0.459 | 0.448 | 0.606 |
17
+ | `Phi-3-mini-128k-instruct ` | 0.586 | 0.342 | 0.785 | 0.731 | 0.596 | 0.512 | 0.782 | 0.851 | 0.401 | 0.547 | 0.464 | 0.432 | 0.594 |
18
+
19
+ ### 5-shot evaluation
20
+ | **Repository (Phi Family)** | **Avg Acc.** | **OpenBQ** | **ARC-E** | **Winogr.** | **HellaS.** | **ARC-C** | **PIQA** | **BoolQ** | **RACE** | **ANLI-R1** | **ANLI-R2** | **ANLI-R3** | **WiC** |
21
+ |:---------------------------------------------|:------------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:---------:|:--------:|:-----------:|:-----------:|:-----------:|:-------:|
22
+ | `Phi-3-mini-128k-instruct-layer-mix-bpw-2.2` | 0.534 | 0.302 | 0.738 | 0.659 | 0.487/0.636 | 0.438 | 0.744 | 0.793 | 0.408 | 0.421 | 0.404 | 0.439 | 0.583 |
23
+ | `Phi-3-mini-128k-instruct-layer-mix-bpw-2.5` | 0.543 | 0.310 | 0.771 | 0.671 | 0.501/0.657 | 0.441 | 0.763 | 0.799 | 0.405 | 0.453 | 0.427 | 0.443 | 0.534 |
24
+ | `Phi-3-mini-128k-instruct-layer-mix-bpw-3.0` | 0.563 | 0.346 | 0.796 | 0.687 | 0.528/0.694 | 0.500 | 0.782 | 0.809 | 0.410 | 0.473 | 0.394 | 0.474 | 0.565 |
25
+ | `Phi-3-mini-128k-instruct-layer-mix-bpw-4.0` | 0.602 | 0.374 | 0.817 | 0.725 | 0.598/0.768 | 0.542 | 0.766 | 0.864 | 0.428 | 0.523 | 0.456 | 0.497 | 0.658 |
26
+ | `Phi-3-mini-128k-instruct ` | 0.608 | 0.408 | 0.825 | 0.725 | 0.608/0.781 | 0.534 | 0.768 | 0.866 | | 0.538 | 0.483 | 0.515 | 0.627 |
27
+