Update README.md
Browse files
README.md
CHANGED
|
@@ -6,3 +6,22 @@ license: apache-2.0
|
|
| 6 |
This is GreenBitAI's pretrained **low-bit** LLMs with extreme compression yet still strong performance.
|
| 7 |
|
| 8 |
Please refer to our [Github page](https://github.com/GreenBitAI/green-bit-llm) for the code to run the model and more information.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
This is GreenBitAI's pretrained **low-bit** LLMs with extreme compression yet still strong performance.
|
| 7 |
|
| 8 |
Please refer to our [Github page](https://github.com/GreenBitAI/green-bit-llm) for the code to run the model and more information.
|
| 9 |
+
|
| 10 |
+
### zero-shot evaluation
|
| 11 |
+
| **Repository (Phi Family)** | **Avg Acc.** | **OpenBQ** | **ARC-E** | **Winogr.** | **HellaS.** | **ARC-C** | **PIQA** | **BoolQ** | **RACE** | **ANLI-R1** | **ANLI-R2** | **ANLI-R3** | **WiC** |
|
| 12 |
+
|:---------------------------------------------|:------------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:---------:|:--------:|:-----------:|:-----------:|:-----------:|:-------:|
|
| 13 |
+
| `Phi-3-mini-128k-instruct-layer-mix-bpw-2.2` | 0.510 | 0.270 | 0.706 | 0.648 | 0.479 | 0.411 | 0.736 | 0.783 | 0.381 | 0.393 | 0.38 | 0.399 | 0.536 |
|
| 14 |
+
| `Phi-3-mini-128k-instruct-layer-mix-bpw-2.5` | 0.514 | 0.290 | 0.719 | 0.656 | 0.488 | 0.401 | 0.750 | 0.778 | 0.401 | 0.392 | 0.410 | 0.407 | 0.493 |
|
| 15 |
+
| `Phi-3-mini-128k-instruct-layer-mix-bpw-3.0` | 0.548 | 0.318 | 0.761 | 0.663 | 0.519 | 0.453 | 0.777 | 0.798 | 0.393 | 0.473 | 0.404 | 0.442 | 0.579 |
|
| 16 |
+
| `Phi-3-mini-128k-instruct-layer-mix-bpw-4.0` | 0.582 | 0.346 | 0.779 | 0.708 | 0.582 | 0.495 | 0.787 | 0.840 | 0.412 | 0.529 | 0.459 | 0.448 | 0.606 |
|
| 17 |
+
| `Phi-3-mini-128k-instruct ` | 0.586 | 0.342 | 0.785 | 0.731 | 0.596 | 0.512 | 0.782 | 0.851 | 0.401 | 0.547 | 0.464 | 0.432 | 0.594 |
|
| 18 |
+
|
| 19 |
+
### 5-shot evaluation
|
| 20 |
+
| **Repository (Phi Family)** | **Avg Acc.** | **OpenBQ** | **ARC-E** | **Winogr.** | **HellaS.** | **ARC-C** | **PIQA** | **BoolQ** | **RACE** | **ANLI-R1** | **ANLI-R2** | **ANLI-R3** | **WiC** |
|
| 21 |
+
|:---------------------------------------------|:------------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:---------:|:--------:|:-----------:|:-----------:|:-----------:|:-------:|
|
| 22 |
+
| `Phi-3-mini-128k-instruct-layer-mix-bpw-2.2` | 0.534 | 0.302 | 0.738 | 0.659 | 0.487/0.636 | 0.438 | 0.744 | 0.793 | 0.408 | 0.421 | 0.404 | 0.439 | 0.583 |
|
| 23 |
+
| `Phi-3-mini-128k-instruct-layer-mix-bpw-2.5` | 0.543 | 0.310 | 0.771 | 0.671 | 0.501/0.657 | 0.441 | 0.763 | 0.799 | 0.405 | 0.453 | 0.427 | 0.443 | 0.534 |
|
| 24 |
+
| `Phi-3-mini-128k-instruct-layer-mix-bpw-3.0` | 0.563 | 0.346 | 0.796 | 0.687 | 0.528/0.694 | 0.500 | 0.782 | 0.809 | 0.410 | 0.473 | 0.394 | 0.474 | 0.565 |
|
| 25 |
+
| `Phi-3-mini-128k-instruct-layer-mix-bpw-4.0` | 0.602 | 0.374 | 0.817 | 0.725 | 0.598/0.768 | 0.542 | 0.766 | 0.864 | 0.428 | 0.523 | 0.456 | 0.497 | 0.658 |
|
| 26 |
+
| `Phi-3-mini-128k-instruct ` | 0.608 | 0.408 | 0.825 | 0.725 | 0.608/0.781 | 0.534 | 0.768 | 0.866 | | 0.538 | 0.483 | 0.515 | 0.627 |
|
| 27 |
+
|