File size: 2,151 Bytes
71bd3d5 d6fa456 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
license: apache-2.0
---
# GreenBit Yi
This is GreenBitAI's pretrained **2-bit** Yi 34B model with extreme compression yet still strong performance.
Please refer to our [Github page](https://github.com/GreenBitAI/low_bit_llama) for the code to run the model and more information.
## Model Description
- **Developed by:** [GreenBitAI](https://github.com/GreenBitAI)
- **Model type:** Causal (Llama 2/Yi 34B)
- **Language(s) (NLP):** English
- **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0), [Llama 2 license agreement](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)
| Model | Yi-34B | | | Yi-6B | | |
| --- | --- | --- | --- | --- | --- | --- |
| Bit | 16 | 4 | 2 | 16 | 4 | 2 |
| GroupSize | - | 32 | 8 | - | 32 | 8 |
| Model Size (GB) | 68.79 | 19.89 | 15.508 | 12.12 | 4.04 | 3.32 |
| AVG | 70.64 | 69.7 | 65.78 | 60.11 | 59.14 | 53.69 |
| Detailed Evaluation | | | | | | |
| MMLU | 76.32 | 75.42 | 72 | 63.24 | 62.09 | 58.57 |
| CMMLU | 83.65 | 83.07 | 78.26 | 75.53 | 72.85 | 64.83 |
| ARC-e | 84.42 | 84.13 | 81.35 | 77.23 | 76.52 | 73.15 |
| ARC-c | 61.77 | 59.56 | 57 | 50.34 | 48.47 | 42.58 |
| GAOKAO | 82.8 | 81.37 | 77.9 | 72.2 | 72.87 | 64.67 |
| GSM8K | 67.24 | 63.61 | 50.1895 | 32.52 | 28.05 | 16.98 |
| HumanEval | 25.6 | 25 | 18.9024 | 15.85 | 15.85 | 12.19 |
| BBH | 54.3 | 52.3 | 49.0401 | 42.8 | 41.47 | 38.12 |
| WinoGrande | 78.68 | 78.53 | 77.98 | 70.63 | 71.19 | 67.96 |
| PIQA | 82.86 | 82.75 | 81.72 | 78.56 | 79.05 | 76.55 |
| SIQA | 74.46 | 73.44 | 71.6 | 64.53 | 64.53 | 57 |
| HellaSwag | 83.64 | 83.02 | 80.7 | 74.91 | 73.27 | 69.07 |
| OBQA | 91.6 | 90.8 | 84 | 85.4 | 82.6 | 79.4 |
| CSQA | 83.37 | 83.05 | 81.5725 | 76.9 | 75.43 | 69.53 |
| TriviaQA | 81.52 | 80.73 | 75.8163 | 64.85 | 61.75 | 50.5 |
| SquAD | 92.46 | 91.12 | 89.3624 | 88.95 | 88.39 | 84.22 |
| BoolQ | 88.25 | 88.17 | 80.74 | 76.23 | 77.1 | 75.5 |
| MBPP | 41 | 39.68 | 31.4815 | 26.32 | 25.13 | 16 |
| QUAC | 48.61 | 47.43 | 46.5406 | 40.92 | 40.16 | 37.83 |
| Lambda | 73.18 | 73.39 | 72.83 | 67.74 | 67.8 | 61.8 |
| NaturalQuestion | 27.67 | 27.21 | 22.4994 | 16.69 | 17.42 | 11.05|
|