Commit
•
3796393
1
Parent(s):
969d794
add MMLU results
Browse files
README.md
CHANGED
@@ -95,6 +95,44 @@ print(f'Elapsed time: {end - start:.2f}s')
|
|
95 |
This model was trained on commercially viable data. For use of our model, refer to the [license information](https://www.nexa4ai.com/licenses).
|
96 |
|
97 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
98 |
## References
|
99 |
We thank the Microsoft team for their amazing model!
|
100 |
```
|
|
|
95 |
This model was trained on commercially viable data. For use of our model, refer to the [license information](https://www.nexa4ai.com/licenses).
|
96 |
|
97 |
|
98 |
+
## Performance
|
99 |
+
### Model Selection
|
100 |
+
We leverage the latest Language Large Models for a variety of domains. Below is a summary of the chosen models for each category. In cases where no specialized model exists for a subject, we utilize generic models like Llama3-8b.
|
101 |
+
|
102 |
+
|
103 |
+
| **Model** | **Category** | **Subjects** |
|
104 |
+
|-----------------------------------------|--------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
105 |
+
| `jondurbin/bagel-8b-v1.0` | Biology | `college_biology`, `high_school_biology` |
|
106 |
+
| `Weyaxi/Einstein-v6.1-Llama3-8B` | Physics | `astronomy`, `college_physics`, `conceptual_physics`, `high_school_physics` |
|
107 |
+
| `meta-llama/Meta-Llama-3-8B-Instruct` | Business | `business_ethics`, `management`, `marketing` |
|
108 |
+
| `meta-llama/Meta-Llama-3-8B-Instruct` | Chemistry | `college_chemistry`, `high_school_chemistry` |
|
109 |
+
| `abacusai/Llama-3-Smaug-8B` | Computer Science | `college_computer_science`, `computer_security`, `high_school_computer_science`, `machine_learning` |
|
110 |
+
| `Open-Orca/Mistral-7B-OpenOrca` | Math | `abstract_algebra`, `college_mathematics`, `elementary_mathematics`, `high_school_mathematics`, `high_school_statistics` |
|
111 |
+
| `meta-llama/Meta-Llama-3-8B-Instruct` | Economics | `econometrics`, `high_school_macroeconomics`, `high_school_microeconomics` |
|
112 |
+
| `AdaptLLM/medicine-chat` | Health | `anatomy`, `clinical_knowledge`, `college_medicine`, `human_aging`, `medical_genetics`, `nutrition`, `professional_medicine`, `virology` |
|
113 |
+
| `STEM-AI-mtl/phi-2-electrical-engineering` | Engineering | `electrical_engineering` |
|
114 |
+
| `meta-llama/Meta-Llama-3-8B-Instruct` | Philosophy | `formal_logic`, `logical_fallacies`, `moral_disputes`, `moral_scenarios`, `philosophy`, `world_religions` |
|
115 |
+
| `microsoft/Phi-3-mini-128k-instruct` | Other | `global_facts`, `miscellaneous`, `professional_accounting` |
|
116 |
+
| `meta-llama/Meta-Llama-3-8B-Instruct` | History | `high_school_european_history`, `high_school_us_history`, `high_school_world_history`, `prehistory` |
|
117 |
+
| `meta-llama/Meta-Llama-3-8B-Instruct` | Culture | `human_sexuality`, `sociology` |
|
118 |
+
| `AdaptLLM/law-chat` | Law | `international_law`, `jurisprudence`, `professional_law` |
|
119 |
+
| `meta-llama/Meta-Llama-3-8B-Instruct` | Psychology | `high_school_psychology`, `professional_psychology` |
|
120 |
+
|
121 |
+
### MMLU Benchmark Results (5-shot learning)
|
122 |
+
Here are the comparative MMLU scores for various models tested under a 5-shot learning setup:
|
123 |
+
|
124 |
+
| **Model** | **MMLU Score** |
|
125 |
+
|-----------------------------------|----------------|
|
126 |
+
| Octopus-V4 | **74.6%** |
|
127 |
+
| GPT-3.5 | 70.0% |
|
128 |
+
| Phi-3-mini-128k-instruct | 68.1% |
|
129 |
+
| OpenELM-3B | 26.7% |
|
130 |
+
| Lamma3-8b-instruct | 68.4% |
|
131 |
+
| Gemma-2b | 42.3% |
|
132 |
+
| Gemma-7b | 64.3% |
|
133 |
+
|
134 |
+
|
135 |
+
|
136 |
## References
|
137 |
We thank the Microsoft team for their amazing model!
|
138 |
```
|