Workbench
Collection
Untested and unfinished models. Works in progress.
•
8 items
•
Updated
TeeZee/GALAXY-XB-v1.03
Experiment, can DUS be taken one or more steps further?
Technical notes:
- model v03 finetuned on 50k entries from SlimOrca dataset
- 12 layers removed from both models, 4 more than in original paper but its 1/4 of all layers(48) as per original paper.
- base version of upstage/SOLAR-10.7B-v1.0 used for merge
To evaluate
- model performance after finetuning, did it recover initial performance loss after merge?
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 57.04 |
| AI2 Reasoning Challenge (25-Shot) | 62.71 |
| HellaSwag (10-Shot) | 84.58 |
| MMLU (5-Shot) | 65.17 |
| TruthfulQA (0-shot) | 47.30 |
| Winogrande (5-shot) | 82.48 |
| GSM8k (5-shot) | 0.00 |
- Downloads last month
- 927
Dataset used to train TeeZee/GALAXY_v03_slimorca_1_epoch_50k
Collection including TeeZee/GALAXY_v03_slimorca_1_epoch_50k
Evaluation results
- normalized accuracy on AI2 Reasoning Challenge (25-Shot)test set Open LLM Leaderboard62.710
- normalized accuracy on HellaSwag (10-Shot)validation set Open LLM Leaderboard84.580
- accuracy on MMLU (5-Shot)test set Open LLM Leaderboard65.170
- mc2 on TruthfulQA (0-shot)validation set Open LLM Leaderboard47.300
- accuracy on Winogrande (5-shot)validation set Open LLM Leaderboard82.480
- accuracy on GSM8k (5-shot)test set Open LLM Leaderboard0.000