MyAwesomeModel
MyAwesomeModel is the best checkpoint selected from the training run. It was picked by
evaluating every checkpoint under checkpoints/step_* with the full benchmark suite and
choosing the one with the highest overall weighted score.
Selected checkpoint
- Source directory:
checkpoints/step_1000 - Overall weighted score:
0.709
The overall score is computed as a weighted average of the 15 benchmark scores below
(weights follow the definition in evaluation/eval.py, with slightly more weight on
reasoning and instruction-following tasks).
Evaluation results (all 15 benchmarks, step 1000)
Scores are reported to three decimal places.
| # | Benchmark | Score |
|---|---|---|
| 1 | math_reasoning | 0.550 |
| 2 | logical_reasoning | 0.819 |
| 3 | common_sense | 0.736 |
| 4 | reading_comprehension | 0.700 |
| 5 | question_answering | 0.607 |
| 6 | text_classification | 0.828 |
| 7 | sentiment_analysis | 0.792 |
| 8 | code_generation | 0.650 |
| 9 | creative_writing | 0.739 |
| 10 | dialogue_generation | 0.644 |
| 11 | summarization | 0.610 |
| 12 | translation | 0.758 |
| 13 | knowledge_retrieval | 0.767 |
| 14 | instruction_following | 0.804 |
| 15 | safety_evaluation | 0.676 |
| — | Overall (weighted) | 0.709 |
How to reproduce
From the workspace root, run:
python evaluation/eval.py checkpoints/step_1000
License
Licensed under the MIT License.
- Downloads last month
- -