Qwen2.5-0.5B-Instruct-CrashCourse-dropout
This model is a fine-tuned version of Qwen/Qwen2.5-0.5B-Instruct, specifically adapted for enhanced performance on instructional and multitask scenarios. It leverages two datasets: agentlans/crash-course and vicgalle/configurable-system-prompt-multitask to improve its capabilities in handling diverse tasks and responding to various instruction formats.
Update: Despite the poor benchmark, the model seems OK at slightly complex prompts. There's more finetuning potential here.
Intended Use
This model is designed for:
- Answering questions related to crash course materials
- Handling configurable system prompts for multitask scenarios
- General instruction-following tasks
Training Procedure
The model was fine-tuned on the specified datasets using the Qwen2.5-0.5B-Instruct as the base model. More details on the training process will be added here later.
Limitations
- The model's performance may be biased towards the specific domains covered in the training datasets.
- As with all language models, it may occasionally produce inaccurate or inconsistent outputs.
- The model's knowledge is limited to the information available in its training data and the base model's knowledge cutoff.
Ethical Considerations
Users should be aware that this model, like all AI models, may reflect biases present in its training data. It's crucial to use the model responsibly and to verify important information from authoritative sources.
Additional Information
For more details on the base model, please refer to the Qwen/Qwen2.5-0.5B-Instruct model card. For information about the datasets used in fine-tuning, check the respective dataset cards on the Hugging Face Hub.
Open LLM Leaderboard Evaluation Results
Detailed results can be found here! Summarized results can be found here!
| Metric | Qwen2.5-0.5B-Instruct-CrashCourse-dropout | Qwen2.5-0.5B-Instruct |
|---|---|---|
| Average | 7.74 % | 8.38 % |
| IFEval (0-Shot) | 29.49 % | 31.53 % |
| BBH (3-Shot) | 7.23 % | 8.17 % |
| MATH Lvl 5 (4-Shot) | 0.08 % | 0.00 % |
| GPQA (0-shot) | 1.79 % | 1.23 % |
| MuSR (0-shot) | 1.11 % | 1.37 % |
| MMLU-PRO (5-shot) | 6.76 % | 8.00 % |
- Downloads last month
- 18
Model tree for agentlans/Qwen2.5-0.5B-Instruct-CrashCourse-dropout
Datasets used to train agentlans/Qwen2.5-0.5B-Instruct-CrashCourse-dropout
Evaluation results
- averaged accuracy on IFEval (0-Shot)Open LLM Leaderboard29.490
- normalized accuracy on BBH (3-Shot)test set Open LLM Leaderboard7.230
- exact match on MATH Lvl 5 (4-Shot)test set Open LLM Leaderboard0.080
- acc_norm on GPQA (0-shot)Open LLM Leaderboard1.790
- acc_norm on MuSR (0-shot)Open LLM Leaderboard1.110
- accuracy on MMLU-PRO (5-shot)test set Open LLM Leaderboard6.760