File size: 2,602 Bytes
90e2973 cd3c500 90e2973 2b717a3 90e2973 cd3c500 90e2973 4b742c5 90e2973 0997768 636c56e 8c143f9 9aae007 64f84f9 8c143f9 64f84f9 8c143f9 6b8ef47 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 |
---
base_model:
- unsloth/SmolLM2-1.7B-Instruct-bnb-4bit
- HuggingFaceTB/SmolLM2-1.7B-Instruct
tags:
- text-generation-inference
- transformers
- unsloth
- trl
- sft
license: apache-2.0
language:
- en
datasets:
- AI-MO/NuminaMath-TIR
---
# Uploaded model
- **Developed by:** Qurtana
- **License:** apache-2.0
- **Finetuned from model :** unsloth/SmolLM2-1.7B-Instruct-bnb-4bit
This model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
Trained using rank-stablized QLoRA with r = 64 and alpha = 5 for one epoch using the "ChatML" data prep.
The following heads were targeted: "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "embed_tokens", and "lm_head".
I strongly believe that this should achieve better performance than the original, particularly in math and reasoning. Hopefully the MUSR and MATH Lvl 5 evaluations reflect this.
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
This is what I used to load the model using AutoClasses:
```!pip install -q transformers accelerate
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
# Define the revision you want to use
revision = "main" # or any other valid revision/branch name
# Load the config, model, and tokenizer with the specified revision
config = AutoConfig.from_pretrained("Qurtana/SmolLM2-1.7B-Instruct-NuminaMath-TIR", revision=revision)
# Use AutoModelForCausalLM instead of AutoModel for this model
model = AutoModelForCausalLM.from_pretrained("Qurtana/SmolLM2-1.7B-Instruct-NuminaMath-TIR",
revision=revision,
config=config,
ignore_mismatched_sizes=True) # Add this line to ignore mismatched sizes
tokenizer = AutoTokenizer.from_pretrained("Qurtana/SmolLM2-1.7B-Instruct-NuminaMath-TIR", revision=revision)
```
Dataset Citation:
@misc{numina_math_datasets,
author = {Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu},
title = {NuminaMath TIR},
year = {2024},
publisher = {Numina},
journal = {Hugging Face repository},
howpublished = {\url{[https://huggingface.co/AI-MO/NuminaMath-TIR](https://github.com/project-numina/aimo-progress-prize/blob/main/report/numina_dataset.pdf)}}
} |