|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- open-thoughts/OpenThoughts-114k |
|
- prithivMLmods/Deepthink-Reasoning-Ins |
|
base_model: |
|
- Qwen/Qwen2.5-0.5B-Instruct |
|
--- |
|
|
|
**Model Training Update** |
|
|
|
We are currently in the process of training our model, with an official release scheduled for **February 23, 2025**. |
|
|
|
Introducing **SaplingDream**, a compact GPT model with 0.5 billion parameters, based on the [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) architecture. This model has been fine-tuned on reasoning datasets with meticulous attention to detail, ensuring the highest quality—hence the name "SaplingDream." |
|
|
|
To enhance generalization, we are fine-tuning the base model using Stochastic Gradient Descent (SGD) alongside a "Polynomial" learning rate scheduler, starting with a learning rate of 1e-4. Our goal is to ensure that the model not only learns the tokens but also develops the ability to reason through problems effectively. |
|
|
|
For training, we are utilizing the [open-thoughts/OpenThoughts-114k](https://huggingface.co/datasets/open-thoughts/OpenThoughts-114k) and [prithivMLmods/Deepthink-Reasoning-Ins](https://huggingface.co/datasets/prithivMLmods/Deepthink-Reasoning-Ins) datasets across the entire epoch. |
|
|
|
**Stay tuned! Until training is complete, we will be uploading every (last three) 200th checkpoint from a total of 14,275 optimization steps. Check out the `Files and versions` section for updates!** |
