license: apache-2.0
datasets:
- open-thoughts/OpenThoughts-114k
- prithivMLmods/Deepthink-Reasoning-Ins
base_model:
- Qwen/Qwen2.5-0.5B-Instruct
Model Training Update
We are currently in the process of training our model, with an official release scheduled for February 23, 2025.
Introducing SaplingDream, a compact GPT model with 0.5 billion parameters, based on the Qwen/Qwen2.5-0.5B-Instruct architecture. This model has been fine-tuned on reasoning datasets with meticulous attention to detail, ensuring the highest quality—hence the name "SaplingDream."
To enhance generalization, we are fine-tuning the base model using Stochastic Gradient Descent (SGD) alongside a "Polynomial" learning rate scheduler, starting with a learning rate of 1e-4. Our goal is to ensure that the model not only learns the tokens but also develops the ability to reason through problems effectively.
For training, we are utilizing the open-thoughts/OpenThoughts-114k and prithivMLmods/Deepthink-Reasoning-Ins datasets across the entire epoch.
Stay tuned! Until training is complete, we will be uploading every (last three) 200th checkpoint from a total of 14,275 optimization steps. Check out the Files and versions section for updates!