Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,12 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: apache-2.0
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
datasets:
|
| 4 |
+
- open-thoughts/OpenThoughts-114k
|
| 5 |
+
- prithivMLmods/Deepthink-Reasoning-Ins
|
| 6 |
+
base_model:
|
| 7 |
+
- Qwen/Qwen2.5-0.5B-Instruct
|
| 8 |
+
---
|
| 9 |
+
Model currently under training. Official publish will be on (DD/MM/YYYY) 20/02/2025.
|
| 10 |
+
SaplingDream is a 0.5B parameter small GPT based on [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) finetuned on reasoning datasets with very high caution to ensure a high-quality sapling model—hence "SaplingDream".
|
| 11 |
+
The base model is finetuned using SGD to ensure better generalisation, in combination with the lr scheduler with a starting lr of 1e-4. Better safe than sorry, we hope the model picks up on not only the tokens but also on how to actually reason through a problem.
|
| 12 |
+
We're using [open-thoughts/OpenThoughts-114k](https://huggingface.co/datasets/open-thoughts/OpenThoughts-114k) and [prithivMLmods/Deepthink-Reasoning-Ins](https://huggingface.co/datasets/prithivMLmods/Deepthink-Reasoning-Ins) for training through an entire epoch.
|