XeTute commited on
Commit
d931547
·
1 Parent(s): e5d5582

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -8,5 +8,5 @@ base_model:
8
  ---
9
  Model currently under training. Official publish will be on (DD/MM/YYYY) 23/02/2025.
10
  SaplingDream is a 0.5B parameter small GPT based on [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) finetuned on reasoning datasets with very high caution to ensure a high-quality sapling model—hence "SaplingDream".
11
- The base model is finetuned using SGD to ensure better generalisation, in combination with the lr scheduler with a starting lr of 1e-4. Better safe than sorry, we hope the model picks up on not only the tokens but also on how to actually reason through a problem.
12
  We're using [open-thoughts/OpenThoughts-114k](https://huggingface.co/datasets/open-thoughts/OpenThoughts-114k) and [prithivMLmods/Deepthink-Reasoning-Ins](https://huggingface.co/datasets/prithivMLmods/Deepthink-Reasoning-Ins) for training through an entire epoch.
 
8
  ---
9
  Model currently under training. Official publish will be on (DD/MM/YYYY) 23/02/2025.
10
  SaplingDream is a 0.5B parameter small GPT based on [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) finetuned on reasoning datasets with very high caution to ensure a high-quality sapling model—hence "SaplingDream".
11
+ The base model is finetuned using SGD to ensure better generalisation, in combination with the lr scheduler "Polynomial" with a starting lr of 1e-4. Better safe than sorry, we hope the model picks up on not only the tokens but also on how to actually reason through a problem.
12
  We're using [open-thoughts/OpenThoughts-114k](https://huggingface.co/datasets/open-thoughts/OpenThoughts-114k) and [prithivMLmods/Deepthink-Reasoning-Ins](https://huggingface.co/datasets/prithivMLmods/Deepthink-Reasoning-Ins) for training through an entire epoch.