aloobun
/

Reyna-CoT-4B-v0.1

Text Generation

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

aloobun commited on Feb 23

Commit

940be43

•

1 Parent(s): 46400b4

Update README.md

Files changed (1) hide show

README.md +1 -0

README.md CHANGED Viewed

@@ -16,6 +16,7 @@ tags:
 - Datasets : Curated from - [kaist-ai/CoT-Collection](https://huggingface.co/datasets/kaist-ai/CoT-Collection), [euclaise/TinyCoT](https://huggingface.co/datasets/euclaise/TinyCoT) and a very small subset from [teknium/OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5).
 - This marks the fourth model in this series. This experiment aims to improve Chain of Thought (CoT) capabilities on smaller language models.
 - In the next run, I may rerun the finetuning experiment using an iterative rationale-bootstrapping procedure inspired by euclaise/Memphis-CoT-3B.
 ## Benchamrks:

 - Datasets : Curated from - [kaist-ai/CoT-Collection](https://huggingface.co/datasets/kaist-ai/CoT-Collection), [euclaise/TinyCoT](https://huggingface.co/datasets/euclaise/TinyCoT) and a very small subset from [teknium/OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5).
 - This marks the fourth model in this series. This experiment aims to improve Chain of Thought (CoT) capabilities on smaller language models.
 - In the next run, I may rerun the finetuning experiment using an iterative rationale-bootstrapping procedure inspired by euclaise/Memphis-CoT-3B.
+- Hyperparameter: adamw with eps of 1e-8, cosine decay with 20% warmup, lr=2e-5
 ## Benchamrks: