CodeSoft/slm-arena-data
1.99 MB
CodeSoft PRO
CodeSoft
AI & ML interests
Working on small language models
Recent Activity
liked a model about 19 hours ago
internlm/Intern-S2-397B updated a bucket 1 day ago
CodeSoft/slm-arena-data upvoted an article 1 day ago
How Did We Get Here?Organizations
Posts 6
Post
2663
Sorbet Mini Experimental is out!
It's not the best model in the 5M parameter range, but it's going to be a really useful model to train on top of. This release was mostly to prove that the model actually works. I trained it on 150M tokens from TinyStories in about 12 minutes.
Does anyone have any tips on training models in this size range? I want to make the full Sorbet Mini release as good as it can be.
It's not the best model in the 5M parameter range, but it's going to be a really useful model to train on top of. This release was mostly to prove that the model actually works. I trained it on 150M tokens from TinyStories in about 12 minutes.
Does anyone have any tips on training models in this size range? I want to make the full Sorbet Mini release as good as it can be.
Post
155
Over the past week or so, I've been working on some models, those released being CodeSoft/sorbet-25m and CodeSoft/sorbet-v2-25m. In general, I'm a little confused because no matter what hyperparameters I change or datasets I add/remove, the benchmarks never move up.
In a recent project, where I attached a TN-gram block to Sorbet-v2-25M, it still stayed the same on benchmarks despite the TN-gram clearly learning (due to the perplexity being lower with the TN-gram attached). When I changed the corpus to favor higher density text (the first paragraphs of Wikipedia articles and synthetic math), the benchmarks either stayed flat or went down.
Does anyone have ideas on what I can do to improve my models? I'd really appreciate any feedback!
In a recent project, where I attached a TN-gram block to Sorbet-v2-25M, it still stayed the same on benchmarks despite the TN-gram clearly learning (due to the perplexity being lower with the TN-gram attached). When I changed the corpus to favor higher density text (the first paragraphs of Wikipedia articles and synthetic math), the benchmarks either stayed flat or went down.
Does anyone have ideas on what I can do to improve my models? I'd really appreciate any feedback!
models 7
CodeSoft/sorbet-mini-experimental
Text Generation • 4.92M • Updated • 1.3k • 1
CodeSoft/sorbet-v2-25m
Text Generation • 25.2M • Updated • 1.59k • 1
CodeSoft/sorbet-25m-gguf
Text Generation • 25.2M • Updated • 119
CodeSoft/sorbet-25m
Text Generation • 25.2M • Updated • 1.44k • 4
CodeSoft/MetaDiffusion-150M-ChatBase
Text Generation • 0.2B • Updated • 645 • 2
CodeSoft/MetaDiffusion-600M-ChatBase
Text Generation • 0.8B • Updated • 653 • 1
CodeSoft/MetaDiffusion-150M-exp
Text Generation • 0.2B • Updated • 199 • 4
datasets 0
None public yet