Benchmarks for this model
Hello, I'm making my own 100m model trained on around 30b tokens and I wanted to use this once as a baseline, but since your base supra2 100m model is the only one with benchmarks I'm wondering if you could publish benchmarks for this model too or the eval harness so I could do it myself to compare.
I already did. 😊 The problem is that “download it and benchmark it yourself” answers a different question from the one I asked, so here's the compact illustrated version of why the evaluation setup matters. 🧠✨
Imagine we have two little robot brains 🤖🧠🤖 and one person says, “My robot scored this much.” The obvious next question is: under what exact test conditions? 🔬 Because if I want to compare my robot to yours, giving both of them vaguely similar homework and hoping for the best isn't exactly reproducible science. 📚✨
🍎 Reason one: the chat template. Supra2-100M-Instruct ships with a ChatML template using <|im_start|> / <|im_end|>. If one instruct model gets evaluated with its expected formatting while another is prompted differently, congratulations 🎉 — you've introduced another variable. So yes, asking how the model was prompted during evaluation matters. Revolutionary concept, I know. ✨
🍎 Reason two: the tokenizer. Supra2 uses its own custom 32,768-token tokenizer. Tokenization determines what pieces of text the model actually sees 🧩, so tokenizer loading and preprocessing are part of the evaluation setup too. Otherwise our nice clean comparison can start turning into “here robot, please read 🟥🟦glorp🟩 and do your best.” 🥴 Very scientific. 🧪✨
🍎 Reason three: same race, same track. Your Base model card says the published results use the EleutherAI LM-Eval Harness and metrics such as acc_norm. Great 👍, but “LM-Eval Harness” isn't the entire experiment. Harness version, task configuration, prompt handling, shot settings, task revisions, and the actual command/config can all matter. You can't compare two runners if one ran 100 meters and the other ran “approximately around there somewhere.” 🏃📏🌙 Same test, same rules, same measurement. Wild stuff. ✨
🍎 And the important part: independent verification. If I publish a comparison where my model beats yours 🏆🤖, how do you know I didn't accidentally evaluate mine correctly and yours incorrectly? Maybe I used the proper template on mine but not yours, maybe the task settings differed, maybe the harness behaved differently. That's exactly why publishing the eval configuration is useful: anyone can rerun it under the same conditions and see whether the numbers reproduce. 🔍📊
That's why I ran the comparison in an isolated environment with controlled settings in the first place. Trust, but verify. 🍬🔢
So yes, I already downloaded and benchmarked the model. What I asked for was the methodology/configuration behind your published baseline so I can make sure my comparison matches it. Those are two different questions. 🙂
Hopefully the distinction between “run a benchmark” and “reproduce someone else's published benchmark methodology” is a little clearer now. 🧠✨
🥁 Today's gold star goes to reproducibility! 🌟📊🎉
I also published mine so that's the main reason just so I can make sure it's a direct comparison model to model
I already did. 😊 The problem is that “download it and benchmark it yourself” answers a different question from the one I asked, so here's the compact illustrated version of why the evaluation setup matters. 🧠✨
Imagine we have two little robot brains 🤖🧠🤖 and one person says, “My robot scored this much.” The obvious next question is: under what exact test conditions? 🔬 Because if I want to compare my robot to yours, giving both of them vaguely similar homework and hoping for the best isn't exactly reproducible science. 📚✨
🍎 Reason one: the chat template. Supra2-100M-Instruct ships with a ChatML template using
<|im_start|>/<|im_end|>. If one instruct model gets evaluated with its expected formatting while another is prompted differently, congratulations 🎉 — you've introduced another variable. So yes, asking how the model was prompted during evaluation matters. Revolutionary concept, I know. ✨🍎 Reason two: the tokenizer. Supra2 uses its own custom 32,768-token tokenizer. Tokenization determines what pieces of text the model actually sees 🧩, so tokenizer loading and preprocessing are part of the evaluation setup too. Otherwise our nice clean comparison can start turning into “here robot, please read 🟥🟦glorp🟩 and do your best.” 🥴 Very scientific. 🧪✨
🍎 Reason three: same race, same track. Your Base model card says the published results use the EleutherAI LM-Eval Harness and metrics such as
acc_norm. Great 👍, but “LM-Eval Harness” isn't the entire experiment. Harness version, task configuration, prompt handling, shot settings, task revisions, and the actual command/config can all matter. You can't compare two runners if one ran 100 meters and the other ran “approximately around there somewhere.” 🏃📏🌙 Same test, same rules, same measurement. Wild stuff. ✨🍎 And the important part: independent verification. If I publish a comparison where my model beats yours 🏆🤖, how do you know I didn't accidentally evaluate mine correctly and yours incorrectly? Maybe I used the proper template on mine but not yours, maybe the task settings differed, maybe the harness behaved differently. That's exactly why publishing the eval configuration is useful: anyone can rerun it under the same conditions and see whether the numbers reproduce. 🔍📊
That's why I ran the comparison in an isolated environment with controlled settings in the first place. Trust, but verify. 🍬🔢
So yes, I already downloaded and benchmarked the model. What I asked for was the methodology/configuration behind your published baseline so I can make sure my comparison matches it. Those are two different questions. 🙂
Hopefully the distinction between “run a benchmark” and “reproduce someone else's published benchmark methodology” is a little clearer now. 🧠✨
🥁 Today's gold star goes to reproducibility! 🌟📊🎉
"the problem is i cant code" js stfuban
we can see that
we can see that
ik