Ryan ROBBE

Théo, I’ll admit I don’t get all the technical details since I’m new to AI, but what really draws me in is the honesty of it all. You tested four ideas, you had four failures, and you’re still showing them off instead of hiding them. And the part where your initial results looked too good to be true, so you dug deeper and found out it was just noise—that’s the kind of rigor we never see, but which is so useful for a (TSSR). Most people would have kept the flattering number and published it.

I have a curious question: since your model “loops” back on itself, does that make it take longer to respond?
In any case, keep it up.

It's impressive that you're running an LLM from scratch, all by yourself, on a 1080 Ti that crashes at night. Just you, all on your own—that's true dedication. The instinct to check your code before jumping to the conclusion that a result is wrong—that's the true mindset of a researcher, not the pursuit of headlines. I'm following your work closely. Keep it up—you're showing that it's possible to do serious AI research without lab resources.

Respect—because starting from scratch with a 1080 Ti was no small feat.

I finally understand why you didn’t want to just fine-tune an existing LLM. You really wanted to understand the entire process.

But I’m wondering something. If you scale up to 1 billion tokens, do you think the 15 million parameters will still cut it, or are you already considering a bigger model? I get the feeling the dataset won’t be the only limiting factor anymore.

In any case, keep at it. I can’t wait to see how V2 turns out.

Translated with DeepL.com (free version)

Hi Théo,

First of all, congratulations on the work you’re sharing. It’s really interesting to see the challenges you’ve faced, rather than just the results.
In any case, don’t give up. Infrastructure issues are often the most frustrating, but they’re also what make projects robust once they’re resolved. Good luck with the rest of the project—I can’t wait to see the next results!