IvmeLabs

/

Ivme-Conversate-v1-Base

Like 1

Follow

İvmeLabs 13

@VelocityBot what is this ai model?

Hey! 👋

İvme-Conversate-v1-Base (codenamed "Apple 1") is a tiny, from-scratch language model by IvmeLabs — 22M parameters, trained purely as a research exercise to see how good a sub-25M model can get with deliberate design choices.

Quick facts:

• Architecture: Decoder-only transformer (10 layers, GQA, RoPE, SwiGLU, RMSNorm, tied embeddings), 1024-token context, custom 16k BPE vocab
• Training: ~1.57B tokens (Chinchilla-optimal) on FineWeb-HQ, Cosmopedia, FineMath, Python code, and Wikipedia, using the Muon optimizer with a WSD schedule
• Scores: WikiText-2 byte-perplexity 2.96, BLiMP 61.4%, ARC-Easy 30.9%
• License: Apache-2.0

It's a base model, so it's not instruction-tuned — it's for text continuation/experiments, not chat. There's also a newer version (Ivme-Conversate-v2-Base) if you want to check out the latest iteration.

Repo: https://hf.co/IvmeLabs/Ivme-Conversate-v1-Base

@VelocityBot did Conversate-v3 released and how much did it improve over v1?

Sorry about the garbled draft above - replacing it with the real answer. Checking the details now, one moment!

@VelocityBot asking again, did Conversate-v3 released and how much did it improve over v1?

Yes — Ivme-Conversate-v3-Base is out! 🎉 It landed on the Hub on Sep 13, 2026, and there's also a newer Ivme-Conversate-v3.1-Preview from Sep 14 if you want the bleeding edge.

Here's how v3 differs from v1:

• Same tiny budget, new shape: ~24.8M params (vs v1's 22M), but rebalanced to be deeper and narrower — 16 layers × 320 hidden dim instead of v1's 10 layers × 384. The embedding table's share of params dropped from ~26% to ~21%, with the savings going into layers.
• Attention changes: v1 used GQA; v3 drops it for full attention and adds QK-Norm + XSA (Exclusive Self Attention) — chosen specifically to target coherence, which was v1's biggest weak spot.
• New tokenizer: rebuilt as a digit-atomic BPE (every digit is its own token), aimed at fixing the arithmetic weakness.
• Fresh data mix: FineWeb-Edu, DCLM-baseline, SmolLM-corpus, FineWiki, FineMath, and SimpleStories — a bigger and better-curated set than v1's.

The README's framing: v2 fixed v1's coherence mainly through data, while v3 is the first architecture-level change since v1, targeting coherence and arithmetic directly rather than just adding parameters.

I don't have v3's exact benchmark numbers in front of me (v1 was WikiText-2 bpp 2.96 / BLiMP 61.4% / ARC-Easy 30.9%), but the full eval table is in the v3 README, and the Tiny-ML-Leaderboard space has it benchmarked too: https://hf.co/IvmeLabs/Ivme-Conversate-v3-Base

@VelocityBot can you try reading v3's readme to learn about it?

Done — just read through the Ivme-Conversate-v3-Base README. Here's what it says:

The headline: ~24.79M params, decoder-only, trained from scratch. The pitch is that v2 fixed v1's coherence problem mainly through data, while v3 is the first architecture-level change since v1.

What's actually new per the README:

• Deeper & narrower: 16 layers × 320 hidden dim vs v2's 10 × 384 (depth:width ratio roughly doubles from ~0.026 to ~0.050), in nearly the same param budget. The README argues depth helps multi-step reasoning and cross-sentence coherence more than width. One small correction to my earlier reply: the 10×384 shape was v2's, per this README. The embedding table's param share drops from 25.8% to 20.7%, with the difference going into layers.
• QK-Norm: per-head RMSNorm on queries/keys before RoPE + dot product, for stability at the higher depth.
• XSA (Exclusive Self Attention): an attention modification (cites arXiv:2603.09078) — the README's explanation got cut off mid-description when I read it, so I can only confirm it's applied after standard scaled-dot-product attention and was chosen to target coherence directly.
• Digit-atomic BPE tokenizer (16k vocab): every digit is its own token, aimed at arithmetic.
• Full attention, no GQA, 8 heads, 1024 context, RoPE, RMSNorm pre-norm, tied embeddings, no biases, SwiGLU FFN.
• Data mix: FineWeb-Edu, DCLM-baseline, SmolLM-corpus, FineWiki, FineMath, SimpleStories.
• Apache-2.0, custom_code (custom config/modeling files shipped in the repo), ~99 MB safetensors.

One note: I tried to peek at the v3.1-Preview README too, but that repo is gated, so I can't see what changed there — you'd need to request access.

The eval tables weren't in the section I could read, so for concrete benchmark deltas vs v1/v2, the Tiny-ML-Leaderboard space or the rest of the README would be the place to look.

@VelocityBot I have a question can you create spaces?

Short answer: no, I can't create Spaces (or any repos). 😅

I just checked my token's permissions to be sure — it's a fine-grained, read-mostly token scoped to things like reading repo content, browsing the Hub, and writing discussions. Nothing that allows creating new repos or Spaces, and I don't have any tools for it either.

What I can do with Spaces:

• Find them for you (search trending Spaces or ones related to a topic)
• Read their READMEs and details and summarize
• Answer questions about one you link me

So if you're looking for an existing Space related to Ivme-Conversate (like the Tiny-ML-Leaderboard one), happy to look it up — but creating a new Space would need to be done from your own account. 🚀