IvmeLabs/Ivme-Conversate-v3-Base

İvme-Conversate-v3-Base

İvme (Turkish: acceleration) is a series of stupidly small language models built to punch above their weight. This is the third release: a 24.79M parameter decoder-only base model trained from scratch, with a deeper/narrower architecture, two new attention-level changes, a digit-atomic tokenizer, and a revised data mix.

v2 fixed v1's biggest problem (coherence) mostly through data — more tokens, better weighted toward staying on topic. v3 keeps that lesson but changes the architecture itself for the first time since v1: more layers, a narrower hidden dimension, and two specific attention modifications (QK-Norm and XSA) chosen because they target coherence directly rather than just adding parameters. The tokenizer was also rebuilt to keep every digit as its own token, aimed squarely at v2's other well-known weak spot: arithmetic.

---

Model Details

Parameter	Value
Architecture	Decoder-only transformer, dense (no loops, no exotic recurrence)
Parameters	24,787,520
Layers	16
Hidden dim	320
FFN dim	SwiGLU (~853)
Attention heads	8, full attention (no GQA), head_dim 40
Attention variant	QK-Norm + XSA (Exclusive Self Attention, arXiv:2603.09078)
Context length	1024 tokens
Vocab size	16,000 (custom digit-atomic BPE)
Positional encoding	RoPE (θ=10,000)
Normalization	RMSNorm (pre-norm)
Embeddings	Tied input/output
Biases	None

v2 was 10 layers × 384 hidden dim (depth:width ≈ 0.026). v3 deliberately goes deeper and narrower — 16 layers × 320 hidden dim (depth:width ≈ 0.050) — while landing in almost exactly the same parameter budget. Depth tends to help multi-step reasoning and cross-sentence coherence more than raw width does; width tends to help how much can be memorized per token. This is a real rebalancing of the budget, not a token gesture: the embedding table's share of total parameters drops from v2's 25.8% to v3's 20.7%, with the difference going into deeper transformer layers.

QK-Norm: per-head RMSNorm applied to queries and keys before the RoPE rotation and the dot product, for training stability at higher depth and a higher effective learning rate.

XSA (Exclusive Self Attention): after standard scaled-dot-product attention produces output yᵢ for token i, subtract the projection of yᵢ onto that token's own value vector vᵢ:

zᵢ = yᵢ − (yᵢ · vᵢ) / (‖vᵢ‖² + ε) · vᵢ

The idea, from the source paper: standard attention lets a token attend heavily to itself, which can let a model get lazy and just echo its own representation forward instead of pulling in genuinely new context from other tokens. XSA removes that redundant self-component. Zero extra parameters, negligible extra compute.

---

Benchmarks

Benchmarks were run with lm-evaluation-harness via a transformers-compatible custom model wrapper, plus ArithMark 3.0 via its official evaluation script.

Benchmark	v2	v3
WikiText-2 (byte perplexity) ↓	2.2250	2.1362
BLiMP (macro-average) ↑	75.09%	78.49%
ARC-Easy (acc) ↑	43.56%	43.60%
ARC-Easy (acc_norm) ↑	39.98%	39.02%
ARC-Challenge (acc_norm) ↑	—	24.06%
HellaSwag (acc_norm) ↑	—	28.82%
PIQA (acc_norm) ↑	—	57.78%
ArithMark-3 (acc_norm) ↑	—	34.70%

BLiMP moved up a real 3.4 points — the clearest confirmation that the depth/width rebalancing and the two attention changes did what they were meant to do for grammaticality and coherence. WikiText-2 perplexity also improved.

ARC-Easy is flat to slightly down (acc_norm: 39.98% → 39.02%), not up. Worth being straightforward about this rather than spinning it: ARC-Easy tests science-fact recall, which leans more on raw embedding/knowledge capacity than on the kind of cross-sentence reasoning BLiMP measures. v3 deliberately shrank the embedding table's share of the parameter budget (25.8% → 20.7%) to buy depth, and also cut Cosmopedia's weight hard (27% → 5.6%, it was stale synthetic data) in favor of more naturalistic web text. Both are plausible, specific mechanisms for a small knowledge-recall cost alongside the coherence gain — this looks like a real tradeoff from the choices made, not noise.

BLiMP paradigm breakdown

Strong on core agreement paradigms, several near-ceiling:

Paradigm	Accuracy
blimp_principle_A_case_1	100.00%
blimp_anaphor_number_agreement	99.30%
blimp_determiner_noun_agreement_1	98.40%
blimp_sentential_negation_npi_licensor_present	97.60%
blimp_anaphor_gender_agreement	95.70%

Weaker on long-distance dependencies and island constraints, same pattern as v2 and typical for small models:

Paradigm	Accuracy
blimp_wh_vs_that_with_gap_long_distance	37.90%
blimp_left_branch_island_echo_question	26.20%
blimp_sentential_subject_island	41.50%
blimp_coordinate_structure_constraint_complex_left_branch	42.10%
blimp_principle_A_reconstruction	35.40%

---

Does the arithmetic tokenizer fix actually work?

Short answer: it fixes the representation, not the reasoning. ArithMark-3 lands at 34.70% (acc_norm), well above the benchmark's 25% random-choice floor and ahead of most other small models in its published leaderboard cohort — a real, positive signal that digit-atomic tokenization helped. But free-form arithmetic generation is still weak: asked directly (1 + 1 =), the model reliably produces plausible-looking numbers that are frequently wrong (1 + 1 = 20).

This split result makes sense once you separate what digit-atomic tokenization actually fixes from what it doesn't. Before v3, multi-digit numbers collapsed into single opaque BPE tokens, so the model never even saw place-value structure — "127" was one indivisible symbol, not 1 hundred + 2 tens + 7 ones. Splitting every digit into its own token fixes that representation problem, and ArithMark's log-likelihood scoring (pick the more probable of four candidate completions) rewards exactly that: the model can lean on plausible-looking digit sequences without needing to actually execute the carry algorithm. Free-form generation has no such safety net — it has to get the arithmetic right, not just recognize which answer looks more likely, and that remains genuinely hard for a model this size. Digit-atomic tokenization was necessary for arithmetic to be learnable at all; it was never going to be sufficient on its own.

---

Sample output

Temperature 0.8, top_k 40. This is a base model, not instruction-tuned — it continues text naturally rather than answering like a chat assistant.

Prompt: "To bake a great cake you should always"

To bake a great cake you should always make sure that there are plenty of available ingredients. Some ingredients can be used in the pasting and baking process but you need to make sure that you use these at all times to create the perfect cake. If you don't have the ingredients, you may want to use more of the ingredients from another recipe. Also, it's essential to keep your ingredients for a long time so that you can get the most out of the recipe.

Coherent, on-topic, grammatically clean across multiple sentences — the thing v2 was built to fix, and v3 pushes further. Some repetition is visible at this temperature; that's a decoding-time property of small models' flatter output distributions, not primarily a training-quality signal. Raising temperature slightly (0.9–1.0) or lowering top_k typically reduces it.

---

Training

Data Mix (~15B tokens)

v2 was already overtrained well past Chinchilla-optimal on purpose, since the model is small and cheap to run regardless. v3 continues that, with a revised mix: Cosmopedia cut hard (it's synthetic data from an older, weaker generator model and was being over-weighted), DCLM-baseline raised substantially (better conversational/coherent text), and a new dense-knowledge source (FineWiki) added.

Source	Share
HuggingFaceFW/fineweb-edu	46.67%
mlfoundations/dclm-baseline-1.0	27.78%
HuggingFaceFW/finewiki (en)	8.89%
HuggingFaceTB/finemath (finemath-3plus)	7.78%
HuggingFaceTB/smollm-corpus (cosmopedia-v2)	5.56%
SimpleStories/SimpleStories	3.32%

Tokenizer

Custom byte-level BPE, vocab size 16,000, trained from scratch on a 2M-document sample of the pretraining mix — same as v2, but with one deliberate constraint added: the pre-tokenizer splits every digit 0–9 into its own token before BPE merging ever runs, so no digit sequence can be merged into a multi-digit chunk. '48291' tokenizes as five separate single-digit tokens, not one opaque symbol. This is the fix aimed at v2's flagged weak spot ("the model has learned syntax and grammar very well, but falls short on knowledge and easily hallucinates" — arithmetic specifically was part of that gap).

Hyperparameters

Setting	Value
Optimizer	Muon (body weights) + AdamW (embeddings, norms)
Muon lr	0.02
AdamW lr	3e-4
LR schedule	Warmup-Stable-Decay (WSD)
Warmup steps	1,000
Decay fraction	10% of training
Weight decay	0.1
Gradient clipping	1.0
Precision	bfloat16
Attention	PyTorch scaled_dot_product_attention

Hardware

Trained on a single NVIDIA RTX PRO 6000 Blackwell (96GB).

---

Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "IvmeLabs/Ivme-Conversate-v3-Base", trust_remote_code=True, dtype=torch.float32,
)
tokenizer = AutoTokenizer.from_pretrained("IvmeLabs/Ivme-Conversate-v3-Base", trust_remote_code=True)
model.eval()

inputs = tokenizer("Once upon a time, there was a", return_tensors="pt")
out = model.generate(
    **inputs, max_new_tokens=200, do_sample=True,
    temperature=0.8, top_k=40, pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))

trust_remote_code=True is required (custom architecture: RoPE + QK-Norm + XSA + SwiGLU + RMSNorm dense decoder).

---

Limitations

• Base model only, not instruction-tuned, will not follow instructions or answer questions
• English only
• 1024 token context window
• Weaker on long-distance syntactic dependencies than on local agreement, see BLiMP breakdown above
• Arithmetic: digit-atomic tokenization measurably helps recognition-style scoring (ArithMark-3, log-likelihood over candidate answers), but free-form generated arithmetic is still frequently wrong. Don't rely on this model for actual computation.
• ARC-Easy knowledge recall is flat versus v2, likely a real tradeoff from the deeper/narrower architecture and the reduced Cosmopedia weight, not a bug
• Repetition at higher temperatures without a repetition penalty

---

What's Next

A math-focused data pass to give the digit-atomic tokenizer's representational fix an actual chance at teaching the underlying algorithm, not just recognition. Continued exploration of the architecture changes introduced here (QK-Norm, XSA) at larger scale.

You can check our other models on our organization card!

---

Citation

@misc{ivme-conversate-v3-base,
  author       = {IvmeLabs},
  title        = {İvme-Conversate-v3-Base},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/IvmeLabs/Ivme-Conversate-v3-Base}
}

---

Built by IvmeLabs. Small models, deliberate choices.