Ivme-Conversate-U-v1-Base

Codename: Ultra Apple 1

A 316,723,200-parameter dense decoder-only transformer, trained entirely from scratch by IvmeLabs. Part of the Conversate family; see the IvmeLabs organization page for related models.

What this model actually is, and why it exists

Every model in the Conversate line so far has been trained on some blend of real web text: FineWeb-Edu, DCLM, Wikipedia, and so on, filtered and mixed in different proportions. The U in this release stands for a deliberately narrower experiment than any of those: what happens if you strip that out entirely and train a small model on nothing but synthetic data, generated end-to-end by a single open-weights teacher model?

Concretely: every one of the 1.50 billion training tokens here was written by Mixtral-8x7B-Instruct-v0.1, not scraped from the web. The corpus is Cosmopedia v2, a collection of textbook-style explanations, stories, blog posts, and WikiHow-style articles that Mixtral was prompted to write across a wide spread of topics, audiences, and formats. Nothing in the pipeline touched a closed-source model at any point, by design; the entire point was to see how much a small student can absorb from a strong-but-open teacher's own writing, at a token budget small enough to run on a single consumer-class GPU rather than a datacenter.

This is also the first Conversate release built on a real published architecture rather than the custom Ivme design used in v1 and v2: the model is a downscaled Qwen3 (the native Qwen3ForCausalLM class from transformers, not a reimplementation), sized down from Qwen3-0.6B's 28-layer/152K-vocab configuration to 316,723,200 parameters with a much smaller, English-only, 32,768-token vocabulary. That choice was deliberate: Qwen3-0.6B's full vocabulary alone eats over 20% of its total parameter budget, most of it on multilingual tokens this model would never see in an English-only synthetic corpus. Shrinking the vocabulary keeps the parameter budget where it's actually useful: in the transformer layers doing the learning, not in embedding rows that would sit almost untrained.

The tokenizer, and why every digit matters

The tokenizer was trained from scratch on a sample of the pretraining mix, byte-level BPE, 32,768 tokens. The one deliberate, load-bearing design choice in it: every single digit 0 through 9 is forced to be its own atomic token, via an explicit pre-tokenizer rule that splits on digit boundaries before BPE merges are ever learned. A number like 2024 is guaranteed to tokenize as 2 0 2 4, never as a single opaque 2024 token or some arbitrary two-digit chunk that happened to be frequent in the training sample.

This matters because ordinary BPE tokenizers learn digit groupings from frequency alone, with no regard for arithmetic structure, which is a big part of why small language models are notoriously bad at basic arithmetic: the model never actually sees the individual digits it would need to add or carry. Digit-isolation is a known mitigation (used in GPT-NeoX, Llama, and PaLM, among others), and it's applied here in full. Whether it was enough to actually produce working arithmetic at this scale and token budget is a separate question, and the honest answer, covered below, is: not on its own.

Results

Benchmarked with lm-evaluation-harness, zero-shot, no fine-tuning of any kind.

Benchmark	Score
BLiMP (macro-average, 67 English paradigms) ↑	70.91%
ARC-Easy (acc) ↑	45.41%
ARC-Easy (acc_norm) ↑	41.08%
HellaSwag (acc) ↑	29.91%
HellaSwag (acc_norm) ↑	32.19%
PIQA (acc) ↑	63.98%
PIQA (acc_norm) ↑	63.76%
WikiText-2 (byte perplexity) ↓	2.4306
WikiText-2 (word perplexity) ↓	115.51
Held-out Cosmopedia validation perplexity ↓	6.07

A quick read of what these numbers actually mean, task by task:

• PIQA (63.98%) is comfortably above the 50% chance baseline for this binary physical-commonsense task. The model has picked up real, usable intuitions about how physical objects and everyday actions relate to each other, likely because Cosmopedia's textbook-style content is full of exactly this kind of explanatory, cause-and-effect writing.
• ARC-Easy (45.41%) sits clearly above chance for a multi-choice science-question task. Some real factual/conceptual association was learned, though as the qualitative sample below shows, "some association" and "reliable" are not the same thing.
• HellaSwag (29.91%) is close to the ~25% chance floor for this 4-way task. This is the clearest negative result in the set: predicting which of several possible continuations of a short narrative scenario is most plausible requires the kind of grounded commonsense reasoning that textbook-style explanatory prose, however fluent, does not teach well. This is a data-mix gap, not a parameter-count wall; a broader mix including more narrative/scenario-style text would likely move this number.
• BLiMP (70.91% average across 67 paradigms) is, by a wide margin, the strongest result here, and worth its own section below.
• WikiText-2 perplexity is measured on out-of-domain text; the model was never trained on Wikipedia, only on Cosmopedia, so some gap here versus a model trained directly on Wikipedia-style text is expected and not itself a sign of a problem.

BLiMP paradigm breakdown: strong grammar, weaker long-range structure

BLiMP tests grammaticality judgment directly: for each of 67 minimal-pair paradigms, the model is shown a grammatical sentence and a minimally-different ungrammatical one, and scored on whether it assigns higher likelihood to the correct one. No world knowledge is needed, only syntax.

Strongest paradigms, almost all of them local-agreement phenomena (subject-verb agreement, determiner-noun agreement, reflexive/anaphor binding):

Paradigm	Accuracy
blimp_principle_A_case_1	99.40%
blimp_anaphor_number_agreement	95.80%
blimp_principle_A_domain_1	95.20%
blimp_existential_there_quantifiers_1	95.10%
blimp_determiner_noun_agreement_1	95.10%
blimp_irregular_past_participle_adjectives	93.20%

Weakest paradigms, almost all of them long-distance dependencies or syntactic island constraints, where a model has to track a relationship across many intervening words or clause boundaries:

Paradigm	Accuracy
blimp_wh_vs_that_with_gap_long_distance	23.80%
blimp_sentential_negation_npi_scope	33.60%
blimp_left_branch_island_echo_question	38.50%
blimp_matrix_question_npi_licensor_present	39.40%
blimp_npi_present_1	40.20%

This split, strong on local grammatical agreement, weaker on long-range structural dependencies, is a well-documented signature of small language models generally, not something specific to this run. It suggests the model built a genuinely solid local grammar (which shows up immediately in how fluent its writing reads) well before it built the kind of deeper structural tracking that long-distance phenomena require. That second capability tends to need either more scale, more data, or both.

---

Does it actually make sense, though?

None of the benchmarks above test the thing that arguably matters most for how a small model feels to use: whether its writing holds together as connected, coherent text across multiple sentences, and whether what it says is actually true. Here's an unedited sample, temperature 0.8, top_p 0.95:

Prompt: "Once upon a time, there was a scientist who"

Once upon a time, there was a scientist who loved learning about the world around us. She had discovered something amazing - the number of babies in the United States when she was just a child! This meant that, in 1906, there were around 30 million little ones that could be born toddlers. That's like a lot of babies being born every day!

Now, let's talk about how these babies

Grammatically, this is clean. It stays on-topic, the sentence structure never breaks down, and it reads like it's building toward an explanatory point, the exact voice Cosmopedia's synthetic data was written in. But look closely at the actual content: the "30 million babies in 1906" claim is invented, stated with complete confidence, and never flagged as uncertain in any way. This is a textbook case of what's sometimes called fluent hallucination: a model that has learned the shape of confident explanatory prose without learning to track whether the specific facts inside that shape are true.

A second, more targeted probe, aimed directly at whatever the digit-aware tokenizer bought this model:

Prompt: "In mathematics, the number 4827 plus 199 equals"

In mathematics, the number 4827 plus 199 equals 3459, while the number 8127 is also 255. These numbers represent the number of hours that a worker spends their time working in a factory and the amount of time they spend doing other activities.

The arithmetic is simply wrong (4827 + 199 = 5026, not 3459), and the response doesn't even stay inside the arithmetic frame, it drifts into an unrelated, invented claim about work hours within the same sentence. This is the honest, disappointing half of the digit-aware tokenizer story: giving the model atomic access to individual digits is necessary for arithmetic capability, but it is very clearly not sufficient on its own. Cosmopedia is textbook-narrative content, not arithmetic drill data, so the model was never actually shown enough worked examples of addition to learn the operation, no matter how cleanly the digits themselves were tokenized. The tokenizer did its job; the data didn't ask it to do arithmetic often enough for the model to learn it.

---

Architecture

Parameter	Value
Base architecture	Qwen3 (Qwen3ForCausalLM), downscaled from Qwen3-0.6B
Total parameters	316,723,200
Embedding parameters	33,554,432 (10.6% of total)
Vocabulary	32,768 tokens, custom byte-level BPE, digit-aware (see above)
Context length	2,048 tokens
Embeddings	Tied input/output
Precision	bfloat16 (autocast, not stored weights)

The embedding-share number above is worth calling out on its own: at 10.6% of total parameters, embeddings are a much smaller share of this model's budget than the roughly 20%+ that Qwen3-0.6B's own full 152K-token multilingual vocabulary would occupy at this hidden size. That difference is entirely the point of shrinking the vocabulary down for an English-only synthetic corpus: more of the parameter budget goes to the transformer layers that actually do the reasoning, rather than to embedding rows for languages and scripts this model never saw a single training example of.

Training

Data: 100% synthetic, 1.50 billion tokens, streamed and packed from HuggingFaceTB/smollm-corpus (cosmopedia-v2 config) without ever downloading the full dataset to disk. Cosmopedia v2 itself was generated by Mixtral-8x7B-Instruct-v0.1 across a deliberately wide spread of topics, target audiences, and writing formats (textbook, story, blog post, WikiHow-style guide, and more).

Optimization: AdamW, cosine learning-rate schedule with linear warmup, gradient clipping at norm 1.0, bf16 automatic mixed precision, torch.compile.

Steps: 12,207 optimizer steps.

Hardware: a single NVIDIA RTX PRO 6000 Blackwell (96GB), with the training batch size auto-probed at startup (binary search for the largest batch that fits in memory, rather than a hand-guessed constant) to use the card efficiently without manual tuning. Total training time approximately 3.33 hours, run across two sessions with a full resume (model weights, optimizer momentum, and learning-rate schedule position all restored) in between.

Final training loss: 1.8033 (held-out validation perplexity 6.07).

Tokenizer

Custom byte-level BPE, trained from scratch on a sample of the pretraining mix, 32,768 tokens. The one non-default choice: an explicit digit-isolation pre-tokenizer rule forces every digit 0-9 to be split off as its own token before BPE merges are learned, so multi-digit numbers can never collapse into a single opaque token regardless of how frequently a particular digit sequence appeared in training. See "The tokenizer, and why every digit matters" above for the full reasoning and its limits in practice.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("IvmeLabs/Ivme-Conversate-U-v1-Base")
tokenizer = AutoTokenizer.from_pretrained("IvmeLabs/Ivme-Conversate-U-v1-Base")

inputs = tokenizer("Once upon a time,", return_tensors="pt")
outputs = model.generate(
    **inputs, max_new_tokens=100, do_sample=True,
    temperature=0.8, top_p=0.95, pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

No trust_remote_code needed here, unlike some earlier Conversate releases: this model uses the native Qwen3ForCausalLM class already built into transformers, so a stock install is enough.

Limitations

• Base model only. Not instruction-tuned in any way. It will continue text in the style it was trained on; it will not follow instructions, answer questions directly, or engage in dialogue.
• English only. The tokenizer and training data contain effectively no other languages.
• 2,048 token context window. Nothing beyond that length was ever seen during training.
• Weak factual reliability. The model has learned local grammar very well (see the BLiMP results above) but frequently states specific, confident-sounding claims that are simply invented. Do not trust any factual claim from this model without independent verification.
• Arithmetic does not work reliably, despite the digit-aware tokenizer. This is discussed in detail above: digit-level tokenization is necessary but not sufficient without training data that actually demonstrates the arithmetic operation itself.
• Weaker on long-distance syntactic dependencies than on local agreement. See the BLiMP paradigm breakdown above for specifics.
• HellaSwag performance sits close to chance. Grounded narrative commonsense reasoning did not meaningfully emerge from this data mix at this scale.
• Trained on synthetic data only, which means the model's knowledge and style are bounded by whatever Mixtral-8x7B-Instruct-v0.1 chose to generate when producing Cosmopedia v2, not by the broader diversity of real-world text.

What's Next

Several open threads from this experiment are worth pursuing directly: a larger and more diverse data mix beyond Cosmopedia alone (to address the HellaSwag and factual-reliability gaps), a data mix that actually includes worked arithmetic examples (to see whether the digit-aware tokenizer pays off once the training signal matches it), and a like-for-like comparison against the custom Ivme architecture used in v1 and v2, to separate out how much of this model's behavior comes from the architecture change versus the pure-distillation data strategy.

You can check our other upcoming models on our organization card!

Citation

@misc{ivme-conversate-u-v1-base,
  author       = {IvmeLabs},
  title        = {Ivme-Conversate-U-v1-Base},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/IvmeLabs/Ivme-Conversate-U-v1-Base}
}

---

Built by IvmeLabs. Small models, deliberate choices.