arm-gemma-e4b

arm-gemma-e4b is Gemma-4-E4B adapted to Armenian by continued pretraining. To our knowledge, it is the first open Armenian LLM released together with its complete training corpus and recipe — every training token is either public (ArmWeb, ArmSTEM, FineWeb-Edu, Stack-smol) or reproducible from the released pipeline.

This is a base model: no instruction tuning, no chat template. Use it for Armenian text continuation, likelihood scoring, or as a starting point for Armenian SFT.

Recipe

• Base: Gemma-4-E4B, stock tokenizer (best Armenian fertility among CPT candidates: 4.15 tokens/word; vocabulary extension was sharply harmful in our ablation at this budget).
• Continued pretraining: 10B tokens, sequence-packed at length 4096, LR 3×10⁻⁵ cosine. The architectural context window is unchanged from the base (128K, max_position_embeddings=131072); CPT only exercised positions up to 4096, and long-context behavior beyond that is inherited from the base and not specifically evaluated.
• Mixture (epoch-capped): 69% ArmWeb, 4% ArmSTEM-HY, 2% ArmSTEM-EN, 20% English web replay (FineWeb-Edu), 5% code (Stack-smol) — every ArmWeb token seen 2× and every ArmSTEM token 7–9× (Gemma-tokenizer accounting). The ArmSTEM stream is a 109,885-item subset of the corpus (30%, sampled at random with balanced stratification across the math and science pools); its manifest ships with ArmSTEM (stats/cpt_training_subset_ids.txt, 104,630 items verbatim in the release, 5,255 superseded before release). A control run trained on the full 373K corpus (~1.3 epochs instead of 7–9) scores within noise of the released model (paper §5), so the repetition is costless.
• Details: global batch 512 sequences (~2.1M tokens/step), 4,770 steps, 100 warmup steps, AdamW (β₂=0.95), weight decay 0.1, bf16, seed 42.

The mixture is the headline finding of the accompanying paper: news-only CPT catastrophically forgets (−21.2pp Belebele at LR 10⁻⁴); a gentler LR recovers two-thirds; 6% verified translated STEM data reverses forgetting entirely, ending +2.2pp above the unadapted base while keeping the fluency gains.

Results

Six-task Armenian likelihood suite (accuracy; harness: lm-eval):

Task	Gemma-4-E4B (base)	arm-gemma-e4b
Belebele-hye	0.619	0.716
INCLUDE-Armenian	0.416	0.456
m-MMLU-hy	0.343	0.337
ARC-hy	0.227	0.229
HellaSwag-hy	0.266	0.262
MultiBLiMP-hye	0.989	0.992
Mean	0.477	0.499

This is the highest six-task mean among all open Armenian models we evaluated — the best prior models score 0.471 (ArmenianGPT-1.0-3B) and 0.436 (HyGPT-10b), both below the unadapted base.

ArmBench-LLM generative tasks (base-model-appropriate metrics; full results in the paper appendix):

Task	base	arm-gemma-e4b
SynDARin (EM)	0.04	0.92
Hartak (EM)	0.02	0.82
DREAM (EM)	0.48	0.84
Belebele gen. (EM)	0.66	0.90
Scientific MCQA (EM)	0.86	1.00*
MMLU-Pro-Hy	0.154	0.251
Topic (14-class)	0.004	0.482

*Audited: zero shared 8-grams between the benchmark items and the ArmSTEM training corpus (paper, Appendix).

Honest negatives: POS tagging regresses under CPT (0.18→0.01); exam mathematics is flat for the released model (1.75 points), and the full-corpus control run reaching 2.75 suggests data diversity rather than difficulty is the binding factor; instruction-dependent ArmBench tasks (judged generation, BLEU QA) are low for all base-style models including this one — they measure formatting, and we defer them to an instruction-tuned variant.

Training data & contamination

Training data was decontaminated by 13-gram overlap against ten Armenian evaluation sets and all ArmBench items, on both the English and Armenian side for translated data. The two remaining reported benchmarks, m-MMLU-hy and ARC-hy, were scanned post hoc: zero of 4.31M ArmWeb training documents and 4 of 372,907 ArmSTEM pairs share any 13-gram with their items. Existing public Armenian corpora overlap these benchmarks at 7.9–17.4% (see the ArmWeb card).

Limitations

Inherits Gemma-4's biases and terms of use. News-domain-concentrated Armenian exposure. Machine-translated STEM data verified for answer preservation and language identity, not stylistic fluency (though a two-annotator native-speaker audit rated 299 of 300 sampled problems valid, Cohen's κ = 1.0). No safety tuning.

Citation

@article{arakelyan2026armweb,
  title  = {From Zero to Hero: An Open LLM Ecosystem for Armenian},
  author = {Arakelyan, Erik and Avetisyan, Khatun and Davtyan, Meri and Grigoryan, Heghine and Khachatryan, Nane and Shahsuvaryan, Hayk and Sergoyan, Henrik and Martirosyan, Vahan},
  year   = {2026},
  journal = {arXiv preprint arXiv:2609.03350}
}