AdithyanAI/Mythos_translation

Mythos-2.04B-MoE

This repository contains model weights and configuration files for the pre-trained model in the PyTorch and Hugging Face format.

These artifacts are compatible with PyTorch, Hugging Face Transformers, vLLM, SGLang, TokenSpeed, etc.

For users seeking managed, scalable translation inference without infrastructure maintenance, the official Mythos API service is in active development. In particular, Mythos-2.04B-MoE will be available as a high-throughput multilingual translation engine with production features, e.g., 8k context length by default, multi-stage translation from single sentences to full multi-page documents, and native 500+ language support. The service is coming soon. Stay tuned for updates.

Following the development of modern multilingual architectures, we are pleased to introduce Mythos-2.04B-MoE, the most capable generation in the Mythos open-model family to date.

Built on the architectural foundation of Sparse Mixture-of-Experts (SMoE), Mythos-2.04B-MoE delivers substantial gains across translation quality, low-resource language understanding, zero-shot cross-lingual transfer, and long-horizon document translation. Mythos-2.04B-MoE brings these advances to a compact, deployment-friendly sparse model: a native multilingual translation model that understands 500+ global languages, designed to carry complex sentence, paragraph, and multi-page document translations through to completion with greater reliability.

---

Mythos-2.04B-MoE Highlights

Mythos-2.04B-MoE features the following enhancements:

• Core Translation Capabilities: Comprehensive improvements across cross-lingual semantic alignment, high-resource language fluency, and zero-shot translation paths across low-resource dialects.
• Sparse MoE Efficiency: 8 specialized SwiGLU expert feedforwards per block with Top-2 dynamic routing, delivering 2.04B total parameter knowledge capacity while requiring only 678M active parameters per token.
• Long-Horizon Context Handling: Dedicated 8,192-token context window with Block-Diagonal Document Packing, effortlessly handling single sentences, multi-sentence paragraphs, and large documents without truncation.
• Vocabulary & Script Coverage: 128,000-token Byte-Level BPE subword vocabulary natively supporting all major writing systems (Latin, Cyrillic, Arabic, Devanagari, Hanzi, Kana, Hangul, African, and Indigenous scripts).
• Downstream Compatibility: Standard Encoder-Decoder interface compatible with popular inference harnesses and distributed frameworks.

---

Model Overview

• Type: Encoder-Decoder Multilingual Translation Model with Sparse Mixture-of-Experts (SMoE)
• Training Stage: Pre-training
• Language Model Architecture:
  • Number of Parameters: 2.04B (2,037,643,264 Total Parameters)
  • Active Parameters / Token: 678.7M (678,688,768 Active Parameters)
  • Hidden Dimension: 1,024
  • Token Embedding: 128,000 (Byte-Level BPE)
  • Number of Layers: 24 Layers (12 Encoder Layers + 12 Decoder Layers)
  • Hidden Layout: 24 × (GQA Attention $\to$ SwiGLU MoE FFN)
  • Gated Attention (GQA):
    • Number of Attention Heads: 16 for Q and 4 for KV (4× KV-Cache Compression)
    • Head Dimension: 64
    • Rotary Position Embedding (RoPE) Dimension: 64 ($\theta = 100,000.0$)
  • Feed-Forward Network (Sparse MoE):
    • Number of Experts: 8 Experts per block
    • Routing Mechanism: Top-2 Active Experts with Load Balancing + Router Z-Loss ($1\times 10^{-4}$)
    • Intermediate Dimension (per Expert): 3,072 (SwiGLU)
  • LM Output: 128,000 (Triton Liger-Fused Cross-Entropy)
  • Context Length: 8,192 tokens natively with Block-Diagonal Document Packing

---

Quickstart & Inference

1. Installation

pip install torch transformers tokenizers sacrebleu unbabel-comet

2. Python Inference Code

import torch
from tokenizers import Tokenizer

# Load Multilingual Tokenizer
tokenizer = Tokenizer.from_file("multilingual_tokenizer.json")
sos_id = tokenizer.token_to_id("[SOS]")
eos_id = tokenizer.token_to_id("[EOS]")
pad_id = tokenizer.token_to_id("[PAD]")

# Source Text and Target Language Prompt
src_text = "Artificial intelligence and multilingual language models are transforming global human communication."
target_language = "fra" # ISO-639-3 Language Code (e.g., 'fra', 'deu', 'hin', 'ara', 'zho', 'spa', 'rus', 'jpn')

prompt_text = f"<2{target_language}>"
enc_tokens = [sos_id] + tokenizer.encode(src_text).ids + [eos_id]
dec_tokens = [sos_id] + tokenizer.encode(prompt_text).ids

encoder_input = torch.tensor([enc_tokens], dtype=torch.long)
decoder_input = torch.tensor([dec_tokens], dtype=torch.long)

print(f"Source Text : {src_text}")
print(f"Target Lang : {target_language}")
print("Translating with Mythos-2.04B-MoE...")

---

Training Methodology & Infrastructure

• Pre-Training Dataset: MaLA-LM Bilingual Translation Corpus (16.5 Billion tokens across 3,759 distributed Parquet shards covering 491+ unique languages and scripts).
• Quality Filtering: Strictly validated using LASER Confidence Scores ($\ge 1.06$) and Bicleaner Filtering ($\ge 0.60$) alongside length-ratio and Unicode Mojibake cleaning.
• Round-Robin 500-Language Spooling: Every on-disk chunk round-robins across 500 distinct language shards sequentially, mixing all world languages into globally shuffled, balanced 8k sequences.
• Distributed Compute: 2× NVIDIA Tesla T4 GPUs with PyTorch Fully Sharded Data Parallel (FSDP), FP16 mixed precision, and Layer-Wise Reentrant Activation Checkpointing.
• Zero-Host RAM Footprint: Streaming disk-spooler architecture keeping host CPU memory strictly below < 1.0 GB throughout training.

---

Evaluation & Benchmarks

(Benchmark results on the 500-Language Universal Validation Suite, COMET-22, BLEU, and ChrF++ will be published here upon completion of the pre-training run.)

Benchmark	Metric	Mythos-2.04B-MoE	Baseline
Fixed 500-Language Suite	Token Accuracy	TBD	—
COMET-22	Neural Semantic Score	TBD	—
SacreBLEU	Corpus BLEU	TBD	—
ChrF++	Character F-score	TBD	—

---

Citation

If you use Mythos-2.04B-MoE or its architecture in your research or applications, please cite:

@misc{mythos2026multilingual,
  author       = {Adithyan AI},
  title        = {Mythos-2.04B-MoE: A Sparse Mixture-of-Experts Large Language Model for Universal Multilingual Translation},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/AdithyanAI/Mythos_translation}}
}