Mihretu-dev/ethiopic-llm-tokenization-blindspot

Lived-Experience AI Capability Evaluation: Ethiopic (Ge'ez) Morpho-Syntactic Fragmentation & Transliteration Blind Spot

Evaluator / Author: Mihretu Hizkel Shuke
Target Program: Fatima Fellowship Fall 2026 Technical Challenge
Evaluated Architecture: Qwen/Qwen2.5-1.5B-Instruct (Open-weight 1.5B parameter frontier compact LLM)

---

1. The Blind Spot: Lived Experience vs. Standard Benchmarks

Standard multilingual evaluations (such as translated MMLU, Flores-200, or XNLI) evaluate models on formal, syntactically sanitized, and standalone sentences. However, living and working in Ethiopia exposes a severe, neglected failure mode in everyday human-AI communication:

1. Subword & Byte-Fallback Explosion: Modern BPE and WordPiece tokenizers allocate negligible vocabulary capacity to Ethiopic (Ge'ez) script. As a result, standard Amharic words undergo catastrophic byte-level fragmentation, expanding a single semantic word into 4 to 8 tokens. This artificially consumes the context window, multiplies inference latency, and degrades cross-layer attention cohesion.
2. Fidel-Latin Code-Switching & Morphological Compounding: Everyday communication frequently switches between formal Ge'ez script and informal Latin transliteration ("Fidel-Latin" phonetics) within the same dialogue. Furthermore, Amharic is heavily agglutinative and root-pattern driven (prefixes and suffixes encode prepositions, pronouns, negation, and tense simultaneously). Frontier compact models (0.6B–6B) consistently fail to parse these bound morphemes, producing nonsensical hallucinated suffixes or reverting to English responses.

---

2. Systematic Evaluation Setup & Artifacts

Methodology

We evaluated Qwen/Qwen2.5-1.5B-Instruct across three parallel linguistic representations covering 20 curated daily tasks (factual QA, instructional steps, arithmetic reasoning, and polite discourse):

• Set A (Pure Ethiopic/Ge'ez Script): Native script reflecting natural Ethiopian business and educational requests.
• Set B (Latin Transliterated Amharic): Informal phonetic Romanization used in SMS and messaging.
• Set C (English Equivalent): Ground-truth semantic control.

Key Empirical Findings

• Token Inflation Ratio: Ethiopic script averaged 4.6 tokens per word, compared to 1.1 tokens per word for English equivalents.
• Semantic Bleed & Syntactic Breakdown: While the model preserved basic instruction-following in English, native Ge'ez prompts frequently triggered grammatical degeneration, repetitive sub-character loops, or abrupt switches to simplified English.
• Transliteration Gap: On Latin-transliterated Amharic, the model exhibited near-complete semantic failure, interpreting phonetically transliterated phrases as obscure Romance or Germanic cognates rather than Semitic syntax.

---

3. Proposed Path Forward

To close this capability divide without the prohibitive cost of training foundation models from scratch, we propose:

1. Morphologically-Aware Tokenization Adaptation: Expand the compact model's embedding matrix with a domain-specialized Ethiopic subword vocabulary (approx. 8,000–12,000 tokens) pre-trained on root-pattern morphological segmentations rather than naive byte-pair frequency.
2. Parallel Fidel-Latin Bimodal Distillation: Curate a synthetically augmented parallel alignment corpus mapping raw Ge'ez script directly to common Latin transliteration phonetics, fine-tuning the model using Parameter-Efficient Fine-Tuning (LoRA / QLoRA) on attention projection layers.
3. Bandwidth-Aware Edge Quantization: Quantize adapted checkpoints to 4-bit AWQ/GGUF representations to permit low-latency local inference on consumer-grade mobile devices across regional connectivity constraints.

---

Evaluation Artifacts in This Repository

• evaluation_prompts.json: The test suite of parallel Ethiopic, Transliterated, and English prompt pairs.
• eval_script.py: The evaluation harness for measuring token length dilation, latency, and response consistency.