Emeritus-21/yoruba-codeswitch-diacritics-long-context

Yorùbá-English Code-Switching Diacritic Restoration (Long-Context)

This model restores missing tone marks and underdots (diacritics) in Yorùbá text within code-switched Yorùbá-English contexts. It is explicitly designed and optimized to handle long-context sequences up to 2048 tokens, enabling the accurate restoration of full paragraphs, documents, and extended conversational text without the truncation or coherence loss typical of standard 512-token models.

📈 Training Dynamics

Figure 1: Phase 3 training and validation loss curves over 5,398 steps. The model achieved a best validation loss of 0.03448 at step 5398, demonstrating stable convergence and successful adaptation to the 2048-token context window without catastrophic forgetting.

---

📋 Model Details

• Model Architecture: ByT5-small (Byte-level T5)
• Base Model: google/byt5-small
• Total Parameters: ~300 million
• Maximum Context Window: 2048 tokens
• Supported Languages: Yorùbá (primary), English (code-switching support)
• Primary Task: Sequence-to-sequence automatic diacritic restoration (ADR)
• License: MIT

---

🏗️ Training Methodology

Progressive Context Expansion (Curriculum Fine-Tuning)

To enable the stable training of a 2048-token context window on consumer-grade hardware (NVIDIA RTX 3070 Laptop, 8GB VRAM), this model was trained using a rigorous 3-Phase Curriculum Fine-Tuning strategy. This methodology prevents catastrophic forgetting of foundational orthographic rules while gradually adapting the model's attention mechanisms to longer, more complex dependencies.

Phase	Dataset Size	Context Length	Primary Objective
Phase 1: Foundation	~700,000 samples	256 tokens	Learn core Yorùbá orthographic rules, tone marks, and subdots from high-frequency short sentences. Establish baseline character-level mapping.
Phase 2: Bridging	~70,000 samples	512 tokens	Adapt to medium-length paragraphs. Learn cross-sentence grammatical dependencies and maintain diacritic consistency across clause boundaries.
Phase 3: Long-Context	~60,000 samples	2048 tokens	Master document-level coherence, long-range dependencies, and complex code-switching contexts while retaining strict orthographic precision.

Phase 3 Training Configuration

The final phase was trained under the following strict hyperparameter configuration to ensure maximum stability and performance:

• Hardware: NVIDIA RTX 3070 Laptop GPU (8GB VRAM)
• Precision: Mixed Precision (BF16/FP16) for memory efficiency
• Batch Size: 1 (with gradient accumulation to simulate larger effective batch sizes)
• Learning Rate: 2e-5
• Learning Rate Scheduler: Linear decay with warmup
• Warmup Steps: 500
• Optimizer: AdamW
• Weight Decay: 0.01
• Maximum Training Steps: 5,398 steps (equivalent to 2 full epochs over the 60,000 sample dataset)
• Evaluation Strategy: Evaluated every 500 steps
• Save Strategy: Checkpoint saved every 1,000 steps
• Best Checkpoint Selected: Step 5,398 (Validation Loss: 0.03448)
• Total Training Time: ~11.8 hours
• Total Floating Point Operations (FLOPs): 1.41 × 10¹⁷

Strict Dataset Filtering Protocol

A critical methodological requirement for long-context evaluation is preventing silent truncation. If a model is trained on a maximum of 2048 tokens, evaluating it on sequences longer than 2048 tokens will result in the tokenizer silently dropping the end of the sequence, artificially inflating error rates (CER/WER/DER) due to forced deletion errors.

To ensure scientific rigor and accurate metric reporting, the Phase 3 test set was strictly filtered based on true token length:

• Original Unfiltered Test Set: 6,391 samples
• Filtered Test Set (≤ 2048 tokens): 5,388 samples (84.3% coverage)
• Excluded Samples: 1,003 samples (15.7%) that exceeded the 2048-token limit after tokenization.

All reported evaluation metrics below are computed exclusively on this filtered 5,388-sample test set to guarantee that the model was evaluated fairly within its designed operational limits.

---

📊 Evaluation Results

Evaluation Protocol

All evaluations were conducted on the filtered test set (5,388 samples ≤ 2048 tokens). The following orthogonal metrics were computed to provide a comprehensive view of model performance:

• CER (Character Error Rate): Levenshtein edit distance at the character level. Measures overall typographical accuracy.
• WER (Word Error Rate): Levenshtein edit distance at the word level. Measures usability and grammatical integrity.
• DER (Diacritic Error Rate): Custom metric measuring the error rate specifically on characters that require diacritics (tone marks and underdots), ignoring plain ASCII characters.
• WDER (Word Diacritic Error Rate): The percentage of diacritic-containing words in the reference that have ≥1 diacritic error in the prediction.
• chrF (Character n-gram F-score): Measures character-level n-gram overlap. Highly sensitive to diacritic restoration quality, as it rewards partial matches better than exact word matching.
• EM (Exact Match): The percentage of sequences where the prediction matches the reference 100% identically.

Custom Long-Context Evaluation (5,388 samples)

Decoding Strategy	Samples	CER ↓	WER ↓	DER ↓	WDER ↓	chrF ↑	EM ↑
Greedy Search	5,388	1.69%	4.30%	8.47%	5.01%	95.33	2.38%
Beam Search (4)	5,388	[Pending]	[Pending]	[Pending]	[Pending]	[Pending]	[Pending]

Note: Beam Search (num_beams=4) evaluation is currently in progress. Final metrics will be updated in this table upon completion.

---

🚀 Usage

Method 1: Simple Transformers Pipeline

from transformers import pipeline

# Load the model pipeline
diacritizer = pipeline(
    "text2text-generation", 
    model="Emeritus-21/yoruba-codeswitch-diacritics-long-context",
    device=0  # Set to 0 for GPU acceleration, -1 for CPU
)

# Input code-switched text without diacritics
text = "mo ri oko ni ilu Eko because I was driving yesterday"

# Generate restored text (Beam search recommended for maximum accuracy)
result = diacritizer(
    text, 
    max_new_tokens=2048, 
    num_beams=4,  
    early_stopping=True
)

print(result[0]['generated_text'])
# Expected output: "mo rí okò ní ìlú Èkó because I was driving yesterday"