50M-Class Comparative Architecture Benchmark (Amharic)
This repository contains trained checkpoints, benchmark logs, and evaluation reports for 4 distinct ~50M parameter language model architectures trained from scratch on the Amharic Wikipedia corpus using an RL-trained subword tokenizer (3,919 tokens).
📊 Models & Architectures
| Architecture | Model Class | Parameters | Distinct Features |
|---|---|---|---|
| Qwen3.5 (Transformer) | TransformerLM |
50.79M | 3:1 Gated DeltaNet / Gated Attention, Zero-Centered RMSNorm, Partial RoPE (0.25), Multi-Token Prediction (MTP) Head |
| HRM-Text | HRMLM |
49.02M | Dual-timescale H2L3 Recurrence, MagicNorm parameterless RMSNorm, PrefixLM masking, Warmup TBPTT ($2 \rightarrow 5$) |
| Mamba SSM | MambaLM |
49.47M | Selective State Space Model, dt_rank=32, S4D $A_{\text{log}}$, specialized log-uniform $\Delta$ initialization |
| Hybrid Mamba-Transformer | HybridMambaTransformerLM |
52.04M | Interleaved 2:1 Mamba to Qwen3.5 Attention, Unified SwiGLU FFN, Idempotent Depth-Scaling |
📁 Repository Structure
├── models/ # PyTorch architecture implementations
│ ├── transformer_lm.py # Qwen3.5 3:1 DeltaNet + Attention + MTP
│ ├── hrm_lm.py # HRM-Text H2L3 recurrence
│ ├── mamba_lm.py # Mamba selective SSM
│ └── hybrid_lm.py # Hybrid Mamba + Qwen attention
├── tokenizer/ # Amharic subword tokenizer
│ ├── vocab.txt
│ └── config.json
├── checkpoints/ # Best model weights for each architecture
│ ├── transformer/best_model.pt
│ ├── hrm/best_model.pt
│ ├── mamba/best_model.pt
│ └── hybrid/best_model.pt
└── analysis/ # Comparative benchmark results & plots
├── report.md
├── results_table.tex
├── loss_curves.png
├── throughput_scaling.png
└── pareto_frontier.png
🚀 How to Load and Use
import torch
from models import create_model
# Load model architecture
model = create_model("transformer", vocab_size=3919)
# Load checkpoint
checkpoint = torch.load("checkpoints/transformer/best_model.pt", map_location="cpu")
model.load_state_dict(checkpoint["model_state"], strict=False)
model.eval()
# Generate tokens
tokens = torch.tensor([[2, 45, 128, 902]], dtype=torch.long)
with torch.no_grad():
logits, _ = model(tokens)
next_token = torch.argmax(logits[:, -1, :], dim=-1)
print("Next token ID:", next_token.item())
📜 Citation & Attribution
If you use these models or comparative benchmarks in your research, please cite:
- Qwen3.5: Alibaba Qwen Team (2025/2026)
- HRM-Text: Sapient Intelligence (Wang et al., 2026)
- Mamba: Gu & Dao (2023)