SOMA-S1: A Somali Language Model

SOMA-S1 is a Somali-first foundation language model (LLM) built to advance natural language processing for Somali. It is a decoder-only, autoregressive Transformer with a sparse Mixture-of-Experts (MoE) feed-forward architecture and Multi-head Latent Attention (MLA). The model was pretrained from scratch on a dedicated Somali-language corpus rather than adapted from an existing pretrained checkpoint.

Somali remains underrepresented in modern AI research and large-scale language-model development. SOMA-S1 is an experimental effort to establish foundational modeling capabilities for the language through a dedicated tokenizer, a Somali-focused training corpus, and a custom neural architecture.

• Model: SOMA-S1
• Developer: Boore
• Type: Base foundation model (pretrained, not instruction-tuned)
• Architecture: Sparse MoE Transformer with Multi-head Latent Attention
• Primary language: Somali
• Status: Research preview

This is a base model, not a conversational assistant. It is trained to continue text rather than follow instructions or answer questions. Generation quality is currently limited, with repetition often emerging after approximately 10–20 generated tokens under greedy decoding. See Generation Settings.

---

Model overview

SOMA-S1 uses a sparse Mixture-of-Experts (MoE) Transformer architecture with Multi-head Latent Attention (MLA). Its attention mechanism employs a compressed latent key-value representation, while its feed-forward architecture uses routed experts, activating a subset of the available expert parameters for each token.

The model contains approximately 126.19 million total parameters and 69.57 million active parameters per token, based on the configured top-2 expert routing.

Each Transformer layer contains 18 routed experts and one shared expert. Two routed experts are selected per token, while the shared expert is also used.

The active parameter count includes the shared model parameters and the selected routed experts. The routed expert parameters alone account for approximately 7.08 million active parameters across the 12 layers.

Technical architecture

Component	Configuration
Model type	Decoder-only Transformer
Training approach	Pretraining from scratch
Total parameters	126.19 million
Active parameters per token	69.57 million
Active routed expert parameters	7.08 million
Hidden dimension	768
Transformer layers	12
Attention mechanism	Multi-head Latent Attention (MLA)
Attention heads	12
KV latent rank	128
Q latent rank	128
QK no-PE head dimension	32
QK RoPE head dimension	32
V head dimension	64
Feed-forward architecture	Mixture of Experts (MoE)
Routed experts	18 per layer
Shared experts	1 per layer
Active routed experts	2 per token
MoE expert intermediate size	128
Vocabulary size	60,000
Tokenizer	Custom byte-level BPE
Maximum context length	2,048 tokens
Normalization	RMSNorm
Activation	SiLU
Positional encoding	RoPE (θ = 10,000)
Tied embeddings	Yes

Parameter-counting note: Active parameters are calculated by counting all non-routed model parameters, including shared experts, together with the two selected routed experts in each of the 12 layers. The 7.08M figure represents only the selected routed expert weights, not the entire active model.

---

Training Data and Limitations

SOMA-S1 was pretrained from scratch using the Somali NLP Corpus, developed by SOMAST. The dataset brings together Somali-language text from multiple publicly available sources to support natural language processing and language-model research.

While this corpus provides a valuable foundation for Somali language modeling, its coverage, linguistic diversity, data quality, and representation of different types of Somali text require further improvement.

SOMA-S1 currently has limitations in text coherence, factual accuracy, repetitive generation, and long-form text generation. Further development will require broader and higher-quality Somali datasets, more comprehensive data filtering, and additional training and evaluation.

SOMA-S1 should therefore be considered an early-stage research model rather than a mature, general-purpose Somali language model.

Training Methodology

The model was trained using a custom data-processing and distributed training pipeline.

• Tokenizer: Custom byte-level Byte Pair Encoding (BPE) tokenizer with a 60,000-token vocabulary.
• Data preparation: Sequence packing to improve training efficiency and reduce padding overhead.
• Sequence length: 2,048 tokens per training sequence.
• Precision: BF16 mixed-precision training.
• Hardware: Distributed training across 2 × NVIDIA GPUs using Distributed Data Parallel (DDP).
• Optimization: Gradient accumulation and cosine learning-rate scheduling.

This pipeline was designed to support efficient pretraining and make effective use of available GPU resources.

---

Files

File	Size	Notes
model.safetensors	504,852,656 B	FP32 weights
config.json	1.2 KB	model_type: deepseek_v2
generation_config.json	234 B	EOS = 3, BOS = 2, PAD = 1
tokenizer.json	4.3 MB	60,000-token byte-level BPE
tokenizer_config.json	320 B	Special tokens and content-only chat template
LICENSE	11 KB	Apache-2.0

---

Quickstart — Transformers

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "boore1/SOMA-S1"

tok = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.float32
)

prompt = "Soomaaliya waa dal"

ids = tok(prompt, return_tensors="pt").input_ids

out = model.generate(
    ids,
    max_new_tokens=120,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
    top_k=50,
    repetition_penalty=1.2,
    pad_token_id=tok.pad_token_id,
)

print(
    tok.decode(
        out[0][ids.shape[1]:],
        skip_special_tokens=True
    )
)

Generation Settings

The bundled chat template is a content-only passthrough. It concatenates message content without role markers, meaning the model behaves as a raw text-completion model rather than a conversational assistant.

For best results, provide a natural Somali text prefix that the model can continue.

Example:

Soomaaliya waa dal

Recommended generation parameters:

• repetition_penalty: 1.2 or higher.
• temperature: 0.7–0.8.
• top_p: 0.9–0.95.

Greedy decoding can produce repetitive sequences. Sampling and repetition penalties may help reduce repetition, but they do not eliminate the model's underlying limitations.

Intended Use

• Research on Somali language modeling.
• A starting point for Somali instruction tuning.
• Tokenizer and architecture experiments for low-resource languages.
• Experimental research into sparse MoE and MLA architectures.

Limitations

• No instruction tuning: The model was not trained to follow instructions or function as a conversational assistant. Direct questions may produce empty responses or irrelevant continuations.
• Repetitive generation: Longer continuations can develop repetitive patterns, particularly under greedy decoding.
• Limited factual knowledge: The model may generate inaccurate or fabricated information about Somalia, history, geography, and other subjects. It should not be treated as a reliable factual source.
• Limited long-form coherence: The model may struggle to maintain context, consistency, and logical flow across longer passages.
• Corpus bias: The training corpus includes publicly available Somali text from different sources and domains. Its linguistic coverage, orthographic consistency, and representation of different registers remain limitations.
• Preliminary evaluation: Comprehensive benchmarks and systematic evaluations are still needed.

Future Development

Future work includes Somali instruction tuning, improved long-context coherence, reduced repetition and hallucination, expanded evaluation, and downstream research in conversational AI, summarization, question answering, and Somali information retrieval.

License

Apache License 2.0 — see LICENSE.

Citation

@misc{soma-s1,
  title  = {SOMA-S1: A Somali Language Model},
  author = {Boore},
  year   = {2026},
  note   = {Sparse Mixture-of-Experts Transformer with Multi-head Latent Attention, pretrained from scratch on Somali text},
  url    = {https://huggingface.co/boore1/SOMA-S1}
}