Nebulixlabs/Horizon-10M-Base

🌌 Horizon-10M

A Tiny, Modern, From-Scratch Causal Language Model

Built with 🖤 by Nebulixlabs

---

🧠 Model Overview

Horizon-10M-Base is a compact, modern-architecture causal language model trained completely from scratch on a 500M-token mixture of four open datasets — fineweb-edu, smollm-corpus (Cosmopedia v2), finemath (4+), and FineWeb-HQ. It uses a GPT-style decoder stack with RoPE positional embeddings, RMSNorm, and a SwiGLU MLP — the same core design patterns used in modern LLMs like LLaMA, just scaled way down (~10M params) for fast experimentation, education, and low-resource environments.

---

🏗️ Architecture Details

Component	Value
🔢 Parameters	~10.06M
📚 Vocabulary size	8,192
📐 Hidden size (d_model)	320
🧱 Layers	6
👁️ Attention heads	5 (head_dim = 64)
🔁 MLP hidden size	864
🪟 Context length	128 tokens
🎯 Positional encoding	Rotary (RoPE)
⚖️ Normalization	RMSNorm (pre-norm)
🚀 MLP type	SwiGLU (gated, SiLU activation)
🔗 Weight tying	Input embedding ↔ LM head
🎲 Dropout	0.0

Attention: Multi-head self-attention with RoPE, using PyTorch's fused scaled_dot_product_attention (causal masking).

MLP block: SwiGLU gated feed-forward (w2(silu(w1(x)) * w3(x))), same family used in LLaMA/Mistral.

---

📦 Training Data Mixture

Horizon-10M-Base is trained on a 500,000,000-token blend of four sources, weighted toward general educational web text with smaller slices of synthetic textbook-style content and math-focused text for broader coverage.

Source	Config	Tokens	Share	Composition
🌐 HuggingFaceFW/fineweb-edu	sample-10BT	440,000,000	88.0%	██████████████████████████░░
📘 HuggingFaceTB/smollm-corpus	cosmopedia-v2	25,000,000	5.0%	██░░░░░░░░░░░░░░░░░░░░░░░░░░░
➗ HuggingFaceTB/finemath	finemath-4plus	30,000,000	6.0%	██░░░░░░░░░░░░░░░░░░░░░░░░░░░
⭐ epfml/FineWeb-HQ	—	5,000,000	1.0%	█░░░░░░░░░░░░░░░░░░░░░░░░░░░░
🪙 Total		500,000,000	100%

• fineweb-edu — broad, high-quality educational web text; the backbone of the mix.
• smollm-corpus (Cosmopedia v2) — synthetic textbook, blog, and story-style content for stylistic and topical diversity.
• finemath (4+) — the high-quality math-filtered subset, for light exposure to numerical/mathematical text.
• FineWeb-HQ — a small high-quality web slice used as an additional quality signal.

All four sources were pre-tokenized with the model's own custom Byte-Level BPE tokenizer and concatenated into a single flat uint16 memmap binary before training, so no source is up- or down-weighted beyond its raw token share above.

---

📊 Training Details

Setting	Value
📦 Datasets	4 sources — see Training Data Mixture
🔤 Tokenizer	Custom Byte-Level BPE, vocab size 8,192 (trained from scratch on fineweb-edu)
🪙 Total training tokens	~500,000,000
📏 Sequence length	128
🧮 Batch size	512 sequences (65,536 tokens/step)
📈 Peak LR	6e-4 (cosine decay → 6e-5)
🔥 Warmup steps	200
⚙️ Optimizer	AdamW (fused), betas=(0.9, 0.95), weight decay 0.1
✂️ Grad clipping	1.0
🖥️ Hardware	2× NVIDIA T4 (Kaggle)
⏱️ Precision	Mixed precision (fp16 autocast + GradScaler)
🧵 Data pipeline	Pre-tokenized to a flat uint16 memmap binary for near-zero dataloading overhead

The tokenizer is a custom-trained Byte-Level BPE (not reused from any existing model), trained on ~40k documents sampled from fineweb-edu, with special tokens <pad>, <bos>, <eos>, <unk>.

---

💻 How to Use

from transformers import AutoTokenizer, AutoModelForCausalLM
repo_id = "Nebulixlabs/Horizon-10M-Base"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
output = model.generate_text(
    tokenizer,
    prompt="The history of artificial intelligence",
    max_new_tokens=60,
    temperature=0.75,
    top_k=40,
    top_p=0.9,
    repetition_penalty=1.18
)
print(output)

Generation tips:

• Keep prompts short — the model only has a 128-token context window.
• Lower temperature (0.6–0.8) for more stable, less erratic output.
• repetition_penalty around 1.1–1.2 helps reduce looping on such a small model.

---

🧪 Sample Raw Output

Unedited generation from the base model, prompt "The history of artificial intelligence":

The history of artificial intelligence and machine learning, and how it was
used to produce a new knowledge.

The information on machine learning is based on the concept of AI that is
not inherently sensitive and is not explicitly applicable in scientific
studies. It is generally not possible for people who have developed the
algorithm or...

This is characteristic of a ~10M-parameter base model: locally fluent phrasing, but it drifts off-topic and loses coherence beyond a sentence or two, with no factual grounding. See Limitations & Biases.

---

🎯 Intended Use

• ✅ Educational purposes — understanding how modern decoder-only LLMs (RoPE, RMSNorm, SwiGLU) are built and trained end-to-end.
• ✅ Fast local experimentation on limited hardware (CPU-friendly inference).
• ✅ Baseline/reference model for architecture or training-pipeline experiments.

🚫 Out-of-Scope Use

• ❌ Any production, factual, medical, legal, or safety-critical use.
• ❌ Long-form or knowledge-intensive generation — 10M params / 128-token context can't support this.
• ❌ Use as a source of factual information — the model will hallucinate.

---

⚠️ Limitations & Biases

• Trained on a mixture dominated (88%) by fineweb-edu, with smaller slices of synthetic and math-focused text; inherits whatever biases and gaps exist in those web-scraped and synthetic corpora.
• Small vocabulary (8,192 tokens) and short context (128 tokens) significantly limit fluency, coherence, and factual grounding.
• No RLHF / instruction-tuning / safety-alignment has been applied — this is a raw base model, not a chat/assistant model.
• Outputs can be repetitive, ungrammatical, or nonsensical, especially beyond a few sentences, as shown in the sample output above.

---

📜 Citation

@misc{horizon10m2026,
  title  = {Horizon-10M-Base: A Small From-Scratch Causal Language Model},
  author = {Nebulixlabs},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Nebulixlabs/Horizon-10M-Base}}
}

---

🔗 Connect with Nebulixlabs

Made with 🖤 by Nebulixlabs