Climb-Ai/Climb-1-Toki

Climb 1 Toki (Micro-LLaMA for Toki Pona)

Climb 1 Toki is an ultra-compact language model (2.85M tied parameters / 3.05M un-tied GGUF tensors) engineered natively for Toki Pona by BBTeam (Climb AI).

For pre-quantized GGUF binaries ready for llama.cpp and Open WebUI, visit Climb-Ai/Climb-1-Toki-GGUF.

---

Research Hypothesis & Experimental Findings

Research Question

Is training a causal neural language model fundamentally faster, more parameter-efficient, and sample-compact when built upon the world's most minimal and isolating language (Toki Pona, ~120–137 root words, strictly regular syntax, zero inflectional morphology) compared to natural languages like English or Russian?

1. The Breakthrough: Radical Parameter Efficiency

• 30–50x Vocabulary Compression: Standard BPE tokenizers for natural languages require 32,000–64,000 tokens. At standard hidden dimensions, an embedding matrix alone consumes 50–100+ MB of memory, rendering sub-15M parameter models practically impossible. In Toki Pona, the complete language—including punctuation and bracketed foreign names [ ]—fits within 1,024 Byte-Level BPE tokens. The resulting embedding table occupies just 196k parameters (~0.7 MB).
• Instant Syntax Convergence: Because Toki Pona has no grammatical conjugations, cases, or irregular forms (jan li moku e telo), the network stabilized valid sentence structures within just 3.26 MB of training text (~1M tokens).

2. The Bottleneck: The Distillation Limit & Data Ceiling

• Combinatorial Polysemy: Because Toki Pona uses a tiny root inventory, complex ideas rely on compound metaphors (telo seli = tea/coffee, telo nasa = alcohol, ilo suno = screen/flashlight, ilo lukin = microscope/glasses). Disentangling subtle contextual meanings from identical roots proved significantly harder for a 2.85M model than learning distinct dedicated tokens.
• The Hard Ceiling Without Distillation: The entire global corpus of written Toki Pona literature across the web totals approximately 3.5–4 MB (the entire Wikipesija is ~3 MB). This experiment demonstrates the fundamental limit of tiny language models trained from scratch: without knowledge distillation from frontier teacher models, training SLMs from scratch on niche or synthetic domains inevitably caps their world knowledge.

---

Model Architecture

• Parameters: 2,853,312 (~2.85M unique tied parameters; 3,049,920 tensors in GGUF format)
• Architecture: Micro-LLaMA (Rotary Embeddings, SwiGLU, RMSNorm, Weight Tying)
• Layers: 6
• Hidden Dimension: 192
• Attention Heads: 6 ($d_{head} = 32$)
• MLP Dimension: 512
• Context Length: 256 tokens
• Vocabulary: 1,024 Byte-Level BPE tokens (with native Toki Pona cartouche brackets [ ])

---

Training Procedure

1. Stage 1 — Pre-training (Causal LM):
   • Pretrained on 3.26 MB (~1,006,000 tokens) of continuous Toki Pona literature:
     • Climb-Ai/toki-pona-tiny-dialogues
     • NetherQuartz/wikipesija (2,901 Wikipedia articles)
     • finnnnnnnnnnnn/toki-pona-sentences (19,094 sentences from lipu tenpo)
2. Stage 2 — SFT & Multi-turn Dialogue Alignment:
   • Multi-turn conversational chains with history loss masking.
   • Identity grounding (Climb 1 Toki, BBTeam (Climb AI)), grammatical calibration, sentence capitalization, and strict punctuation spacing.

---

Python Inference (PyTorch)

import torch
from tokenizers import Tokenizer
from modeling_micro_llama import MicroLlamaForCausalLM

tokenizer = Tokenizer.from_file("tokenizer.json")
model = MicroLlamaForCausalLM(vocab_size=1024, dim=192, n_head=6, n_layer=6, hidden_dim=512, max_len=256)
state = torch.load("pytorch_model.bin", map_location="cpu", weights_only=True)
model.load_state_dict(state)
model.eval()

prompt = "<|user|> toki! sina jan seme? <|bot|>"
input_ids = tokenizer.encode(prompt).ids
generated = list(input_ids)

with torch.no_grad():
    for _ in range(40):
        inp = torch.tensor([generated[-256:]], dtype=torch.long)
        logits, _ = model(inp)
        next_tok = torch.argmax(logits[0, -1]).item()
        if next_tok == 0:  # <|endoftext|>
            break
        generated.append(next_tok)

reply = tokenizer.decode(generated[len(input_ids):])
print("Bot:", reply.strip())

---

License

Apache 2.0. Developed by BBTeam (Climb AI).