KPTPona: Toki Pona Language Model
KPTPona is a lightweight PyTorch LSTM model trained for text generation in Toki Pona using data cleaned from Toki Pona Wikipedia (wikipesija).
Model Description
- Developed by: aroderg
- Model Type: Char/Word-level Recurrent Neural Network (LSTM)
- Language: Toki Pona (
tok) - License: MIT
Architecture & Hyperparameters
| Parameter | Value |
|---|---|
| Embedding Dim | 128 |
| Hidden Dim | 256 |
| LSTM Layers | 1 |
| Max Sequence Length | 32 tokens |
| Vocabulary | Character/word regex filter (min_freq=20) |
Input Tokens ──> Embedding (128) ──> LSTM (256) ──> Linear Layer ──> Logits
Quickstart
import torch
from tokenizer import TokiPonaTokenizer
from model import KPTPona
from generate import generate_text
from safetensors.torch import load_file
# Load tokenizer & model
tokenizer = TokiPonaTokenizer()
tokenizer.load_vocab('vocab.json')
model = KPTPona(vocab_size=len(tokenizer.word2idx), embedding_dim=128, hidden_dim=256)
model.load_state_dict(load_file("model.safetensors", device="cpu"))
# Generate text
prompt = "jan li"
output = generate_text(prompt, model, tokenizer, max_new_tokens=20, temperature=0.7, top_p=0.85)
print(f"Result: {output}")
Dataset Details
Source: Cleaned text from Toki Pona Wikipedia (wikipesija).
Preprocessing: Standardized regex filtering with a minimum frequency threshold (min_freq=20).
Target Context Length: 32 tokens.
Training Setup
Optimizer: Adam (lr=0.001, CrossEntropyLoss ignoring )
Batch Size / Epochs: 24 / 6 epochs
Optimizations: AMP (torch.amp), TF32 support, and torch.compile() enabled.
Evaluation
Model quality is evaluated via Perplexity (PPL) on dataset sequences:
PPL < 15: Excellent grammar prediction
15–35: Good / Moderate uncertainty
35: Under-trained or noisy data
Validation Results
Final Training PPL: 10.97
Validation PPL: 12.54
Sample Outputs
Prompt: jan
Generated: jan li wile sona e ni : ona li wile sona e ijo lon
Prompt: ma
Generated: ma li ma lon ma elopa
Intended Use & Limitations
Direct Use
Lightweight text generation experiments in Toki Pona.
Educational demonstrations of small LSTM language architectures.
Limitations
Context Length: Constrained to short sequences (32 tokens max context).
Out-of-Vocabulary: Low-frequency words (< 20 occurrences) are filtered out.
Size: Very small parameter size, intended for basic syntax modeling rather than complex reasoning.
Repository Structure
data_prep.py: Wikipedia XML parsing & JSONL dataset prep
tokenizer.py: TokiPonaTokenizer and PyTorch TokiPonaDataset
model.py: KPTPona PyTorch LSTM model architecture
train.py: Training script with Automatic Mixed Precision (AMP)
generate.py: Inference script (Temperature, Top-K, Top-P, Repetition Penalty)
Citation & License
License: MIT License
Citation: If you use this model or code in your work, please cite this repository:
@misc{kptpona2024,
author = {aroderg},
title = {KPTPona: Toki Pona Language Model},
year = {2026},
publisher = {Hugging Face},
journal = {Hugging Face Model Hub},
howpublished = {\url{[https://huggingface.co/aroderg/KPT-pona](https://huggingface.co/aroderg/KPT-pona)}}
}