bnaz12/kambaata-bpe-tokenizer

Kambaata (ktb) Byte-Level BPE Tokenizer

A 12,000-token byte-level BPE tokenizer for Kambaata (also spelled Kambatta, ISO 639-3 ktb), a Cushitic language spoken in southern Ethiopia. It is trained from scratch on cleaned Kambaata text and is intended as a starting point for language models and NLP tools for Kambaata.

Quick start

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("bnaz12/kambaata-bpe-tokenizer")
enc = tok("Wonaa Maganu samaahaa uullataa qoocceei.")
print(enc["input_ids"])
print(tok.convert_ids_to_tokens(enc["input_ids"]))
print(tok.decode(enc["input_ids"], skip_special_tokens=True))

With the tokenizers library only:

from tokenizers import Tokenizer
t = Tokenizer.from_pretrained("bnaz12/kambaata-bpe-tokenizer")
print(t.encode("Wonaa Maganu samaahaa").tokens)

Tokenizer specification

Property	Value
Algorithm	Byte-level BPE (Hugging Face tokenizers)
Vocabulary size	12,000
Normalization	NFKC
Pre-tokenization	Custom regex split, then byte-level mapping (no prefix space)
Special tokens	<pad>=0, <unk>=1, <s>=2, </s>=3
Post-processing	<s> $A </s> (pairs: <s> $A </s> $B </s>)
Decoder	ByteLevel
Max length	512 (set in config; the tokenizer itself does not limit length)

Because the model is byte-level, any input can be encoded without <unk>.

How it was built

1. Corpus cleaning. Starting from 69,837 raw lines, I:

   • applied NFKC normalization;
   • unified curly quotes (’ ‘ ′) to ' and (“ ” „) to ", and en/em dashes to -;
   • collapsed dot leaders (....) to ... and removed rules made of _, -, =, *;
   • dropped lines containing Ethiopic or Greek script (other languages and noise);
   • dropped lines shorter than 2 words;
   • removed exact duplicate lines.

2. Split. After shuffling with seed 42, 2% was held out for evaluation: 54,919 training lines and 1,120 validation lines. Duplicates were removed before splitting, so no line appears in both sets.

3. Pre-tokenization. The default byte-level regex is built for English and splits at apostrophes (it treats 'm, 's as contractions). Kambaata uses the apostrophe inside words (for example xa'mmees), so a custom regex keeps word-internal apostrophes inside the word and isolates every other symbol:

    ?[\p{L}\p{M}]+(?:'[\p{L}\p{M}]+)*| ?\p{N}| ?[^\s\p{L}\p{M}\p{N}]|\s+(?!\S)|\s+
   

   Each match is then mapped to bytes with the standard byte-level alphabet.

4. Training. BPE with a target vocabulary of 12,000 on the training split only. The size was chosen after an earlier 16k run left about a third of the vocabulary rarely used.

5. Checks. Special-token IDs, batching, padding, truncation, exact round-trip on held-out text, vocabulary hygiene (no punctuation-only merges), and comparison against xlm-roberta-base.

Training data

Property	Value
Source	educational texts,religious book and some personal texts
Training lines	54,919
Words (training)	593,435
Unique words (lowercased)	76,968
Characters	5,217,623

Character set after cleaning (112 characters): !"#%'()*+,-./0123456789:;=>?ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]_abcdefghijklmnopqrstuvwxyz|¡«»×áóĀāħĩĪī̳‖•‹›→↔`

The corpus is not distributed with this repository.

Evaluation

Measured on the 1,120 held-out lines (not used in training).

Metric	This tokenizer	xlm-roberta-base
Tokens per word	1.57	3.24
Characters per token	5.51
Exact round-trip	1120/1120

Segmentation of the 5,875 unique validation words:

Tokens per word	Share
1	42.4%
2	41.1%
3-4	15.6%
5+	0.9%

Example splits (validation words):

Word	Tokens
woluagiin	wolu + agiin
bareenin	bareen + in
abbata	abbata
heechchaachchissaa	heechch + aachch + issaa
gooni	gooni
gaitaau	gaitaau
caakkissaau	caakkissaau
bahiru	bahiru

Tokens per word is counted with a simple letter-and-apostrophe word regex.

Intended use

• Pre-training or fine-tuning language models for Kambaata
• Text statistics, search, and other NLP pipelines for Kambaatissa
• Comparing tokenization efficiency across Ethiopian languages
• to degitalize kembaatissa language and make it suitable for NLP tasks

Limitations

• Domain bias. The training text is mostly religious and educational text-style writing. Everyday speech, news, social media, and technical text will tokenize less efficiently. Please test on your own domain.
• Small corpus. About 0.6M words. Rare words are often split at arbitrary points and not at morpheme boundaries; the splits are statistical, not linguistic.
• Orthography. The tokenizer follows the spelling conventions of the training text. Other spelling systems or dialect variants may be split poorly.
• Normalization. NFKC changes some compatibility characters. Decoding returns the normalized text, not necessarily the exact original characters.
• Not linguistically validated. The segmentations have not been reviewed by a full linguistic study.
• Changing the vocabulary breaks models trained with an earlier version. Pin a revision if you depend on it.

Citation

@misc{kambaata_bpe_tokenizer,
  author = BINIYAM SOLOMON ,
  title  = {Kambaata Byte-Level BPE Tokenizer},
  year   = {2026},
  url    = {https://huggingface.co/bnaz12/kambaata-bpe-tokenizer}
}

Contact

PHONE:0960258919 EMAIL:biniyamsolomon1994@gmail.com