Costikooo-32K

Costikooo-32K is a byte-level BPE tokenizer trained from scratch for a trilingual English, Spanish, and Romanian language model.

Summary

Property	Value
Vocabulary	32,768 tokens
Training text	1,000,000,000 characters
English share	55%
Spanish share	25%
Romanian share	20%
Normalization	Unicode NFC
Unknown tokens in validation	0

Romanian is intentionally upsampled for tokenizer training relative to a typical model-pretraining mixture. This reduces excessive fragmentation of Romanian words and diacritics.

Special tokens

ID	Token
0	`<
1	`<
2	`<
3	`<
4	`<
5	`<
6	`<
7	`<

The tokenizer does not automatically add BOS or EOS to raw pretraining text. The included chat template uses the role tokens and <|end|> explicitly.

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Pacific-i64/Costikooo-32K")

encoded = tokenizer("Știința, la ciencia, and science.")
print(encoded.input_ids)
print(tokenizer.decode(encoded.input_ids))

Chat formatting:

messages = [
    {"role": "system", "content": "Answer clearly."},
    {"role": "user", "content": "Explică pe scurt fotosinteza."},
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

Corpus and provenance

The tokenizer vocabulary was learned from streamed, deterministic samples of:

• English: HuggingFaceFW/fineweb-edu, sample-10BT;
• Spanish: HuggingFaceFW/fineweb-2, spa_Latn;
• Romanian: HuggingFaceFW/fineweb-2, ron_Latn.

manifest.json records exact source revisions, character/document counts, recipe hashes, artifact hashes, runtime versions, and validation results. No training documents are redistributed in this repository.

The source datasets are made available under ODC-By 1.0 and remain subject to their dataset cards and Common Crawl terms.

Validation

The release passed exact encode/decode round-trip tests for English, Spanish, and Romanian, including Romanian and Spanish diacritics. All 256 byte values are represented through the byte-level alphabet.