Toivo0/turkish-news-tokenizer

Turkish Tokenizer (BPE)

A byte-level BPE tokenizer for Turkish, trained with the Hugging Face tokenizers library and compatible with transformers.AutoTokenizer.

Note on the name: this repo is named turkish-news-tokenizer, but the current version (v1) was trained only on Turkish Wikipedia text, not news data. News data may be added in a future version — see Training Data below.

Model Details

• Type: Byte-level BPE (Byte-Pair Encoding)
• Vocabulary size: 30,000
• Special tokens: <unk>, <pad>, <bos>, <eos>
• Framework: built with tokenizers, wrapped as a PreTrainedTokenizerFast for transformers compatibility
• Pre-tokenization: byte-level (no prefix space added)

This repo contains only the tokenizer (vocabulary + merges). It is not paired with any pretrained model weights — it can be used to prepare data for training a new model from scratch, or wherever a standalone Turkish tokenizer is needed.

Training Data

The tokenizer was trained on ~20,000 sentence-level lines extracted from Turkish Wikipedia (wikimedia/wikipedia, dump 20231101.tr, streamed via 🤗 Datasets).

Processing steps:

1. Wikipedia articles were streamed and their text normalized (whitespace collapsed).
2. Each article was split into sentences using a simple punctuation-based rule (splitting after ., !, or ?).
3. Sentences shorter than 30 characters were discarded.
4. The first 20,000 resulting lines were kept as the training corpus.

Because the source is Wikipedia, the text is largely encyclopedic / formal register Turkish. See Limitations below.

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Toivo0/turkish-news-tokenizer")

text = "Merhaba! Bu küçük bir tokenizer örneğidir."
ids = tokenizer.encode(text)

print("Tokens:", tokenizer.convert_ids_to_tokens(ids))
print("Token IDs:", ids)
print("Decoded:", tokenizer.decode(ids))

Intended Use

This is a general-purpose, community-facing Turkish tokenizer. It's intended for anyone who needs a compact (30k vocab) Turkish BPE tokenizer for experimentation, coursework, or as a starting point for training a Turkish language model — not tied to any specific downstream model.

Limitations and Bias

• Domain: trained solely on Wikipedia, so it's optimized for formal/encyclopedic Turkish. It may under-tokenize (split into more subwords than ideal) informal text, social media language, dialectal variation, slang, or domain-specific jargon (e.g. legal, medical, colloquial news headlines).
• Coverage: 20,000 lines is a relatively small training corpus for a 30k vocabulary; rare words and proper nouns outside Wikipedia's coverage may be split into many subword pieces.
• Not a model: this repo provides tokenization only — no language understanding or generation capability comes with it.

License

This tokenizer is released under CC BY-SA 4.0, matching the license of its Turkish Wikipedia training data. If you use this tokenizer, please retain attribution and share derivative tokenizers/datasets under a compatible license.

Citation

If you use this tokenizer, please consider citing this repository and the source dataset:

@misc{turkish-news-tokenizer,
  title  = {Turkish Tokenizer (BPE)},
  author = {Toivo0},
  year   = {2026},
  url    = {https://huggingface.co/Toivo0/turkish-news-tokenizer}
}

Training data: Wikimedia Wikipedia dataset, Turkish dump 20231101.tr.