MadTokenizer-250K

Byte-level BPE tokenizer for English + Russian, trained from scratch on wikitext-103, multi_news and the Russian news corpus gazeta. Vocabulary size: 250,000 tokens.

About

• Type: byte-level BPE (same recipe as GPT-2)
• Vocab size: 250,000
• Languages: English, Russian
• Corpora: wikitext-103, multi_news, gazeta (ru)
• Special tokens: <s>, <pad>, </s>, <unk>, <mask>
• Use case: tokenizer-only model (no language-model weights yet)

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("MadSensiro/MadTokenizer-250K")

for s in ("Привет, мир! Как дела сегодня?",
          "Hello, world! How are you today?"):
    ids = tok(s).input_ids
    print(tok.convert_ids_to_tokens(ids))

The tokenizer is byte-level, so it handles arbitrary text (unicode, spaces, emojis) without producing <unk>, because every byte sequence is decodable.

Files

config.json              # model config + metadata
tokenizer.json           # serialized fast tokenizer (loaded by HF)
tokenizer_config.json    # tokenizer options
special_tokens_map.json  # special token mapping
vocab.json               # token -> id
merges.txt               # BPE merge rules

Training

Trained with the Hugging Face tokenizers library (BpeTrainer, ByteLevel pre-tokenizer, min_frequency=2).

Note

BPE uses a leading whitespace marker (Ġ) for beginning-of-word tokens, so decode of a standalone string may prepend a space — this is standard GPT-2 behavior and expected.