SimpleStories BPE 2048 Tokenizer
This is a 2,048-token byte-level BPE tokenizer trained on the complete training split of
SimpleStories/SimpleStories.
Training details
- Dataset revision:
e63b8adc3b1a1bdc7cac5b500d150b71346b0628 - Training examples: 2,115,696 stories
- Text column:
story - Pre-tokenizer and decoder: byte level, without an added prefix space
- Vocabulary size: 2,048, including three special tokens
- Model maximum length: 1,024 tokens
- Normalization: none; capitalization and whitespace are preserved
- UTF-8 input is represented without unknown-token fallbacks
| Token | ID |
|---|---|
<unk> |
0 |
| `< | start_story |
| `< | end_story |
<unk> is also used for padding. This matches the effective special-token
mapping exposed by AutoTokenizer.from_pretrained("C10X/tenny").
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("C10X/simplestories-bpe-2048")
encoding = tokenizer("Once upon a time, a little fox found a blue key.")
print(encoding.tokens())
plain = tokenizer(
"Once upon a time, a little fox found a blue key.",
add_special_tokens=False,
)
print(tokenizer.decode(plain.input_ids))
The tokenizer works with AutoTokenizer, batched padding and truncation,
PyTorch tensor output, and Hugging Face language-modeling data collators.
add_special_tokens=True prepends <|start_story|>, matching C10X/tenny.
The end-of-story token is registered but is not appended automatically;
training pipelines can append it according to their sequence-packing policy.
The package is tested with Transformers 4.31 / Tokenizers 0.13.3 and with Transformers 5.16 / Tokenizers 0.23.1.
Plain vocabulary exports are included as bpe_2048.txt and basic_vocab.txt.