tokenizers
Collection
trained and adapted tokenizers - various
•
14 items
•
Updated
BEE-spoke-data/MiniTokenizer-20480
This is a ByteLevelBPETokenizer trained on the JeanKaddour/minipile dataset with the aim to create a compact English-only tokenizer.
Usage
load with AutoTokenizer, i.e.:
from transformers import AutoTokenizer
tk = AutoTokenizer.from_pretrained('BEE-spoke-data/MiniTokenizer-20480')
tk
Inference Providers
NEW
This model is not currently available via any of the supported third-party Inference Providers, and
HF Inference API was unable to determine this model’s pipeline type.