BEE-spoke-data
/

MiniTokenizer-20480

Inference Endpoints

Model card Files Files and versions Community

Edit model card

BEE-spoke-data/MiniTokenizer-20480

This is a ByteLevelBPETokenizer trained on the JeanKaddour/minipile dataset with the aim to create a compact English-only tokenizer.

Usage

load with AutoTokenizer, i.e.:

from transformers import AutoTokenizer
tk = AutoTokenizer.from_pretrained('BEE-spoke-data/MiniTokenizer-20480')
tk

Downloads last month: -; Downloads are not tracked for this model. How to track

Unable to determine this model’s pipeline type. Check the docs .

Dataset used to train BEE-spoke-data/MiniTokenizer-20480