README.md

metadata

library_name: transformers
license: mit
datasets:
  - JeanKaddour/minipile
language:
  - en

BEE-spoke-data/MiniTokenizer-20480

This is a ByteLevelBPETokenizer trained on the JeanKaddour/minipile dataset with the aim to create a compact English-only tokenizer.

Usage

load with AutoTokenizer, i.e.:

from transformers import AutoTokenizer
tk = AutoTokenizer.from_pretrained('BEE-spoke-data/MiniTokenizer-20480')
tk