Hugging Face Space: anveshplus/BPE-Tokenizer
Author: anveshplus
SDK: streamlit
title: BPE Tokenizer
tokens length: 5000000
by paired tokens length (Tokens): 4817
compression ratio: 3.48X
license: mit
short_description: Tailored for Telugu, considering its unique script.