Hugging Face Space: xu-song/tokenizer-arena
Author: xu-song
SDK: gradio
title: Tokenizer Arena
datasets: cc100
tags: tokenizer
short_description: Compare different tokenizers in char-level and byte-level.