Hugging Face model: ggunio/intelligent-tokenizer-v6
Author: ggunio
Created: 2025-09-13 19:09:28
Last modified: 2025-09-14 15:57:27
Likes: 0 | Downloads: 6
Tags: pytorch, intelligent_tokenizer, tokenization, byte-level, neural-tokenizer, pattern-learning, vocabulary-free, multilingual, ko, en, zh, ja, es, fr, de, ru, ar, hi, dataset:flores200, license:mit, model-index, region:us
Card metadata:
language: multilingual, ko, en, zh, ja, es, fr, de, ru, ar, hi
tags: tokenization, byte-level, neural-tokenizer, pattern-learning, vocabulary-free
license: mit
datasets: flores200
metrics: accuracy