Model: ggunio/intelligent-tokenizer-v6
Author: ggunio
Tags: pytorch, intelligent_tokenizer, tokenization, byte-level, neural-tokenizer, pattern-learning, vocabulary-free, multilingual, ko, en, zh, ja, es, fr, de, ru, ar, hi, dataset:flores200, license:mit, model-index, region:us
Card metadata: language: multilingual, ko, en, zh, ja, es, fr, de, ru, ar, hi
tags: tokenization, byte-level, neural-tokenizer, pattern-learning, vocabulary-free
license: mit
datasets: flores200
metrics: accuracy
model-index: name: intelligent-tokenizer-v6
results: task: type: token-classification
name: Language Pattern Learning
dataset: name: flores200
type: flores200
metrics: type: accuracy
value: 0.623
name: Character Accuracy (Major Languages)
verified: no, type: accuracy
value: 0.472
name: Character Accuracy (Minor Languages)
verified: no
Likes: 0 | All-time downloads: 88
Created: 2025-09-13 19:09:28