Hugging Face model: ggunio/B2NL-IntelligentTokenizer-v6.2.1
Author: ggunio
Created: 2025-10-06 16:21:42
Last modified: 2025-10-07 12:25:56
Likes: 0 | Downloads: 16
Tags: pytorch, safetensors, b2nl_tokenizer, tokenizer, byte-to-nl, compression, multilingual, embeddings, semantic-boundaries, ko, en, zh, ja, dataset:flores200, license:apache-2.0, model-index, region:us
Card metadata:
language: ko, en, zh, ja, multilingual
license: apache-2.0
tags: tokenizer, byte-to-nl, compression, multilingual, pytorch, embeddings, semantic-boundaries
datasets: flores200
metrics: compression_ratio