Model: ggunio/B2NL-IntelligentTokenizer-v6.2.1
Author: ggunio
Tags: pytorch, safetensors, b2nl_tokenizer, tokenizer, byte-to-nl, compression, multilingual, embeddings, semantic-boundaries, ko, en, zh, ja, dataset:flores200, license:apache-2.0, model-index, region:us
Card metadata: language: ko, en, zh, ja, multilingual
license: apache-2.0
tags: tokenizer, byte-to-nl, compression, multilingual, pytorch, embeddings, semantic-boundaries
datasets: flores200
metrics: compression_ratio
model-index: name: B2NL-IntelligentTokenizer-v6.2.1
results: task: type: text-compression
name: Text Compression
metrics: type: compression_ratio
value: 16
name: Fixed Compression Ratio
verified: no, type: accuracy
value: 95
name: Korean Reconstruction (single chunk)
verified: no, type: accuracy
value: 92.3
name: English Reconstruction
verified: no
Likes: 0 | All-time downloads: 117
Created: 2025-10-06 16:21:42