Hugging Face model: dawncr0w/KorByte-128K
Author: dawncr0w
Created: 2026-08-04 05:54:26
Last modified: 2026-08-06 03:13:23
Library: tokenizers
Likes: 0 | Downloads: 0
Tags: tokenizers, tokenizer, korean, byte-level-bpe, lossless, ko, en, dataset:HuggingFaceFW/fineweb-2, dataset:wikimedia/wikipedia, dataset:eduagarcia/multilingual_tokenizer_benchmark, dataset:HAERAE-HUB/KMMLU, license:apache-2.0, region:us
Card metadata:
language: ko, en
license: apache-2.0
library_name: tokenizers
datasets: HuggingFaceFW/fineweb-2, wikimedia/wikipedia, eduagarcia/multilingual_tokenizer_benchmark, HAERAE-HUB/KMMLU
tags: tokenizer, korean, byte-level-bpe, lossless