Hugging Face model: konko/bornomala-bengali-tokenizer
Author: konko
Created: 2026-07-23 14:08:13
Last modified: 2026-08-08 20:18:55
Pipeline: token-classification
Library: tokenizers
Likes: 0 | Downloads: 0
Tags: tokenizers, bengali, bangla, tokenizer, tokenization, grapheme, indic, unicode, bornomala, token-classification, bn, en, dataset:wikimedia/wikipedia, dataset:wikimedia/wikisource, dataset:ai4bharat/sangraha, dataset:csebuetnlp/xlsum, license:apache-2.0, region:us
Card metadata:
license: apache-2.0
language: bn, en
library_name: tokenizers
pipeline_tag: token-classification
tags: bengali, bangla, tokenizer, tokenization, grapheme, indic, unicode, bornomala
datasets: wikimedia/wikipedia, wikimedia/wikisource, ai4bharat/sangraha, csebuetnlp/xlsum
metrics: name: Fertility (tokens per word, held-out Bengali Wikipedia)
type: fertility
value: 1.524, name: Conjunct fragmentation
type: fragmentation
value: 0.0001