Hugging Face model: theschoolofai/BrahmicTokenizer-131K
Author: theschoolofai
Created: 2026-05-19 10:08:05
Last modified: 2026-05-29 14:38:43
Pipeline: feature-extraction
Library: transformers
Likes: 5 | Downloads: 0
Tags: transformers, tokenizer, byte-level-bpe, brahmic, indic, multilingual, hindi, bengali, tamil, telugu, kannada, malayalam, marathi, gujarati, punjabi, odia, assamese, feature-extraction, en, hi, bn, ta, te, kn, ml, mr, gu, pa, or, as, arxiv:2605.29379, license:apache-2.0, endpoints_compatible, region:us
Card metadata:
language: en, hi, bn, ta, te, kn, ml, mr, gu, pa, or, as
library_name: transformers
license: apache-2.0
pipeline_tag: feature-extraction
tags: tokenizer, byte-level-bpe, brahmic, indic, multilingual, hindi, bengali, tamil, telugu, kannada, malayalam, marathi, gujarati, punjabi, odia, assamese