Hugging Face model: mokataria/hindi-hinglish-bpe-tokenizer-v0.1-experimental
Author: mokataria
Created: 2026-06-15 16:33:29
Last modified: 2026-06-15 16:57:44
Pipeline: feature-extraction
Library: tokenizers
Likes: 0 | Downloads: 0
Tags: tokenizers, tokenizer, bpe, hindi, hinglish, romanized-hindi, code-mixed, mixed-script, byte-fallback, indic, experimental, research, feature-extraction, hi, en, dataset:ai4bharat/Aksharantar, license:other, region:us
Card metadata:
language: hi, en
library_name: tokenizers
license: other
license_name: research-and-non-commercial-experimental-release
tags: tokenizer, bpe, hindi, hinglish, romanized-hindi, code-mixed, mixed-script, byte-fallback, indic, experimental, research
datasets: ai4bharat/Aksharantar
pipeline_tag: feature-extraction