Hugging Face model: vaibhavmaurya/hindi-slm-tokenizer-v001
Author: vaibhavmaurya
Created: 2026-05-18 08:08:23
Last modified: 2026-05-18 08:53:36
Pipeline: text-generation
Library: tokenizers
Likes: 0 | Downloads: 0
Tags: tokenizers, tokenizer, hindi, devanagari, unigram, sentencepiece, subword, nlp, indic, slm, text-generation, hi, en, dataset:ai4bharat/sangraha, arxiv:1804.10959, arxiv:1808.06226, license:apache-2.0, region:us
Card metadata:
language: hi, en
license: apache-2.0
tags: tokenizer, hindi, devanagari, unigram, sentencepiece, subword, nlp, indic, slm
datasets: ai4bharat/sangraha
metrics: unk_rate
pipeline_tag: text-generation
library_name: tokenizers