Hugging Face model: vivekvar/indictok-256k
Author: vivekvar
Created: 2026-07-25 12:25:21
Last modified: 2026-07-25 12:25:38
Pipeline: feature-extraction
Library: transformers
Likes: 0 | Downloads: 0
Tags: transformers, tokenizer, indic, multilingual, bpe, feature-extraction, hi, bn, ta, te, kn, ml, mr, gu, pa, or, as, ur, ne, sa, sat, mni, arxiv:2310.08754, license:apache-2.0, endpoints_compatible, region:us
Card metadata:
license: apache-2.0
library_name: transformers
language: hi, bn, ta, te, kn, ml, mr, gu, pa, or, as, ur, ne, sa, sat, mni
tags: tokenizer, indic, multilingual, bpe
pipeline_tag: feature-extraction