Hugging Face Space: sagargurujula/telugu-tokenizer
Author: sagargurujula
SDK: docker
title: Telugu Tokenizer App
description: A tokenizer app for tokenizing Telugu text. It uses BPE (Byte Pair Encoding) to tokenize Telugu text. 5k is the vocab size.
tags: telugu
tokenizer
NLP
transformers
license: apache-2.0
model: telugu-tokenizer-model
datasets: telugu-dataset
isPrivate: no