Models
Datasets
Spaces
Docs
Pricing
Log In
Sign Up

SaulLu
/

bengali-tokenizer

Model card Files Files and versions Community

bengali-tokenizer

History: 4 commits

SaulLu's picture

set normalized to true 524ccfa about 1 year ago

.gitattributes 690 Bytes initial commit about 1 year ago
special_tokens_map.json 321 Bytes set normalized to true about 1 year ago
tokenizer.json 1.72 MB set normalized to true about 1 year ago
tokenizer_config.json 467 Bytes tokenizer v2- include normalization discussed with Bengali community about 1 year ago