SaulLu
/

bengali-tokenizer

Model card Files Files and versions Community

bengali-tokenizer / special_tokens_map.json

Commit History

set normalized to true

524ccfa

SaulLu commited on May 11, 2021

tokenizer v2- include normalization discussed with Bengali community

9ff1d40

SaulLu commited on May 11, 2021

add tokenizer

f472ef8

SaulLu commited on May 7, 2021