Kanberta / added_tokens.json
anonymouse03052002's picture
Upload tokenizer
7c8157d
{
"</s>": 64001,
"<2as>": 64003,
"<2bn>": 64004,
"<2en>": 64005,
"<2gu>": 64006,
"<2hi>": 64007,
"<2kn>": 64008,
"<2ml>": 64009,
"<2mr>": 64010,
"<2or>": 64011,
"<2pa>": 64012,
"<2shuf>": 64002,
"<2ta>": 64013,
"<2te>": 64014,
"<s>": 64000
}