malbert-base-cased-64k / tokenizer_config.json
cservan's picture
Updated tokenizer config
7e72a15
{"keep_accents": true, "do_lower_case": false, "model_max_length": 512, "tokenizer_class": "AlbertTokenizer"}