paddle-fnet-base / tokenizer_config.json
HJHGJGHHG's picture
Upload tokenizer_config.json
5c02270
{"do_lower_case": false,
"remove_space": true,
"keep_accents": true,
"unk_token": "<unk>",
"sep_token": "[SEP]",
"pad_token": "<pad>",
"cls_token": "[CLS]",
"mask_token": "[MASK]",
"sp_model_kwargs": {},
"model_max_length": 512,
"tokenizer_class": "FNetTokenizer"}