TorchMoji / tokenizer_config.json
Pendrokar's picture
Create tokenizer_config.json
374b54c verified
raw
history blame
156 Bytes
{"do_lower_case": true, "unk_token": "CUSTOM_UNKNOWN", "sep_token": "CUSTOM_BREAK", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "CUSTOM_MASK"}