SaulLu
/

dummy-tokenizer-bytelevel-bpe

Model card Files Files and versions Community

dummy-tokenizer-bytelevel-bpe / tokenizer_config.json

SaulLu's picture

add tokenizer

2ba79ba almost 2 years ago

raw history blame contribute delete

No virus

46 Bytes

{"tokenizer_class": "PreTrainedTokenizerFast"}