dummy-tokenizer-bytelevel-bpe / tokenizer_config.json
SaulLu's picture
add tokenizer
2ba79ba
raw
history blame contribute delete
No virus
46 Bytes
{"tokenizer_class": "PreTrainedTokenizerFast"}