JYYong
/

korean_tokenizer

Model card Files Files and versions Community

korean_tokenizer / special_tokens_map.json

JYYong's picture

add tokenizer

8b07eeb over 1 year ago

raw history blame contribute delete

No virus

173 Bytes

	{
	"bos_token": "[BOS]",
	"cls_token": "[CLS]",
	"eos_token": "[EOS]",
	"mask_token": "[MASK]",
	"pad_token": "[PAD]",
	"sep_token": "[SEP]",
	"unk_token": "[UNK]"
	}