christopher
/

arxiv-pyserini-tokenizer

Model card Files Files and versions Community

arxiv-pyserini-tokenizer / tokenizer_config.json

cakiki's picture

Upload tokenizer

37dae64 over 1 year ago

raw history blame contribute delete

No virus

129 Bytes

	{
	"model_max_length": 1000000000000000019884624838656,
	"tokenizer_class": "PreTrainedTokenizerFast",
	"vocab_size": 70000
	}