gpt2-small-italian-embeddings / tokenizer_config.json
wietsedv's picture
add model files
b4c7c40
{
"unk_token": "<|endoftext|>",
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"add_prefix_space": false
}