santacoder / tokenizer_config.json
RaymondLi's picture
add special tokens
1354bdd
raw
history blame
102 Bytes
{
"errors": "replace",
"tokenizer_class": "PreTrainedTokenizerFast",
"model_max_length": 2048
}