tokenizer_max_length not set

#1
by AngledLuffa - opened

Thanks for providing this model!

There is a slight issue with the tokenizer config, though: tokenizer.model_max_length is not set correctly, so pre-filtered sentences to be <= the maximum length of the tokenizer does not work.

Sign up or log in to comment