tokenizer_max_length not set
#1
by
AngledLuffa
- opened
Thanks for providing this model!
There is a slight issue with the tokenizer config, though: tokenizer.model_max_length
is not set correctly, so pre-filtered sentences to be <= the maximum length of the tokenizer does not work.