Update to tokenizers > 0.13

#4
by gregoriopellegrino - opened

Hi,
thanks for this great work. Unfortunately tokenizers 0.12.1 cannot be installed on mac with M1 or M2 chips. Would it be possible to update tokenizers dependencies to at least version 0.13?

Thank you,
Gregory

Sign up or log in to comment