load error in transformers?

#2
by bdambrosio - opened

ValueError: Wrong index found for [PAD]: should be 32001 but found 32000.

I'll admit I didn't go far in trying to debug this, always find token dictionary size problems confusing.
Do I need to add a pad token to tokenizer or something?

Please try downloading again. I just fixed tokenizer.json

Sign up or log in to comment