Mismatch b/w tokenizer and model embedding. What to use?
#12
by
dexter89kp
- opened
the tokenizer has a vocab size of 32004, but the token embedding of the model only has dimension of 32000. What should be used? Should we be editing the tokenizer?
Same question, it was working fine before though