Issue if you try to resize position embeddings
#26
by
njbrake
- opened
During my finetuning, I had loaded the tokenizer with a pad_token set, and then thought I had to resize the model position embeddings using
model.resize_token_embeddings(len(tokenizer))
However, since the model vocab_size is larger than the tokenizer vocab size, doing so actually shrinks the model vocab size, which I'm guessing contributed to why my training flopped. Should an error or something be thrown to prevent us from trying to resize/shrink the model vocab_size?