Inconsistent number of vocab from pretrained T5Tokenizer and T5ForConditionalGeneration
Author: cstorm125
Pull request: no

# ❓ Questions & Help
Pretrained `T5Tokenizer ` has vocab size of 32100 (32000 tokens plus 100 extra_ids) but the shared embedding layer of `T5ForConditionalGeneration` has size of (32128, 768). I checked the google-research implementation of T5 and also found that they have vocab size of 32100 also.

Where did the extra 28 embeddings come from and how can we map it to the tokenizer?

## To reproduce

```
from transformers import (
    T5Tokenizer, 
    T5ForConditionalGeneration,
)

tokenizer_pretrained = T5Tokenizer.from_pretrained('t5-base')
model_pretrained = T5ForConditionalGeneration.from_pretrained('t5-base')
len(tokenizer_pretrained.get_vocab()), model_pretrained.state_dict()['shared.weight'].shape
```

Output:
```
(32100, torch.Size([32128, 768]))
```


Comments:
Hey @cstorm125, 

I think, those `28` leftover embeddings are simply not used. The reason why the embedding matrix is of length 32128 as far as I know is simply because 32128 is a more GPU  friendly number `32128 = 128 * 251` than `32100 = 4 * 8025`. That means that the GPU is probably more efficient if it can directly deal with a power of 2 shape. 

Also see: https://www.quora.com/Why-should-I-choose-a-mini-batch-size-of-32-64-128-256-etc-i-e-a-power-of-two-and-not-a-size-of-50-100-500-1000-Is-there-any-benefit-of-choosing-power-of-two-mini-batch-sizes



Hi all, I ran into this too. But I did find a bug as a result of this mismatch. I try to resize the embedding to be smaller and got a Cuda assert error.  See bug report. 

https://github.com/huggingface/transformers/issues/8643


I found this mismatch recently and I think this may result in many bugs. Wish someone can fix it.

> Hey @cstorm125,
> 
> I think, those `28` leftover embeddings are simply not used. The reason why the embedding matrix is of length 32128 as far as I know is simply because 32128 is a more GPU friendly number `32128 = 128 * 251` than `32100 = 4 * 8025`. That means that the GPU is probably more efficient if it can directly deal with a power of 2 shape.
> 
> Also see: https://www.quora.com/Why-should-I-choose-a-mini-batch-size-of-32-64-128-256-etc-i-e-a-power-of-two-and-not-a-size-of-50-100-500-1000-Is-there-any-benefit-of-choosing-power-of-two-mini-batch-sizes

This is wrong. It shouldn't be this way. In case model predicts wong index and when you calculate loss, it will cause serious issues. Its hard to believe no one cares this. 

Hey @s4sarath, 

During training all input_ids and labels are defined by the tokenizer. If the tokenizer has a vocab_size of 32000 there is no way that it will tokenize to an id >= 32000 neither for `input_ids` nor for `labels`. Because no label ever has an id >= 32000 the model learns to never predict those ids. I don't really see a problem with this to be honest

Hi Patrick,

Thanks for the reply.
If the embedding matrix is 32128 x d , for an example if the predicted id
is say 32099, if we are using Sentencepiece tokenizer ( not huggingface ),
it will fail to decode that.

And special tokens ( 100 tokens ) are added extra, right. Which are
actually not a part of official sentecepice model. That's why I told, it
shouldn't be that way.

Thanks anyway, I really appreciate your reply.:-)

On Fri, 10 Dec, 2021, 7:11 pm Patrick von Platen, ***@***.***>
wrote:

> Hey @s4sarath <https://github.com/s4sarath>,
>
> During training all input_ids and labels are defined by the tokenizer. If
> the tokenizer has a vocab_size of 32000 there is no way that it will
> tokenize to an id >= 32000 neither for input_ids nor for labels. Because
> no label ever has an id >= 32000 the model learns to never predict those
> ids. I don't really see a problem with this to be honest
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/4875#issuecomment-990983491>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACRE6KAZNTAF4DX5ZXOHUH3UQH7P5ANCNFSM4NZOPKUQ>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>


Upvoting this. Another subtle bug this causes is when doing prompt tuning. The common way to do it is to call `add_tokens` to add some special prompt tokens, and also create a special embedding class that consists of two embedding matrics, the original one + one for the prompt tokens, and the forward call simply indexes into the two matrices concatenated together. Then all parameters but the prompt token embedding matrix are frozen. The expected behavior is that the IDs of the added tokens correspond to the prompt token embeddings when concatenated with the original. However, this mismatch causes the tokenizer to assign IDs starting from 32100, which are still a part of the original embedding matrix, which doesn't get gradients.

Temporary Solution : `model.resize_token_embeddings(len(tokenizer))`


I just found that it sometimes generates > 32100 input ids in generate function. Especially that happens if I evaluate a fine-tuned model in the very early step while training. Thanks, @Darshan2104 ! model.resize_token_embeddings(len(tokenizer))  temporally resolves my issue.

I am also facing the `IndexError: index out of range in self` issue due to this difference between the vocab size in the t5 tokenizer and the model for ConditionalGeneration. Should I resize model token_embeddings?

> model.resize_token_embeddings(len(tokenizer)

I tried this but not helping.

> > model.resize_token_embeddings(len(tokenizer)
> 
> I tried this but not helping.

@kanak8278 , could you double-check that you are using the right tokenizer for the model?

For the model, could you show me what happens when you run this code?
```python
{n:p.shape for n, p in model.named_parameters() if "embedding" in n}
```

For the tokenizer, could you do `len(tokenizer)` and report what it says?

And then could you do this on your input ids? `torch.tensor(input_ids).max()`

This is a bit troubling, especially because I'm only interested in using a model for inference. I'm generating some sequences using multinomial sampling from `pythia-70M` model. When I attempt to obtain the corresponding scores to this sequence, I obtain a CUDA assertion (which, when running in the CPU, reveals itself as an indexing error). Upon checking the size of the model and the tokenizer, I find these are different, and although I understand @patrickvonplaten's justification, I am not sure how to proceed in terms of replacing these tokens, the fact is that they are being selected during the random sampling (even though they shouldn't since they were learned...). The other troubling problem of having a model head greater than the vocab size is that, by definition, these tokens will still contain some probability mass.

@PastelBelem8 

The model was never incentivized to predict those tokens so the weights for the tokens with ids > len(tokenizer) will have extraordinarily low scores. I did a quick test and the scores for those extra tokens summed together to be on the order of 1e-30 for each token. That is basically 0. 

Could you share your sampling approach?

Never mind, it was an error on my end! I apologize for the confusion! I thought I had tried everything and was desperate. 

I have the same problem (continuous pre-training of mT5 with Flax) with this code: https://github.com/huggingface/transformers/blob/main/examples/flax/language-modeling/run_t5_mlm_flax.py and I get this error: 
ValueError: Trying to load the pretrained weight for ('lm_head', 'kernel') failed: checkpoint has shape (768, 250112) which is incompatible with the model shape (768, 250200)

I tried this: model.resize_token_embeddings(len(tokenizer))

But I still get the same error, how can I fix this? 

@Anmalutgen , change the vocab size in the config.json

> [@Anmalutgen](https://github.com/Anmalutgen) , change the vocab size in the config.json

Thank you so much, it works now!