Token embedding resizing does not work for TFGPT2Model
Author: visionscaper
Pull request: no

### System Info

- `transformers` version: 4.25.1
- Platform: Linux-5.15.0-57-generic-x86_64-with-glibc2.35
- Python version: 3.9.16
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.11.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@gante and @Rocketknight1

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

After `add_special_tokens` to tokenizer and `resize_token_embeddings` on `TFGPT2Model`, evaluating the model results in an error that indicates that the embeddings are not resized as expected.

Please see the example code and the execution output below: 

```
from transformers import GPT2Tokenizer, TFGPT2Model

SPECIAL_TOKENS_MAPPING = {
    'bos_token': '<bos>',
    'eos_token': '<eos>',
    'pad_token': '<pad>',
    'additional_special_tokens': ['<speaker1>', '<speaker2>']
}

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = TFGPT2Model.from_pretrained("gpt2")

print("Evaluating TFGPT2Model BEFORE extending the tokenizer and model with additional tokens ...")

inputs = tokenizer("Hello, my dog is cute", return_tensors="tf")
print(f"inputs = \n{inputs}\n")

outputs = model(inputs)
print(f"DONE!")

print("Adding tokens...")
orig_num_tokens = len(tokenizer.get_vocab())
num_special_tokens = tokenizer.add_special_tokens(SPECIAL_TOKENS_MAPPING)
print(f"orig_num_tokens = {orig_num_tokens}, num_special_tokens={num_special_tokens}")

model.resize_token_embeddings(new_num_tokens=orig_num_tokens + num_special_tokens)

print("Evaluating TFGPT2Model AFTER extending the tokenizer and model with additional tokens ...")

inputs = tokenizer("<speaker1>Hello, my dog is cute<speaker2>I agree!", return_tensors="tf")
print(f"inputs = \n{inputs}\n")

outputs = model(inputs)
print(f"DONE!")
```

```
Evaluating TFGPT2Model BEFORE extending the tokenizer and model with additional tokens ...
inputs = 
{'input_ids': <tf.Tensor: shape=(1, 6), dtype=int32, numpy=array([[15496,    11,   616,  3290,   318, 13779]], dtype=int32)>, 'attention_mask': <tf.Tensor: shape=(1, 6), dtype=int32, numpy=array([[1, 1, 1, 1, 1, 1]], dtype=int32)>}

DONE!

Adding tokens...
orig_num_tokens = 50257, num_special_tokens=5

Evaluating TFGPT2Model AFTER extending the tokenizer and model with additional tokens ...
inputs = 
{'input_ids': <tf.Tensor: shape=(1, 11), dtype=int32, numpy=
array([[50260, 15496,    11,   616,  3290,   318, 13779, 50261,    40,
         4236,     0]], dtype=int32)>, 'attention_mask': <tf.Tensor: shape=(1, 11), dtype=int32, numpy=array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], dtype=int32)>}

Traceback (most recent call last):
  File "/home/freddy/workspace/Nuhame/mlpug/examples/chatbot/tensorflow/test_tf_resize_token_size.py", line 33, in <module>
    outputs = model(inputs)
  File "/home/freddy/.virtualenvs/mlpug-tf/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/freddy/.virtualenvs/mlpug-tf/lib/python3.9/site-packages/transformers/modeling_tf_utils.py", line 432, in run_call_with_unpacked_inputs
    return func(self, **unpacked_inputs)
  File "/home/freddy/.virtualenvs/mlpug-tf/lib/python3.9/site-packages/transformers/models/gpt2/modeling_tf_gpt2.py", line 773, in call
    outputs = self.transformer(
  File "/home/freddy/.virtualenvs/mlpug-tf/lib/python3.9/site-packages/transformers/modeling_tf_utils.py", line 432, in run_call_with_unpacked_inputs
    return func(self, **unpacked_inputs)
  File "/home/freddy/.virtualenvs/mlpug-tf/lib/python3.9/site-packages/transformers/models/gpt2/modeling_tf_gpt2.py", line 447, in call
    tf.debugging.assert_less(
tensorflow.python.framework.errors_impl.InvalidArgumentError: Exception encountered when calling layer 'transformer' (type TFGPT2MainLayer).

input_ids must be smaller than the embedding layer's input dimension (got 50261 >= 50257)
Condition x < y did not hold.
First 3 elements of x:
[50260 15496    11]
First 1 elements of y:
[50257]

Call arguments received by layer 'transformer' (type TFGPT2MainLayer):
  • input_ids=tf.Tensor(shape=(1, 11), dtype=int32)
  • past_key_values=None
  • attention_mask=tf.Tensor(shape=(1, 11), dtype=int32)
  • token_type_ids=None
  • position_ids=None
  • head_mask=None
  • inputs_embeds=None
  • encoder_hidden_states=None
  • encoder_attention_mask=None
  • use_cache=True
  • output_attentions=False
  • output_hidden_states=False
  • return_dict=True
  • training=False
```

### Expected behavior

The model should have 50257 + 5 = 50262 embeddings after resizing and thus an input ID with value 50261 should not result in any errors. The above code should run without errors.

Comments:
@visionscaper thank you for raising the issue! It is a generalized problem with this check, which should only rely on the config's vocab size (which is the only reliable source of the actual vocabulary size at any given moment).

@susnato opened a fix for GPT2, but other models will also need a fix as well

(@susnato -- I've assigned this issue to me so it doesn't get forgotten, but I'm counting on your aid 😉  )

Hi @gante 
I have been hit by the same issue! Namely, after having added new tokens to the tokenizer (GPT2Tokenizer), and resized the token_embeddings of the model (TFGPT2LMHeadModel), the model.fit(...) throw errors the same as @visionscaper reported.
When could you release a fix patch? Or is there a workaround solution for now?

You guys are doing a great job! And your support is highly appreciated! Cheers~

Hello @gante ! Thanks for your support. I also has faced the same issue as is commented by @visionscaper and @tqye2000. Especially, I tried to check almost every TFGPT2 based pretrained models released by huggingface and figured it out that resize_token_embeddings() does not work for all of them, even including the example code written in huggingface document. Hope this error gets fixed as soon as possible ! :)


EDIT)
After reading the comment below:
> @visionscaper thank you for raising the issue! It is a generalized problem with this check, which should only rely on the config's vocab size (which is the only reliable source of the actual vocabulary size at any given moment).
> 
> @susnato opened a fix for GPT2, but other models will also need a fix as well

I installed the source version of transformers library, which the most latestes on-going code handled by huggingface.co, rather than installing a stable distribution version. Then, resize_token_emeddings() successfully worked with TFGPT2 module ! Thanks to @gante @susnato for fixing crucial errors to Tensorflow users. :)



@tqye2000 @CHLEE-Leo Hey 👋 

Yes, the current source version has the issue fixed for TFGPT2. A new release of `transformers` should happen late next week, which will include this fix. The issue is present in other models, but hopefully will be sorted out soon as well. 

FYI, this issue appeared because we noticed a dangerous pattern in our embedding layers -- in TF, we can request to embed integers outside the bounds of the embedding layer and the code won't crash (returns a vector of zeros), which is extremely dangerous. I've added an out-of-bounds check, but forgot to account for the case with resized vocabulary 🙃  

Fixed on all models, thanks to @susnato 🧡 

Thanks @gante!

> Hello @gante ! Thanks for your support. I also has faced the same issue as is commented by @visionscaper and @tqye2000. Especially, I tried to check almost every TFGPT2 based pretrained models released by huggingface and figured it out that resize_token_embeddings() does not work for all of them, even including the example code written in huggingface document. Hope this error gets fixed as soon as possible ! :)
> 
> EDIT) After reading the comment below:
> 
> > @visionscaper thank you for raising the issue! It is a generalized problem with this check, which should only rely on the config's vocab size (which is the only reliable source of the actual vocabulary size at any given moment).
> > @susnato opened a fix for GPT2, but other models will also need a fix as well
> 
> I installed the source version of transformers library, which the most latestes on-going code handled by huggingface.co, rather than installing a stable distribution version. Then, resize_token_emeddings() successfully worked with TFGPT2 module ! Thanks to @gante @susnato for fixing crucial errors to Tensorflow users. :)

Hi
Could you please show me where or how could I get the latest source version of transformers? Can I get it with pip upgrade?
Many thanks!

Hey @tqye2000 👋  You can upgrade your `transformers` installation to match the current source version with `pip install --upgrade git+https://github.com/huggingface/transformers.git`

Thank you very much, @gante! After having upgraded to the current source version, the resize_token_emeddings() seems to be working now. However I get "Allocation of 740033280 exceeds 10% of free system memory" messages. I guess this is my PC's issue.



Hi @gante 
May I ask another question. For fine tuning the gpt-2 model, should I pass the labels exactly the same as the inputs or should I shift the inputs by one token to create the labels? I get mixed information on the internet, some said the labels should be a copy of inputs, some examples showed the labels should be one-token shifted of the inputs.
I apologise if here is not the right place for asking such questions!
Many thanks! 

Hey @tqye2000 -- using the best possible reference, [the code itself](https://github.com/huggingface/transformers/blob/31336dcf3f93dee19cd13c981f16982d612040d2/src/transformers/models/gpt2/modeling_gpt2.py#L1068), you can see that you *don't* need to shift the inputs. In other words, labels = inputs, all shifting happens inside the model. I hope this helps 🤗 

Hi @gante 
Thank you very much for replying! 
Indeed I eventually dived into the code to see what's going on there and found:
`        if labels is not None:
            # Shift so that tokens < n predict n
            shift_logits = lm_logits[..., :-1, :].contiguous()
            shift_labels = labels[..., 1:].contiguous()
`
But nevertheless it is good to have your confirmation!


Hi @gante 
I think I still need to shift the labels by 1 token by myself. I guess this may be to do with the way I am passing the dataset to the transformer model.
`dataset= tf.data.Dataset.from_tensor_slices((inputs, labels))
dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE)

hist = model.fit(dataset, epochs=4)
`
I just tested. If I didn't shift the labels myself, the fine tuning failed.

Perhaps only if the labels is passed explicitly "labels=labels" to the model, then no need to shift beforehand. 

@tqye2000 that should not be needed -- with HF models, if the label is not provided, [we try to infer it](https://github.com/huggingface/transformers/blob/73a2ff69740123ef85343580cbfa9ee8ce5e6fd5/src/transformers/modeling_tf_utils.py#L1521) (which is the case for GPT2, where labels = inputs).

I'd recommend seeing our example to fine-tune models like GPT2: https://github.com/huggingface/transformers/blob/main/examples/tensorflow/language-modeling/run_clm.py

(and, if it still fails, to open a new issue with a snippet where we can reproduce the problem :) )

Hi. I faced with a similar issue with TFBertForSequenceClassification.
The maximum value of input_ids (Tensor(\"tf_bert_for_sequence_classification/bert/embeddings/Max:0\", shape=(), dtype=int32)) must be smaller than the embedding layer\'s input dimension (30522).
I have both added the new token in the tokenizer and the model's embedding layer. Moreover, I made sure that the inputs are tokenized correctly with the new token and the embedding layer's size actually increased with the new embedding.
I am pretty sure that the model is referencing tokenizer.vocab_size, which is not getting updated. It would be great if tokenizer.vocab_size dynamically updated based on the shape of the tokenizer.

Dear all, I am facing the same issue with transformers 4.42.4 and tokenizers 0.19.1, in particular with TFBartEncoder