Token embedding resizing does not work for TFBertForSequenceClassification
Author: doniobidov
Pull request: no
Labels: name: Good Second Issue

### System Info

Hi. I faced with a similar issue using TFBertForSequenceClassification as [Token embedding resizing does not work for TFGPT2Model · Issue #21053 · huggingface/transformers (github.com)](https://github.com/huggingface/transformers/issues/21053).
transformers version 4.39.0.

The maximum value of input_ids (Tensor("tf_bert_for_sequence_classification/bert/embeddings/Max:0", shape=(), dtype=int32)) must be smaller than the embedding layer's input dimension (30522).
I have both added the new token in the tokenizer and the model's embedding layer. Moreover, I made sure that the inputs are tokenized correctly with the new token and the embedding layer's size actually increased with the new embedding.
I am pretty sure that the model is referencing tokenizer.vocab_size, which is not getting updated. It would be great if tokenizer.vocab_size dynamically updated based on the shape of the tokenizer.

### Who can help?

@gante

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

assertion failed: [The maximum value of input_ids (Tensor(\"tf_bert_for_sequence_classification/bert/embeddings/Max:0\", shape=(), dtype=int32)) must be smaller than the embedding layer\'s input dimension (30522). The likely cause is some problem at tokenization time.] [Condition x < y did not hold element-wise:] [x (IteratorGetNext:0) = ] [[101 2919 2143...]...] [y (tf_bert_for_sequence_classification/bert/embeddings/Cast/x:0) = ] [30522]
	 [[{{node tf_bert_for_sequence_classification/bert/embeddings/assert_less/Assert/Assert}}]] [Op:__inference_predict_function_4874]

### Expected behavior

Problem with this check, which relies on the config's vocab size.

Comments:
Hi @ArthurZucker !

May I work in this issue? The `good first issues` all seem to be taken, but this one looks doable. :)

@CKeibel Sure! Feel free to open a PR and ping us for review when ready 🤗 


Hi @amyeroberts !

Maybe I didn't understand the issue correctly. At least I could not reproduce the problem with the following code. Am I missing something?

```
from transformers import TFBertForSequenceClassification, BertTokenizer

# init model
model_id = "distilbert-base-uncased"
model = TFBertForSequenceClassification.from_pretrained(model_id)
tokenizer = BertTokenizer.from_pretrained(model_id)

# add new tokens
old_vocab_size = model.config.vocab_size
tokenizer.add_tokens(["NewTestToken"], special_tokens=False) # id = 30522
tokenizer.add_tokens(["NewTestSpecialToken"], special_tokens=True) # id = 30523


# update model
model.resize_token_embeddings(len(tokenizer))

# assert update (30524 > 30522)
assert model.config.vocab_size > old_vocab_size

# inference with new tokens
inputs = tokenizer("Look at my NewTestToken and my NewTestSpecialToken", return_tensors="tf")
model(inputs)

print(inputs.input_ids)
_______________________________________
> tf.Tensor([[  101  2298  2012  2026 30522  1998  2026 30523   102]], shape=(1, 9), dtype=int32)
```

Kind regards,

Christopher

Hi Christopher, try this:
TFBertForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

Hi @doniobidov !

Thanks for the quick reply!

I'll try it and hopefully fix it 🙂 

Mhh unfortunately I still cannot reproduce the error. 😕 

It looks to me like someone has fixed this [here](https://github.com/huggingface/transformers/pull/21164/files#diff-d018970643bceeeec17b51a822b1c8765f805d9a2d95f8031ec2e1b11d09ce39) before.

@doniobidov  can you possibly share some more code about what you did?

@CKeibel I cannot reproduce the error either. Somehow the problem got fixed. I think you can close the issue. Thank you!

I'm glad to hear that your problem has been solved! 🙂

Closing! 🤗 thanks all for this