Error while loading finetuned distilbert model: embedding dimension mismatch
Author: rohanshingade
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.3.1
- Python version: 3.6.9
- Tensorflow version (GPU?): 2.3.0
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik @patil-suraj @jplu
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information
I am using `TFAutoModelForSequenceClassification` with `distilbert-base-multilingual-cased` model. For finetuning i have freezed the embedding layer. Finetuning is successful and i have saved the weights using `save_pretrained`. However after finetuning when i load the model for inference using `TFAutoModelForSequenceClassification` or `TFDistilBertForSequenceClassification` it throws error.

However, I did not face any issues with `TFXLMRobertaForSequenceClassification` and `jplu/tf-xlm-roberta-base` while trying the same thing.

## To reproduce
```
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-multilingual-cased")
tokenizer_output = tokenizer.batch_encode_plus(train_texts, max_length=100, padding="max_length", truncation=True,return_attention_mask=True, add_special_tokens=True)
input_ids, attention_mask = tokenizer_output["input_ids"], tokenizer_output["attention_mask"]            
            
config = AutoConfig.from_pretrained("distilbert-base-multilingual-cased", num_labels=num_classes,label2id=label2id, id2label=id2label,finetuning_task="text-classification")
model = TFAutoModelForSequenceClassification.from_pretrained("distilbert-base-multilingual-cased", config=config)

# freezing embedding layers
model.layers[0].embeddings.trainable = False

loss = tf.keras.losses.CategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.Accuracy()
optimizer = tf.keras.optimizers.Adam(learning_rate=2e-6, epsilon=1e-08)
model.compile(loss=loss, optimizer=optimizer, metrics=[metric])
            
model.fit([input_ids, attention_mask], train_labels, epochs=10, batch_size=16)
model.save_pretrained("model")

#Throws error for both cases
#loaded_model = TFAutoModelForSequenceClassification.from_pretrained("model")
loaded_model = TFDistilBertForSequenceClassification.from_pretrained("model")

```
Error while loading saved model: 
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-29-0ad4a4ce38ca> in <module>
      2 
      3 #config = AutoConfig.from_pretrained(os.path.join("/home/Rajat/Rohan/models/xlmr104", "model"))
----> 4 model = TFDistilBertForSequenceClassification.from_pretrained(os.path.join("/home/Rajat/Rohan/models/dbert103", "model"))
      5 
      6 t2 = time.time()

/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    614         # see https://github.com/tensorflow/tensorflow/blob/00fad90125b18b80fe054de1055770cfb8fe4ba3/tensorflow/python/keras/engine/network.py#L1339-L1357
    615         try:
--> 616             model.load_weights(resolved_archive_file, by_name=True)
    617         except OSError:
    618             raise OSError(

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in load_weights(self, filepath, by_name, skip_mismatch, options)
   2207       if by_name:
   2208         hdf5_format.load_weights_from_hdf5_group_by_name(
-> 2209             f, self.layers, skip_mismatch=skip_mismatch)
   2210       else:
   2211         hdf5_format.load_weights_from_hdf5_group(f, self.layers)

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/saving/hdf5_format.py in load_weights_from_hdf5_group_by_name(f, layers, skip_mismatch)
    784                                symbolic_weights[i])) +
    785                            ', but the saved weight has shape ' +
--> 786                            str(weight_values[i].shape) + '.')
    787 
    788         else:

ValueError: Layer #0 (named "distilbert"), weight <tf.Variable 'tf_distil_bert_for_sequence_classification_9/distilbert/embeddings/word_embeddings/weight:0' shape=(119547, 768) dtype=float32, numpy=
array([[ 0.00207364,  0.01255192,  0.01065131, ...,  0.0182375 ,
        -0.01671835, -0.02844721],
       [ 0.0333954 ,  0.03589885, -0.03751937, ..., -0.01915496,
        -0.00888181, -0.00063128],
       [ 0.01174717,  0.00945629, -0.01179059, ...,  0.03340805,
        -0.00715566, -0.02317093],
       ...,
       [ 0.01775699, -0.01719745, -0.03220321, ...,  0.00817569,
        -0.00393617, -0.00730391],
       [ 0.03056052, -0.00136884, -0.02507194, ...,  0.01245719,
        -0.00362111, -0.01495665],
       [ 0.03703629,  0.01664717, -0.01278388, ...,  0.02537051,
         0.02492457,  0.01191532]], dtype=float32)> has shape (119547, 768), but the saved weight has shape (768, 768).

```
                   

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Hello!

First of all, can you try with the source version in order to see if the problem still occurs.

Hello @jplu ,
Just running these 4 lines throws error. 

```
model = TFDistilBertForSequenceClassification.from_pretrained("distilbert-base-multilingual-cased")
model.layers[0].embeddings.trainable = False
model.save_pretrained("model")
loaded_model = TFDistilBertForSequenceClassification.from_pretrained("model")
```
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-8-aa416dc4d078> in <module>
----> 1 loaded_model = TFDistilBertForSequenceClassification.from_pretrained("model")

/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    614         # see https://github.com/tensorflow/tensorflow/blob/00fad90125b18b80fe054de1055770cfb8fe4ba3/tensorflow/python/keras/engine/network.py#L1339-L1357
    615         try:
--> 616             model.load_weights(resolved_archive_file, by_name=True)
    617         except OSError:
    618             raise OSError(

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in load_weights(self, filepath, by_name, skip_mismatch, options)
   2207       if by_name:
   2208         hdf5_format.load_weights_from_hdf5_group_by_name(
-> 2209             f, self.layers, skip_mismatch=skip_mismatch)
   2210       else:
   2211         hdf5_format.load_weights_from_hdf5_group(f, self.layers)

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/saving/hdf5_format.py in load_weights_from_hdf5_group_by_name(f, layers, skip_mismatch)
    784                                symbolic_weights[i])) +
    785                            ', but the saved weight has shape ' +
--> 786                            str(weight_values[i].shape) + '.')
    787 
    788         else:

ValueError: Layer #0 (named "distilbert"), weight <tf.Variable 'tf_distil_bert_for_sequence_classification_1/distilbert/embeddings/word_embeddings/weight:0' shape=(119547, 768) dtype=float32, numpy=
array([[ 0.00801877, -0.01047559, -0.03101005, ...,  0.02595956,
        -0.01114979,  0.0103603 ],
       [ 0.00097553, -0.00474179, -0.0065623 , ...,  0.03424093,
        -0.0189246 ,  0.01545161],
       [-0.02869349, -0.03147252, -0.02191292, ...,  0.00606783,
         0.0091517 ,  0.00140686],
       ...,
       [ 0.00324067,  0.01025188, -0.0173355 , ...,  0.00799547,
         0.00298822, -0.00772437],
       [ 0.00393043,  0.02751113,  0.00989435, ...,  0.00630352,
        -0.01590282,  0.00017761],
       [-0.02440546, -0.02454552,  0.01318205, ..., -0.02244014,
         0.02798119, -0.006583  ]], dtype=float32)> has shape (119547, 768), but the saved weight has shape (768, 768).

```
​



With which transformers version?

I'm using this docker image `huggingface/transformers-tensorflow-gpu:3.3.1`

Ok I just tried your code snipped on the source version and it works as expected, so it looks like this issue has already been fixed. Then I suggest you to update your container to the last release.

Thanks

I'm getting the same error, with code that hasn't changed since it worked.  I guess the model downloaded by from_pretrained() is no longer compatible with older tokenizers or transformers code.

tokenizers==0.8.1.rc1
transformers==3.0.2

But what to upgrade to so that it works?  I'm trying to maintain compatibility with javascript tokenizers 0.6.2 because there are other version issues there on the nodejs side.  However it seems this may not be possible.

Just got same error with:
tokenizers==0.8.1.rc2
transformers==3.3.1


Can't even get working with latest tokenizers and transformers.  Although upgrading change the error to:
`ValueError: cannot reshape array of size 22268928 into shape (30522,768)`

Reproduction script:
```
from transformers import DistilBertConfig, TFDistilBertModel
config = DistilBertConfig(dropout=0.2, attention_dropout=0.2)
config.output_hidden_states = False
print('loaading')
transformer_model = TFDistilBertModel.from_pretrained(
    "distilbert-base-cased", config=config
)
print('loaded')
```

Python: 3.8.4

absl-py==0.10.0
astunparse==1.6.3
datasets==1.8.0
filelock==3.0.12
gast==0.3.3
h5py==2.10.0
keras==2.4.3
keras-applications==1.0.8
keras-preprocessing==1.1.2
numpy==1.19.2
opt-einsum==3.3.0
protobuf==3.12.2
regex==2020.7.14
requests==2.24.0
sacremoses==0.0.43
sentencepiece==0.1.94
six==1.15.0
scikit-learn==0.24.2
scipy==1.4.1
tensorboard==2.4.0
tensorflow==2.4.0
termcolor==1.1.0
tokenizers==0.10.3
tqdm==4.48.0
transformers==4.7.0
wrapt==1.12.1




Interestingly `distibert-base-uncased` works just not `distilbert-base-cased`. Maybe missing some config?

+1