Error in loading reduced multilingual layoutxlm model: RuntimeError: Error(s) in loading state_dict for LayoutLMv2ForTokenClassification:
Author: Merchaoui
Pull request: no

### System Info

- `transformers` version: 4.33.3
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.9.18
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.3.3
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cpu (False)
- Tensorflow version (GPU?): 2.10.1 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

model = LayoutLMv2ForTokenClassification.from_pretrained("./layoutxlm_reduced/microsoft/layoutxlm-base",
                                                                      num_labels=len(labels))

### Expected behavior

I followed the official tuto  to reduce the size of the model because I only need english and spanish languages for layoutxlm. (https://medium.com/@coding-otter/reduce-your-transformers-model-size-by-removing-unwanted-tokens-and-word-embeddings-eec08166d2f9). 
When I load the reduced model I have this error: "RuntimeError: Error(s) in loading state_dict for LayoutLMv2ForTokenClassification: size mismatch for classifier.weight: copying a param with shape torch.Size([2, 768]) from checkpoint, the shape in current model is torch.Size([6, 768]). size mismatch for classifier.bias: copying a param with shape torch.Size([2]) from checkpoint, the shape in current model is torch.Size([6]). You may consider adding ignore_mismatched_sizes=True` in the model `from_pretrained` method."
and when I set `ignore_mismatched_sizes=True`, I get the error "IndexError: index out of range in self" when it starts training

Comments:
Hi @Merchaoui, thanks for raising this issue! 

Loading the model with `ignore_mismatched_sizes=True` is the correct way to go i.e.:

```py
model = LayoutLMv2ForTokenClassification.from_pretrained(
    "./layoutxlm_reduced/microsoft/layoutxlm-base",
    num_labels=len(labels),
    ignore_mismatched_sizes=True
)
```

The other error shouldn't be happening though. Could you share a minimal code snippet we could run which reproduces the error when training? 

Could you also share the error message with the full traceback? 


Thank you so much, I just figured it out. When I reduced the model, I removed unnecessary tokens from LayoutXLMTokenizerFast and for the training I was processing with LayoutXLMTokenizer. Now it´s working thanks

@Merchaoui Great - glad to hear it's working! Thanks for following up on the issue to confirm :) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.