TFCLIPModel.resize_token_embeddings() not working
Author: Iyeleon
Pull request: no
Labels: name: TensorFlow

name: bug

name: Multimodal

### System Info

- `transformers` version: 4.43.4
- Platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.12.0
- Huggingface_hub version: 0.23.1
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.17.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@amyeroberts @ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

 I am attempting to extend the token size and embedding size of TFCLIPModel. I successfully added the new tokens to the tokenizer, but I can't resize the token embedding in the model. I keep getting TFCLIPMainLayer does not have attribute "get_input_embeddings. 
I have tried subclassing and writing a method to get the embeddings manually but the parent class does not expose the text_model (or vision_model) attributes under TFCLIPMainLayer. I'm probably not going about this in the right way and was wondering if there is a better way to do this?

```
from transformers import TFCLIPModel
# Load the pretrained CLIP model
clip_model = TFCLIPModel.from_pretrained("openai/clip-vit-base-patch32")

# Resize the token embeddings to match the new vocabulary size
clip_model.resize_token_embeddings(50000)
```

![image](https://github.com/user-attachments/assets/0ef996a5-27dc-4c21-9bbd-fa7a0d94ba1a)

### Expected behavior

Updated input embedding to match tokenizer.

Comments:
cc @Rocketknight1 

Hi @Iyeleon, unfortunately this isn't supported in the TF port of CLIP! I'd recommend trying it with the PyTorch version instead, where it should work.

Hi @Rocketknight1, many thanks for looking into this. I did try with the pytorch version and that raises `NotImplementedError`. I guess it is not supported in general. I eventually figured out how to access the text and vision models and got the tensorflow version to work by subclassing the TFCLIPModel class and implementing a resize_token_embedding method following the logic in TFPretrainedModel. I could be wrong but I think this can be solved by adding a 'get_input_embedding' method that returns the text embeddings in TFCLIPMainLayer class as that's where the disconnect comes from.

Ah, that's interesting! If you got it working, feel free to submit a PR including those changes to the `transformers` codebase, and I'd be happy to review it.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.