Add I-JEPA
Author: jmtzt
Pull request: yes
Labels: name: New model

name: Vision

name: run-slow

name: Modular

# What does this PR do?

This PR adds [I-JEPA](https://github.com/facebookresearch/ijepa).

To-Do's:
- [x]  convert remaining checkpoints `ijepa_vith14_22k`, `ijepa_vith16_1k`, `ijepa_vitg16_22k`.
- [ ]  transfer checkpoints to the meta org

Comments:
cc @amyeroberts and @qubvel 

Thanks for reviewing the code, @amyeroberts! :)

I've just pushed the adjustments, and now the fx support seems to be working fine.

Thanks again for checking @amyeroberts :) I've just pushed the adjustments, however some tests appear to be failing due to some OS errors inside CircleCI, do you know why's that? 

Thanks for pushing again! Hmmmm - I'm not sure why the slow model tests aren't being picked up or run here cc @ydshieh 

Yih-Dar found the issue - the actions won't be triggered whilst there are merge conflicts in the PR. Could you resolve these, then push another `[run-slow] ijepa` commit? This should hopefully run the workflow! 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33125). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for looking into it. I fixed the merge conflicts, now the tests seem to have run without any issues. cc @amyeroberts 

@jmtzt Great! Final step is to move the checkpoints to be under the official organization and updating the paths in the PR. Could you share a link to all the checkpoints for this model under your name on hf.co? 

Sure, here you can find them:

- [ijepa_vith14_1k](https://huggingface.co/jmtzt/ijepa_huge_patch14_1k)
- [ijepa_vith14_22k](https://huggingface.co/jmtzt/ijepa_huge_patch14_22k)
- [ijepa_vith16_1k](https://huggingface.co/jmtzt/ijepa_huge_patch16_1k)
- [ijepa_vitg16_22k](https://huggingface.co/jmtzt/ijepa_giant_patch16_22k)

@amyeroberts I've also updated the model cards :)

Hi @jmtzt! Thanks for working on the model and sorry for the long wait, sometimes we are overwhelmed with issues and PRs! Don't hesitate to ping a few times!

No problem @qubvel, is there anything else I should do from my end? Thanks :)

Hi @jmtzt let us know if you need any help to push this one to the finish line!

> Hi @jmtzt let us know if you need any help to push this one to the finish line!

Hi @NielsRogge, I just need some time to go over the new conversion script and modelling pattern, I believe this weekend I can make some progress on it and keep you posted. Thanks!

hey @qubvel just pushed the adjustments according to the new modular structure, as suggested. Will add the snippets to the model cards soon... :)

Thanks for iterating on the PR and addressing the comments!

While trying to use the converted checkpoint with the snippet from the model card I got the message that some weights were not initialized.

```python
import requests
from PIL import Image
from torch.nn.functional import cosine_similarity

from transformers import AutoModel, AutoProcessor

url_1 = "http://images.cocodataset.org/val2017/000000039769.jpg"
url_2 = "http://images.cocodataset.org/val2017/000000219578.jpg"
image_1 = Image.open(requests.get(url_1, stream=True).raw)
image_2 = Image.open(requests.get(url_2, stream=True).raw)

model_id = "jmtzt/ijepa_vith14_1k"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id)

def infer(image):
    inputs = processor(image, return_tensors="pt")
    outputs = model(**inputs)
    return outputs.pooler_output

embed_1 = infer(image_1)
embed_2 = infer(image_2)

similarity = cosine_similarity(embed_1, embed_2)
print(similarity)
```

Message:
Some weights of IJepaModel were not initialized from the model checkpoint at jmtzt/ijepa_vith14_1k and are newly initialized: ['pooler.dense.bias', 'pooler.dense.weight']

Were these weights missed? Without the full model initialized I suppose the output similarity is incorrect, isn't it?

the original I-JEPA model doesn't have the pooling layer, so I think to get around this we might need to default the `add_pooling_layer` to `False` in its initialization, and modify this snippet accordingly to get the last hidden states rather than the `pooler_output`

Ok, sounds good 👍 

hi @qubvel, thanks for the support and reviewing the PR! Just pushed the suggested changes, let me know if it's alright now.

@ArthurZucker please review whenever you have bandwidth! The model is similar to ViT, so the `Modular` is used here to remove the CLS token.

Checkpoints can be found here:
https://huggingface.co/jmtzt

(we will need to transfer them to [facebook](https://huggingface.co/facebook) org as soon as we ensure the code is in the final stage + rename all occurrences in code and model cards)

Feel free to ping me again for another review! 🫡 

> Feel free to ping me again for another review! 🫡

@ArthurZucker I've pushed the adjustments according to your comments, let me know if anything is missing. Thanks! :)

@ArthurZucker can you transfer checkpoints to Facebook org? Also, code snippets need to be adjusted.
Checkpoints are here: https://huggingface.co/jmtzt


@jmtzt congratulations on the model merged 🎉  was glad to collaborate with you on this!
Can you please confirm if we can transfer checkpoints to Facebook org from your account?

thanks for the support @qubvel and @ArthurZucker :)

Sure, go ahead!