Add AltCLIP
Author: jongjyh
Pull request: yes

# Adding AltCLIP
We add AltCLIP model.
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
@jongjyh @shunxing1234

If there is anything unclear, or you need help regarding the commit conflict, don't hesitate!

@sgugger  @ydshieh  Thanks! I'll fix it. :)
By the way, is there only processor need to be redefine?  Do we need to check out all classes?

model, config and processor should be redefined, but not tokenizer, image processor, or feature extractor.
(@sgugger Is this correct?)

You can see the file structure in `models/x_clip` to get some idea

Hi @jongjyh @shunxing1234 

I had to fix two files that contain `<<<< HEAD` (from previous conflicts when you merged HF's `main` into your `main`).


@sgugger 

Currently, the PR authors don't implement `AltCLIPVisionModel`, as the vision component is just the same as `CLIPVisionModel`. They do implement the necessary modules like `AltCLIPVisionTransformer`, as this is required in `AltCLIPModel`.

This seems fair to me.

However,

In the model tester file, they do
```
from ...models.clip.test_modeling_clip import CLIPVisionModelTester
```
and there is no `AltCLIPVisionModelTest` being implemented (same reason, it's just `CLIPVisionModelTest`).

~Do you think this is OK - I don't like the dependency though.~

@jongjyh @shunxing1234 

IMO, despite the vision model is just `CLIPVisionModel`, I think for the completeness and max independency, it's good to have `AltCLIPVisionModel`. It should be very quick to add I believe. 

No, the test files of models should be independent from each other, like the modeling files, or it will make our lives harder down the road for maintenance :-)

Hi @jongjyh @shunxing1234

Please go ahead with adding `AltCLIPVisionModel`, removing the usage of `CLIPVisionModelTester`, adding `AltCLIPVisionModelTester`.

This could reduce a few more test failures :-)

Hi, @ydshieh 

thank you for your checking! I added alt_clipvision model right now, please let me know if there are other requests. :)

_The documentation is not available anymore as the PR was closed or merged._

Finally the CIs are green! I will take a review in more details!

Hey, @ydshieh @sgugger  

Here is my new pr. :) Please help to check whether it meets the requirements

Hi, now we still need to convert the weights, I loaded the previous weight file under the new `modeling_altclip.py`:

> Some weights of the model checkpoint at BAAI/AltCLIP were not used when initializing AltCLIPModel: ['text_model.roberta.pooler.dense.bias', 'text_model.roberta.pooler.dense.weight']

It seems only pooler in Roberta need to be removed. Could I just upload a new `pytorch.bin` with no pooler to replace original one after results of the down-steam task were reproduced using this new modeling file?

I also notice processor file need to renew. 

Thank you for your long-term follow-up and help!

> Could I just upload a new pytorch.bin with no pooler to replace original one after results of the down-steam task were reproduced using this new modeling file?

Good for me, as long as the results from the original model/checkpoint & from the added model calss/converted checkpoint matches.


@jongjyh 

It looks like the CI doesn't run anymore, but it indeed ran for some commits you pushed previously. I think you have followed this before, but could you try again 🙏 to trigger the CI. Thanks.

> It seems there is an issue with your CircleCI permissions, the tests won't run.
Could you try refreshing your permissions as shown [here](https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-)?

cc @sgugger It's ready for merge IMO (other than doctesting - which the author and me will take care). But before merge, would like to have you final look/approve 🙏 Thanks.

Hi @ydshieh,
I am worried that there may be many dirty commits in this PR at present. Is there any way to merge these more than 100 commits before it is officially merged into the main branch? :)

Hi @jongjyh! No worry for the dirty commits, the GitHub page will `Squash and merge`, as shown in the button below :-)

Hi. the doctest for the config/modeling files pass now. Thank you a lot!
However, the doctest for `docs/source/en/model_doc/clip.mdx` has some problem, it won't be run.

To check
```bash
python3 -m pytest -v --make-reports doc_tests_gpu --doctest-modules docs/source/en/model_doc/altclip.mdx -sv --doctest-continue-on-failure --doctest-glob="*.mdx"
```

I checked the file permission 
```bash
ls -l docs/source/en/model_doc/
```
which shows that file `altclip.mdx` is executable file `-rwxr-xr-x`, but other files are not. See
```bash
-rw-r--r-- 1 root root  4795 Dec 21 01:32 albert.mdx
-rwxr-xr-x 1 root root  4829 Dec 23 10:01 altclip.mdx
-rw-r--r-- 1 root root  3491 Dec 21 01:32 audio-spectrogram-transformer.mdx
-rw-r--r-- 1 root root  7523 Dec 23 10:01 auto.mdx
-rw-r--r-- 1 root root  9454 Dec 23 10:01 bart.mdx
```

Could you fix this permission issue? Potentially replace that file with a newly created file with the content copied.
Thanks

@ydshieh @sgugger Hi, we update the pr, but there are some unexcepted issues(tf error) occured

Thanks a lot for all your work on this! There just needs to be a rebase on main and we can merge this.

Thank you again @jongjyh @shunxing1234 🚀 !