Add Chinese-CLIP implementation
Author: yangapku
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds Chinese-CLIP model into Transformers repo. The Chinese-CLIP model was introduced in [Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese](https://arxiv.org/abs/2211.01335). Chinese CLIP is an implementation and adaptation of CLIP (Radford et al., 2021) on a large-scale dataset of Chinese image-text pairs. It is capable of performing Chinese-based cross-modal retrieval and also playing as a vision backbone for vision tasks like zero-shot image classification, open-domain object detection, etc. This model was contributed by [OFA-Sys](https://huggingface.co/OFA-Sys). The original Github repo of Chinese-CLIP can be found [at this link](https://github.com/OFA-Sys/Chinese-CLIP). Currently we have released our model weights on the [Huggingface ModelHub](https://huggingface.co/OFA-Sys/chinese-clip-vit-base-patch16). Compared with original OpenAI CLIP, we changed the text encoder to Chinese Roberta encoder, thus we reimplemented the config, modeling and preprocessor modules of Chinese-CLIP. Necessary unit tests and documents have been added.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Regarding modeling, it somehow breaks our philosophy `one model - one file philosophy`. We usually copy the modeling code and add `# copied from`. cc @sgugger 



> Regarding modeling, it somehow breaks our philosophy `one model - one file philosophy`. We usually copy the modeling code and add `# copied from`. cc @sgugger

So in conclusion, all I need to is to copy all the modules imported from other models and add `# Copied from` statements. The use of feature extractor is still acceptable at this point. Am I right? @ydshieh 

> > Regarding modeling, it somehow breaks our philosophy `one model - one file philosophy`. We usually copy the modeling code and add `# copied from`. cc @sgugger
> 
> So in conclusion, all I need to is to copy all the modules imported from other models and add `# Copied from` statements. The use of feature extractor is still acceptable at this point. Am I right? @ydshieh

Yes :-)

Hi, @ydshieh @sgugger I have updated my code to remove the import of the configuration and modeling files from other models. Is that able to be merged now? Meanwhile, happy Thanksgiving and best wishes ❤️  !

> Hi @yangapku Thank you for adding this model! 非常感謝您!
> 
> I left a few comments, but there might be more comments next week, as I haven't review all the current files.

Excuse me, apart from the proposed comments on model implementation codes and documents, are there any more comments on the other code changes? Let me fix them together 😄. @ydshieh  

Hi @yangapku 

I haven't continued the review process yet after back from the weekend. I will review what you have addressed according the our comments, as well as what I haven't reviewed last time.

Hope you can understand it's not super easy to have a full review in one-go for large PRs like this 🙏. Even for small PR, it's also normal for the review being a iterative process :-)

 (and we sometime get distracted to other tasks that pop up  😅).

> Hi @yangapku
> 
> I haven't continued the review process yet after back from the weekend. I will review what you have addressed according the our comments, as well as what I haven't reviewed last time.
> 
> Hope you can understand it's not super easy to have a full review in one-go for large PRs like this 🙏. Even for small PR, it's also normal for the review being a iterative process :-)
> 
> (and we sometime get distracted to other tasks that pop up 😅).

I fully understand 👍 . Okay I will manage to address the existing comments and try to work out them all before the tomorrow's review.

@ydshieh  All the comments mentioned above have been addressed. 

@ydshieh Hi, is the PR able to be merged now? Thank you very much! ❤️ 

@sgugger I feel there must be very good reason we want to have `CLIPTextTransformer` and `CLIPVisionTransformer`, and use these components in `CLIPTextModel`, `CLIPVisionModel` and `CLIPModel`. 

(Potentially to avoid `CLIPPreTrainedModel` in another `CLIPPreTrainedModel` which might cause some issues - at least if we ever want to have a TF port).

Do you think here we need to avoid this line

```
self.text_model = ChineseCLIPTextModel(text_config, add_pooling_layer=False)
```
and to create `ChineseCLIPTextTransformer` and use it? 


Hi @yangapku , other than the above comment, LGTM! But let's wait @sgugger 's response.

There are a few places where I believe we can still use `# copied from` (probably need some tweaks) - I can help on this before merge.

@ydshieh I think this was done this way just to be more aligned with the original checkpoints in the CLIP case. Here it works fine with the checkpoint, so I wouldn't over-complexify things

@yangapku Before we merge, could you run

```python
RUN_SLOW=1 python -m pytest -v tests/models/chinese_clip/
```

I got 5 failures. You can focus on the first/last ones at this moment. Let me know if you need help on fixing them 🙏 

```
FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPTextModelTest::test_model_from_pretrained - AttributeError: 'ChineseCLIPConfig' object has no attribute 'vocab_size'
FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_output_attentions - AssertionError: Items in the second set but not the first:
FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_output_hidden_state - AssertionError: Items in the second set but not the first:
FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_simple - AssertionError: Items in the second set but not the first:
FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelIntegrationTest::test_inference - OSError: Can't load tokenizer for 'OFA-Sys/chinese-clip-vit-base-patch16'. If you were trying to load it
```


> @yangapku Before we merge, could you run
> 
> ```python
> RUN_SLOW=1 python -m pytest -v tests/models/chinese_clip/
> ```
> 
> I got 5 failures. You can focus on the first/last ones at this moment. Let me know if you need help on fixing them 🙏
> 
> ```
> FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPTextModelTest::test_model_from_pretrained - AttributeError: 'ChineseCLIPConfig' object has no attribute 'vocab_size'
> FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_output_attentions - AssertionError: Items in the second set but not the first:
> FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_output_hidden_state - AssertionError: Items in the second set but not the first:
> FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_simple - AssertionError: Items in the second set but not the first:
> FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelIntegrationTest::test_inference - OSError: Can't load tokenizer for 'OFA-Sys/chinese-clip-vit-base-patch16'. If you were trying to load it
> ```

Okay I will try to fix them today.

> > @yangapku Before we merge, could you run
> > ```python
> > RUN_SLOW=1 python -m pytest -v tests/models/chinese_clip/
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > I got 5 failures. You can focus on the first/last ones at this moment. Let me know if you need help on fixing them 🙏
> > ```
> > FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPTextModelTest::test_model_from_pretrained - AttributeError: 'ChineseCLIPConfig' object has no attribute 'vocab_size'
> > FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_output_attentions - AssertionError: Items in the second set but not the first:
> > FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_output_hidden_state - AssertionError: Items in the second set but not the first:
> > FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_simple - AssertionError: Items in the second set but not the first:
> > FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelIntegrationTest::test_inference - OSError: Can't load tokenizer for 'OFA-Sys/chinese-clip-vit-base-patch16'. If you were trying to load it
> > ```
> 
> Okay I will try to fix them today.

@ydshieh The first and last failed cases have been fixed. Now only the failed test cases with Torchscript still remain. Meanwhile, to fix the first failed case, I have to remove the copied from comment for `ChineseCLIPTextModel`, since it has diverged from `BertModel` with our customed config_class `ChineseCLIPTextConfig`.

@ydshieh Hi, is the PR able to be merged now? Do I have to fix the test cases related with Torchscript? If so, more help is needed since I am not so familiar with it 😢 .

I will take a look on those 3 tests @yangapku . 

@yangapku I pushed the remaining fix. Will merge once the final CI is green 🚀 🚀 🚀 

非常感謝您的工作! 💯 

Thank you very much for your brilliant support! @ydshieh @sgugger 

Hi @yangapku Just a follow up. From your branch , I see the file
```
convert_chinese_clip_original_pytorch_to_hf.py
```
is last modified on Nov 22. (The change on Nov 29 doesn't count). However, the modeling file changed quite a lot since then due to our review comments. I just want to make sure the conversion script still works correctly, and the original checkpoints and the the converted HF checkpoints still have the same outputs on some test examples.

It would be super nice if you can double check, but it's your call, it's just a suggestion.
(It's always good to make sure the users get the right checkpoints to use :-))



@ydshieh Hi, I have ensured that this conversion script works correctly 😄 . In fact, today we have also updated the other 3 model scales (ViT-L/14, ViT-L/14@336px, ViT-H/14) on our HF model hub, during which I have used this script to convert our original model to HF format. After the conversion, I have tested all the converted HF checkpoints (the `pytorch_model.bin`), all of them works in expectation.

Thank you @yangapku !