CLIP
Author: patil-suraj
Pull request: yes
Labels: name: PR for Model Addition

# What does this PR do?

This PR adds the [CLIP](https://github.com/openai/CLIP) model.

CLIP is a multi-modal vision+language model which uses a transformer model for encoding both the images and text.

- The model here is designed such that both `CLIPTextModel` and `CLIPVisionModel` can be loaded independently, and composed together to get the `CLIPModel`.
- Both `CLIPTextModel` and `CLIPVisionModel` use the shared encoder class `CLIPEncoder`.
- The config classes are also kept in separate i.e `CLIPTextConfig` and `CLIPVisionConfig`.  This could be kept in one config class but then we would have to add two arguments for each config value i.e `text_hidden_size` for text model `vision_hidden_size` for vision model etc.
One issue here is that when we load an individual model, like `CLIPTextModel` using the weights of the whole `CLIPModel`
the config ends up containing both text and vision config dicts, this does not cause any issue but could be confusing to look at. 

One important thing to note here is that CLIP's tokenizer does have a pad token defined for it, but they use 0 as `pad_token_id`  to pad the text, but the token, but the token associated with 0 is not a pad token. So here, to able to do padding I've added `pad_token_id` as a `property` which returns 0. I would be happy to hear if there is some other way to achieve this.

Also, I've added a processor class here but not sure if we really need it for this model. We could easily use the extractor for the vision model and tokenizer for the text model.

Would love your review about the design @LysandreJik , @patrickvonplaten , @sgugger.

Comments:
@sgugger 

> is it possible to add a fast version of the tokenizer?

Yes, will add the fast version as well.


> why are there two classes for the vision model and the text model? They have the exact same forward, so we should only have one class IMO.

Added two versions so that one could just load `CLIPTextModel` or `CLIPVisionModle` directly from `CLIPModel`'s weights. If we just keep single modules then it's not possible to load the weights from the full model because then the keys won't match. The `CLIPModel` has these extra keys `text_model` and `vision_model`, hence the two extra modules with the same keys.

This would allow one to use the vision model in some other downstream tasks like adding a liner layer on top or using it as an image encoder in some other settings. Not sure if users will actually want this, but this does not really add much complexity to the code IMO.

All green!! 
I've addressed most of the suggestions, notably

- new processor API => as discussed with @patrickvonplaten and @LysandreJik processor's `__call__` now accepts both the text and/or images and returns a single encoding dict. `as_target_processor` is now removed. The API is as follows

```python3
model = CLIPModel.from_pretrained(checkpoint)
inputs = CLIPProcessor(texts=..., images=..., some_other_kwargs)
outputs = model(**inputs)
```
- the `encode_text` and `encode_image` methods are renamed to `get_text_features` and `get_image_features`
- Added fast tokenizer.

Ready for second review @LysandreJik @sgugger @patrickvonplaten 

> All green!!
> I've addressed most of the suggestions, notably
> 
> * new processor API => as discussed with @patrickvonplaten and @LysandreJik processor's `__call__` now accepts both the text and/or images and returns a single encoding dict. `as_target_processor` is now removed. The API is as follows
> 
> ```python
> model = CLIPModel.from_pretrained(checkpoint)
> inputs = CLIPProcessor(texts=..., images=..., some_other_kwargs)
> outputs = model(**inputs)
> ```
> 
> * the `encode_text` and `encode_image` methods are renamed to `get_text_features` and `get_image_features`
> * Added fast tokenizer.
> 
> Ready for second review @LysandreJik @sgugger @patrickvonplaten


How to use processor in __getitem()__? I got an error"RuntimeError: stack expects each tensor to be equal size, but got [1, 11] at entry 0 and [1, 13] at entry 1" ,as follow:
def __getitem__(self, idx):
        img_id = self.img_ids[idx]
        # randomly pick one caption from the image captions
        text = random.choice(self.img_id_to_captions[img_id])
        img_filename = self.img_id_to_filename[img_id]
        img_path = op.join(self.img_dir, img_filename)
        img = Image.open(img_path) 
        input = self.processor(text = text, images = img, return_tensors = "pt", padding = True)
        return input
I thought processor might need other args, inherited from pretraintokenizerbase,such as padding.But I couldn't find it at processor's __call__ in doc.

Hi @lycfight  could you please open an issue with a minimal code snippet so we could take a look. Thanks :) 

> Hi @lycfight could you please open an issue with a minimal code snippet so we could take a look. Thanks :)

of course