ValueError: You have to specify pixel_values in CLIP for ver >= 4.18.0 
Author: naoto0804
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.20.1
- Platform: Linux-4.19.0-21-cloud-amd64-x86_64-with-debian-10.12
- Python version: 3.7.12
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.8.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@patil-suraj

I try to run TFCLIPModel.get_image_features example [here](https://huggingface.co/docs/transformers/model_doc/clip#transformers.TFCLIPModel.get_image_features.example), which is also pasted in Reproduction section.
When I use `transformers >= 4.18.0`, it throws an error "ValueError: You have to specify pixel_values" (details pasted below).
Is there any way to fix this?

```
Traceback (most recent call last):
  File "dummy.py", line 13, in <module>
    image_features = model.get_image_features(**inputs)
  File "/opt/conda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 383, in run_call_with_unpacked_inputs
    return func(self, **unpacked_inputs)
  File "/opt/conda/lib/python3.7/site-packages/transformers/models/clip/modeling_tf_clip.py", line 1318, in get_image_features
    return_dict=return_dict,
  File "/opt/conda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 383, in run_call_with_unpacked_inputs
    return func(self, **unpacked_inputs)
  File "/opt/conda/lib/python3.7/site-packages/transformers/models/clip/modeling_tf_clip.py", line 796, in get_image_features
    raise ValueError("You have to specify pixel_values")
ValueError: You have to specify pixel_values
```

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from PIL import Image
import requests
from transformers import CLIPProcessor, TFCLIPModel

model = TFCLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(images=image, return_tensors="tf")
image_features = model.get_image_features(**inputs)
```

### Expected behavior

Finish without throwing an error described above.

Comments:
cc @NielsRogge, or @amyeroberts @alaradirik if you have any pointers

Thanks for raising @naoto0804 ! 

Doing a bit of digging, this is because of the behaviour of the `unpack_inputs` decorator and the fact `TFCLIPModel` is being used. `unpack_inputs` tries to get the name of the `main_input_name` to the function (see [here](https://github.com/huggingface/transformers/blob/d4ebd4e112034b4a429ab7f813d7e168e7bb63c3/src/transformers/modeling_tf_utils.py#L429))

`TFCLIPModel` inherits from `TFPreTrainedModel` which has `main_input_name` set to `input_ids`. However, the `main_input_name` needed for this function `pixel_values`.  

@naoto0804 If all you want are the image features, the fastest and cleanest way you'll be able to get them is by using `TFCLIPVisionModel`:
```
from PIL import Image
import requests
from transformers import CLIPProcessor, TFCLIPVisionModel

model = TFCLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(images=image, return_tensors="tf")
image_features = model.get_image_features(**inputs)
```

However, this still leaves unexpected behaviour in the code. I can see the `unpack_inputs` decorator was added in https://github.com/huggingface/transformers/pull/16128 as part of https://github.com/huggingface/transformers/pull/15907. 

One thing I'm unsure of is the logic for `input_ids` in `input_processing`.  [It seems there's lots of processing to handle the different possible formats for `input_ids`](https://github.com/huggingface/transformers/blob/981714efe12c5fc481ad38632ca0db88cd85004c/src/transformers/modeling_tf_utils.py#L508). `unpack_inputs` [can pass in any argument as the main_input](https://github.com/huggingface/transformers/blob/981714efe12c5fc481ad38632ca0db88cd85004c/src/transformers/modeling_tf_utils.py#L431), including e.g. `pixel_values`. In the `call` method for `TFCLIPModel` [both `input_ids` and `pixel_values` can be passed](https://github.com/huggingface/transformers/blob/981714efe12c5fc481ad38632ca0db88cd85004c/src/transformers/models/clip/modeling_tf_clip.py#L1344) i.e. it seems the processing logic in `input_processing` isn't necessary for the `pixel_values` input, even if it's set as the `main_input_name`, as in `TFCLIPVisionModel`.  

Would a reasonable solution be to move this logic, such the signature for `input_processing` becomes `input_processing(func, config, **kwargs)` and then apply the processing logic to the input ids if `input_ids` is in `parameter_names`? 
 @gante 

@amyeroberts good finding! 

Looking in hindsight, my design of `unpack_inputs` was sub-optimal here -- #18110 simplifies it and makes it independent from `main_input_name`, which solves the issue raised by @naoto0804 

As for `input_processing`, it is an old function that I intend to rewrite and simplify soon 🙏 It is the cause of many bugs that are not easy to identify unless you're familiar with the function.

@amyeroberts thank you for the answer, which saved my day!