CLIPProcessor using only single core
Author: nreimers
Pull request: no

## Environment info
- `transformers` version: 4.11.3
- Platform: Linux-5.4.0-54-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyTorch version (GPU?): 1.9.1 (True)


### Who can help
 @patil-suraj

## Information

The CLIPProcessor uses only a single core (on my machine) to process images. This makes encoding images with the CLIP model quite slow (as reported by https://github.com/UKPLab/sentence-transformers/issues/1200), as the image processing takes usually by far the longest.

The original CLIP processor (https://github.com/openai/CLIP/blob/c13005fd422b20dcd11774e9fff46370887218e4/clip/clip.py#L75) in contrast uses multiple cores, leading to an image encoding speed-up of factor 3-4.

## Example
```python

import time
import requests
from PIL import Image
import torch
from transformers import CLIPProcessor, CLIPModel
import numpy as np
torch.set_num_threads(4)

img = Image.open(requests.get("https://cdn.pixabay.com/photo/2015/04/23/22/00/tree-736885__480.jpg", stream=True).raw)
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

# Repeat the same image 256 times
images = [img] * 256


print("Transformers processor")
runtimes = []
for _ in range(5):
    start_time = time.time()
    inputs_hf = processor(images=images, return_tensors="pt", padding=True)
    runtimes.append(time.time()-start_time)
    print("Time:", runtimes[-1])
    
print("Avg. runtime:", np.mean(runtimes))

#######

print("Original CLIP code")
from torchvision.transforms import Compose, Resize, CenterCrop, ToTensor, Normalize

def _transform(n_px):
    return Compose([
        Resize(n_px, interpolation=Image.BICUBIC),  #InterpolationMode.BICUBIC
        CenterCrop(n_px),
        lambda image: image.convert("RGB"),
        ToTensor(),
        Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711)),
    ])
    
clip_processor = _transform(224)
runtimes = []
for _ in range(5):
    start_time = time.time()
    inputs_org = torch.stack([clip_processor(img) for img in images])
    runtimes.append(time.time()-start_time)
    print("Time:", runtimes[-1])
 
print("Avg. runtime:", np.mean(runtimes))
    
print(inputs_hf.pixel_values.shape)
print(inputs_org.shape)

# Ensure the same output
assert torch.allclose(inputs_hf.pixel_values, inputs_org)
```

Output of the script:
```
Transformers processor
Time: 3.9625985622406006
Time: 3.9464197158813477
Time: 3.854238748550415
Time: 3.7350003719329834
Time: 3.6931965351104736
Avg. runtime: 3.838290786743164

Original CLIP code
Time: 1.2502388954162598
Time: 1.2544281482696533
Time: 1.2220158576965332
Time: 1.2993192672729492
Time: 1.1974220275878906
Avg. runtime: 1.2446848392486571
torch.Size([256, 3, 224, 224])
torch.Size([256, 3, 224, 224])
```
As we see, the original `_transform` code is 3 times faster than `CLIPProcessor`

Performance was measured on a DGX-2 machine with 96 cores and Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz.

 `CLIPProcessor` just utilizes one core, while the original CLIP _transform code utilizes multiple threads (restricted to 4, but can be changed to any number).

# Questions

Is this behavior intended, that CLIPProcessor just utilizes a single core?

If not, should I create a PR that fixes this issue so that CLIPProcessor uses multiple cores?



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

This has significant implications if using `CLIP` models for production applications with GPUs. Essentially image preprocessing takes significant part of inference time for batches of data since it uses a single core as can be seen here:
https://github.com/huggingface/transformers/blob/61d3928bfb3029bceb5be3e68ca3d4bf8456758f/src/transformers/models/clip/image_processing_clip.py#L324-L342

This is still a problem. Can this issue be reopened?

@asydorchuk @thesofakillers A bit of context around the design of the image processors: they've been created to enable users to go quickly from data -> predictions, preparing a batch of images to be ready for the model. The aim is that they can be quickly created using the `AutoImageProcessor` API and easy to understand and configure e.g. setting `do_resize=False`. 

They aren't, however, written to be fast or perform additional operations such as augmentations e.g. flip, rotate, change colours. The reason for this is that there are many great existing libraries for image processing that already do these and are extremely fast. In addition, our image processors need to support multiple frameworks: jax, tensorflow, pytorch. As such, we don't want to use one of these underlying libraries like torchvision directly as that would force e.g. TF users to install torch. You'll see in our framework specific examples, [that torchvision is used to process](https://github.com/huggingface/transformers/blob/main/examples/pytorch/image-classification/run_image_classification.py#LL302C5-L318C5), rather than our image processors.

@amyeroberts thank you very much. The framework-specific examples are very useful. Perhaps a warning/mention of this can be added to the [CLIP docs](https://huggingface.co/docs/transformers/model_doc/clip).

@thesofakillers I would add it somewhere more general, as this is true for all models' image processors, not just CLIP's. Perhaps as a note in the vision examples README or in the [docs for the image processor](https://huggingface.co/docs/transformers/v4.30.0/en/main_classes/image_processor#image-processor)? If you'd like to open a PR with these changes I'd be happy to review.  

@amyeroberts Makes sense, but I didn't even know that that docs page existed. I think most people will be reading what is in the CLIP docs. If you think it should still be in the docs for the image processor, the CLIP docs should at least point to it explicitly. Lmk what you think, and please point me to the github file I'd edit for submitting PR's for doc edits, it's not super clear atm.