Preserving gradient flow through Clip Processor
Author: ErwannMillon
Pull request: no

### Feature request

Hi, 
I was using the HF CLIP implementation to build a VQGAN CLIP Implementation and noticed that the CLIPProcessor forces conversion to PIL Images for efficiency. However, when inputting torch tensor images to the processor, this breaks gradient flow. 




### Motivation

Would like to be able to backpropagate through CLIP image processing steps

### Your contribution


This was my quick and hacky fix, using torchvision to do the same transformations and processing steps. I'd be happy to properly code up a better equivalent and submit a pull request if you think this is a feature worth adding. 

```
class ProcessorGradientFlow():
    """
    This wraps the huggingface CLIP processor to allow backprop through the image processing step.
    The original processor forces conversion to numpy then PIL images, which is faster for image processing but breaks gradient flow. 
    """
    def __init__(self, device="cuda") -> None:
        self.device = device
        self.processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
        self.image_mean = [0.48145466, 0.4578275, 0.40821073]
        self.image_std = [0.26862954, 0.26130258, 0.27577711]
        self.normalize = torchvision.transforms.Normalize(
            self.image_mean,
            self.image_std
        )
        self.resize = torchvision.transforms.Resize(224)
        self.center_crop = torchvision.transforms.CenterCrop(224)
    def preprocess_img(self, images):
        images = self.center_crop(images)
        images = self.resize(images)
        images = self.center_crop(images)
        images = self.normalize(images)
        return images
    def __call__(self, images=[], **kwargs):
        processed_inputs = self.processor(**kwargs)
        processed_inputs["pixel_values"] = self.preprocess_img(images)
        processed_inputs = {key:value.to(self.device) for (key, value) in processed_inputs.items()}
        return processed_inputs

```


Comments:
cc @ArthurZucker and @amyeroberts 

Sorry, here's a properly formatted snippet :) These are just a few of the transformations in the hf implementation, but would be happy to properly implement all of the transforms in the CLIPImageProcessor class
```
class ProcessorGradientFlow():
    """
    This wraps the huggingface CLIP processor to allow backprop through the image processing step.
    The original processor forces conversion to numpy then PIL images, which is faster for image processing but breaks gradient flow. 
    """
    def __init__(self, device="cuda") -> None:
        self.device = device
        self.processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
        self.image_mean = [0.48145466, 0.4578275, 0.40821073]
        self.image_std = [0.26862954, 0.26130258, 0.27577711]
        self.normalize = torchvision.transforms.Normalize(
            self.image_mean,
            self.image_std
        )
        self.resize = torchvision.transforms.Resize(224)
        self.center_crop = torchvision.transforms.CenterCrop(224)
    def preprocess_img(self, images):
        images = self.center_crop(images)
        images = self.resize(images)
        images = self.center_crop(images)
        images = self.normalize(images)
        return images
    def __call__(self, images=[], **kwargs):
        processed_inputs = self.processor(**kwargs)
        processed_inputs["pixel_values"] = self.preprocess_img(images)
        processed_inputs = {key:value.to(self.device) for (key, value) in processed_inputs.items()}
        return processed_inputs

```



Hi @ErwannMillon, thanks for raising this issue! 

Unfortunately, you're right and the gradient flow won't be preserved when passing images through the image processor. This will occur even if the images aren't cast to `PIL.Image.Image` i.e. if `do_resize=False`, as all input images are converted to numpy arrays. This is to ensure all supported inputs (PIL images, and numpy, tensorflow, jax and pytorch arrays) are processed in the same way. 

Training VQGAN CLIP is a great use case for our CLIP models and seems like a good fit for a [research project example](https://github.com/huggingface/transformers/tree/main/examples/research_projects). If you would like to contribute this we'd be very happy to have it added to the repo and review any PRs. 

Great, thanks for getting back to me. Would be happy to work on this in my spare time and submit a PR. 

But just to be clear, would you just be interested in having a VQGAN-CLIP specific research project that works around the issue with the HF Processor class, or a pull request that also modifies this class directly? (for example, with a preserve_gradient or convert_to_pil parameter that would use the torchvision transforms)

For the VQGAN-CLIP, I already have this repo that uses the HF clip model: 
https://github.com/ErwannMillon/Simple-VQGAN-CLIP

I can clean this up some more to get it to the standard of the other projects in the research project examples you sent me, but was just wondering if you would be interested in extending the CLIPProcessor class

> Would be happy to work on this in my spare time and submit a PR.

Great! Excited to have this added to the repo and seeing the PR :) 

> would you just be interested in having a VQGAN-CLIP specific research project that works around the issue with the HF Processor class, or a pull request that also modifies this class directly? 

A specific research project that works around the issue. For the processor class, you can choose what that looks like within the research project i.e. is it completely independent or an extension. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.