Make `CLIPFeatureExtractor` accept batch of images as `torch.Tensor`.
Author: fcakyon
Pull request: no
Labels: name: Good First Issue

name: Feature extractors

# 🚀 Feature request

Make `CLIPFeatureExtractor` (or any FeatureExtractor in general) accept batch of images as `torch.Tensor`.

## Motivation

Currently batch of images as `torch.Tensor` are not treated as a batch, it has to be a `List[torch.Tensor]` but it is not the case when using native Pytorch DataLoader. Can we update this line so that it accepts batches as `torch.Tensor`. Maybe we can check if the tensor has 4 dimension then assume it is a batch?
 https://github.com/huggingface/transformers/blob/75ae287aecf20a37c232a41e25443a3421a8b5e2/src/transformers/models/clip/feature_extraction_clip.py#L136

@patil-suraj @TobiasNorlund

Comments:
@fcakyon, why was I mentioned here? The only time I edited this file was to add a missing space in an error message.

My apologies @aphedges. Saw your name on PR.

Thanks for the issue!

Since a batch of images is a very common input I think we should support this. Pinging @NielsRogge, @sgugger 

@fcakyon Also note that when resizing, the feature extractors convert the tensors to PIL images, so the most efficient way to use these is to pass a list of PIL images.


@patil-suraj I was using as:

`PIL Images` > `Pytorch DataLoader` > `batch as torch.Tensor` > `CLIPFeatureExtractor` > `CLIPModel`

But in this workflow `CLIPFeatureExtractor` treats `batch as torch.Tensor` as a single image tensor and as a result `CLIPModel` gives error.

Is the intended usage different?

We could support batch as tensors but as explained in the above comment it won't be efficient. So I would recommend passing a list of PIL images.

@patil-suraj Do you still see supporting batched tensors as necessary/beneficial? It seems pretty straightforward (e.g. checking `ndim`), but I thought I'd ask for confirmation before jumping into a draft PR.

I found a way to input a batched tensor by first converting it into a list of tensors. However, it is still not properly:

I think that supporting batched tensors is necessary. In my case, I want to use a feature extractor (ViTFeatureExtractor) during a forward pass, where I may have my tensor ready to use (with gradients). What also happens is that I can't use the Feature Extractors because they convert tensors first into a PIL image, hence loosing the gradients.

The support should be similar to the one used in Pytorch. For instance in https://pytorch.org/vision/main/models/generated/torchvision.models.vgg16_bn.html#torchvision.models.VGG16_BN_Weights.

Please correct me if something is wrong, and let me know if there are other possibilities.

Hi, I see that the issue is still up for grabs.
Is it possible that I can take it up? 
Pinging @NielsRogge and @patil-suraj 

@amyeroberts do you think we should support this?

Hi, I fixed the problem by directly using ViTModel for that same task and ignoring the FeatureExtractor.

A batch of images can now be passed into an image processor after the merging of #21144. 

As @patil-suraj points out, it won't be efficient, as the batch is transformed to a list of images but it is now an accepted input. @joanrod the gradients will still be lost unfortunately too. A related issue #21064 has been raised and @ErwannMillon will be contributing a research example with a custom image processor (I imagine with a similar solution to the one you found).