Return type of `ViTFeatureExtractor` does not match `return_tensors` parameter when input is `torch.Tensor` or `PIL.Image.Image`
Author: liyufan
Pull request: no
Labels: name: WIP

name: Feature extractors

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.15.0
- Platform: Ubuntu 16.04
- Python version: 3.9.7
- PyTorch version (GPU?): 1.10.1+cu113
- Tensorflow version (GPU?): not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- T5, BART, Marian, Pegasus, EncoderDecoder: @patrickvonplaten
- Blenderbot, MBART: @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet, BigBird: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten @narsil
- Tokenizers: @LysandreJik
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): ViT

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
Classification on ImageNet

## To reproduce
**Note**: Since `typing.List` is deprecated since python 3.9, I am using `builtins.list` in the following contents.

Steps to reproduce the behavior:

1. Set `do_normalize` and `do_resize` parameter of a `ViTFeatureExtractor`
2. Try different combinations
3. We call like this
```python
from transformers import ViTFeatureExtractor

extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")
img = torch.randn(3, 256, 256)
input = extractor(img, return_tensors="pt")
# or
input = extractor([img, img], return_tensors="pt")
```
As the `__call__` function of `ViTFeatureExtractor` accepts `(PIL.Image.Image, np.ndarray, torch.Tensor, list[PIL.Image.Image], list[np.ndarray], list[torch.Tensor])` as its first parameter, it does't matter whether to call `extractor(img)` or `extractor([img])`. I also tried this:
```jupyter
>>> ViTFeatureExtractor()
ViTFeatureExtractor {
  "do_normalize": true,
  "do_resize": true,
  "feature_extractor_type": "ViTFeatureExtractor",
  "image_mean": [
    0.5,
    0.5,
    0.5
  ],
  "image_std": [
    0.5,
    0.5,
    0.5
  ],
  "resample": 2,
  "size": 224
}

>>> ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")
 ViTFeatureExtractor {
   "do_normalize": true,
   "do_resize": true,
   "feature_extractor_type": "ViTFeatureExtractor",
   "image_mean": [
     0.5,
     0.5,
     0.5
   ],
   "image_std": [
     0.5,
     0.5,
     0.5
   ],
   "resample": 2,
   "size": 224
 }
```
which indicates the two extractors are exactly same. And the results are very weird, shown in table below:
|  `do_resize`  |  `do_normalize`  |  `return_tensors`  |  actual return type  |
|  :----------:  |  :---------------:  |  :----------------:  |  :-----------:  |
|  ✅  |  ✅  |  not specified  |  `list[np.ndarray]`  |
|  ✅  |  ✅  |  "pt"  |  4-D `torch.Tensor` <br> with shape `(B, C, H, W)`  | 
|  ❌  |  ✅  |   not specified |  `list[torch.Tensor]`  |
|  ❌  |  ✅  |  "pt"  |  ValueError  |
|  ✅  |  ❌  |   not specified  |  `list[PIL.Image.Image]`  |
|  ✅  |  ❌  |  "pt"  |  ValueError  |
|  ❌  |  ❌  |   not specified  |  `list[torch.Tensor]`  |
|  ❌  |  ❌  |  "pt"  |  ValueError  |

When return type is `list[torch.Tensor]`, each element in the list is a 3-D `torch.Tensor` with shape `(C, H, W)`. The `ValueError` in table refers to
>ValueError: Unable to create tensor, you should probably activate padding with 'padding=True' to have batched tensors with the same length.

I cannot understand why `__call__` returns a list of `PIL.Image.Image` when `do_normalize = False` and `do_resize = True`, which is the most weird thing. It seems that tensors are only converted to PIL images, resized but no more operations.
According to the doc, the default value of `return_tensors` is `"np"` when not specified. But it does not correspond to the real type of return value when `do_normalize` or `do_resize` is changed. The doc also says
>NumPy arrays and PyTorch tensors are converted to PIL images when resizing, so the most efficient is to pass PIL images.

As we can do resize and normalization using `torchvision.transforms`, there are 3 solutions:
### Solution 1
Do feature extract before using `torch.utils.data.DataLoader` (pass PIL images to `__call__`).
```python
from torchvision.datasets import ImageFolder

dataset = ImageFolder(root)
extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")
extractor.do_normalize = True
extractor.do_resize = True
# X is list of PIL.Image.Image
X = [x for x, _ in dataset]  # This will consume all of your memory
X = extractor(X, return_tensors="pt")["pixel_values"]
# or
for x, _ in dataset:
    x = extractor(x, return_tensors="pt")["pixel_values"].squeeze()  # This is very slow and inefficient
```
### Solution 2
Do feature extract after `torch.utils.data.DataLoader` and `torchvision.datasets.ImageFolder` in a small batch (pass PIL images to `__call__`).
```python
from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader

dataset = ImageFolder(root)
extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")
extractor.do_normalize = True
extractor.do_resize = True
loader = DataLoader(dataset, batch_size=64)
X, y = next(iter(loader))
X = extractor(X, return_tensors="pt")["pixel_values"]
```
This will raise an error:
>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'PIL.Image.Image'>

### Solution 3
#### Warning
It is not recommended to use `ViTFeatureExtractor` along with `torchvision.transforms`. Inappropriate combinations can lead to a decrease in accuracy. Generally, never do resize after normalization. Examples:
* Use `torchvision.transforms` only
```python
import torch
from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader
from torchvision import transforms
from transformers.image_utils import (
    IMAGENET_STANDARD_MEAN,
    IMAGENET_STANDARD_STD
)
from transformers import ViTForImageClassification

img_size = 224

normalize = transforms.Normalize(mean=IMAGENET_STANDARD_MEAN, std=IMAGENET_STANDARD_STD)

tf = transforms.Compose([
    transforms.Resize((img_size, img_size)),
    transforms.ToTensor(), normalize])

dataset = ImageFolder(root=root, transform=tf)
loader = DataLoader(dataset, batch_size=64)
model = ViTForImageClassification.from_pretrained("google/vit-base-patch16-224")
outputs = model(dataset[0][0].unsqueeze(0))
# or
outputs = model(next(iter(loader))[0])
```
* Use `ViTFeatureExtractor` only (not recommended, images are converted to tensors then to PIL images again)
```python
import torch
from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader
from torchvision import transforms
from transformers import ViTFeatureExtractor, ViTForImageClassification

tf = transforms.ToTensor()

dataset = ImageFolder(root=root, transform=tf)
loader = DataLoader(dataset, batch_size=64)
extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")
model = ViTForImageClassification.from_pretrained("google/vit-base-patch16-224")
inputs = extractor(images=dataset[0][0], return_tensors="pt")
outputs = model(**inputs)
```
`DataLoader` cannot be used here as it requires that each tensor in the mini-batch has the same shape.
* Do resize using `torchvision.transforms`, do normalization using `ViTFeatureExtractor`
```python
import torch
from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader
from torchvision import transforms
from transformers import ViTFeatureExtractor, ViTForImageClassification

img_size = 224

tf = transforms.Compose([
    transforms.Resize((img_size, img_size)),
    transforms.ToTensor()])

dataset = ImageFolder(root=root, transform=tf)
loader = DataLoader(dataset, batch_size=64)
extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")
model = ViTForImageClassification.from_pretrained("google/vit-base-patch16-224")
extractor.do_resize = False
inputs = extractor(images=dataset[0][0])
outputs = model(torch.stack(inputs["pixel_values"]))
# or
images = list(next(iter(loader))[0].unbind())
inputs = extractor(images=images)
outputs = model(torch.stack(inputs["pixel_values"]))
```
And I proposed a flexible workaround for those who want to use `ViTModel` or `ViTForImageClassification` with `torch.utils.data.DataLoader`.
```python
import torch
from torch import Tensor
from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader
from torchvision import transforms
from transformers.image_utils import (
    IMAGENET_STANDARD_MEAN,
    IMAGENET_STANDARD_STD
)

img_size = 224

normalize = transforms.Normalize(mean=IMAGENET_STANDARD_MEAN, std=IMAGENET_STANDARD_STD)

tf = transforms.Compose([
    transforms.Resize((img_size, img_size)),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(), normalize])

dataset = ImageFolder(root=root, transform=tf)
loader = DataLoader(dataset, batch_size=64)

extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")

# X must be a 4-D tensor with shape (B, C, H, W)
def feature_extract(X: Tensor, extractor: ViTFeatureExtractor, do_resize=False, do_normalize=False) -> Tensor:
    X = list(X.unbind())
    extractor.do_resize = do_resize
    extractor.do_normalize = do_normalize
    if do_resize:
        if do_normalize:
            batch_feature = extractor(images=X, return_tensors="pt")
            return batch_feature["pixel_values"]
        else:
            batch_feature = extractor(images=X)
            imgs = [transforms.ToTensor()(img) for img in batch_feature["pixel_values"]]
            return torch.stack(imgs)
    else:
        batch_feature = extractor(images=X)
        return torch.stack(batch_feature["pixel_values"])
```
Usage:
```python
from torch import nn

def model_fn(batch: list[Tensor], extractor: ViTFeatureExtractor, model: nn.Module, device: str, criterion: nn.Module) -> tuple[Tensor, Tensor]:
    X, y = batch
    X = feature_extract(X, extractor)
    X, y = X.to(device), y.to(device)

    o = model(X)

    if hasattr(o, "logits"):  # Use for ViTForImageClassification
        outs: Tensor = o.logits
    else:  # Use for other model containing ViTModel
        outs: Tensor = o

    loss = criterion(outs, y)
    preds = outs.argmax(-1)
    accuracy = torch.mean((preds == y).float())

    return loss, accuracy

loss, accuracy = model_fn(next(iter(loader)), extractor, model, "cuda", nn.CrossEntropyLoss())
```
### Update
Tested on official docs, the return type is not functioning either, see table below. I modified on code provided by offical docs:
```python
from transformers import ViTFeatureExtractor, ViTForImageClassification
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

feature_extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")
model = ViTForImageClassification.from_pretrained("google/vit-base-patch16-224")

feature_extractor.do_resize = False
feature_extractor.do_normalize = True
inputs = feature_extractor(images=image, return_tensors="pt")
# or
inputs = feature_extractor(images=[image, image], return_tensors="pt")
```
type of `image` is
```
>>> type(image)
PIL.JpegImagePlugin.JpegImageFile
```
and results are
|  `do_resize`  |  `do_normalize`  |  `return_tensors`  |  actual return type  |
|  :----------:  |  :---------------:  |  :----------------:  |  :-----------:  |
|  ✅  |  ✅  |  not specified  |  `list[np.ndarray]`  |
|  ✅  |  ✅  |  "pt"  |  4-D `torch.Tensor` <br> with shape `(B, C, H, W)`  | 
|  ❌  |  ✅  |   not specified |  `list[np.ndarray]`  |
|  ❌  |  ✅  |  "pt"  | 4-D `torch.Tensor` <br> with shape `(B, C, H, W)` |
|  ✅  |  ❌  |   not specified  |  `list[PIL.Image.Image]`  |
|  ✅  |  ❌  |  "pt"  |  ValueError  |
|  ❌  |  ❌  |   not specified  |  `list[PIL.JpegImagePlugin.JpegImageFile]`  |
|  ❌  |  ❌  |  "pt"  |  ValueError  |

I have only tested when input is `torch.Tensor | list[torch.Tensor]` or `PIL.Image.Image | list[PIL.Image.Image]`. Not sure if other conditions work properly.
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Return type of `ViTFeatureExtractor.__call__` should match `return_tensors`, and there should be no error in table above.
<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Thanks for this table, it beautifully summarizes all possible cases and it should indeed work in any case. Here are my thoughts:

## Single image

If you are providing a single image (or a list containing a single image) and not specifying the `return_tensors` argument/setting it to 'np', it should in any case return a 4D Numpy array of shape `(1, num_channels, height, width)` - feature extractors should always include a batch dimension similar to the tokenizers. Whether or not to place the channel dimension last is a debate, but as all models use channels=first, we should probably default to this.

## Batch of images

In case a batch of images is provided to the feature extractor, it should work (i.e., return a 4D tensor) in any case _unless_ one provides a list of images which don't have the same resolution and one sets `do_resize=False`. In that case, the following error should be returned:

> ValueError: Unable to create tensor, you should probably activate padding with 'padding=True' to have batched tensors with the same length.

Note that the feature extractors use PIL behind the scenes, rather than torchvision (image transformations are defined in [image_utils.py](https://github.com/huggingface/transformers/blob/master/src/transformers/image_utils.py))

We will work on a PR that fixes this.

Edit: updated to make sure channels are first.

@NielsRogge Updated results when input are PIL images.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@liyufan Thanks for reporting these issues. They should now be resolved with the recent refactoring of the feature extractors (now called image processors). Following from your snippet, I ran the below example

```python
from transformers import ViTFeatureExtractor, ViTForImageClassification
from PIL import Image
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

feature_extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224")

settings = [
    {"do_resize": True, "do_normalize": True, "return_tensors": None},
    {"do_resize": True, "do_normalize": True, "return_tensors": "pt"},
    {"do_resize": False, "do_normalize": True, "return_tensors": None},
    {"do_resize": False, "do_normalize": True, "return_tensors": "pt"},
    {"do_resize": True, "do_normalize": False, "return_tensors": None},
    {"do_resize": True, "do_normalize": False, "return_tensors": "pt"},
    {"do_resize": False, "do_normalize": False, "return_tensors": None},
    {"do_resize": False, "do_normalize": False, "return_tensors": "pt"},
]
for kwargs in settings:
    input_single = feature_extractor(images=image, **kwargs)['pixel_values']
    input_batch = feature_extractor(images=[image, image], **kwargs)['pixel_values']
    print("\n" + str(kwargs))
    print(f"Single image    - type: {type(input_single)}, shape: {[x.shape for x in input_single] if isinstance(input_single, list) else input_single.shape}")
    print(f"Batch of images - type: {type(input_batch)}, shape: {[x.shape for x in input_batch] if isinstance(input_batch, list) else input_batch.shape}")
```

And got the following output:
```
{'do_resize': True, 'do_normalize': True, 'return_tensors': None}
Single image    - type: <class 'list'>, shape: [(3, 224, 224)]
Batch of images - type: <class 'list'>, shape: [(3, 224, 224), (3, 224, 224)]

{'do_resize': True, 'do_normalize': True, 'return_tensors': 'pt'}
Single image    - type: <class 'torch.Tensor'>, shape: torch.Size([1, 3, 224, 224])
Batch of images - type: <class 'torch.Tensor'>, shape: torch.Size([2, 3, 224, 224])

{'do_resize': False, 'do_normalize': True, 'return_tensors': None}
Single image    - type: <class 'list'>, shape: [(3, 480, 640)]
Batch of images - type: <class 'list'>, shape: [(3, 480, 640), (3, 480, 640)]

{'do_resize': False, 'do_normalize': True, 'return_tensors': 'pt'}
Single image    - type: <class 'torch.Tensor'>, shape: torch.Size([1, 3, 480, 640])
Batch of images - type: <class 'torch.Tensor'>, shape: torch.Size([2, 3, 480, 640])

{'do_resize': True, 'do_normalize': False, 'return_tensors': None}
Single image    - type: <class 'list'>, shape: [(3, 224, 224)]
Batch of images - type: <class 'list'>, shape: [(3, 224, 224), (3, 224, 224)]

{'do_resize': True, 'do_normalize': False, 'return_tensors': 'pt'}
Single image    - type: <class 'torch.Tensor'>, shape: torch.Size([1, 3, 224, 224])
Batch of images - type: <class 'torch.Tensor'>, shape: torch.Size([2, 3, 224, 224])

{'do_resize': False, 'do_normalize': False, 'return_tensors': None}
Single image    - type: <class 'list'>, shape: [(3, 480, 640)]
Batch of images - type: <class 'list'>, shape: [(3, 480, 640), (3, 480, 640)]

{'do_resize': False, 'do_normalize': False, 'return_tensors': 'pt'}
Single image    - type: <class 'torch.Tensor'>, shape: torch.Size([1, 3, 480, 640])
Batch of images - type: <class 'torch.Tensor'>, shape: torch.Size([2, 3, 480, 640])
```

Disabling either `do_resize` or `do_normalize` should no longer have an effect on the output type and `return_tensors=None` will return a list of numpy arrays. However, if `do_resize=False` and `return_tensors="pt"` an error may be raised if the images are of different shapes as the images can't be batched together. 

Note that `do_resize` and `do_normalize` can now also be passed to the image processor call directly, rather than modifying the instance properties. 

Thanks a lot @amyeroberts!

Closing this issue as it's resolved.