Fast image processor
Author: amyeroberts
Pull request: yes
Labels: name: WIP

name: run-slow

# What does this PR do?

Adds a ViTImageProcessorFast class, which uses a torchvision backend to speed up image processor

```py
import requests
from PIL import Image
from transformers import AutoImageProcessor

# Load the image
image = Image.open(requests.get("http://images.cocodataset.org/val2017/000000039769.jpg", stream=True).raw)

# Set use_fast=True to select the fast implementation
image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k", use_fast=True)

outputs = image_processor(images=image, return_tensors="pt")
```

Benchmark comparing the two image processors: 
![2024_06_06_benchmark_fast_image_processor](https://github.com/huggingface/transformers/assets/22614925/3d1f61ec-5423-4b27-b4b0-6c5e92f28030)


```
List of PIL image input
Fast mean time: 0.005302833318710327 std: 0.0007948700795509047
Slow mean time: 0.010124003410339356 std: 0.0002542689804230619
1.9541 times faster

Torch tensor batch input
Fast mean time: 0.0016665704250335694 std: 5.2973509597780026e-05
Slow mean time: 0.05901132440567017 std: 0.0005533502207234909
35.4395 times faster. 35.4395 times faster

List of PIL image input
Fast mean time: 0.0017093017101287842 std: 4.127424976607243e-05
Slow mean time: 0.005274987697601318 std: 6.580574283983749e-05
3.0876 times faster

Torch tensor batch input
Fast mean time: 0.0009776098728179932 std: 1.986684460031034e-05
Slow mean time: 0.005883036375045776 std: 5.80707152345902e-05
6.0196 times faster. 6.0196 times faster
```

<details>
  <summary>Script for replicating benchmark</summary>

```py
import time

import matplotlib.pyplot as plt
import numpy as np
import requests
import torch
from PIL import Image

from transformers import ViTImageProcessorFast, ViTImageProcessor

# Load the image
image = Image.open(requests.get("http://images.cocodataset.org/val2017/000000039769.jpg", stream=True).raw)

image_processor_fast = ViTImageProcessorFast.from_pretrained("google/vit-base-patch16-224-in21k")
image_processor = ViTImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k")

processed_image_fast = image_processor_fast(images=image, do_rescale=True, do_normalize=True, return_tensors="pt")
processed_image = image_processor(images=image, do_rescale=True,  do_normalize=True, return_tensors="pt")

print(processed_image['pixel_values'].dtype)  # torch.float32
print(processed_image_fast['pixel_values'].dtype)  # torch.float32
print((processed_image_fast['pixel_values'] - processed_image['pixel_values']).abs().max())  # tensor(0.)


N_ITERATIONS = 1000
DO_RESCALE = True

# # Benchmarking
fast_times = []
for _ in range(N_ITERATIONS):
    start = time.time()
    image_processor_fast(images=[image, image, image], return_tensors="pt")
    end = time.time()
    fast_times.append(end - start)

slow_times = []
for _ in range(N_ITERATIONS):
    start = time.time()
    image_processor(images=[image, image, image], return_tensors="pt")
    end = time.time()
    slow_times.append(end - start)

fast_times = np.array(fast_times)
slow_times = np.array(slow_times)
print("List of PIL image input")
print("Fast mean time:", np.mean(fast_times), "std:", np.std(fast_times))
print("Slow mean time:", np.mean(slow_times), "std:", np.std(slow_times))
times_faster = np.mean(slow_times/fast_times)
print(f"{times_faster:.4f} times faster")

fig, ax = plt.subplots(2, 2, figsize=(10, 5), sharex=True)

# Plotting
ax[0][0].hist(fast_times, bins=max(N_ITERATIONS // 10, 10), label=f"fast {np.mean(fast_times):.4f} ± {np.std(fast_times):.4f}")
ax[0][0].hist(slow_times, bins=max(N_ITERATIONS // 10, 10), label=f"slow {np.mean(slow_times):.4f} ± {np.std(slow_times):.4f}")
ax[0][0].set_xlabel("Time (s)")
ax[0][0].set_ylabel("Frequency")
ax[0][0].legend()
ax[0][0].set_title(f"List of PIL image input.\n{times_faster:.4f} times faster")

# Benchmarking - torch
image_pt = torch.tensor(np.array(image)).permute(2, 0, 1).unsqueeze(0).float()
# FIXME - how to handle the rescaling?
if not DO_RESCALE:
    image_pt /= 255
images = torch.vstack([image_pt, image_pt, image_pt])

fast_times = []
for _ in range(N_ITERATIONS):
    start = time.time()
    image_processor_fast(images=images, return_tensors="pt", do_rescale=DO_RESCALE)
    end = time.time()
    fast_times.append(end - start)

slow_times = []
for _ in range(N_ITERATIONS):
    start = time.time()
    image_processor(images=images, return_tensors="pt", do_rescale=DO_RESCALE)
    end = time.time()
    slow_times.append(end - start)

fast_times = np.array(fast_times)
slow_times = np.array(slow_times)
print("Torch tensor batch input")
print("Fast mean time:", np.mean(fast_times), "std:", np.std(fast_times))
print("Slow mean time:", np.mean(slow_times), "std:", np.std(slow_times))
times_faster = np.mean(slow_times/fast_times)
print(f"{times_faster:.4f} times faster.")

# Plotting
ax[0][1].hist(fast_times, bins=max(N_ITERATIONS // 10, 10), label=f"fast {np.mean(fast_times):.4f} ± {np.std(fast_times):.4f}")
ax[0][1].hist(slow_times, bins=max(N_ITERATIONS // 10, 10), label=f"slow {np.mean(slow_times):.4f} ± {np.std(slow_times):.4f}")
ax[0][1].set_xlabel("Time (s)")
ax[0][1].set_ylabel("Frequency")
ax[0][1].legend()
ax[0][1].set_title(f"Torch tensor batch input.\n{times_faster:.4f} times faster")

# # Benchmarking PIL images, different sizes
image_small = image.resize((64, 48), Image.Resampling.NEAREST)
image_medium = image.resize((128, 96), Image.Resampling.NEAREST)
image_large = image.resize((256, 192), Image.Resampling.NEAREST)

images = [image_small, image_medium, image_large]

fast_times = []
for _ in range(N_ITERATIONS):
    start = time.time()
    image_processor_fast(images=images, return_tensors="pt")
    end = time.time()
    fast_times.append(end - start)

slow_times = []
for _ in range(N_ITERATIONS):
    start = time.time()
    image_processor(images=images, return_tensors="pt")
    end = time.time()
    slow_times.append(end - start)

fast_times = np.array(fast_times)
slow_times = np.array(slow_times)
print("List of PIL image input")
print("Fast mean time:", np.mean(fast_times), "std:", np.std(fast_times))
print("Slow mean time:", np.mean(slow_times), "std:", np.std(slow_times))
times_faster = np.mean(slow_times/fast_times)
print(f"{times_faster:.4f} times faster")

ax[1][0].hist(fast_times, bins=max(N_ITERATIONS // 10, 10), label=f"fast {np.mean(fast_times):.4f} ± {np.std(fast_times):.4f}")
ax[1][0].hist(slow_times, bins=max(N_ITERATIONS // 10, 10), label=f"slow {np.mean(slow_times):.4f} ± {np.std(slow_times):.4f}")
ax[1][0].set_xlabel("Time (s)")
ax[1][0].set_ylabel("Frequency")
ax[1][0].legend()
ax[1][0].set_title(f"List of PIL different sized images input.\n{times_faster:.4f} times faster")

# Benchmarking - torch different sizes
images = [
    torch.tensor(np.array(image_small)).permute(2, 0, 1).float(),
    torch.tensor(np.array(image_medium)).permute(2, 0, 1).float(),
    torch.tensor(np.array(image_large)).permute(2, 0, 1).float(),
]

fast_times = []
for _ in range(N_ITERATIONS):
    start = time.time()
    image_processor_fast(images=images, return_tensors="pt", do_rescale=DO_RESCALE)
    end = time.time()
    fast_times.append(end - start)

slow_times = []
for _ in range(N_ITERATIONS):
    start = time.time()
    image_processor(images=images, return_tensors="pt", do_rescale=DO_RESCALE)
    end = time.time()
    slow_times.append(end - start)

fast_times = np.array(fast_times)
slow_times = np.array(slow_times)
print("Torch tensor batch input")
print("Fast mean time:", np.mean(fast_times), "std:", np.std(fast_times))
print("Slow mean time:", np.mean(slow_times), "std:", np.std(slow_times))
times_faster = np.mean(slow_times/fast_times)
print(f"{times_faster:.4f} times faster. {times_faster:.4f} times faster")

# Plotting
ax[1][1].hist(fast_times, bins=max(N_ITERATIONS // 10, 10), label=f"fast {np.mean(fast_times):.4f} ± {np.std(fast_times):.4f}")
ax[1][1].hist(slow_times, bins=max(N_ITERATIONS // 10, 10), label=f"slow {np.mean(slow_times):.4f} ± {np.std(slow_times):.4f}")
ax[1][1].set_xlabel("Time (s)")
ax[1][1].set_ylabel("Frequency")
ax[1][1].legend()
ax[1][1].set_title(f"Torch tensor list of different sized images input.\n{times_faster:.4f} times faster")

fig.tight_layout()
fig.savefig("benchmark_fast_image_processor.png")
```

</details>


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28847). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks everyone for your extensive reviews! I think I've addressed everything and have updated the graph on the description to reflect the current benchmarks. 

@qubvel Would be great to have a final review before merging

@qubvel Thanks for your review! 

Regarding your questions: 

> Do we claim that fast and slow image processor's results are somehow equivalent? Probably, they will not match exactly due to some difference in resizing algorithm, but we can test mean absolute error.

I don't think we can claim they'll be exactly the same, but they should be similar. I can add an equivalence test that has a certain tolerance. 

> Worth it to add a slow test to benchmark slow vs fast image processors, just to be sure that fast is really faster and we will not break anything in the future?

Sure - good idea. I can add a test to make sure a single pass (or maybe series of passes as the fast benefits from the cache) are faster for the fast class.