Fuyu: improve image processing
Author: molbap
Pull request: yes

# What does this PR do?

This PR aims at aligning the FuyuImageProcessor class with other vision/language models within transformers. Fuyu model expects a tensor of token ids, a tensor of patch embeddings, and an indexing tensor indicating where to put rows of patch embeddings into the token embeddings, separated by the input ids. Currently the image processor does not separate the steps necessary to achieve this output in the Processor. It also limits the inference size to batches of size 1. It also aims at improving readability and code quality of the processor to possibly enable pipelining later on.

Pending tasks:

- [x] Return a `BatchFeature` with arbitrary batch size
- [x] add `do_rescale`, `do_normalize`, `do_pad` arguments in the `ImageProcessor` constructor
- [x] align patch-ification methods to ViTMAE and possibly pix2struct
- [x] rework and refactor method `process_images_for_model_input`, currently hard to read
- [x] test long images, stretched images, usual processor edge cases
- [x] test images and no text, text and no image in Processor class leveraging tokenizer + ImageProcessor 

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts




Comments:
_The documentation is not available anymore as the PR was closed or merged._

This version of the processor now correctly supports batching, dtype casting, and the left-padded batch generation yields the same results as single-input generation. 

```python
from PIL import Image
import requests
import io
from transformers import FuyuForCausalLM, FuyuProcessor, FuyuImageProcessor, AutoTokenizer
from PIL import Image

pretrained_path = "adept/fuyu-8b"

tokenizer = AutoTokenizer.from_pretrained(pretrained_path, pad_token_id=0)
image_processor = FuyuImageProcessor()
processor = FuyuProcessor(image_processor=image_processor, tokenizer=tokenizer)

text_prompt = "Answer the following DocVQA question based on the image. \n Which is the metro in California that has a good job Outlook?"
jobs_image_url = "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/jobs.png"
jobs_image_pil = Image.open(io.BytesIO(requests.get(jobs_image_url).content))

second_text_prompt = "Answer the following DocVQA question based on the image. \n What if the maximum male life expectancy?"
chart_image_url = "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/chart.png"
chart_image_pil = Image.open(io.BytesIO(requests.get(chart_image_url).content))

third_text_prompt = "Answer the following DocVQA question based on the image. \n What sport is that?"
skate_image_url = "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/skateboard.png"
skate_image_pil = Image.open(io.BytesIO(requests.get(skate_image_url).content))

fourth_text_prompt = "Answer the following DocVQA question based on the image. \n What was the fair amount of paid vacation days in the United Kingdom?"
vacations_image_url = "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/vacation_days_hr.png"
vacations_image_pil = Image.open(io.BytesIO(requests.get(vacations_image_url).content)).convert('RGB')

texts = [text_prompt, second_text_prompt, third_text_prompt, fourth_text_prompt]
images = [jobs_image_pil, chart_image_pil, skate_image_pil, vacations_image_pil]

model_inputs = processor(text=texts, images=images).to('cuda')


model = FuyuForCausalLM.from_pretrained(pretrained_path, device_map='auto')

generation = processor.tokenizer.batch_decode(model.generate(
    **model_inputs, max_new_tokens=10)[:, -10:], skip_special_tokens=True)

single_generations = ['Los Angeles', '80.7',
                      'skateboarding', '28']


for single_generation, batched_generation in zip(single_generations, generation):
    answer = batched_generation.split('\x04 ', 1)[1] if '\x04' in batched_generation else ''
    assert (single_generation == answer)

```

I think current version of image processing and tokenization does not support the usage sample code in the original release, right?

```
from transformers import FuyuProcessor, FuyuForCausalLM
from PIL import Image

# load model and processor
model_id = "adept/fuyu-8b"
processor = FuyuProcessor.from_pretrained(model_id)
model = FuyuForCausalLM.from_pretrained(model_id, device_map="cuda:0")

# prepare inputs for the model
text_prompt = "Generate a coco-style caption.\n"
image_path = "bus.png"  # https://huggingface.co/adept-hf-collab/fuyu-8b/blob/main/bus.png
image = Image.open(image_path)

inputs = processor(text=text_prompt, images=image, return_tensors="pt")
for k, v in inputs.items():
    inputs[k] = v.to("cuda:0")

# autoregressively generate text
generation_output = model.generate(**inputs, max_new_tokens=7)
generation_text = processor.batch_decode(generation_output[:, -7:], skip_special_tokens=True)
assert generation_text == ['A bus parked on the side of a road.']

```

I am trying to run the above code, and the error occurs that the inputs['image_patches'] is now a list and cannot be put to device.

I suggested that either you can also support this type of processing or you can directly update the sample code on the huggingface release page in [link](https://huggingface.co/adept/fuyu-8b)

Hi,

I've updated the code snippet on the model card, it works for me as expected (note that you need to install Transformers from the main branch: pip install -q git+https://github.com/huggingface/transformers.git)

Have there been any updates to this? Still running into the same issue. Thanks!

Hello @cyrilzakka 👋 I've read the thread and according to Niels the sample code in the model card should work, can you please give more details about the issue you are facing? Thank you! :) 

Hey @pcuenca! Sorry for the trouble but having issues running Fuyu inference on a multi-GPU (4x25GB GPUs) setup: `RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cuda:3)`: 
```
from transformers import FuyuProcessor, FuyuForCausalLM
from PIL import Image
import requests
import torch

# load model and processor
model_id = "adept/fuyu-8b"
processor = FuyuProcessor.from_pretrained(model_id)
model = FuyuForCausalLM.from_pretrained(model_id, device_map="auto", torch_dtype=torch.bfloat16)

# # prepare inputs for the model
text_prompt = "What do you see in the image?\n"
image = Image.open("/home/cyril/Downloads/image.jpg").convert("RGB")

inputs = processor(text=text_prompt, images=image, return_tensors="pt").to('cuda')
generation_output = model.generate(**inputs, max_new_tokens=7)
```

> Hey @pcuenca! Sorry for the trouble but having issues running Fuyu inference on a multi-GPU (4x25GB GPUs) setup: `RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cuda:3)`:
> 
> ```
> from transformers import FuyuProcessor, FuyuForCausalLM
> from PIL import Image
> import requests
> import torch
> 
> # load model and processor
> model_id = "adept/fuyu-8b"
> processor = FuyuProcessor.from_pretrained(model_id)
> model = FuyuForCausalLM.from_pretrained(model_id, device_map="auto", torch_dtype=torch.bfloat16)
> 
> # # prepare inputs for the model
> text_prompt = "What do you see in the image?\n"
> image = Image.open("/home/cyril/Downloads/image.jpg").convert("RGB")
> 
> inputs = processor(text=text_prompt, images=image, return_tensors="pt").to('cuda')
> generation_output = model.generate(**inputs, max_new_tokens=7)
> ```
same here


cc @SunMarc maybe a device placement issue, anyway we need the full traceback @oops343 not the `same here` traceback 😈 

The code I used :
```
from transformers import FuyuForCausalLM, FuyuProcessor, FuyuImageProcessor, AutoTokenizer
from PIL import Image
import torch
torch.manual_seed(1234)

# load model and processor
model_id = "adept/fuyu-8b"

tokenizer = AutoTokenizer.from_pretrained(model_id, pad_token_id=0)
image_processor = FuyuImageProcessor()
processor = FuyuProcessor(image_processor=image_processor, tokenizer=tokenizer)

model = FuyuForCausalLM.from_pretrained(model_id, device_map="auto",torch_dtype=torch.float16).eval()

def baseline_for_fuyu(model, processor, batch):
    img_path = batch['img_path'][0]
    text = batch['text'][0]

    query = f"Given the meme, with the text [{text}] accompanied by the image, is this meme {adj}?\n."
    image = Image.open(img_path)
    if image.mode != "RGB":
        image = image.convert("RGB")
    inputs = processor(text=[query], images=[image]).to(model.device)
    with torch.no_grad():  
        generation_output = model.generate(**inputs, max_new_tokens=20)
        generation_text = processor.batch_decode(generation_output[:, -20:], skip_special_tokens=True)
        print("Generated text: ", generation_text[0].split('\x04', 1)[-1])
        torch.cuda.empty_cache()
        return generation_text[0].split('\x04', 1)[-1]
```
Traceback here: 

> Traceback (most recent call last):
>   File "/public/home/baseline/fuyu_baseline.py", line 151, in <module>
>     response = baseline_for_fuyu(model, processor, batch)
>   File "/public/home/baseline/fuyu_baseline.py", line 142, in baseline_for_fuyu
>     generation_output = model.generate(**inputs, max_new_tokens=20)
>   File "/public/home/.conda/envs/baseEnv/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
>     return func(*args, **kwargs)
>   File "/public/home/.conda/envs/baseEnv/lib/python3.9/site-packages/transformers/generation/utils.py", line 1527, in generate
>     result = self._greedy_search(
>   File "/public/home/.conda/envs/baseEnv/lib/python3.9/site-packages/transformers/generation/utils.py", line 2411, in _greedy_search
>     outputs = self(
>   File "/public/home/.conda/envs/baseEnv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
>     return forward_call(*args, **kwargs)
>   File "/public/home/.conda/envs/baseEnv/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
>     output = old_forward(*args, **kwargs)
>   File "/public/home/.conda/envs/baseEnv/lib/python3.9/site-packages/transformers/models/fuyu/modeling_fuyu.py", line 296, in forward
>     inputs_embeds = self.gather_continuous_embeddings(
>   File "/public/home/.conda/envs/baseEnv/lib/python3.9/site-packages/transformers/models/fuyu/modeling_fuyu.py", line 207, in gather_continuous_embeddings
>     output_embeddings[batch_idx, dst_indices] = continuous_embeddings[batch_idx][src_indices]
> RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cuda:1)


Sry about the simple traceback lol
This happened with 2 x RTX3090 GPU, CUDA version 11.7, transformers 4.39.0, torch 2.0.1  
@ArthurZucker  please check , I think this won't happen for the other transformer models I'm using, with the same .to(model.device)

@oops343 would it be possible to open a new issue for this?

@NielsRogge  sure, pls check