Fuyu Finetuning Example
Author: ncoop57
Pull request: yes

# What does this PR do?

This PR is about contributing an example for finetuning the recent Adept Fuyu model architecture. It adds the ability to calculate the loss for the model as well as add a new folder for image-text examples. It is currently in DRAFT model as it still has a bug due to the data collator padding and an easy way of performing evaluation. I'm opening the PR in this state so that I can get help.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@molbap would love some help since you initially created the model!

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts

Library:

- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100


Comments:
The issue I am getting right now is that I am unable to pad the image patches correct it seems for batch training. The image processor doesn't seem to be doing it correct.

I thought it would be as simple as modifying variable_size var to false: https://github.com/huggingface/transformers/blob/main/src/transformers/models/fuyu/processing_fuyu.py#L508C1-L508C37

And that seems to patch the image patches correct, but the `image_patches_indices` var doesn't seem to be updated properly

> The issue I am getting right now is that I am unable to pad the image patches correct it seems for batch training. The image processor doesn't seem to be doing it correct.

Yeah, the current image processor seems only outputs batch-size=1 result

Hey! Thanks a lot for wanting to contribute! We usually share links to training scripts like in the ressource section (like [ this one](https://twitter.com/m_olbap/status/1715362812757999991)), but we don't add them to the repo for every new model! @molbap is working on a fix for the image processor see #27007

Heyya @ArthurZucker ! In that case, should I close this PR or should this PR be the first of multi-modal model training examples? Seems like the space is getting super interesting and so it might be nice for people to see example scripts in the official repo for huggingface

Hi Is this working now?


This is working for me, but only with batch size 1. Using 4-bit Lora training this does fit on a 24GB GPU.

Using higher than batch 1 gives the error I reported in: #27255





Hi @ncoop57! It was mentioned before not sure if this PR should be merged here but it's a great resource. We don't have official training scripts, but we updated the `FuyuProcessor` has been updated with batching + attention masks with left-padding, see https://github.com/huggingface/transformers/blob/88832c01c8a962b653874c4ce4ed8df5783ac5cd/src/transformers/models/fuyu/processing_fuyu.py#L335C1-L383C1. Is that factored in for your loss calculation? I'm not 100% sure about the code but I think it would need to be adjusted a bit, such as 

```python
def collate_fn(examples):
    texts = [e[text_column_name] for e in examples]
    images = [e[image_column_name] for e in examples]
    output = processor(
        text=texts,
        images=images,
        padding="max_length",
        truncation=True
    )
    first_non_padded = (output["input_ids"] != tokenizer.pad_token_id).nonzero(as_tuple=False)
    first_non_padded_indices = first_non_padded[:, 1].view(-1, 1)
    labels = torch.full_like(output["input_ids"], -100)
    for i, index in enumerate(first_non_padded_indices):
        labels[i, index:] = output["input_ids"][i, index:]
    output["labels"] = labels
    return output

```
let me know what you think!

@molbap @ArthurZucker I would be pro adding a finetuning script for Fuyu to our examples because we yet have an example for this modality / task. 

Super happy to see more excitement around this example script! Thanks for the loss pointer, I'm looking into it. I had the same error as @Carolinabanana mentioned but only when using the Trainer object, not when doing it myself. I did try to account for padding/image patches because I don't think we want to train on the prediction of the image patch indices, but I did forget to take into account the left padding!

The script doesn't support VQA, does it?

Can it fit in a 80GB card with bfloat16 for tuning? I faced a CUDA memory issue when running `run_fuyu_no_trainer.py` on a single A100 card.

Can anyone provide a train script? Thanks!


Hi @ncoop57 Any update on the script and readiness for PR review? Already looks very good! Let us know if you need any help adding it to the library

AH sadly haven't had a chance to look back into this. Luckily Ill be off all next week so gonna add this to my list to complete and will reach out with help that I need, ty!

Thanks @ncoop57 ! I would love an example training script to try finetuning Fuyu-8B for a domain specific image captioning dataset 

@amyeroberts think I got it fully in and ready for review. @molbap if you have some time, I'd love for you to checkout how I am handling the left padding:

```python
def collate_fn(examples):
        texts = [e[text_column_name] for e in examples]
        images = [e[image_column_name] for e in examples]
        output = processor(
            text=texts, images=images, padding="max_length", truncation=True
        )
        position = (output["input_ids"] == tokenizer.vocab["
"]).nonzero(
            as_tuple=True
        )[0][
            0
        ]  # This gets the index of the first '1' in the tensor by passing the left padding
        output["labels"] = torch.full_like(
            output["input_ids"], -100
        )  # This creates a tensor filled with -100
        output["labels"][position:] = output["input_ids"][position:]
        return output
```

Let me know what else is needed to get this merged in 🤓 

I wasn't able to get the Trainer working so right now this is just with a standard training loop and uses accelerate to handle all the FSDP stuff

@ncoop57 Great! For the quality checks, make sure toe run `make fixup` and resolve any issues it flags locally and push those changes. This should make the CI green. For example, I can see from [the recent CI](https://app.circleci.com/pipelines/github/huggingface/transformers/82174/workflows/f3b64855-fc57-4c9d-b74f-d9d6c8c03241/jobs/1056941) theres a few objects imported by never used in the script. 

@amyeroberts looks like I got all the CI stuff working 🤓 lemme know if any other stuff are needed

maybe one tweak would be renaming the folder to "visual-language-modeling"?

@amyeroberts any updates on this?

Awesome, thanks for the review and recommendations! 🤓 I've been working on finetuning using the https://huggingface.co/datasets/HuggingFaceM4/WebSight dataset, so once I have the model trained I'll make sure to add it here and to the README!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.