Adding new zero-shot examples
Author: SangbumChoi
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes https://github.com/huggingface/transformers/issues/32459

Following two PR should be merged before merging this example!
https://github.com/huggingface/transformers/pull/31828
https://github.com/huggingface/transformers/pull/31964

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
@qubvel Hi pavel! Is there any standard batch size that hugginface suggest? (In my experience maximum batch size was 2 with 24GB) The reason why I am asking is unlike standard detr, GroundingDino cannot handle batch size 8 for normal single GPU.

@qubvel Hi Pavel this PR is ready for review. However, there is one problem with using evaluation process in `Trainer`.

In order to use the `Trainer` from the transformers we have to get `input_ids` for the post_processing pipeline which is from the tokenizer output. However, transformers `TrainingArgument` `include_input_for_metric` is set to get input variable of first index of class variable which is `pixel_values`. This is hard-coded so I cannot get the input_ids for the input. Also cc @muellerzr.

https://github.com/huggingface/transformers/blob/aefd3e2ae1e2ae5ae555028c0409118bae9e12ee/src/transformers/trainer.py#L3865

There is three ways for this
1. having additional argument to set the name
2. change the order of `forward()` (not preferred)
3. change to main_input_name = getattr(self.model, "input_ids", "input_ids")
WDYT?


Hi @SangbumChoi, thanks for working on this!

> Is there any standard batch size that hugginface suggest? (In my experience maximum batch size was 2 with 24GB) The reason why I am asking is unlike standard detr, GroundingDino cannot handle batch size 8 for normal single GPU.

I think it is fine to have batch size 2 if the model can be trained with such a batch size. In README and a comment we can mention how much memory one would require to run the script and also advise using gradient accumulation, lower precision, and multi-GPU setup (if it is supported).



> However, there is one problem with using evaluation process in Trainer.
In order to use the Trainer from the transformers we have to get input_ids for the post_processing pipeline which is from the tokenizer output. However, transformers TrainingArgument include_input_for_metric is set to get input variable of first index of class variable which is pixel_values. This is hard-coded so I cannot get the input_ids for the input. 

We can probably move these lines
```python
main_input_name = getattr(self.model, "main_input_name", "input_ids")
inputs_decode = self._prepare_input(inputs[main_input_name]) if args.include_inputs_for_metrics else None
```
to a separate method of `Trainer`, for example, `_get_inputs(...)`. Then we will be able to override it in a new `ZeroShotTrainer` class in a script to return required inputs.

agreed re; `get_inputs`

Let me work on pavel's suggestion and ping again. Thanks for the direction!

@qubvel Hi requesting for first review. I will state changes from the original object_detection.py

1. Instead of wandb logging graph, I just added table of GPU consumption (since there is no other architecture to compare)
2. Also as you might know calculating mAP from zero-shot is not accurate. Currently I have made workaround which is the logic that if the output text is directly in `label2id` then match otherwise treat as background object as 0. (We can discuss this if it seems inappropriate). See `def convert_zero_shot_to_coco_format`
3. Added ZeroShotTrainer to override Trainer with `def _get_input_by_name`

Also kudos to @EduardoPach for `backbone_freeze` and `text_backbone_freeze`. Which make the output more stable :)

Before finetune
<img width="1113" alt="Screenshot 2024-08-09 at 2 15 04 PM" src="https://github.com/user-attachments/assets/fc8c8cc6-e426-4f1d-a49b-fa5ec03ec7d2">
After finetune
<img width="1118" alt="Screenshot 2024-08-09 at 2 15 15 PM" src="https://github.com/user-attachments/assets/57bd8ec4-0645-45ca-8bbf-fb9d1ea19119">

@qubvel Sounds good. I can wait until the loss PR get merged.
> Btw, did you try to fine-tune OwlV2?

I didn't aware that we can fine-tune OwlV2, let me see while waiting loss PR!

Hi @qubvel It is now ready to review this new example. For the CI error it says `unauthorized` I'm not sure how to handle this so it would be appreciated to let me know how to fix this.

<img width="1447" alt="Screenshot 2024-09-27 at 4 36 25 PM" src="https://github.com/user-attachments/assets/89ce9d7c-5076-409d-ac38-593d74db259c">


> <img alt="Screenshot 2024-09-27 at 4 36 25 PM" width="1447" src="https://private-user-images.githubusercontent.com/34004152/371439481-89ce9d7c-5076-409d-ac38-593d74db259c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzI4NDkzODQsIm5iZiI6MTczMjg0OTA4NCwicGF0aCI6Ii8zNDAwNDE1Mi8zNzE0Mzk0ODEtODljZTlkN2MtNTA3Ni00MDlkLWFjMzgtNTkzZDc0ZGIyNTljLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDExMjklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMTI5VDAyNTgwNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWY0YmM3MDQ0NjQ5NmFjYTk0YjcxMmM1MTZjNDE0MzcxNzM1NDNlNzc1MTIxYzQ0NmNlYjJhOWMyZDNlODQ5N2YmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.XYK2e6G1_QXaLhqCCIaPNnruKLI8I2vFDsizAdkoiME">

Thanks for share the finetune code.

I test with the code https://github.com/SangbumChoi/transformers on branch grounding_examples.
and only get the lower MAP, it is same as the tensorboard logs in your model [here](https://huggingface.co/danelcsb/grounding-dino-tiny-finetuned-cppe-5-10k-steps-no-trainer/tensorboard)

![image](https://github.com/user-attachments/assets/8a60696b-15e1-4fbb-99c0-34ae4e49ec16)

Looks like I missed something...

@junxnone Thanks for the interest. This is because due to difference between zero-shot to coco converted AP. I have fixed in my code but not updated in here :(

@qubvel @EduardoPach thanks for your work on this so far (and in general for the wider HF ecosystem :) ). Ended up here after digging around for a train script entrypoint to use for [finetuning grounding dino tiny](https://huggingface.co/IDEA-Research/grounding-dino-tiny). Any updates on why this PR is not merged to the main branch? @EduardoPach could you comment on what training script you used to finetune grounding dino tiny to get [this model](https://huggingface.co/EduardoPacheco/grounding-dino-tiny-aquarium-fine-tune)? Thanks in advance!!!

@yonigozlan @molbap anybody interested in this PR?

https://github.com/SangbumChoi/transformers/pull/8#issuecomment-4644749929

Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/27895892765) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: omdet_turbo, owlv2, owlvit