Object detection training/fine-tuning for Owl-vit/Owlv2
Author: catalys1
Pull request: no
Labels: name: Good Second Issue

name: Feature request

name: Vision

### Feature request

Currently the Owl-vit models support inference and CLIP-style contrastive pre-training, but don't provide a way to train (or fine-tune) the detection part of the model. According to [the paper](https://arxiv.org/pdf/2205.06230), detection training is similar to Detr.

### Motivation

It would be really awesome to be able to train or fine-tune one of these already-existing open-vocabulary object detection models.

### Your contribution

I may be able to help some with this, not sure at present

Comments:
cc @molbap @qubvel 

Hi @catalys1, thank you so much for your feature request! We agree that the ability to fine-tune Owl-vit/Owlv2 would be a great addition. If you have the time and are interested in contributing, we would love to collaborate with you on this! Your help would be greatly appreciated 🤗 

This PR might be also helpful
 - https://github.com/huggingface/transformers/pull/31828

@qubvel Is https://github.com/huggingface/transformers/pull/34057 maybe the solution here? I would also be happy to see the ability to finetune (some) ZeroShotObjectDetection models.

Hi @daniel-bogdoll, not sure linked PR solved the issue, there is no object detection loss added there.
cc @yonigozlan

Hi @qubvel and team 👋

Thanks so much for the discussion and for flagging this as a "Good Second Issue" — I'd love to contribute to this feature!


Hi all, I found a related discussion https://github.com/huggingface/transformers/issues/28778, but I could not find a complete script or instructions.

Hi @aryanchauhan31 are you actively working on this. I have been working on a PR part-time for the past few weeks because I needed to a way to fine-tune detection models. I should have mentioned this in the discussion weeks ago, my mistake. I just want to know if you are actively working on it

@SuccessMoses No, I was kinda waiting for someone to confirm if i can work on this. Please go ahead and if u need any help do let me know, I would love to collaborate 

> > 嗨，您是否正在积极处理此事。过去几周我一直在兼职做 PR，因为我需要一种方法来微调检测模型。我应该在几周前的讨论中提到这一点，我的错误。我只想知道您是否正在积
> > 你现在能进行owl v2的微调了吗?我改代码一直没改出来
> 
> Hi, are you actively working on this? Over the past few weeks, I’ve been working part-time on the PR because I needed a way to fine-tune the detection model. I should have mentioned this in the discussion a few weeks ago — that was my mistake. I just want to know if you're still actively working on it. Were you able to fine-tune OWL-V2 now? I’ve been trying to modify the code, but still haven’t gotten it to work.


Hello, I am currently doing the same thing and want to fine-tune the effect of owlv2. Have you made any progress?

@qubvel @LysandreJik if no ones working on this can i try my hand at it? 

Hey @0x-avi, you can work on it!

hey sorry something had come up , actively working on this now 

hey @qubvel sorry for the ping i needed a bit of help understanding certain things, please correct me anywhere if im wrong, so i read up the entire owl-vit paper firstly and broke down the tasks , firstly i need to implement their adaption of bipartite matching loss that they copeid from DETR into the modeling file for [owlv2](https://github.com/huggingface/transformers/blob/main/src/transformers/models/owlv2/modeling_owlv2.py) and the [original owl-vit](https://github.com/huggingface/transformers/blob/main/src/transformers/models/owlvit/modeling_owlvit.py) then return those losses from the [objectdetection](https://github.com/huggingface/transformers/blob/a61fc6a0b985548c681df89c15f8053dafe322bf/src/transformers/models/owlv2/modeling_owlv2.py#L174C1-L174C47) model. after this it gets kind of confusing but from what i infer, as each loss returned is for a single time step so i need to handle how the losses are processed over an entire epoch and just call accelerate.Accelerator.backward() to get grads which then i use to update the weights till desired number of epoch, and all of this is already handled by the trainer class and i just need to pass in the object detection model?

Now i had two questions:
1 - the original paper seems to have a custom jax training loop, do i use that or try to adhere to how every other model uses hf trainer?
2 - i tried searching through other relevant PRs for implementing a training loop for a model but i didn't manage to find one, could i possibly get a link to one such closed PR?

@molbap @qubvel still waiting on a reply , sorry for the ping

Is this assigned to someone... If no can I work on this ?? @molbap @qubvel 

@anidoesdev you can work on it!