GroundingDino - Loss calculation exceptions
Author: Nitaym
Pull request: no

### System Info

transformers==4.40.2
Python 3.10.14
Ubuntu WSL  under Windows 10

### Who can help?

@amyeroberts 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I've been trying to fine tune GroundingDino with transformers' GroundingDinoForObjectDetection. To ease things I've been using batch_size = 1.
(I haven't tried with any other batch sizes)

When running the model, I got this exception:
```
Exception has occurred: RuntimeError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
split_with_sizes expects split_sizes to sum exactly to 2700 (input tensor's size at dimension -1), but got split_sizes=[3]
  File "/home/nitay/.local/lib/python3.10/site-packages/torch/_tensor.py", line 921, in split
    return torch._VF.split_with_sizes(self, split_size, dim)
  File "/home/nitay/.local/lib/python3.10/site-packages/transformers/models/grounding_dino/modeling_grounding_dino.py", line 2723, in forward
    indices = [linear_sum_assignment(c[i]) for i, c in enumerate(cost_matrix.split(sizes, -1))]
  File "/home/nitay/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/nitay/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nitay/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/nitay/.local/lib/python3.10/site-packages/transformers/models/grounding_dino/modeling_grounding_dino.py", line 2866, in forward
    indices = self.matcher(outputs_without_aux, targets)
  File "/home/nitay/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nitay/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/nitay/.local/lib/python3.10/site-packages/transformers/models/grounding_dino/modeling_grounding_dino.py", line 3091, in forward
    loss_dict = criterion(outputs_loss, labels)
  File "/home/nitay/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nitay/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/mnt/folder/main.py", line 84, in train
    outputs = model(input_ids=input_ids, pixel_values=pixel_values, pixel_mask=pixel_mask, labels=labels)
  File "/mnt/folder/main.py", line 98, in <module>
    train()
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main (Current frame)
    return _run_code(code, main_globals, None,
RuntimeError: split_with_sizes expects split_sizes to sum exactly to 2700 (input tensor's size at dimension -1), but got split_sizes=[3]
```

(There were indeed 3 bounding boxes in the label data)

### Expected behavior

Loss should be calculated with no errors

Comments:
Hey @amyeroberts, are you the relevant person for this bug?

I have further questions, if possible:
1. Regarding labels - What should the "class_labels" tensor be filled in? Where should I get the right class indices from? Since this is an open-set detection model, I assume there's not a simple class index dictionary.

2. Is there example code somewhere for fine-tuning this GroundingDino model with huggingface / custom datasets?

Thanks!
Nitay

cc @EduardoPach 

> Hey @amyeroberts, are you the relevant person for this bug?
> 
> 
> 
> I have further questions, if possible:
> 
> 1. Regarding labels - What should the "class_labels" tensor be filled in? Where should I get the right class indices from? Since this is an open-set detection model, I assume there's not a simple class index dictionary.
> 
> 
> 
> 2. Is there example code somewhere for fine-tuning this GroundingDino model with huggingface / custom datasets?
> 
> 
> 
> Thanks!
> 
> Nitay

TL;DR

I will work to fix this during this week :)

Hey, thanks for the opening the issue! The implementation of GroundingDinoLoss is not actually correct and when adding the model I didn't focus that much on making it right as the original repo doesn't have training code or the loss calculation. 

That being said I found an issue in the original repo where authors point to other repos that implement the training for Grounding DINO so I will use that and check with the paper to fix this :)

Thanks @EduardoPach!

I'll be happy to assist as needed. Could you point me to the reference implementations you've mentioned?


Any update @EduardoPach?

> Any update @EduardoPach?

I have added the corrections (haven't created the PR yet) I just need to test them know. I will probably do that during the weekend 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Waiting final approval https://github.com/huggingface/transformers/pull/31828

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

still ongoing...

Any progress on the minimum example on training? Would love to be able to finetune grounding dino on HF.

Should https://github.com/huggingface/transformers/issues/31434
be merged somehow?

@stevenwudi #31828 is very close to being merged in. You'll see on the PR there were just a few small outstanding comments to address so expect to see this available soon! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I see it's completed, but is it merged?