Multi-GPU Training Object Detection
Author: SangbumChoi
Pull request: no
Labels: name: bug

name: Vision

### System Info

- `transformers` version: 4.45.0.dev0
- Platform: Linux-5.4.0-167-generic-x86_64-with-glibc2.35
- Python version: 3.10.14
- Huggingface_hub version: 0.24.5
- Safetensors version: 0.4.4
- Accelerate version: 0.33.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA TITAN RTX

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://github.com/huggingface/transformers/blob/main/examples/pytorch/object-detection/run_object_detection.py

Simply running this script with two or more GPUs

### Expected behavior

While investigating https://github.com/huggingface/transformers/issues/31677 (cc. @SunMarc )
Former issue 
https://github.com/huggingface/transformers/issues/28740
https://github.com/huggingface/transformers/issues/31461
https://github.com/huggingface/transformers/issues/13197
needed to be resolved so I have digged into it.

Found out the issue was no longer related to @NielsRogge comments (which was related to normalize the num_boxes). Now it is related to the targets in `accelerate` and `Trainer` with concat to the multi-gpu circumstances so following error was occured.
```
    indices = [linear_sum_assignment(c[i]) for i, c in enumerate(cost_matrix.split(sizes, -1))]
IndexError: index 2 is out of bounds for dimension 0 with size 2
```

If we do not use `Trainer` class it resolves by follows (no bug)
```
cost_matrix torch.Size([2, 100, 34])
sizes [22, 12]
```
However, when we use `Trainer` class it shows -> which is the circumstances of 3 multi-GPUs concat the individual targets.
```
cost_matrix torch.Size([2, 100, 17])
sizes [3, 2, 3, 1, 3, 5]
```

I am more investigating how to fundamentally fix this problem (not modifying model files just simply add some argument such as `do_train_concat`) but issue this for also other people who might be interested in. (cc. @qubvel )

Comments:
Hi @SangbumChoi, thanks for opening the issue! It will be great to have it fixed 🙂 

Might be also relevant
 - https://github.com/huggingface/transformers/issues/33165
 - https://github.com/huggingface/transformers/issues/32525

#32525 This error is related to torchmetrics of cpu/gpu problem https://github.com/Lightning-AI/torchmetrics/issues/2477

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I'm running into the same issue with code that was working fine before I upgraded a few libraries. Unfortunately I don't remember which libraries I upgraded: does anyone know the library versions that don't cause this issue so I can down/upgrade accordingly?