DETR: IndexError: Caught IndexError in replica 0 on device 0. IndexError: index 8 is out of bounds for dimension 0 with size 8
Author: michaelgruner
Pull request: no
Labels: name: Good Second Issue

name: Vision

### System Info

- `transformers` version: 4.37.1
- Platform: Linux-6.2.0-32-generic-x86_64-with-glibc2.37
- Python version: 3.10.12
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.26.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Not explicitly, but Trainer is picking up 2 GPUs

### Who can help?

@amyeroberts Hi. I'm getting the error in the title trying to reproduce [this example](https://huggingface.co/docs/transformers/tasks/object_detection). The error is real. I don't know what caused it, but I've narrowed the cause to DETR receiving `BatchSize x NumGPUs` number of targets, but expecting only `BatchSize` which causes the overflow. If I limit the amount of GPUs to 1 (via `CUDA_VISIBLE_DEVICES=0`, for example), it runs ok.

Here's the stack trace:
```
Traceback (most recent call last):
  File "/home/mgruner/cellphones-in-the-wild/./train.py", line 116, in <module>
    trainer.train()
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/trainer.py", line 1869, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/trainer.py", line 2768, in training_step
    loss = self.compute_loss(model, inputs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/trainer.py", line 2791, in compute_loss
    outputs = model(**inputs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 185, in forward
    outputs = self.parallel_apply(replicas, inputs, module_kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 200, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/parallel/parallel_apply.py", line 110, in parallel_apply
    output.reraise()
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/_utils.py", line 694, in reraise
    raise exception
IndexError: Caught IndexError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in _worker
    output = module(*input, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/models/detr/modeling_detr.py", line 1603, in forward
    loss_dict = criterion(outputs_loss, labels)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/models/detr/modeling_detr.py", line 2202, in forward
    indices = self.matcher(outputs_without_aux, targets)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/models/detr/modeling_detr.py", line 2330, in forward
    indices = [linear_sum_assignment(c[i]) for i, c in enumerate(cost_matrix.split(sizes, -1))]
  File "/opt/pyenv/versions/cellphones-in-the-wild/lib/python3.10/site-packages/transformers/models/detr/modeling_detr.py", line 2330, in <listcomp>
    indices = [linear_sum_assignment(c[i]) for i, c in enumerate(cost_matrix.split(sizes, -1))]
IndexError: index 8 is out of bounds for dimension 0 with size 8
```

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Follow this tutorial: https://huggingface.co/docs/transformers/tasks/object_detection


### Expected behavior

I expect the model to train.

Comments:
@amyeroberts can I work on this?

@adityagandhamal Sure! 

same issue

I encountered the same issue, it also reproduces with `TableTransformerForObjectDetection` (only when trying to train with multiple gpus as OP stated).
@michaelgruner have you found any workaround until this gets fixed?

yes, in my case i have several GPUs, but training on a single one of them avoids the problem for me:CUDA_VISIBLE_DEVICES=0 ./train.pyOnly a workaround but got me unstuck.On 23 Apr 2024, at 10:40, Alex Mellado ***@***.***> wrote:﻿
I encountered the same issue, it also reproduces with TableTransformerForObjectDetection (only when trying to train with multiple gpus as OP stated).
@michaelgruner have you found any workaround until this gets fixed?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

@michaelgruner Thanks for the quick reply. I will try with a single gpu then. Would be nice to have this fixed though, since multi-gpu training speeds up the training quite a lot.

Hi,

As explained in the [docs](https://huggingface.co/transformers/model_doc/detr.html):

> If you want to train the model in a distributed environment across multiple nodes, then one should update the num_boxes variable in the DetrLoss class of modeling_detr.py. When training on multiple nodes, this should be set to the average number of target boxes across all nodes, as can be seen in the original implementation here.


Please refer to NielsRogge's answer: https://github.com/huggingface/transformers/issues/13197#issuecomment-902520054