Deformable DETR Finetuning breaks for any dataset
Author: iamsashank09
Pull request: no
Labels: name: bug

### System Info

- GPU: V100
- torch2.6.0+cu126
- transformers 4.57.1

### Who can help?

Hi @yonigozlan  @molbap  @NielsRogge 

Thanks for the awesome work on vision models!

I've been trying to finetune the Deformable DETR models (SenseTime/deformable-detr-with-box-refine-two-stage) for the past few days on a custom object detection dataset using the finetuning DETR notebook suggested in the Docs (https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/Fine_tuning_DetrForObjectDetection_on_custom_dataset_(balloon).ipynb)  and have swapped out the model name where needed, to the one Deformable DETR model mentioned above , and I have constantly been running into errors, two in particular:

> File /libraries/env/lib/python3.11/site-packages/transformers/loss/loss_deformable_detr.py:55, in <listcomp>(.0) 
52 cost_matrix = cost_matrix.view(batch_size, num_queries, -1).cpu() 
54 sizes = [len(v["boxes"]) for v in targets] ---> 55 indices = [linear_sum_assignment(c[i]) for i, c in enumerate(cost_matrix.split(sizes, -1))] 
56 return [(torch.as_tensor(i, dtype=torch.int64), torch.as_tensor(j, dtype=torch.int64)) for i, j in indices] 
\
ValueError: matrix contains invalid numeric entries

(on several forums this was addressed by turning off AMP, which in your example notebook using Trainer, can be done by passing precision = 32)

and when I can get that to work, I am immediately hit by - 

> "/libraries/env/lib/python3.11/site-packages/transformers/loss/loss_for_object_detection.py", line 418, in generalized_box_iou [rank1]: raise ValueError(f"boxes1 must be in [x0, y0, x1, y1] (corner) format, but got {boxes1}") 
[rank1]: ValueError: boxes1 must be in [x0, y0, x1, y1] (corner) format, but got tensor([[nan, nan, nan, nan], 
[rank1]: [nan, nan, nan, nan], 
[rank1]: [nan, nan, nan, nan], 
[rank1]: ...,
[rank1]: [nan, nan, nan, nan], 
[rank1]: [nan, nan, nan, nan], 
[rank1]: [nan, nan, nan, nan]], device='cuda:1') 
Epoch 0: 0%| | 1/4358 [00:03<4:23:53, 0.28it/s, v_num=20, training_loss_step=nan.0]


I for the life of me can't figure out what is going on. I tried the same notebook code with my dataset using the original DETR model (facebook/detr-resnet-50) listed and it works perfectly well. 

For sanity, I went back and tried to run the balloon dataset as-is in the notebook (https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/Fine_tuning_DetrForObjectDetection_on_custom_dataset_(balloon).ipynb), but with the Deformable DETR model and processor and I run into the same errors, kinda proving that my data wasn't the issue. 

Would love to get your help in understanding why the same notebook doesn't work with the Deformable DETR checkpoints I linked above, since it worked perfectly well on the DETR one's. 

Other Env details:
- GPU: V100
- torch2.6.0+cu126
- transformers 4.57.1

The deformable family of models would suit my usecase well and hence have been trying to make it work. 
Thank you, love the work the team puts in and appreciate the effort. 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

The exact steps followed in https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/Fine_tuning_DetrForObjectDetection_on_custom_dataset_(balloon).ipynb

but with the SenseTime/deformable-detr-with-box-refine-two-stage (or other SenseTime/deformable-detr-*) models. 

### Expected behavior

Expecting it to work similar to the DETR finetuning

Comments:
Hi @iamsashank09, I have tried opening the exact steps followed on the notebook link shared but it seems invalid.

I've been trying to reproduce this as we also saw this crop up on our CI and on a cluster machine (while the same branch locally passed). The only difference at that time was `transformers > 4.57`, but I've not been able to figure out exactly why and reproducing seems difficult. I've now seen this on 4.56.2 as well. It passes on some machines and not others. It doesn't seem to be a CUDA problem, as the tests would fail on CPU too.

I don't think it's AMP as I've also had successful runs with 16-mixed. Looking at my diffs between successful and failing runs, one is possibly the float32 matmul precision ('highest' vs 'high'). The other could be a "normal" explosion due to the learning rate being too high. We use `lightning` to handle the training loop, so can't rule out something there.

I was also unable to reproduce the above notebook error with any of the 4.75.x releases using a much simpler script (a single hand constructed COCO sample + forward pass). The only change I made to the official example was to fix up some an API change to `lightning`.

Best guess I have is the model is very sensitive to learning rates. I'm running 4xB200 + batch size 16 + lr 1e-3 backbone 1e-4. If I run the same on a single B200, I get the crash above (and the loss ticks upwards). @iamsashank09 if you drop it to something very low, like 1e-6, does it still blow up? Please share your hyperparameters if you can.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This should be fixed in https://github.com/huggingface/transformers/pull/41549!
There is indeed an overflow problem in pos embeddings with fp16

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing as this should be fixed by https://github.com/huggingface/transformers/pull/41549.
@iamsashank09 @jveitchmichaelis Don't hesitate to reopen if you find that the issue is still present!