RTDetrLoss crashing with "ValueError: cost matrix is infeasible"
Author: m-matthias
Pull request: no
Labels: name: bug

### System Info

 `transformers` version: 5.13.0.dev0
- Platform: Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 1.7.1
- Safetensors version: 0.8.0
- Accelerate version: 1.13.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.8.0+cu128 (CUDA)
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: yes
- GPU type: NVIDIA RTX 500 Ada Generation Laptop GPU

### Who can help?

@yonigozlan 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

When training a model which uses RTDetrHungarianMatcher from loss_rt_detr.py on a dataset with a large number of small boxes per image (~100) and using AMP, the training crashes after a few epochs with the following error:
``` 
File "/.../transformers/loss/loss_rt_detr.py", line 118, in forward
    indices = [linear_sum_assignment(c[i]) for i, c in enumerate(cost_matrix.split(sizes, -1))] 
ValueError: cost matrix is infeasible
```

### Expected behavior

Maybe a possible solution could be to replace infinite values in the cost matrix with large numbers, e.g. as it was done as a fix for a similar bug in another model (see [here](https://github.com/huggingface/transformers/pull/27897/changes))

Comments:
Hi @m-matthias, I'd like to work on this — could I be assigned?

I can reproduce the failure mode: in `RTDetrHungarianMatcher.forward`, when `use_focal_loss` is enabled, `pos_cost_class`/`neg_cost_class` involve `log()` terms that can overflow to `inf` under AMP (fp16), and unlike Mask2Former, the final `cost_matrix` is passed to `scipy.optimize.linear_sum_assignment` without any guard — hence `ValueError: cost matrix is infeasible`.

Proposed fix, following the pattern already accepted for Mask2Former in #27897:

```python
# eliminate infinite values in cost_matrix to avoid the error ``ValueError: cost matrix is infeasible``
cost_matrix = torch.minimum(cost_matrix, torch.tensor(1e10))
cost_matrix = torch.maximum(cost_matrix, torch.tensor(-1e10))
```

applied in `src/transformers/loss/loss_rt_detr.py` just before the `linear_sum_assignment` call, plus a unit test that feeds a cost matrix containing `inf` through the matcher. The same unguarded pattern also exists in `loss_for_object_detection.py` and `loss_grounding_dino.py` — happy to include those in the same PR or keep the scope to RT-DETR, whichever you prefer.

Thanks for reporting the issue @m-matthias!

This is a typical detr loss problem. Depending on the codebase there are various approaches to solve this issue. In Mask2Former we have:

https://github.com/huggingface/transformers/blob/0ca360bcba153c72cf00b47933d9e0761d38cd07/src/transformers/models/mask2former/modeling_mask2former.py#L470-L472

Others default to `torch.nan_to_num(cost_matrix, 1.0)` (DEIM, D-FINE, RT-DETRv4, LT-DETR):
https://github.com/Intellindust-AI-Lab/DEIMv2/blob/8241e8d79fd1225694445aec347fa39c708cc7e8/engine/deim/matcher.py#L126

And RF-DETR sets all inf/nan values to `max(cost_matrix) + max(abs(cost_matrix)) + 1`:
https://github.com/roboflow/rf-detr/blob/e6d4bdf5a2fee6b03e2477036156edf16d2b8605/src/rfdetr/models/matcher.py#L119-L122

The transformers RF-DETR version replaces all values with `torch.finfo(cost_matrix.dtype).max`:
https://github.com/huggingface/transformers/blob/0ca360bcba153c72cf00b47933d9e0761d38cd07/src/transformers/loss/loss_rf_detr.py#L266-L268

Personally, I think the last version is the clearest and most robust option. If any cost is -inf/+inf/nan we probably don't want to match it, so setting it to the max possible value is a good option. I prefer it over the first option (Mask2Former) which sets nan to 0  and -inf to -1e10 which makes these values possible or even highly likely to be matched as typical ranges for cost matrix values are in `[-1000, +1000]`. So overall I would recommend to go with:
```
max_value = torch.finfo(cost_matrix.dtype).max
cost_matrix = torch.nan_to_num(cost_matrix, nan=max_value, posinf=max_value, neginf=max_value)
```

CC @molbap 

@guarin Thank you for the explanation, I also think the transformers RF-DETR version is the best solution. @AyushDas4890  what do you think?

Thank you @guarin for the detailed comparison of the approaches, and @m-matthias for the follow-up.

I agree that the RF-DETR-style guard is the most robust option: invalid entries should never become attractive matches (which clamping `-inf` to `-1e10` would allow, since typical cost values are orders of magnitude smaller), and it also covers `NaN`, which the clamping approach missed.

I have updated #47016 accordingly:

```python
max_value = torch.finfo(cost_matrix.dtype).max
cost_matrix = torch.nan_to_num(cost_matrix, nan=max_value, posinf=max_value, neginf=max_value)
```

The regression test has also been extended to cover the `NaN` case in addition to the two `inf` saturation cases. Please let me know if anything else is needed.

#47016 handles this for RT-DETR — thanks @AyushDas4890. While looking at it, I noticed the same unguarded `linear_sum_assignment(cost_matrix)` pattern still exists in two sibling matchers that #47016 doesn't touch:

- `HungarianMatcher` in `loss/loss_for_object_detection.py` (DETR / Deformable DETR family)
- `GroundingDinoHungarianMatcher` in `loss/loss_grounding_dino.py`

Both raise the same `ValueError` when the cost matrix contains `inf`/`NaN`. I'm opening a follow-up PR that applies the same `torch.finfo(cost_matrix.dtype).max` guard (@guarin's suggestion above) to both, with a regression test for each. The pattern is also present in `loss_deformable_detr.py` and `loss_lw_detr.py` — happy to extend the guard there in the same PR if you'd prefer a single sweep.


Sounds good @MutugiD — thanks for extending the sweep to the other matchers. The same `torch.finfo(cost_matrix.dtype).max` guard should drop in cleanly for `HungarianMatcher` and `GroundingDinoHungarianMatcher`. Happy to review once your PR is up.

Thanks @AyushDas4890! The PR is already up — #47048. It applies the `torch.finfo(cost_matrix.dtype).max` guard to both `HungarianMatcher` and `GroundingDinoHungarianMatcher`, with a regression test for each (each one fails on `main` with the `ValueError` and passes with the guard). CI is green.

I kept the scope to the two matchers flagged here; the same pattern is also in `loss_deformable_detr.py` and `loss_lw_detr.py`, so I'm glad to fold those into the same PR if a reviewer would prefer a single sweep. Reviews welcome — cc @guarin @yonigozlan.