Different results for PPDocLayoutV3 on CPU and CUDA
Author: 99991
Pull request: no
Labels: name: bug

### System Info

PPDocLayoutV3 seems to have a numerical precision issue that snowballs into larger discrepancies. One cause of those precision issues is the function `_cached_generate_anchors`, which creates a binary mask. The mask may or may not contain the bottom-right boundary, depending on whether the chosen execution device is CPU or CUDA.

The function is duplicated at least 7 times over the transformers repository: https://grep.app/search?q=_cached_generate_anchors

I have not verified whether other models besides PPDocLayoutV3 actually trigger this issue. It requires specific widths and heights, for example 50. I have simplified the code a bit to make the error easier to spot:

The code below will result in the following different masks:

<img width="640" height="480" alt="Image" src="https://github.com/user-attachments/assets/7799753c-847e-4999-bf81-9b7df7e10704" />

The root cause is a bug PyTorch, which has undefined floating point rounding modes: https://github.com/pytorch/pytorch/issues/186618

I would guess that the CPU mask is probably the intended behavior due to symmetry, although I am not entirely certain.

### Who can help?

Perhaps @yonigozlan knows more.

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
# This program creates two masks, one using CPU and one using CUDA. Boundary pixels are handled differently.
import torch

def _cached_generate_anchors(device):
    width = height = 50

    grid_y, grid_x = torch.meshgrid(
        torch.arange(end=height, device=device).float(),
        torch.arange(end=width, device=device).float(),
        indexing="ij",
    )

    grid_xy = torch.stack([grid_x, grid_y], -1)
    grid_xy = grid_xy.unsqueeze(0) + 0.5
    grid_xy[..., 0] /= width # <--- rounding error introduced here
    grid_xy[..., 1] /= height
    anchors = grid_xy
    eps = 1e-2
    valid_mask = ((anchors > eps) * (anchors < 1 - eps)).all(-1, keepdim=True) # <--- rounding error turned into larger error here
    return valid_mask

mask_cuda = _cached_generate_anchors("cuda")
mask_cpu = _cached_generate_anchors("cpu")

if not mask_cpu.equal(mask_cuda.cpu()):
    print("ERROR: Different masks on CPU and GPU")

import matplotlib.pyplot as plt
plt.subplot(1, 2, 1)
plt.title("CUDA")
plt.imshow(mask_cuda.reshape(50, 50).detach().cpu(), cmap="gray")
plt.subplot(1, 2, 2)
plt.title("CPU")
plt.imshow(mask_cpu.reshape(50, 50).detach().cpu(), cmap="gray")
plt.show()
```


### Expected behavior

The masks should be the same on CPU and GPU.

Comments:
Thanks for the detailed report.

I'm investigating this issue locally before proposing any fix. My plan is to:

* locate all duplicated implementations of `_cached_generate_anchors`
* reproduce the CPU/CUDA discrepancy
* trace how the generated mask propagates through PPDocLayoutV3
* determine whether the impact is model-specific or affects all implementations

I'll share findings once I have a clearer picture of the root cause and scope.


## CPU vs CUDA Discrepancy in `_cached_generate_anchors`

I was able to reproduce the CPU vs CUDA discrepancy in `_cached_generate_anchors` using a minimal setup.

For the 50×50 case, I consistently observe a difference in the `valid_mask` (97 mismatched entries), concentrated along the boundary region of the feature map. The behavior is stable across runs, suggesting this is not random noise.

I have only validated the issue at the anchor generation level so far. I have not yet measured downstream effects in `PPDocLayoutV3` or other models due to environment limitations.

Given that this function is duplicated across multiple DETR-style implementations in the repository, it may be worth reviewing cross-device consistency in this specific masking step.

Happy to help further with deeper validation if needed.

@Haris-bin-shakeel Don't waste your tokens on this. This problem has to be solved by someone who actually knows what this function is supposed to do. If you change the tolerance as in the PR, you are not fixing the problem. You are just moving it elsewhere. Also, just consistency is not sufficient. You also have to match the _correct_ implementation.

We need one of the PP-DocLayoutV3 developers to run some images through the original model on their hardware to compute logits, valid_masks and other outputs so we can actually verify that the current implementation on different hardware produces the same values.

Thanks for the clear explanation, that makes sense. A tolerance patch without knowing the intended numerical behaviour would just shift the boundary, not fix it.

Closing this PR. Will leave the reproduction script in the issue thread in case it's useful for whoever does the ground-truth validation.

Hi!
I'm not a PP-DocLayoutV3 developer myself, but here's something that might help the discussions.

From your example, I can see that constructing the grid directly from an equivalent linspace (instead of arange then normalization) gives the same mask (and grid) between CPU and CUDA.

```py
anchors = torch.stack(
    torch.meshgrid(
        torch.linspace(0.5 / height, 1. - 0.5 / height, height, device=device).float(),
        torch.linspace(0.5 / width, 1. - 0.5 / width, width, device=device).float(),
        indexing="ij",
    ), -1
).unsqueeze(0)
```

<img width="543" height="291" alt="Image" src="https://github.com/user-attachments/assets/4132e058-9a42-4634-bac6-221f7262b4c5" />


CPU and CUDA masks are also both equal to the CPU mask in your original implementation (_~note that you mislabeled your plot, it is actually CUDA that is asymmetrical~ fixed_).

I can open a PR to replace the current `_cached_generate_anchors` implementation if you/maintainers think this would be an appropriate fix.

> note that you mislabeled your plot, it is actually CUDA that is asymmetrical

Oops, you are right! Let me fix that real quick.

> if you/maintainers think this would be an appropriate fix

I do not know what the intended behavior is. And even if we knew, it might still make sense to match the "incorrect" behavior if the model has been trained on the incorrect masks.

To determine what the correct behavior should be, we could run a benchmark, but the [PaddleOCR-VL-1.5 paper](https://arxiv.org/pdf/2601.21957) that introduced the PP-DocLayoutV3 model does not seem to benchmark the PP-DocLayoutV3 model individually, only in conjunction with PaddleOCR-VL-1.5-0.9B.

Unfortunately, I do not currently have the time to rebuild their benchmark.

cc @yonigozlan since I think you touched the anchor function in question!

It's a CUDA optimization and expected on Pytorch's side (https://github.com/pytorch/pytorch/issues/186618#issuecomment-4685400381).

@peinguim that approach looks right.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.