Repository: huggingface/transformers
Pull request: RT-Detr correct 2d positional embeddings for non-square images

# What does this PR do?

This PR solves a subtle bug in how 2d positional embeddings are generated in the RT-Detr architecture. After flattening the meshgrid that generated the 2d positional embeddings, and for non-square images, the positional embeddings no longer encode the correct 2d positional information.

For an image with width=4 and height=3 we get:

```python
import torch 
# Create 2d grid
x = torch.tensor([0, 1, 2, 3])
y = torch.tensor([0, 1, 2])

grid_x, grid_y = torch.meshgrid(x, y, indexing='ij')

print(grid_x)
#tensor([[0, 0, 0],
#        [1, 1, 1],
#        [2, 2, 2],
#        [3, 3, 3]])
print(grid_y)
#tensor([[0, 1, 2],
#        [0, 1, 2],
#        [0, 1, 2],
#        [0, 1, 2]])

# Create image
image = torch.range(0,11).reshape((3,4)).int()
print(image)
#tensor([[ 0,  1,  2,  3],
#        [ 4,  5,  6,  7],
#        [ 8,  9, 10, 11]], dtype=torch.int32)

#Print flattened grid and image
print(grid_x.flatten())
# tensor([0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3])
print(grid_y.flatten())
# tensor([0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2])
print(image.flatten())
# tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
```
Note how the first four pixels 0,1,2,3 get x positions 0,0,0,1 instead of the intended 0,0,0,0.

# Solution
The proposed solution is consistent with the current way that positional embeddings are generated, while generalizing correctly to non-square shapes.

Fixes # (issue)
#41379

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Models:

- vision models: @yonigozlan @molbap @qubvel 


Changed files:
filename: src/transformers/models/d_fine/modeling_d_fine.py
patch: @@ -2064,7 +2064,7 @@ def build_2d_sincos_position_embedding(
     ):
         grid_w = torch.arange(torch_int(width), device=device).to(dtype)
         grid_h = torch.arange(torch_int(height), device=device).to(dtype)
-        grid_w, grid_h = torch.meshgrid(grid_w, grid_h, indexing="ij")
+        grid_w, grid_h = torch.meshgrid(grid_w, grid_h, indexing="xy")
         if embed_dim % 4 != 0:
             raise ValueError("Embed dimension must be divisible by 4 for 2D sin-cos position embedding")
         pos_dim = embed_dim // 4
@@ -2074,7 +2074,7 @@ def build_2d_sincos_position_embedding(
         out_w = grid_w.flatten()[..., None] @ omega[None]
         out_h = grid_h.flatten()[..., None] @ omega[None]
 
-        return torch.concat([out_w.sin(), out_w.cos(), out_h.sin(), out_h.cos()], dim=1)[None, :, :]
+        return torch.concat([out_h.sin(), out_h.cos(), out_w.sin(), out_w.cos()], dim=1)[None, :, :]
 
     def forward(
         self,

filename: src/transformers/models/rt_detr/modeling_rt_detr.py
patch: @@ -1151,7 +1151,7 @@ def build_2d_sincos_position_embedding(
     ):
         grid_w = torch.arange(torch_int(width), device=device).to(dtype)
         grid_h = torch.arange(torch_int(height), device=device).to(dtype)
-        grid_w, grid_h = torch.meshgrid(grid_w, grid_h, indexing="ij")
+        grid_w, grid_h = torch.meshgrid(grid_w, grid_h, indexing="xy")
         if embed_dim % 4 != 0:
             raise ValueError("Embed dimension must be divisible by 4 for 2D sin-cos position embedding")
         pos_dim = embed_dim // 4
@@ -1161,7 +1161,7 @@ def build_2d_sincos_position_embedding(
         out_w = grid_w.flatten()[..., None] @ omega[None]
         out_h = grid_h.flatten()[..., None] @ omega[None]
 
-        return torch.concat([out_w.sin(), out_w.cos(), out_h.sin(), out_h.cos()], dim=1)[None, :, :]
+        return torch.concat([out_h.sin(), out_h.cos(), out_w.sin(), out_w.cos()], dim=1)[None, :, :]
 
     def forward(
         self,

filename: src/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py
patch: @@ -1107,7 +1107,7 @@ def build_2d_sincos_position_embedding(
     ):
         grid_w = torch.arange(torch_int(width), device=device).to(dtype)
         grid_h = torch.arange(torch_int(height), device=device).to(dtype)
-        grid_w, grid_h = torch.meshgrid(grid_w, grid_h, indexing="ij")
+        grid_w, grid_h = torch.meshgrid(grid_w, grid_h, indexing="xy")
         if embed_dim % 4 != 0:
             raise ValueError("Embed dimension must be divisible by 4 for 2D sin-cos position embedding")
         pos_dim = embed_dim // 4
@@ -1117,7 +1117,7 @@ def build_2d_sincos_position_embedding(
         out_w = grid_w.flatten()[..., None] @ omega[None]
         out_h = grid_h.flatten()[..., None] @ omega[None]
 
-        return torch.concat([out_w.sin(), out_w.cos(), out_h.sin(), out_h.cos()], dim=1)[None, :, :]
+        return torch.concat([out_h.sin(), out_h.cos(), out_w.sin(), out_w.cos()], dim=1)[None, :, :]
 
     def forward(
         self,