Repository: kornia/kornia

Issue:
RandomJigsaw not working for non-square images
### Describe the bug

When applying the RandomJigsaw augmentation to non-square inputs, a runtime error occurs:

`shape mismatch: value tensor of shape [1, 3, 1600, 1200] cannot be broadcast to indexing result of shape [1, 3, 1200, 1600]`

The bug seems to be in the apply_tranform method of RandomJigsaw:

```
        input = (
            input.reshape(-1, b, self.flags["grid"][1], h, piece_size_w)
            .permute(0, 1, 2, 4, 3)
            .reshape(-1, b, h, w)
            .permute(0, 1, 3, 2)
            .permute(1, 0, 2, 3)
        )
```

It should actually be
```
        input = (
            input.reshape(-1, b, self.flags["grid"][1], h, piece_size_w)
            .permute(0, 1, 2, 4, 3)
            .reshape(-1, b, w, h)
            .permute(0, 1, 3, 2)
            .permute(1, 0, 2, 3)
        )
```

(Switch h and w in the reshape call)

Code to reproduce:
```
import numpy as np

from PIL import Image
from kornia import augmentation
from kornia.utils import image_to_tensor, tensor_to_image

img_path = "<path_to_arbitrary_image>"

img_size = (1600, 1200)
grid_size = (3, 4)

aug = augmentation.RandomJigsaw(
    grid=grid_size,
    data_keys=["image"],
    p=1.0,
)

img = np.array(Image.open(img_path).convert("RGB").resize(img_size)).astype("float32")

img = image_to_tensor(img, keepdim=False)
img = aug(img)
```

### Reproduction steps

```bash
1. Load image with non square shape
2. Create instance of RandomJigsaw
3. Call RandomJigsaw with the image as input
```


### Expected behavior

Output should be the original image with RandomJigsaw applied.

### Environment

```shell
PyTorch version: 1.13.1+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.8.15 (default, Nov 24 2022, 15:19:38)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.4.0-147-generic-x86_64-with-glibc2.17
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   39 bits physical, 48 bits virtual
CPU(s):                          8
On-line CPU(s) list:             0-7
Thread(s) per core:              1
Core(s) per socket:              8
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           158
Model name:                      Intel(R) Core(TM) i7-9700K CPU @ 3.60GHz
Stepping:                        12
CPU MHz:                         4235.919
CPU max MHz:                     4900,0000
CPU min MHz:                     800,0000
BogoMIPS:                        7200.00
Virtualization:                  VT-x
L1d cache:                       256 KiB
L1i cache:                       256 KiB
L2 cache:                        2 MiB
L3 cache:                        12 MiB
NUMA node0 CPU(s):               0-7
Vulnerability Itlb multihit:     KVM: Vulnerable
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT disabled
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Mitigation; Clear CPU buffers; SMT disabled
Vulnerability Retbleed:          Mitigation; IBRS
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Mitigation; Microcode
Vulnerability Tsx async abort:   Mitigation; Clear CPU buffers; SMT disabled
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] torch==1.13.1+cu116
[pip3] torchaudio==0.13.1+cu116
[pip3] torchmetrics==0.11.0
[pip3] torchvision==0.14.1+cu116
[pip3] torchviz==0.0.2
[conda] numpy                     1.24.1                   pypi_0    pypi
[conda] torch                     1.13.1+cu116             pypi_0    pypi
[conda] torchaudio                0.13.1+cu116             pypi_0    pypi
[conda] torchmetrics              0.11.0                   pypi_0    pypi
[conda] torchvision               0.14.1+cu116             pypi_0    pypi
[conda] torchviz                  0.0.2                    pypi_0    pypi
```


### Additional context

_No response_
RandomJigsaw not working for non-square images
### Describe the bug

When applying the RandomJigsaw augmentation to non-square inputs, a runtime error occurs:

`shape mismatch: value tensor of shape [1, 3, 1600, 1200] cannot be broadcast to indexing result of shape [1, 3, 1200, 1600]`

The bug seems to be in the apply_tranform method of RandomJigsaw:

```
        input = (
            input.reshape(-1, b, self.flags["grid"][1], h, piece_size_w)
            .permute(0, 1, 2, 4, 3)
            .reshape(-1, b, h, w)
            .permute(0, 1, 3, 2)
            .permute(1, 0, 2, 3)
        )
```

It should actually be
```
        input = (
            input.reshape(-1, b, self.flags["grid"][1], h, piece_size_w)
            .permute(0, 1, 2, 4, 3)
            .reshape(-1, b, w, h)
            .permute(0, 1, 3, 2)
            .permute(1, 0, 2, 3)
        )
```

(Switch h and w in the reshape call)

Code to reproduce:
```
import numpy as np

from PIL import Image
from kornia import augmentation
from kornia.utils import image_to_tensor, tensor_to_image

img_path = "<path_to_arbitrary_image>"

img_size = (1600, 1200)
grid_size = (3, 4)

aug = augmentation.RandomJigsaw(
    grid=grid_size,
    data_keys=["image"],
    p=1.0,
)

img = np.array(Image.open(img_path).convert("RGB").resize(img_size)).astype("float32")

img = image_to_tensor(img, keepdim=False)
img = aug(img)
```

### Reproduction steps

```bash
1. Load image with non square shape
2. Create instance of RandomJigsaw
3. Call RandomJigsaw with the image as input
```


### Expected behavior

Output should be the original image with RandomJigsaw applied.

### Environment

```shell
PyTorch version: 1.13.1+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.8.15 (default, Nov 24 2022, 15:19:38)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.4.0-147-generic-x86_64-with-glibc2.17
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   39 bits physical, 48 bits virtual
CPU(s):                          8
On-line CPU(s) list:             0-7
Thread(s) per core:              1
Core(s) per socket:              8
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           158
Model name:                      Intel(R) Core(TM) i7-9700K CPU @ 3.60GHz
Stepping:                        12
CPU MHz:                         4235.919
CPU max MHz:                     4900,0000
CPU min MHz:                     800,0000
BogoMIPS:                        7200.00
Virtualization:                  VT-x
L1d cache:                       256 KiB
L1i cache:                       256 KiB
L2 cache:                        2 MiB
L3 cache:                        12 MiB
NUMA node0 CPU(s):               0-7
Vulnerability Itlb multihit:     KVM: Vulnerable
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT disabled
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Mitigation; Clear CPU buffers; SMT disabled
Vulnerability Retbleed:          Mitigation; IBRS
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Mitigation; Microcode
Vulnerability Tsx async abort:   Mitigation; Clear CPU buffers; SMT disabled
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] torch==1.13.1+cu116
[pip3] torchaudio==0.13.1+cu116
[pip3] torchmetrics==0.11.0
[pip3] torchvision==0.14.1+cu116
[pip3] torchviz==0.0.2
[conda] numpy                     1.24.1                   pypi_0    pypi
[conda] torch                     1.13.1+cu116             pypi_0    pypi
[conda] torchaudio                0.13.1+cu116             pypi_0    pypi
[conda] torchmetrics              0.11.0                   pypi_0    pypi
[conda] torchvision               0.14.1+cu116             pypi_0    pypi
[conda] torchviz                  0.0.2                    pypi_0    pypi
```


### Additional context

_No response_


Discussion:
Hi @Atlas8008, thanks for reporting. You may file up to a PR for the fix.
I will try, if I find some time.
On a similar note(maybe can be fixed together) the example image does not render in docs
<img width="863" alt="Screenshot 2023-08-04 at 10 01 04 PM" src="https://github.com/kornia/kornia/assets/64613009/da3e6f88-f5dd-4de2-a539-19c1bc54bddc">

As mentioned in the issue, swapping `w` and `h` works however, I lose pixel colors when saving the image back, any idea what I'm doing wrong?
<img src="https://github.com/kornia/kornia/assets/64613009/ee9582ac-ead3-464b-b8f9-2d618280c6d5" width="240" 
height="320" /> <img src="https://github.com/kornia/kornia/assets/64613009/5d66cf80-6f73-46b6-8cea-c718174d54c1 " width="240" height="320" />

```
import numpy as np

from PIL import Image
from kornia import augmentation
from kornia.utils import image_to_tensor, tensor_to_image

img_path = "panda.jpeg"

img_size = (1200, 1600)
grid_size = (4, 3)

aug = augmentation.RandomJigsaw(
    grid=grid_size,
    data_keys=["image"],
    p=1,
)

img = np.array(Image.open(img_path).resize(img_size)).astype("float32")

img = image_to_tensor(img, keepdim=False)
img = aug(img)
x = np.array(img[0]) * 255
x = x.astype(np.uint8)
Image.fromarray(np.transpose(x, (1, 2, 0))).save('pic2.png')
```
@pri1311 We will fix the doc string.

As for the colour issue, it might be your saving problem. It works with the following code:

```python
import numpy as np
import torch

from PIL import Image
from kornia import augmentation
from kornia.utils import image_to_tensor, tensor_to_image
import matplotlib.pyplot as plt

img_path = "panda.jpg"

img_size = (1200, 1600)
grid_size = (4, 3)

aug = augmentation.RandomJigsaw(
    grid=grid_size,
    data_keys=["image"],
    p=1,
)

img = np.array(Image.open(img_path).resize(img_size)).astype("float32")

img = image_to_tensor(img, keepdim=False)
img = aug(img)
plt.imshow(img[0].permute(1, 2, 0).to(torch.long))
```
Hi @Atlas8008, thanks for reporting. You may file up to a PR for the fix.
I will try, if I find some time.
On a similar note(maybe can be fixed together) the example image does not render in docs
<img width="863" alt="Screenshot 2023-08-04 at 10 01 04 PM" src="https://github.com/kornia/kornia/assets/64613009/da3e6f88-f5dd-4de2-a539-19c1bc54bddc">

As mentioned in the issue, swapping `w` and `h` works however, I lose pixel colors when saving the image back, any idea what I'm doing wrong?
<img src="https://github.com/kornia/kornia/assets/64613009/ee9582ac-ead3-464b-b8f9-2d618280c6d5" width="240" 
height="320" /> <img src="https://github.com/kornia/kornia/assets/64613009/5d66cf80-6f73-46b6-8cea-c718174d54c1 " width="240" height="320" />

```
import numpy as np

from PIL import Image
from kornia import augmentation
from kornia.utils import image_to_tensor, tensor_to_image

img_path = "panda.jpeg"

img_size = (1200, 1600)
grid_size = (4, 3)

aug = augmentation.RandomJigsaw(
    grid=grid_size,
    data_keys=["image"],
    p=1,
)

img = np.array(Image.open(img_path).resize(img_size)).astype("float32")

img = image_to_tensor(img, keepdim=False)
img = aug(img)
x = np.array(img[0]) * 255
x = x.astype(np.uint8)
Image.fromarray(np.transpose(x, (1, 2, 0))).save('pic2.png')
```
@pri1311 We will fix the doc string.

As for the colour issue, it might be your saving problem. It works with the following code:

```python
import numpy as np
import torch

from PIL import Image
from kornia import augmentation
from kornia.utils import image_to_tensor, tensor_to_image
import matplotlib.pyplot as plt

img_path = "panda.jpg"

img_size = (1200, 1600)
grid_size = (4, 3)

aug = augmentation.RandomJigsaw(
    grid=grid_size,
    data_keys=["image"],
    p=1,
)

img = np.array(Image.open(img_path).resize(img_size)).astype("float32")

img = image_to_tensor(img, keepdim=False)
img = aug(img)
plt.imshow(img[0].permute(1, 2, 0).to(torch.long))
```

Resolving patch:
diff --git a/docs/generate_examples.py b/docs/generate_examples.py
--- a/docs/generate_examples.py
+++ b/docs/generate_examples.py
@@ -157,6 +157,7 @@ def main():
         "RandomSolarize": ((0.2, 0.2), 2, 2019),
         "RandomVerticalFlip": ((), 1, 2020),
         "RandomThinPlateSpline": ((), 1, 2020),
+        "RandomJigsaw": ((), 2, 2020),
     }
 
     # ITERATE OVER THE TRANSFORMS
@@ -171,6 +172,8 @@ def main():
 
         # set seed
         torch.manual_seed(seed)
+        if aug_name == "RandomJigsaw":  # make sure the image is dividable
+            img_in = K.geometry.resize(img_in, (1020, 500))
         # apply the augmentation to the image and concat
         out = aug(img_in)
 
diff --git a/kornia/augmentation/_2d/mix/jigsaw.py b/kornia/augmentation/_2d/mix/jigsaw.py
--- a/kornia/augmentation/_2d/mix/jigsaw.py
+++ b/kornia/augmentation/_2d/mix/jigsaw.py
@@ -13,7 +13,7 @@
 class RandomJigsaw(MixAugmentationBaseV2):
     r"""RandomJigsaw augmentation.
 
-    .. image:: https://raw.githubusercontent.com/kornia/data/main/random_jigsaw.png
+    .. image:: _static/img/RandomJigsaw.png
 
     Make Jigsaw puzzles for each image individually. To mix with different images in a
     batch, referring to :class:`kornia.augmentation.RandomMosic`.
@@ -61,11 +61,14 @@ def apply_transform(
 
         b, c, h, w = input.shape
         perm = params["permutation"]
+        # Note: with a 100x100 image and a grid size of 3x3, it could work if
+        #       we make h = piece_size_h * self.flags["grid"][0] with one pixel loss, then resize to 100 x 100.
+        #       Probably worth to check if we should tolerate such "errorness" or to raise it as an error.
         piece_size_h, piece_size_w = input.shape[-2] // self.flags["grid"][0], input.shape[-1] // self.flags["grid"][1]
         # Convert to C BxN H' W'
         input = (
-            input.unfold(2, piece_size_h, piece_size_w)
-            .unfold(3, piece_size_h, piece_size_w)
+            input.unfold(2, piece_size_h, piece_size_h)
+            .unfold(3, piece_size_w, piece_size_w)
             .reshape(b, c, -1, piece_size_h, piece_size_w)
             .permute(1, 0, 2, 3, 4)
             .reshape(c, -1, piece_size_h, piece_size_w)
@@ -75,7 +78,7 @@ def apply_transform(
         input = (
             input.reshape(-1, b, self.flags["grid"][1], h, piece_size_w)
             .permute(0, 1, 2, 4, 3)
-            .reshape(-1, b, h, w)
+            .reshape(-1, b, w, h)
             .permute(0, 1, 3, 2)
             .permute(1, 0, 2, 3)
         )


Test patch:
diff --git a/test/augmentation/test_augmentation_mix.py b/test/augmentation/test_augmentation_mix.py
--- a/test/augmentation/test_augmentation_mix.py
+++ b/test/augmentation/test_augmentation_mix.py
@@ -338,11 +338,20 @@ def test_p(self, p, device, dtype):
 
 
 class TestRandomJigsaw:
-    def test_smoke(self):
+    def test_smoke(self, device, dtype):
         f = RandomJigsaw(data_keys=["input"])
         repr = "RandomJigsaw(grid=(4, 4), p=0.5, p_batch=1.0, same_on_batch=False, grid=(4, 4))"
         assert str(f) == repr
 
+        # Test square and non-square images.
+        f = RandomJigsaw(grid=(2, 2), p=1.0, data_keys=["input"])
+        input = torch.arange(64, device=device, dtype=dtype).reshape(2, 1, 4, 8)
+        f(input)
+        input = torch.arange(64, device=device, dtype=dtype).reshape(2, 1, 8, 4)
+        f(input)
+        input = torch.arange(32, device=device, dtype=dtype).reshape(2, 1, 4, 4)
+        f(input)
+
     def test_numerical(self, device, dtype):
         torch.manual_seed(22)
         f = RandomJigsaw(grid=(2, 2), p=1.0, data_keys=["input"])