Why isn't RandomCrop inserting the padding in pytorch?

Brando_Miranda: <p>I am getting odd images despite printing the transform the data is using:</p>
<pre><code class="lang-auto">-- splits[i]='train'
taskset=&lt;learn2learn.data.task_dataset.TaskDataset object at 0x7fbc38345880&gt;
taskset.dataset.dataset.datasets[0].dataset.transform=Compose(
    ToPILImage()
    RandomCrop(size=(84, 84), padding=8)
    ColorJitter(brightness=[0.6, 1.4], contrast=[0.6, 1.4], saturation=[0.6, 1.4], hue=None)
    RandomHorizontalFlip(p=0.5)
    ToTensor()
    Normalize(mean=[0.47214064400000005, 0.45330829125490196, 0.4099612805098039], std=[0.2771838538039216, 0.26775040952941176, 0.28449041290196075])
)
</code></pre>
<p>but the padding is missing:<br>
<a href="https://i.stack.imgur.com/Pqb4B.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/Pqb4B.png" alt="enter image description here" width="690" height="236"></a></p>
<p>but when I use this instead:</p>
<pre><code class="lang-auto">    train_data_transform = Compose([
        RandomResizedCrop((size - padding*2, size - padding*2), scale=scale, ratio=ratio),
        Pad(padding=padding),
        ColorJitter(brightness=0.4, contrast=0.4, saturation=0.4),
        RandomHorizontalFlip(),
        ToTensor(),
        Normalize(mean=mean, std=std),
    ])
</code></pre>
<p>it seems to work:<br>
<a href="https://i.stack.imgur.com/U5Gcw.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/U5Gcw.png" alt="enter image description here" width="633" height="500"></a><br>
why don’t <strong>both</strong> have the 8 and 8 padding on both sides I expect?</p>
<hr>
<p>cross:</p>
<ul>
<li>
<p><a href="https://github.com/learnables/learn2learn/issues/376" class="inline-onebox" rel="noopener nofollow ugc">Do we really need mini-imagenet padding to be 8? What just having it be 84 without the 8? · Issue #376 · learnables/learn2learn · GitHub</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/74482017/why-isnt-randomcrop-inserting-the-padding-in-pytorch" class="inline-onebox" rel="noopener nofollow ugc">machine learning - Why isn't RandomCrop inserting the padding in pytorch? - Stack Overflow</a></p>
</li>
</ul>

Brando_Miranda: <p>I think this is a bug in pytorch: <a href="https://github.com/pytorch/pytorch/issues/89253" class="inline-onebox" rel="noopener nofollow ugc">Padding for RandomCrop does not match description in documentation · Issue #89253 · pytorch/pytorch · GitHub</a></p>

ptrblck: <p>Answered on <a href="https://github.com/pytorch/pytorch/issues/89253#issuecomment-1319583306">GitHib</a>: it’s expected to see the padding in on corner/edge since the padding is applied before the tensor/image is cropped as seen <a href="https://github.com/pytorch/vision/blob/d710f3d1edc06afa244468cb96603ba6dbd4d9d5/torchvision/transforms/transforms.py#L664-L675">here</a>.</p>

Brando_Miranda: <p>Docs are super confusing</p>

ptrblck: <p>I don’t think the docs are confusing and explain which operations are applied as described here:</p><aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/89253#issuecomment-1320708337">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/89253#issuecomment-1320708337" target="_blank" rel="noopener">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/89253#issuecomment-1320708337" target="_blank" rel="noopener">Padding for RandomCrop does not match description in documentation</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2022-11-18" data-time="01:00:50" data-timezone="UTC">01:00AM - 18 Nov 22 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/brando90" target="_blank" rel="noopener">
          <img alt="brando90" src="https://avatars.githubusercontent.com/u/1855278?v=4" class="onebox-avatar-inline" width="20" height="20">
          brando90
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">### 🐛 Describe the bug

RandomCrop does not behave as expected. It should add <span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">padding to all sides according to the docs but only puts it at the bottom:

I discovered &amp; confirmed that by doing it on cifar. But note this NOT what [the docs say for RandomCrop](https://pytorch.org/vision/stable/generated/torchvision.transforms.RandomCrop.html):
&gt; Optional padding on each border of the image. Default is None. If a single int is provided this is used to pad all borders.

it says something very similar [to pad](https://pytorch.org/vision/main/generated/torchvision.transforms.Pad.html#torchvision.transforms.Pad):
&gt; Padding on each border. If a single int is provided this is used to pad all borders.

```
def check_padding_random_crop_cifar_pure_torch():
    # -
    import sys
    print(f'python version: {sys.version=}')
    import torch
    print(f'{torch.__version__=}')
    # -
    from uutils.plot.image_visualization import visualize_pytorch_tensor_img
    from torchvision.transforms import RandomCrop

    # - for determinism
    import random
    random.seed(0)
    import torch
    torch.manual_seed(0)
    import numpy as np
    np.random.seed(0)

    # -
    from pathlib import Path
    root = Path('~/data/').expanduser()
    import torch
    import torchvision
    # - test tensor imgs
    from torchvision.transforms import Resize
    from torchvision.transforms import Pad
    from torchvision.transforms import ToTensor
    from torchvision.transforms import Compose

    # -- see if pad doubles length
    print(f'--- test padding doubles length with Pad(...)')
    transform = Compose([Resize((32, 32)), Pad(padding=4), ToTensor()])
    train = torchvision.datasets.CIFAR100(root=root, train=True, download=True,
                                          transform=transform,
                                          target_transform=lambda data: torch.tensor(data, dtype=torch.long))
    transform = Compose([Resize((32, 32)), Pad(padding=8), ToTensor()])
    test = torchvision.datasets.CIFAR100(root=root, train=True, download=True,
                                         transform=transform,
                                         target_transform=lambda data: torch.tensor(data, dtype=torch.long))
    # - test padding doubles length
    from torch.utils.data import DataLoader
    loader = DataLoader(train)
    x, y = next(iter(loader))
    print(f'{x[0].size()=}')
    assert x[0].size(2) == 32 + 4 * 2
    assert x[0].size(2) == 32 + 8
    visualize_pytorch_tensor_img(x[0], show_img_now=True)
    #
    loader = DataLoader(test)
    x, y = next(iter(loader))
    print(f'{x[0].size()=}')
    assert x.size(2) == 32 + 8 * 2
    assert x.size(2) == 32 + 16
    visualize_pytorch_tensor_img(x[0], show_img_now=True)

    # -- see if RandomCrop also puts the pad
    print(f'--- test RandomCrop indeed puts padding')
    transform = Compose([Resize((32, 32)), RandomCrop(28, padding=8), ToTensor()])
    train = torchvision.datasets.CIFAR100(root=root, train=True, download=True,
                                          transform=transform,
                                          target_transform=lambda data: torch.tensor(data, dtype=torch.long))
    transform = Compose([Resize((32, 32)), RandomCrop(28), ToTensor()])
    test = torchvision.datasets.CIFAR100(root=root, train=True, download=True,
                                         transform=transform,
                                         target_transform=lambda data: torch.tensor(data, dtype=torch.long))
    # - test that the padding is there visually
    from torch.utils.data import DataLoader
    loader = DataLoader(train)
    x, y = next(iter(loader))
    print(f'{x[0].size()=}')
    assert x[0].size(2) == 28
    visualize_pytorch_tensor_img(x[0], show_img_now=True)
    #
    loader = DataLoader(test)
    x, y = next(iter(loader))
    print(f'{x[0].size()=}')
    assert x.size(2) == 28
    visualize_pytorch_tensor_img(x[0], show_img_now=True
```

### Versions

```
(meta_learning) brandomiranda~/diversity-for-predictive-success-of-meta-learning ❯ wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
--2022-11-17 17:00:22--  https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17278 (17K) [text/plain]
Saving to: ‘collect_env.py’

collect_env.py                                                                                                             100%[=======================================================================================================================================================================================================================================================================================================================================&gt;]  16.87K  --.-KB/s    in 0s      

2022-11-17 17:00:23 (48.3 MB/s) - ‘collect_env.py’ saved [17278/17278]

zsh: command not found: #
Collecting environment information...
PyTorch version: 1.9.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.6 (x86_64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.102)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.7 (default, Sep 16 2021, 08:50:36)  [Clang 10.0.0 ] (64-bit runtime)
Python platform: macOS-10.16-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.1
[pip3] pytorch-transformers==1.2.0
[pip3] torch==1.9.1
[pip3] torchaudio==0.12.1
[pip3] torchmeta==1.8.0
[pip3] torchtext==0.13.1
[pip3] torchvision==0.10.1
[conda] mkl                       2021.4.0           hecd8cb5_637  
[conda] mkl-service               2.4.0            py39h9ed2024_0  
[conda] nomkl                     3.0                           0  
[conda] numpy                     1.21.4                   pypi_0    pypi
[conda] numpy-base                1.23.1           py39hbda7086_0  
[conda] pytorch-transformers      1.2.0                    pypi_0    pypi
[conda] torch                     1.9.1                    pypi_0    pypi
[conda] torchaudio                0.12.1                   pypi_0    pypi
[conda] torchmeta                 1.8.0                    pypi_0    pypi
[conda] torchtext                 0.13.1                   pypi_0    pypi
[conda] torchvision               0.10.1                   pypi_0    pypi

```

pytorch forum: https://discuss.pytorch.org/t/why-isnt-randomcrop-inserting-the-padding-in-pytorch/166244
SO: https://stackoverflow.com/questions/74482017/why-isnt-randomcrop-inserting-the-padding-in-pytorch</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Brando_Miranda: <p>as a side note <a class="mention" href="/u/ptrblck">@ptrblck</a> do you think doing this transform makes sense?</p>
<pre><code class="lang-auto">    train_data_transform = Compose([
        RandomResizedCrop((size, size), scale=scale, ratio=ratio),
        RandomCrop(size=size, padding=padding),
        ColorJitter(brightness=0.4, contrast=0.4, saturation=0.4),
        RandomHorizontalFlip(),
        ToTensor(),
        Normalize(mean=mean, std=std),
    ])
</code></pre>

ptrblck: <p>It’s hard to tell if this transformation would work for your use case.<br>
The <code>RandomCrop</code> transformation will basically add the padding border to the already cropped image (coming from <code>RandomResizedCrop</code>) and will then return a new crop in the same shape as the input from the padded image.<br>
Without knowing about your use case and dataset, I would not expect to see a huge difference between this approach and a transformation which drops the <code>RandomCrop</code> transformation, but in any case you should try both and compare their performance.</p>

edgarriba: <p>Alternatively you can check Kornia that includes more functionality with examples, differentiability, serialise the sampled parameters and compute the inverse of the operation.</p>
<p><a href="https://kornia.readthedocs.io/en/latest/augmentation.module.html#kornia.augmentation.RandomCrop" class="onebox" target="_blank" rel="noopener nofollow ugc">https://kornia.readthedocs.io/en/latest/augmentation.module.html#kornia.augmentation.RandomCrop</a></p>