How to resize and pad in a torchvision.transforms.Compose()?

nansravn: <p>I’m creating a <code>torchvision.datasets.ImageFolder()</code> data loader, adding <code>torchvision.transforms</code> steps for preprocessing each image inside my training/validation datasets.</p>
<p>My main issue is that <strong>each image from training/validation has a different size</strong> (i.e.: 224x400, 150x300, 300x150, 224x224 etc). Since the classification model I’m training is very sensitive to the shape of the object in the image, I can’t make a simple <code>torchvision.transforms.Resize()</code>, <strong>I need to use padding to maintain the proportion of the objects.</strong></p>
<p>Is there a simple way to add a padding step into a <code>torchvision.transforms.Compose()</code> pipeline (ensuring that every image is 224x224, without cropping the image, only doing a resize and padding)? *<em>each image has a different original shape/size</em></p>
<pre><code class="lang-python">data_transforms = {
    'train': transforms.Compose([
        transforms.Resize((224,224)),
        transforms.Grayscale(num_output_channels=3),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize((224,224)),
        transforms.Grayscale(num_output_channels=3),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

data_dir = "./my_data_dir/"
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}

dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=16,
                                             shuffle=True, num_workers=8)
              for x in ['train', 'val']}
</code></pre>
<p>Reading the <code>torchvision.transforms.Pad()</code> documentation, I’ve understood that I need to know the size of the padding beforehand, and know if it will be applied on left/right or top/bottom, before using this transform.</p>
<p><strong>Is there a simple way to add a step on this <code>transforms.Compose()</code> to infer the image size, so I can get the parameters that I need to configure my <code>torchvision.transforms.Pad()</code>?</strong></p>

satrya-sabeni: <p>I think I did something similar where I kept all ratios by making the each making all the images the width, and height of the biggest image. Then set the image in the center and pad the empty spaces. I don’t know if there’s an function that does this automatically but I did it myself. but I made functions like this in my data class, which works for me. After this you should be able resize and keep the ratios with the padding.</p>
<pre><code class="lang-auto">def get_padding(image):
    max_w = 1203 
    max_h = 1479
    
    imsize = image.size
    h_padding = (max_w - imsize[0]) / 2
    v_padding = (max_h - imsize[1]) / 2
    l_pad = h_padding if h_padding % 1 == 0 else h_padding+0.5
    t_pad = v_padding if v_padding % 1 == 0 else v_padding+0.5
    r_pad = h_padding if h_padding % 1 == 0 else h_padding-0.5
    b_pad = v_padding if v_padding % 1 == 0 else v_padding-0.5
    
    padding = (int(l_pad), int(t_pad), int(r_pad), int(b_pad))
    
    return padding
</code></pre>
<pre><code class="lang-auto">def pad_image(self, image):
        padded_im = pad(image, get_padding(image)) # torchvision.transforms.functional.pad
        return padded_im
</code></pre>
<p>Note: I just wrote this only to make it work, so it’s not optimal or ‘ideal way’ but it works for me.</p>

nansravn: <p>Thanks <a class="mention" href="/u/satrya-sabeni">@satrya-sabeni</a>, based on your recommendation, I’ve prepared the following <code>torchvision.transforms.Compose()</code> pipeline:</p>
<pre><code class="lang-python">from torchvision.transforms.functional import pad
from torchvision import transforms
import numpy as np
import numbers

def get_padding(image):    
    w, h = image.size
    max_wh = np.max([w, h])
    h_padding = (max_wh - w) / 2
    v_padding = (max_wh - h) / 2
    l_pad = h_padding if h_padding % 1 == 0 else h_padding+0.5
    t_pad = v_padding if v_padding % 1 == 0 else v_padding+0.5
    r_pad = h_padding if h_padding % 1 == 0 else h_padding-0.5
    b_pad = v_padding if v_padding % 1 == 0 else v_padding-0.5
    padding = (int(l_pad), int(t_pad), int(r_pad), int(b_pad))
    return padding

class NewPad(object):
    def __init__(self, fill=0, padding_mode='constant'):
        assert isinstance(fill, (numbers.Number, str, tuple))
        assert padding_mode in ['constant', 'edge', 'reflect', 'symmetric']

        self.fill = fill
        self.padding_mode = padding_mode
        
    def __call__(self, img):
        """
        Args:
            img (PIL Image): Image to be padded.

        Returns:
            PIL Image: Padded image.
        """
        return F.pad(img, get_padding(img), self.fill, self.padding_mode)
    
    def __repr__(self):
        return self.__class__.__name__ + '(padding={0}, fill={1}, padding_mode={2})'.\
            format(self.fill, self.padding_mode)

data_transforms = {
    'train': transforms.Compose([
        NewPad(),
        transforms.Resize((224,224)),
        transforms.Grayscale(num_output_channels=3),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        NewPad(),
        transforms.Resize((224,224)),
        transforms.Grayscale(num_output_channels=3),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}
</code></pre>

mcPytorch: <pre><code>import torch.functional as F
F.pad 
</code></pre>
<p>has been moved to</p>
<pre><code>from torchvision.transforms.functional import pad
</code></pre>
<p>as far as I can see.</p>
<p>Could you explain the <code>% 1</code> and <code>+0.5</code> part of your code snippet?<br>
Isn’t <code>% 1</code> always 0 ?</p>
<p><em>EDIT</em>: Wait, I can see it. the <code> / 2</code> division is not returning an integer but a float, which results in 0.5 values for image shape length non divisible by 2</p>

weisunding: <p>This post saved my day, thanks! For simplicity, forget about the odd/even value of the image width/height is OK, the simplified code.</p>
<pre><code class="lang-auto">import torchvision.transforms.functional as F

class SquarePad:
	def __call__(self, image):
		w, h = image.size
		max_wh = np.max([w, h])
		hp = int((max_wh - w) / 2)
		vp = int((max_wh - h) / 2)
		padding = (hp, vp, hp, vp)
		return F.pad(image, padding, 0, 'constant')

# now use it as the replacement of transforms.Pad class
transform=transforms.Compose([
    SquarePad(),
    transforms.Resize(image_size),
    transforms.CenterCrop(image_size),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
])
</code></pre>

MaiHai: <p>Your SquarePad class works perfectly. Thank you!!</p>

MaiAnd: <p>Mmmm, what I cannot fully understand is: the variable image_size is missing; how should I deal with the problem?</p>

1chimaruGin: <p>The above code snippet will pad to the maximum of width or height value of the image.<br>
e.g.<br>
input image - 180x240<br>
resulting image will be - padded 240x240</p>
<p>Original image:<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9002e8f860b0757f0d79dc487b398b74bd6b0a5.jpeg" alt="cat" data-base62-sha1="uXG02UHaIXmFwRogfvqFUHQFcup" width="640" height="480"></p>
<p>Padded image:<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e77857ef7deae9c74f3c718d5b03241461d72b4.jpeg" alt="padcat" data-base62-sha1="8UBFXmvTkYzqcSDXUebWev9xPjC" width="640" height="480"></p>

ntomita: <p>I would extend the <a class="mention" href="/u/weisunding">@weisunding</a> 's code to be more precise as follows.</p>
<pre><code class="lang-auto">import torchvision.transforms.functional as F

class SquarePad:
    def __call__(self, image):
        max_wh = max(image.size)
        p_left, p_top = [(max_wh - s) // 2 for s in image.size]
        p_right, p_bottom = [max_wh - (s+pad) for s, pad in zip(image.size, [p_left, p_top])]
        padding = (p_left, p_top, p_right, p_bottom)
        return F.pad(image, padding, 0, 'constant')

target_image_size = (224, 224)  # as an example
# now use it as the replacement of transforms.Pad class
transform=transforms.Compose([
    SquarePad(),
    transforms.Resize(target_image_size),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
])
</code></pre>
<p>The original code fails to return a square image when either side is odd. The code above guarantees to return a square image by computing residuals (See a line start with <code>p_right</code>). It also removes <code>np</code> for less dependency and <code>CenterCrop</code> as it does not do anything after <code>Resize</code> with the same size.</p>

Samil: <p>F.pad function requires input as torch tensor but it is given PIL Image here. It throws error on me.</p>

krishna.g: <aside class="quote" data-post="11" data-topic="71850">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samil/48/12692_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-resize-and-pad-in-a-torchvision-transforms-compose/71850/11">How to resize and pad in a torchvision.transforms.Compose()?</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    F.pad function requires input as torch tensor but it is given PIL Image here. It throws error on me.
  </blockquote>
</aside>
<p>
Use<br>
<code>import torchvision.transforms.functional as F</code><br>
instead of<br>
<code>import torch.nn.functional as F</code></p>

J_Johnson: <p>Good function but needs a little fix:</p>
<pre><code class="lang-auto">import torchvision.transforms.functional as F

class SquarePad:
	def __call__(self, image):
        s = image.size()
		max_wh = np.max([s[-1], s[-2])
		hp = int((max_wh - s[-1]) / 2)
		vp = int((max_wh - s[-2]) / 2)
		padding = (hp, vp, hp, vp)
		return F.pad(image, padding, 0, 'constant')

</code></pre>
<p>It was cropping on the wrong dim.  Also, this deals with both grayscale and color images. Cheers.</p>

Aziz_Ilyosov: <p>It didn’t  solve the issue</p>

MatR: <p>FWIW, after finding this thread in 2024, it seems that using <a href="https://pytorch.org/vision/stable/generated/torchvision.transforms.CenterCrop.html#torchvision.transforms.CenterCrop" rel="noopener nofollow ugc">CenterCrop</a> and setting the size to your desired output dimensions automatically pads the images and keeps it centered.</p>

goofydoge: <p>This is significantly better than everything mentioned above. Thank you for pointing this out. I suggest others use CenterCrop as well.</p>