Torchvision.transfors: How to perform identical transform on both image and target?

zeakey: <p>How can I perform an identical transform on both image and target?</p>
<p>For example, in <strong>Semantic segmentation</strong> and <strong>Edge detection</strong> where the input image and target ground-truth are both 2D images, one must perform the same transform on both input image and target ground-truth.</p>

jdhao: <p>A simple search on Google will give you <a href="https://github.com/pytorch/vision/issues/9">this</a></p>

zeakey: <p>Oh, my fault.<br>
Thank you for the tips.</p>

zeakey: <p>Hi jdhao,</p>
<p>Sorry to bother you again.<br>
I’ve checked related issues and prs, but I’m still confused about how to use the identical transform.</p>
<p>So is there any documentation about this part?</p>
<p>Thank you.</p>

jdhao: <p>PyTorch has an <a href="http://pytorch.org/tutorials/beginner/data_loading_tutorial.html">excellent tutorial</a> on data loading. In that tutorial, the author shows how to do transform for both data and target. You can try to mimic the way in the tutorial.</p>

IssamLaradji: <p>The question hasn’t been answered. The provided references don’t show the ideal practice to do identical transforms for both input image and segmentation label…</p>

ptrblck: <p>Alternatively to the functions from the tutorial, you could use torchvision’s functional API.<br>
Here is a small example for an image and the corresponding mask image:</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, image_paths, target_paths, train=True):
        self.image_paths = image_paths
        self.target_paths = target_paths

    def transform(self, image, mask):
        # Resize
        resize = transforms.Resize(size=(520, 520))
        image = resize(image)
        mask = resize(mask)

        # Random crop
        i, j, h, w = transforms.RandomCrop.get_params(
            image, output_size=(512, 512))
        image = TF.crop(image, i, j, h, w)
        mask = TF.crop(mask, i, j, h, w)

        # Random horizontal flipping
        if random.random() &gt; 0.5:
            image = TF.hflip(image)
            mask = TF.hflip(mask)

        # Random vertical flipping
        if random.random() &gt; 0.5:
            image = TF.vflip(image)
            mask = TF.vflip(mask)

        # Transform to tensor
        image = TF.to_tensor(image)
        mask = TF.to_tensor(mask)
        return image, mask

    def __getitem__(self, index):
        image = Image.open(self.image_paths[index])
        mask = Image.open(self.target_paths[index])
        x, y = self.transform(image, mask)
        return x, y

    def __len__(self):
        return len(self.image_paths)
</code></pre>

Oktai15: <p>what is TF in your code?</p>

ptrblck: <p>Ah sorry, it’s:</p>
<pre><code class="lang-auto">import torchvision.transforms.functional as TF
</code></pre>

kelam_goutam: <p>Assuming both Input and ground truth are images. If we can concatenate input and GT along the axis and then pass the concatenated image through torchvision.transforms.RandomHorizontalFlip() [say]. Then it makes sure that the GT is also flipped when the corresponding input is flipped. I am not sure whether it will work or not practically since I have not tried but theoretically makes sense to me.</p>

ptrblck: <p>The current transformations work with <code>PIL.Image</code>s, so that your concatenated image might not be recognized as a valid image. Besides that, it seems to be a good idea.</p>

kelam_goutam: <p>Thanks for the clarification. I did Random Horizontal flip by generating the random number and suppose it is greater than a threshold just do a normal horizontal flip of both input and GT.</p>

yuqli: <p>Just to add on this thread - the linked PyTorch tutorial on picture loading is kind of confusing. The author does both <code>import skimage import io, transform</code>, and <code>from torchvision import transforms, utils</code>.</p>
<p>For <code>transform</code>, the authors uses a resize() function and put it into a customized Rescale class.  For <code>transforms</code>, the author uses the <code>transforms.Compose</code> function to organize two transformations. <strong>But they are from two different modules!</strong></p>
<p>To add to the confusion, torchvision <code>transforms</code> also has its own API called <code>Resize()</code> which is the same name with the one in <code>skimage</code> module.</p>

Preet_Khaturia: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Where can I get the index from?</p>

ptrblck: <p>Do you mean the <code>index</code> in <code>__getitem__</code>?<br>
You can pass the index directly to your <code>Dataset</code>:</p>
<pre><code class="lang-python">data, target = dataset[0]
</code></pre>
<p>or the sampler will do it for you, if you are using a <code>DataLoader</code>.</p>

sherlock: <aside class="quote" data-post="1" data-topic="31981">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/0ea827/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/custom-dataset-resize-and-padding/31981">Custom DataSet Resize and padding</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    I have images in horizontal and vertical orientation. What is the ideal way to resize and pad them to a common size. I tried padding all the images to maximum height and width available but that doesn’t give good results .
  </blockquote>
</aside>


Ilyajob05: <p>You can also use the standard functions from the library torchvision<br>
example:</p>
<pre><code class="lang-auto">from PIL import Image
from torchvision import transforms
from torchvision.transforms import functional as tvF
...
# create custom class transform
class RRC(transforms.RandomResizedCrop):
    def __call__(self, img1, img2):
        assert img1.size == img2.size
        # fix parameter
        i, j, h, w = self.get_params(img1, self.scale, self.ratio)
        # return the image with the same transformation
        return [tvF.resized_crop(img1, i, j, h, w, self.size, self.interpolation), tvF.resized_crop(img2, i, j, h, w, self.size, self.interpolation)]

imgInput = Image.open('input.png')
imgTarget = Image.open('target.png')   

imgInput, imgTarget = RRC(inputSize, scale=(0.5, 1.0), ratio=(5.0/6.0, 6.0/5.0))(imgInput, imgTarget)

</code></pre>

John1231983: <aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="10606">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>import torchvision.transforms.functional as TF</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a>: Can it work on 3D data likes DxHxW where D is depth or number of slice? Your example is 2D image HxW. Thanks</p>

ptrblck: <p><code>torchvision.transforms</code> often rely on <code>PIL</code> as the underlying library, so you would need to transform each slice separately. (At least I’m not aware of <code>PIL</code> methods working on volumetric data)</p>
<p>That being said, it might be faster to write the transformations, e.g. random cropping, manually and apply them directly on the tensor data.</p>

Ilyajob05: <p>This code for multiple layers</p>
<pre><code class="lang-python">
# random resized crop for two images
class RRC(transforms.RandomResizedCrop):
    def __call__(self, imgs):
        """
        Args:
            img (PIL Image): Image to be cropped and resized.

        Returns:
            PIL Image: Randomly cropped and resized image.
        """

        for im in range(1, len(imgs)):
            assert imgs[im].size == imgs[0].size

        i, j, h, w = self.get_params(imgs[0], self.scale, self.ratio)

        for imgCount in range(len(imgs)):
            imgs[imgCount] = tvF.resized_crop(imgs[imgCount], i, j, h, w, self.size, self.interpolation)

        return imgs

</code></pre>