For segmentation how to perform data augmentation in Pytorch?

ahmediqbal: <p>I am using PyTorch for semantic segmentation, But I am facing a problem, because I am use  images , and their  masks/labels . I want to perform data augmentation such as RandomHorizontalFlip, and RandomCrop, etc.</p>
<p>Here is my code, please check and let me know, how I can embed the following operations in the provided code.</p>
<blockquote>
<p>import torchvision.transforms.functional as F</p>
<p>class ToTensor(object):<br>
def <strong>call</strong>(self, sample):<br>
image, label = sample[‘image’], sample[‘label’]<br>
return {‘image’: F.to_tensor(image), ‘label’: F.to_tensor(label)}</p>
<p>my_transform = transforms.Compose([ ToTensor() ])</p>
<p>dataset = Mydataset(image_dir, label_dir, transform = my_transform)</p>
<h1>Print dataset output</h1>
<p>dataset[1]</p>
</blockquote>
<p><strong>Output</strong></p>
<blockquote>
<p>{‘image’: tensor([[[0.0824, 0.0824, 0.0824,  …, 0.0824, 0.0824, 0.0824],<br>
[0.0824, 0.0824, 0.0824,  …, 0.0824, 0.0824, 0.0824],<br>
[0.0824, 0.0824, 0.0824,  …, 0.0824, 0.0824, 0.0824],<br>
…,<br>
[0.0902, 0.0902, 0.0902,  …, 0.0824, 0.0824, 0.0824],<br>
[0.0824, 0.0824, 0.0824,  …, 0.0824, 0.0824, 0.0824],<br>
[0.0745, 0.0745, 0.0745,  …, 0.0824, 0.0824, 0.0824]],</p>
<pre><code>     [[0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
      [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
      [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
      ...,
      [0.0902, 0.0902, 0.0902,  ..., 0.0824, 0.0824, 0.0824],
      [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
      [0.0745, 0.0745, 0.0745,  ..., 0.0824, 0.0824, 0.0824]],

     [[0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
      [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
      [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
      ...,
      [0.0902, 0.0902, 0.0902,  ..., 0.0824, 0.0824, 0.0824],
      [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
      [0.0745, 0.0745, 0.0745,  ..., 0.0824, 0.0824, 0.0824]]]),
</code></pre>
<p>‘label’: tensor([[[0.0824, 0.0824, 0.0824,  …, 0.0824, 0.0824, 0.0824],<br>
[0.0824, 0.0824, 0.0824,  …, 0.0824, 0.0824, 0.0824],<br>
[0.0824, 0.0824, 0.0824,  …, 0.0824, 0.0824, 0.0824],<br>
…,<br>
[0.0902, 0.0902, 0.0902,  …, 0.0824, 0.0824, 0.0824],<br>
[0.0824, 0.0824, 0.0824,  …, 0.0824, 0.0824, 0.0824],<br>
[0.0745, 0.0745, 0.0745,  …, 0.0824, 0.0824, 0.0824]]])}</p>
</blockquote>

ahmediqbal: <p><strong>Please someone answer <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></strong></p>

Usama_Hasan: <p>Hy there, One way to perform transformation on both data and label is to write your own custom Dataset Class.</p>
<pre><code class="lang-auto">
class CustomDataset(Dataset):
    def __init__(self , tensors , transform = None ):
        assert (tensor.size(0) == tensors[0].size(0) for tensor in tensors)
        self.tensors = tensors
        self.transform = transform
    def __getitem__(self,index):
        x = self.tensors[0][index]  
        y = self.tensors[1][index]
        if self.transform:
            x = self.transform(x) 
            y = self.transform(y)      
        return x , y

    def __len__(self):
        return len(self.tensors[0])  

# Flip tensor horizontally
def hflip(tensor):
    tensor = tensor.flip(2)
    return tensor

traindataset = CustomTensorDataset(tensors=(X_train, y_train), transform=vflip)
</code></pre>

21fa417e3fb06e56040c: <p>Hi. You see here use case transform image argumentation</p>
<p>Sample code<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/d481f2d88f16c6a6c1f143629876bf536072921b/references/segmentation/train.py#L29" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/d481f2d88f16c6a6c1f143629876bf536072921b/references/segmentation/train.py#L29" target="_blank" rel="nofollow noopener">pytorch/vision/blob/d481f2d88f16c6a6c1f143629876bf536072921b/references/segmentation/train.py#L29</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="19" style="counter-reset: li-counter 18 ;">
<li>        "voc": ('/datasets01/VOC/060817/', torchvision.datasets.VOCSegmentation, 21),</li>
<li>        "voc_aug": ('/datasets01/SBDD/072318/', sbd, 21),</li>
<li>        "coco": ('/datasets01/COCO/022719/', get_coco, 21)</li>
<li>    }</li>
<li>    p, ds_fn, num_classes = paths[name]</li>
<li>
</li><li>    ds = ds_fn(p, image_set=image_set, transforms=transform)</li>
<li>    return ds, num_classes</li>
<li>
</li><li>
</li><li class="selected">def get_transform(train):</li>
<li>    base_size = 520</li>
<li>    crop_size = 480</li>
<li>
</li><li>    min_size = int((0.5 if train else 1.0) * base_size)</li>
<li>    max_size = int((2.0 if train else 1.0) * base_size)</li>
<li>    transforms = []</li>
<li>    transforms.append(T.RandomResize(min_size, max_size))</li>
<li>    if train:</li>
<li>        transforms.append(T.RandomHorizontalFlip(0.5))</li>
<li>        transforms.append(T.RandomCrop(crop_size))</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Example transform funtions<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/master/references/segmentation/transforms.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/master/references/segmentation/transforms.py" target="_blank" rel="nofollow noopener">pytorch/vision/blob/master/references/segmentation/transforms.py</a></h4>
<pre><code class="lang-py">import numpy as np
from PIL import Image
import random

import torch
from torchvision import transforms as T
from torchvision.transforms import functional as F


def pad_if_smaller(img, size, fill=0):
    min_size = min(img.size)
    if min_size &lt; size:
        ow, oh = img.size
        padh = size - oh if oh &lt; size else 0
        padw = size - ow if ow &lt; size else 0
        img = F.pad(img, (0, 0, padw, padh), fill=fill)
    return img


class Compose(object):
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/vision/blob/master/references/segmentation/transforms.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


PabloVD: <p>Here is what I do for data augmentation in semantic segmentation.</p>
<p>First I define a composed transform such as</p>
<pre><code class="lang-python">transf_aug = tf.Compose([tf.RandomHorizontalFlip(), tf.RandomResizedCrop((height,width),scale=(0.7, 1.0))])
</code></pre>
<p>Then, during the training phase, I apply the transformation at each image and mask. Given that each time <code>transf_aug</code> is applied it is a different random transformation, to ensure that the same transformation is applied to the image and the mask, we do the following trick (based on <a href="https://discuss.pytorch.org/t/pytorch-how-to-use-torchvision-transforms-augmix-with-torch-float32/161586">this comment</a>):</p>
<pre><code class="lang-python">state = torch.get_rng_state()
img = augmentdata(img)
torch.set_rng_state(state)
mask = augmentdata(mask)
</code></pre>
<p>Hope it is useful,<br>
Pablo.</p>