Normalization in custom Dataset class

username: soloupis
cooked: <p>Hello fellow Pytorchers,</p>
<p>I am trying to add normalization to the custom Dataset class Pytorch provides inside <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="noopener nofollow ugc">this</a> tutorial.</p>
<p>The problem is that it gives always the same error:</p>
<blockquote>
<p>TypeError: tensor is not a torch image.</p>
</blockquote>
<p>As you can see inside ToTensor() method it returns:</p>
<blockquote>
<p>return {‘image’: torch.from_numpy(image),‘masks’: torch.from_numpy(landmarks)} so I think it returns a tensor already.</p>
</blockquote>
<p>I give you my code:</p>
<pre><code class="lang-auto">class Rescale(object):
    """Rescale the image in a sample to a given size.

    Args:
        output_size (tuple or int): Desired output size. If tuple, output is
            matched to output_size. If int, smaller of image edges is matched
            to output_size keeping aspect ratio the same.
    """

    def __init__(self, output_size):
        assert isinstance(output_size, (int, tuple))
        self.output_size = output_size

    def __call__(self, sample):
        image, landmarks = sample['image'], sample['masks']

        h, w = image.shape[:2]
        if isinstance(self.output_size, int):
            if h &gt; w:
                new_h, new_w = self.output_size * h / w, self.output_size
            else:
                new_h, new_w = self.output_size, self.output_size * w / h
        else:
            new_h, new_w = self.output_size

        new_h, new_w = int(new_h), int(new_w)

        img = transform.resize(image, (new_h, new_w))

        # h and w are swapped for landmarks because for images,
        # x and y axes are axis 1 and 0 respectively
        #landmarks = landmarks

        return {'image': img, 'masks': landmarks}


class ToTensor(object):
    """Convert ndarrays in sample to Tensors."""

    def __call__(self, sample):
        image, landmarks = sample['image'], np.array(sample['masks'])

        # swap color axis because
        # numpy image: H x W x C
        # torch image: C X H X W
        image = image.transpose(2,0,1)
        return {'image': torch.from_numpy(image),'masks': torch.from_numpy(landmarks)}
      
transformed_train_dataset = MasksTrainDataset(csv_file='pneumo_input/train/train-rle.csv',
                                           root_dir='pneumo_input/train/images/256/dicom/',
                                           transform=transforms.Compose([
                                               Rescale(224),
                                               ToTensor(),
                                               transforms.Normalize(mean=[0.5, 0.5, 0.5],std=[0.5, 0.5, 0.5])
                                           ]))

for i in range(len(transformed_train_dataset)):
    sample = transformed_train_dataset[i]

    print(i, sample['image'].size(), sample['masks'])

    if i == 3:
        break
        
train_dataloader = DataLoader(transformed_train_dataset, batch_size=4,
                        shuffle=True, num_workers=4)

</code></pre>
<p>I am using grayscale images converted to RGB.</p>
<p>Thank you in advance!</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>It is about the code you have implemented in <code>__getitem()__</code> method in your <code>MasksTrainDataset</code>. Can you post how you return an item of your dataset using this method?</p>

username: soloupis
cooked: <p><a class="mention" href="/u/nikronic">@Nikronic</a></p>
<p>Check this out:</p>
<pre><code class="lang-auto">class MasksTrainDataset(Dataset):
    """Masks Train dataset."""

    def __init__(self, csv_file, root_dir, transform=None):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.masks_frame = pd.read_csv(csv_file, skiprows=1500, nrows=10000)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.masks_frame)

    def __getitem__(self, idx):
        img_name = os.path.join(self.root_dir,
                                self.masks_frame.iloc[idx, 0])
        image = io.imread(img_name + '.png')
        image = cv2.cvtColor(image,cv2.COLOR_GRAY2RGB)
        ## use strip to get exact result
        masks = self.masks_frame.iloc[idx, 1].strip()
        if masks == '-1':
            mark = 0
        else:
            mark = 1
            
        #masks = np.array([masks])
        #masks = masks.astype('float').reshape(-1, 2)
        sample = {'image': image, 'masks': mark}

        if self.transform:
            sample = self.transform(sample)

        return sample
      
      
train_dataset = MasksTrainDataset(csv_file='pneumo_input/train/train-rle.csv',
                                    root_dir='pneumo_input/train/images/256/dicom/')
print(len(train_dataset))
for i in range(len(train_dataset)):
    sample = train_dataset[i]

    print(i, sample['image'].shape, sample['masks'])
    print(type(sample['masks']))

    if i == 4:
        plt.show()
        break
</code></pre>
<p>What do you think?</p>

username: Nikronic
cooked: <p>OK,<br>
It seems your <code>image</code> and <code>masks</code> are CV2 objects. Pytorch’s image backend is <code>Pillow</code> if you want to do some transformation on it. And as you can see in <code>ToTensor</code> class, it expects <code>numpy</code> array or <code>PIL</code> image. So you can solve this issue by converting your <code>image</code> and <code>masks</code> to numpy or Pillow image in <code>__getitem()__</code>.</p>
<p>I have not tried it by <code>np.array(your image or mask)</code> should do the job.</p>

username: soloupis
cooked: <p>Thank you I will definitelly try it.</p>

username: soloupis
cooked: <p><a class="mention" href="/u/nikronic">@Nikronic</a></p>
<p>It seems that I cannot make it work. I have to use a method to turn one channel of grayscale image to 3 channel (RGB).I thought I have managed it with CV but I had problems with the normalize function. I used:</p>
<blockquote>
<p>image = Image.open(img_name + ‘.png’).convert(‘RGB’)</p>
</blockquote>
<p>but then it raises other shape related errors.</p>
<p>Do you think there is an error at the above code instead use CV?</p>

username: Nikronic
cooked: <p>Actually, your problem should not be CV or PIL, because if you provide a numpy, they will have the same result sometimes.</p>
<p>Here your code to convert to RGB is correct and PIL just duplicate the gray channel twice and concatenate them to make it 3 channel image.</p>
<p>Try this code and please print errors (it is hard to track without having errors):</p>
<pre><code class="lang-python">import numpy as np

print(np.array(image = Image.open(img_name + ‘.png’)).shape)
</code></pre>

username: soloupis
cooked: <p><a class="mention" href="/u/nikronic">@Nikronic</a><br>
I changed everything to below code:</p>
<pre><code class="lang-auto">class MasksTrainDataset(Dataset):
    """Masks Train dataset."""

    def __init__(self, csv_file, root_dir, transform=None):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.masks_frame = pd.read_csv(csv_file, skiprows=1500, nrows=10000)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.masks_frame)

    def __getitem__(self, idx):
        img_name = os.path.join(self.root_dir,
                                self.masks_frame.iloc[idx, 0])
        
        image = Image.open(img_name + '.png')
        ## use strip to get exact result
        masks = self.masks_frame.iloc[idx, 1].strip()
        if masks == '-1':
            mark = 0
        else:
            mark = 1
            
        #masks = np.array([masks])
        #masks = masks.astype('float').reshape(-1, 2)
        sample = {'image': np.array(image), 'masks': np.array(mark)}

        if self.transform:
            sample = self.transform(sample)

        return sample
      
      
train_dataset = MasksTrainDataset(csv_file='pneumo_input/train/train-rle.csv',
                                    root_dir='pneumo_input/train/images/256/dicom/')
print(len(train_dataset))
for i in range(len(train_dataset)):
    sample = train_dataset[i]

    print(i, sample['image'].shape, sample['masks'])
    print(type(sample['masks']))

    if i == 4:
        plt.show()
        break
</code></pre>
<p>and Transforms:</p>
<pre><code class="lang-auto">class Rescale(object):
    """Rescale the image in a sample to a given size.

    Args:
        output_size (tuple or int): Desired output size. If tuple, output is
            matched to output_size. If int, smaller of image edges is matched
            to output_size keeping aspect ratio the same.
    """

    def __init__(self, output_size):
        assert isinstance(output_size, (int, tuple))
        self.output_size = output_size

    def __call__(self, sample):
        image, landmarks = sample['image'], sample['masks']

        h, w = image.shape[:2]
        if isinstance(self.output_size, int):
            if h &gt; w:
                new_h, new_w = self.output_size * h / w, self.output_size
            else:
                new_h, new_w = self.output_size, self.output_size * w / h
        else:
            new_h, new_w = self.output_size

        new_h, new_w = int(new_h), int(new_w)

        img = transform.resize(image, (new_h, new_w))

        # h and w are swapped for landmarks because for images,
        # x and y axes are axis 1 and 0 respectively
        #landmarks = landmarks

        return {'image': np.array(img), 'masks': np.array(landmarks)}


class ToTensor(object):
    """Convert ndarrays in sample to Tensors."""

    def __call__(self, sample):
        image, landmarks = np.array(sample['image']), np.array(sample['masks'])

        # swap color axis because
        # numpy image: H x W x C
        # torch image: C X H X W
        image = image.transpose(2,0,1)
        #print(image.shape)
        #normalize = transforms.Normalize(mean=[0.5, 0.5, 0.5],std=[0.5, 0.5, 0.5])
        #return {'image': torch.from_numpy(image).unsqueeze(0),'masks': torch.from_numpy(landmarks)}
        #print(torch.is_tensor(torch.from_numpy(image)) and torch.from_numpy(image).ndimension() == 3)
        return {'image': torch.from_numpy(image),'masks': torch.from_numpy(landmarks)}
      
transformed_train_dataset = MasksTrainDataset(csv_file='pneumo_input/train/train-rle.csv',
                                           root_dir='pneumo_input/train/images/256/dicom/',
                                           transform=transforms.Compose([
                                               Rescale(224),
                                               ToTensor(),
                                               transforms.Normalize(mean=[0.5, 0.5, 0.5],std=[0.5, 0.5, 0.5])
                                           ]))

for i in range(len(transformed_train_dataset)):
    sample = transformed_train_dataset[i]

    print(i,sample['image'].size(),sample['masks'])

    if i == 3:
        break
        
train_dataloader = DataLoader(transformed_train_dataset, batch_size=4,
                        shuffle=True, num_workers=4)
</code></pre>
<p>Still the same error:</p>
<pre><code class="lang-auto">----------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-98-710d244d9279&gt; in &lt;module&gt;()
     61 
     62 for i in range(len(transformed_train_dataset)):
---&gt; 63     sample = transformed_train_dataset[i]
     64 
     65     print(i,sample['image'].size(),sample['masks'])

3 frames
&lt;ipython-input-96-283923ce157c&gt; in __getitem__(self, idx)
     37 
     38         if self.transform:
---&gt; 39             sample = self.transform(sample)
     40 
     41         return sample

/usr/local/lib/python3.6/dist-packages/torchvision/transforms/transforms.py in __call__(self, img)
     59     def __call__(self, img):
     60         for t in self.transforms:
---&gt; 61             img = t(img)
     62         return img
     63 

/usr/local/lib/python3.6/dist-packages/torchvision/transforms/transforms.py in __call__(self, tensor)
    162             Tensor: Normalized Tensor image.
    163         """
--&gt; 164         return F.normalize(tensor, self.mean, self.std, self.inplace)
    165 
    166     def __repr__(self):

/usr/local/lib/python3.6/dist-packages/torchvision/transforms/functional.py in normalize(tensor, mean, std, inplace)
    199     """
    200     if not _is_tensor_image(tensor):
--&gt; 201         raise TypeError('tensor is not a torch image.')
    202 
    203     if not inplace:

TypeError: tensor is not a torch image.
</code></pre>
<p>What do you think?</p>

username: soloupis
cooked: <p><a class="mention" href="/u/nikronic">@Nikronic</a></p>
<p>I think the problem is because ToTensor custom method returns a dictionary.<br>
I changed code to below:</p>
<pre><code class="lang-auto">transformed_train_dataset = MasksTrainDataset(csv_file='pneumo_input/train/train-rle.csv',
                                           root_dir='pneumo_input/train/images/256/dicom/',
                                           transform=transforms.Compose([
                                               Rescale(224),
                                               ToTensor(),
                                               transforms.Lambda(lambda x: x['image'].repeat(1, 1, 1)),
                                               transforms.Normalize(mean=[0.5, 0.5, 0.5],std=[0.5, 0.5, 0.5]),
                                           ]))
</code></pre>
<p>and normalization works. BUT now with Lambda function I lose labels (x[‘masks’]).</p>
<p>I found where is the problem though. So I have to normalize image before returning a dictionary at ToTensor custom method.</p>

username: soloupis
cooked: <p><a class="mention" href="/u/nikronic">@Nikronic</a></p>
<p>Final and working</p>
<pre><code class="lang-auto">class ToTensor(object):
    """Convert ndarrays in sample to Tensors."""

    def __call__(self, sample):
        image, landmarks = sample['image'], np.array(sample['masks'])

        # swap color axis because
        # numpy image: H x W x C
        # torch image: C X H X W
        image = image.transpose(2,0,1)
        print(image.shape)
        image = torch.from_numpy(image).float()
        
        in_transform = transforms.Compose([transforms.Normalize([0.5, 0.5, 0.5],[0.5, 0.5, 0.5])])
        ## discard the transparent, alpha channel (that's the :3)
        image = in_transform(image)[:3,:,:]
        
               
        return {'image': image,'masks': torch.from_numpy(landmarks)}
        #return torch.from_numpy(image).float()
      
transformed_train_dataset = MasksTrainDataset(csv_file='pneumo_input/train/train-rle.csv',
                                           root_dir='pneumo_input/train/images/256/dicom/',
                                           transform=transforms.Compose([
                                               Rescale(224),
                                               ToTensor(),
                                           ]))

for i in range(len(transformed_train_dataset)):
    print(type(transformed_train_dataset))
    print(len(transformed_train_dataset))
    sample = transformed_train_dataset[i]
    

    print(i,sample['image'],sample['masks'])

    if i == 3:
        break
        
train_dataloader = DataLoader(transformed_train_dataset, batch_size=4,
                        shuffle=True, num_workers=4)
</code></pre>

username: Nikronic
cooked: <p>Yes you right, you should not return a dictionary in <code>ToTensor</code> or any of <code>Transforms</code> class.<br>
Sorry if I answered late (time zone differences!).</p>
<p>But I have a suggestion here. It is better to build your classes modular so you can use them in other tasks with different datasets easily. For instance, maybe you need 3 or 4 images to be transformed or using different transforms on them. In this case you have to edit your <code>ToTensor</code> or <code>Rescale</code> class. So I think it is better to implement all transform classes for only a sample of input, actually, this is the approach has been chosen in PyTorch.</p>
<p>If I want to explain scenario, I can say if want to do other transforms for example adding gaussian noise to your image not landmarks, you will be stuck again and you have change your <code>ToTensor</code> code because still you are returning dictionary or even you are using another transform inside another one. But if your classes only take one tensor as input and return the changed tensor, you can use all of your custom classes in any order or in any dataset you want.</p>
<aside class="quote quote-modified" data-post="2" data-topic="30385">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rasbt/48/60_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/custom-dataset-with-some-preprocessing/30385/2">Custom Dataset with some preprocessing</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    When should I do this based on  Dataset  class? Should I do my processing step in  __getitem__() ? If yes, would it be parallel and fast? 

Yes. the __getitem__ calls are the ones run in parallel. Here’s some example: 
class CelebaDataset(Dataset):
    """Custom Dataset for loading CelebA face images"""

    def __init__(self, txt_path, img_dir, transform=None):
    
        df = pd.read_csv(txt_path, sep=" ", index_col=0)
        self.img_dir = img_dir
        self.txt_path = txt_path
        …
  </blockquote>
</aside>

<p>By the way, I use same approach as pytorch so I really did not think about your <code>ToTensor</code> custom implementation.</p>
<blockquote>
<p>usage in preprocessing step</p>
</blockquote>
<aside class="onebox githubblob" data-onebox-src="https://github.com/Nikronic/CoarseNet/blob/master/utils/preprocess.py#L98-L101">
  <header class="source">

      <a href="https://github.com/Nikronic/CoarseNet/blob/master/utils/preprocess.py#L98-L101" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/Nikronic/CoarseNet/blob/master/utils/preprocess.py#L98-L101" target="_blank" rel="noopener nofollow ugc">Nikronic/CoarseNet/blob/master/utils/preprocess.py#L98-L101</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="98" style="counter-reset: li-counter 97 ;">
          <li>if self.transform is not None:</li>
          <li>    X = self.transform(X)</li>
          <li>    random.seed(seed)</li>
          <li>    y_descreen = self.transform_gt(y_descreen)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<blockquote>
<p>usage in DataLoaders</p>
</blockquote>
<aside class="onebox githubblob" data-onebox-src="https://github.com/Nikronic/CoarseNet/blob/master/Train.py#L147-L153">
  <header class="source">

      <a href="https://github.com/Nikronic/CoarseNet/blob/master/Train.py#L147-L153" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/Nikronic/CoarseNet/blob/master/Train.py#L147-L153" target="_blank" rel="noopener nofollow ugc">Nikronic/CoarseNet/blob/master/Train.py#L147-L153</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="147" style="counter-reset: li-counter 146 ;">
          <li># %% get dataset specific mean and std values</li>
          <li>train_dataset = PlacesDataset(txt_path=args.txt,</li>
          <li>                              img_dir=args.img,</li>
          <li>                              transform=ToTensor(),</li>
          <li>                              test=True)</li>
          <li></li>
          <li>mean, std = OnlineMeanStd()(train_dataset, batch_size=1, method='strong')</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<blockquote>
<p>Custom Transform</p>
</blockquote>
<aside class="onebox githubblob" data-onebox-src="https://github.com/Nikronic/CoarseNet/blob/master/utils/preprocess.py#L109-L119">
  <header class="source">

      <a href="https://github.com/Nikronic/CoarseNet/blob/master/utils/preprocess.py#L109-L119" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/Nikronic/CoarseNet/blob/master/utils/preprocess.py#L109-L119" target="_blank" rel="noopener nofollow ugc">Nikronic/CoarseNet/blob/master/utils/preprocess.py#L109-L119</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="109" style="counter-reset: li-counter 108 ;">
          <li>class RandomNoise(object):</li>
          <li>    def __init__(self, p, mean=0, std=0.1):</li>
          <li>        self.p = p</li>
          <li>        self.mean = mean</li>
          <li>        self.std = std</li>
          <li></li>
          <li>    def __call__(self, img):</li>
          <li>        if random.random() &lt;= self.p:</li>
          <li>            noise = torch.empty(*img.size(), dtype=torch.float, requires_grad=False)</li>
          <li>            return img+noise.normal_(self.mean, self.std)</li>
          <li>        return img</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Good luck</p>