Classification using 4-channel images

sampa: <p>I have some gray scale and color images with label. I want to combine this gray and color images (4-channel) and run transfer learning using 4-channel images. How to do that?</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/how-to-transfer-the-pretrained-weights-for-a-standard-resnet50-to-a-4-channel/52252/2">This post</a> gives you an example of your use case. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

sampa: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. What about dataloader part? How I will load two types (Gray &amp; RGB) of image and combine them?</p>

ptrblck: <p>I would recommend to write a custom <code>Dataset</code> class as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a>, load the image pairs in the <code>__getitem__</code> method and concatenate them there as well.</p>
<p>I assume you have some mapping between the RGB and grayscale images.<br>
If so, you could pass the mapping together with the paths for the images to <code>__init__</code> and load the pairs lazily in <code>__getitem__</code>.</p>

sampa: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I did what u said and now got this error<br>
image= torch.cat((image_BW, image_RGB), 1)<br>
TypeError: expected Tensor as element 0 in argument 0, but got BmpImageFile</p>
<pre><code class="lang-auto">class bothDataset(Dataset):
    '''
    Initilize custom Dataset for image analysis
    It should correlate image with label for training dataset
    '''

    def __init__(self, csv_path, root_dir, transform=None):
        """Init function should not do any heavy lifting, but
            must initialize how many items are available in this data set.
            
            Args:
                csv_path : Path to the csv where classes (labels) and image names are located accordingly
                root_dir : Directory with all images
                transform: Optional transform to be applied on a sample
        """
        
        #df = pd.read_csv(csv_path)#, sep=',',index_col=0)
        self.img_names = pd.read_csv(csv_path)
        self.transform = transform
        self.root_dir = root_dir
        #self.images = read_images(root + "/images")
        #self.labels = read_labels(root + "/labels")

    def __len__(self):
        """return number of points in our dataset"""

        return len(self.img_names)

    def __getitem__(self, idx):
        """ Here we have to return the item requested by `idx`
            The PyTorch DataLoader class will use this method to make an iterable for
            our training or validation loop.
        """

        if torch.is_tensor(idx):
            idx = idx.tolist()

        img_name_BW = os.path.join(self.root_dir,
                                self.img_names.iloc[idx, 0])
        image_BW = Image.open(img_name_BW+'.bmp')
        img_name_RGB = os.path.join(self.root_dir,
                                self.img_names.iloc[idx, 1])
        image_RGB = Image.open(img_name_RGB+'.bmp')
        image= torch.cat((image_BW, image_RGB), 1) 
        #image = io.imread(img_name+'.jpg')
        #image = cv2.imread(img_name+'.jpg')
        #image=image_BW
        label = self.img_names.iloc[idx, 2]
        #label = np.array([label])
        #label = label.astype('float')#.reshape(-1, 2)
        #sample = {'image': image, 'label': label}

        if self.transform:
            image = self.transform(image)
            
        return image, label                                   
train_datak = bothDataset(csv_path='RelationBS1.csv',
                               root_dir=test_folder, 
                                  transform=train_transform)
print('Num training images: ', len(train_datak))
batch_size =64
num_workers=0

# prepare data loaders
train_loader = torch.utils.data.DataLoader(train_datak, batch_size=batch_size, 
                                           num_workers=num_workers, shuffle=True)
dataiter = iter(train_loader)
images, labels = dataiter.next()  
print(type(images))
print(images.shape)
</code></pre>

ptrblck: <p>Use <code>torchvision.transforms.functional.to_tensor</code> on both images, as they are currently still instances of <code>PIL.Image</code>, while <code>torch.cat</code> expects a list of tensors.</p>

sampa: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for helping me. you want to say like this?But still same error…<img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<pre><code class="lang-auto">        img_name_BW = os.path.join(self.root_dir,
                                self.img_names.iloc[idx, 0])
        image_BW = Image.open(img_name_BW+'.bmp')
        torchvision.transforms.functional.to_tensor(image_BW)
        img_name_RGB = os.path.join(self.root_dir,
                                self.img_names.iloc[idx, 1])
        image_RGB = Image.open(img_name_RGB+'.bmp')
        torchvision.transforms.functional.to_tensor(image_RGB)
        image= torch.cat((image_BW, image_RGB), 1) 
</code></pre>

ptrblck: <p>You would have to assign the result back:</p>
<pre><code class="lang-python">image_BW = torchvision.transforms.functional.to_tensor(image_BW)
</code></pre>

sampa: <p>BW &amp; RGB image dimensions are different. Still can I use  torch. cat function?<br>
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 1 and 3 in dimension 0<br>
I am trying to concatenate (1,224, 224) and (3, 224, 224)</p>

ptrblck: <p>It should work, if you concatenate these tensors in dim0 via <code>torch.cat((a, b), dim=0)</code>.</p>

sampa: <p>I am using this code torch.cat((a, b), dim=0). But getting the same error as my image sizes are not same. How to resize image size before concatenate?<br>
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 391 and 389 in dimension 1</p>

ptrblck: <p>Oh, based on your previous post, it seemed both images have a spatial size of <code>224</code>.<br>
You could use e.g. <code>torchvision.transforms.Resize</code> on both <code>PIL.Images</code>.</p>

sampa: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks. But now error is<br>
TypeError: pic should be PIL Image or ndarray. Got &lt;class ‘torchvision.transforms.transforms.Resize’&gt;</p>
<pre><code class="lang-auto">image_BW = Image.open(img_name_BW+'.bmp')
        image_BW  =transforms.Resize(224, 224)
        image_BW=torchvision.transforms.functional.to_tensor(image_BW)
</code></pre>
<p>I converted before applying Resize. Still same error</p>
<pre><code class="lang-auto">image_BW=torchvision.transforms.transforms.ToPILImage(image_BW)
</code></pre>

ptrblck: <p><code>Resize</code> is a Python object, so you would need to create an instance:</p>
<pre><code class="lang-python">res = transforms.Resize((224, 224))
image_BW = res(image_BW)
...
</code></pre>
<p>or use the functional API again via: <code>torchvision.transforms.functional.resize</code>.</p>

sampa: <p>It is working now. Thank you so much.</p>

peter_wills: <p>Thanks - very useful. Small questions:</p>
<ul>
<li>
<p>why no_grad. Is there  reason to freeze the weights?<br>
Note that I count trainable parameters before and after making the change and get the same answer in both cases.</p>
</li>
<li>
<p>what are these two lines for - test?<br>
x = torch.randn(10, 4, 224, 224)<br>
output = model(x)</p>
</li>
</ul>

ptrblck: <ol>
<li>
<p>The weights are reassigned, which shouldn’t be tracked by Autograd as a differentiable operation.</p>
</li>
<li>
<p>Yes, these lines of code test, if the manipulation of the parameters was successful or if we get a shape mismatch error.</p>
</li>
</ol>