ERROR: Change input chanel with inceptionv3

Huynh_Viet_Trung: <p>Hi everybody.<br>
My English not good, im from VietNam<br>
I tried change input chanels to inception recive input 4 channel (mycode below) with one picture .png and depth information .npy, I get error<br>
<strong>My code:</strong></p>
<pre><code class="lang-auto">def Inception(in_planes, out_planes, pretrained=False):
    if pretrained is True:       
        model = models.inception_v3(pretrained=True)
        print("Pretrained model is loaded")
    else:        
        model = models.inception_v3(pretrained=False)
    if in_planes == 4:
        model.Conv2d_1a_3x3.conv=nn.Conv2d(4, 64, kernel_size=3, stride=2, padding=2, bias=False)
        nn.init.kaiming_normal_(model.Conv2d_1a_3x3.conv.weight, mode='fan_out', nonlinearity='relu')
    # Parameters of newly constructed modules have requires_grad=True by default
    model.fc = nn.Linear(model.fc.in_features, out_planes)
    return model
</code></pre>
<p><strong>My error:</strong></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train_softmax.py", line 307, in &lt;module&gt;
    pretrained_optim_path=args.pretrained_optim_path)
  File "train_softmax.py", line 155, in train_model
    outputs,aux_outputs = model(inputs)
  File "/home/viet/anaconda3/envs/pythonProject11/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/viet/anaconda3/envs/pythonProject11/lib/python3.7/site-packages/torchvision/models/inception.py", line 101, in forward
    x = self.Conv2d_1a_3x3(x)
  File "/home/viet/anaconda3/envs/pythonProject11/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/viet/anaconda3/envs/pythonProject11/lib/python3.7/site-packages/torchvision/models/inception.py", line 352, in forward
    x = self.conv(x)
  File "/home/viet/anaconda3/envs/pythonProject11/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/viet/anaconda3/envs/pythonProject11/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)
  File "/home/viet/anaconda3/envs/pythonProject11/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size 64 4 3 3, expected input[16, 3, 299, 299] to have 4 channels, but got 3 channels instead
  0%|                                                                                                                                
</code></pre>
<p>My input is one picture .PNG and one .NPY file. I concate and tranform to tensor.</p>
<p>Please help me.</p>

ptrblck: <p>Based on the error message it seems that the manipulation of the model is indeed working, while the input concatenation is not:</p>
<pre><code class="lang-python">RuntimeError: Given groups=1, weight of size 64 4 3 3, expected input[16, 3, 299, 299] to have 4 channels, but got 3 channels instead
</code></pre>
<p>This error indicates that the input is still using <code>3</code> channels, while <code>4</code> are expected.<br>
Could you double check and/or post the code which creates the new input tensor?</p>

Huynh_Viet_Trung: <p>Before: I concate image RGB numpy with depthmap</p>
<pre><code class="lang-auto">    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
        rgb_image_path, dep_image_path, cls_id = self.df.iloc[idx]
        # print(rgb_image_path)
        image = Image.open(rgb_image_path)
        if self.input_channels == 4:
            rgb_image = np.asarray(image)
            misc.imsave("trung.jpg", rgb_image)
            dep_image = np.load(dep_image_path)
            depth_colormap = cv2.applyColorMap(cv2.convertScaleAbs(dep_image, alpha=340), cv2.COLORMAP_JET)
            misc.imsave("trung1.jpg", depth_colormap)
            dep_image = np.expand_dims(dep_image, axis=-1)
            image = np.concatenate((rgb_image, dep_image), axis=-1)
            #print("chua tranform",image)#kieu numpu
            
        if self._transform is not None:
            image = self._transform(image)

            #print("da tranform", image)#kieu tensor
        # There is no need to transfer into One-hot encoding
        # label = torch.zeros(self._num_of_classes, dtype=torch.long).scatter_(0, torch.from_numpy(np.array(cls_id)), 1)
        # label = (np.arange(self._num_of_classes) == cls_id).astype(np.float32)
        #print(image)
        return image, cls_id

</code></pre>
<p>After: I using new define function to normalize</p>
<pre><code class="lang-auto">
class Resize(object):
    """Resize the image in a sample to a given size.
    ** Strongly Recommended use this class only when dealing with RGB-D Images **

    Args:
        output_size (tuple or int): Desired output size. If tuple, output is
            matched to output_size. If int, smaller of image edges is matched
            to output_size keeping aspect ratio the same.
    """

    def __init__(self, output_size):
        assert isinstance(output_size, (int, tuple))
        self.output_size = output_size

    def __call__(self, image):

        #print("co resize:===========================&gt;",image)
        #print("shape:=====================",image.shape)#(256,256,4)
        if isinstance(image, np.ndarray):
            h, w = image.shape[:2]
        else:
            return transforms.Resize(self.output_size)(image)
        if isinstance(self.output_size, int):
            if h &gt; w:
                new_h, new_w = self.output_size * h / w, self.output_size
            else:
                new_h, new_w = self.output_size, self.output_size * w / h
        else:
            new_h, new_w = self.output_size
        new_h, new_w = int(new_h), int(new_w)
        image = transform.resize(image, (new_h, new_w)).astype(np.float32)

        #print("co resize111111:===========================&gt;", image)
        print("shape co resize:=====================",image.shape)
        return image


class RandomHorizontalFlip(object):
    """Horizontally flip the given Image randomly with a given probability.

    Args:
        p (float): probability of the image being flipped. Default value is 0.5
    """

    def __init__(self, p=0.5):
        self.p = p

    def __call__(self, image):
        """
        Args:
            image ( Image): Image to be flipped.

        Returns:
            Image: Randomly flipped image.
        """
        if not isinstance(image, np.ndarray):
            return transforms.RandomHorizontalFlip(p=self.p)(image)
        if random.random() &lt; self.p:
            return np.fliplr(image).astype(np.float32)
        return image

    def __repr__(self):
        return self.__class__.__name__ + '(p={})'.format(self.p)

</code></pre>
<p>I think inceptionv3 cannot receive 4 chanel input here, but how about.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/6/e62b05b367e291b74912b556862db1952b60d61d.png" data-download-href="https://discuss.pytorch.org/uploads/default/e62b05b367e291b74912b556862db1952b60d61d" title="Screenshot from 2021-07-05 09-01-58"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e62b05b367e291b74912b556862db1952b60d61d_2_690x388.png" alt="Screenshot from 2021-07-05 09-01-58" data-base62-sha1="wQ9Zz35coMu0EzpsdGESrjRe4wd" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e62b05b367e291b74912b556862db1952b60d61d_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e62b05b367e291b74912b556862db1952b60d61d_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e62b05b367e291b74912b556862db1952b60d61d_2_1380x776.png 2x" data-dominant-color="ECEBEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-07-05 09-01-58</span><span class="informations">1920×1080 306 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Please help me.</p>

ptrblck: <p>I don’t think the issue is in the model, but still the input.<br>
In your current code you are checking for 4 channels and are only concatenating the image tensors then:</p>
<pre><code class="lang-python">if self.input_channels == 4:
    ...
    image = np.concatenate((rgb_image, dep_image), axis=-1)
</code></pre>
<p>which seems to be wrong. Maybe you want to check for 3 channels to concatenate the images to 4 channels?</p>

Huynh_Viet_Trung: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="125711">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code> image = np.concatenate((rgb_image, dep_image), axis=-1)</code></p>
</blockquote>
</aside>
<p>My code and result</p>
<pre><code class="lang-auto">def normalize_depth_image(image):
  min_depth = np.min(image)
  return (image.astype(float) - min_depth) / (np.max(image) - min_depth)
#RGB image
original = Image.open('31.png')
RGB_image=np.array(original)
#Depthmap of face
my_data = np.load('31.npy')
NN=normalize_depth_image(my_data)
output_data_N = (NN * 255).astype(np.uint8)
dep_image = np.expand_dims(output_data_N, axis=-1)

print('My RGB data:',RGB_image)
print('My depthmap data of face :',dep_image)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5e2cf1530ec730e77c6fd7bbefdde2fbfba6079a.png" data-download-href="https://discuss.pytorch.org/uploads/default/5e2cf1530ec730e77c6fd7bbefdde2fbfba6079a" title="Screenshot from 2021-07-06 10-48-24"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e2cf1530ec730e77c6fd7bbefdde2fbfba6079a_2_517x267.png" alt="Screenshot from 2021-07-06 10-48-24" data-base62-sha1="dr7aD4QugmPGWDnuDjAxRaJ4IYO" width="517" height="267" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e2cf1530ec730e77c6fd7bbefdde2fbfba6079a_2_517x267.png, https://discuss.pytorch.org/uploads/default/original/3X/5/e/5e2cf1530ec730e77c6fd7bbefdde2fbfba6079a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/e/5e2cf1530ec730e77c6fd7bbefdde2fbfba6079a.png 2x" data-dominant-color="2D2D2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-07-06 10-48-24</span><span class="informations">731×378 14.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ffe7f28958f9059dd3013aa0e4e37bdd325579c7.png" data-download-href="https://discuss.pytorch.org/uploads/default/ffe7f28958f9059dd3013aa0e4e37bdd325579c7" title="Screenshot from 2021-07-06 10-49-08"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffe7f28958f9059dd3013aa0e4e37bdd325579c7_2_366x375.png" alt="Screenshot from 2021-07-06 10-49-08" data-base62-sha1="AvQHoMmzbGqekD8OX5jiqkxy9f1" width="366" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffe7f28958f9059dd3013aa0e4e37bdd325579c7_2_366x375.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffe7f28958f9059dd3013aa0e4e37bdd325579c7_2_549x562.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffe7f28958f9059dd3013aa0e4e37bdd325579c7_2_732x750.png 2x" data-dominant-color="2C2C2C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-07-06 10-49-08</span><span class="informations">762×779 19.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I concate got result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e23bbc94f2af045501135be7f3edee0201ba8ed5.png" data-download-href="https://discuss.pytorch.org/uploads/default/e23bbc94f2af045501135be7f3edee0201ba8ed5" title="Screenshot from 2021-07-06 10-53-23"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e23bbc94f2af045501135be7f3edee0201ba8ed5_2_517x282.png" alt="Screenshot from 2021-07-06 10-53-23" data-base62-sha1="whlToBQmXExkC6l9suCFe7gPRXL" width="517" height="282" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e23bbc94f2af045501135be7f3edee0201ba8ed5_2_517x282.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e23bbc94f2af045501135be7f3edee0201ba8ed5_2_775x423.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/2/e23bbc94f2af045501135be7f3edee0201ba8ed5.png 2x" data-dominant-color="2D2D2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-07-06 10-53-23</span><span class="informations">831×455 23.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
My picture to train:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f96b757d3900ca8272795edda01be61381942257.png" alt="Screenshot from 2021-07-06 10-54-43" data-base62-sha1="zAt7qKLpcPyRt9MCaGNpBr8WFf1" width="340" height="121"></p>
<p>How can i finetune inception  to train with 4 input channels instead 3 channel.<br>
Thank for your help.</p>

ptrblck: <p>I’m unsure, if you are still stuck at the previous problem or if you are facing another issue.<br>
Could you describe your current problem a bit more, please?</p>

Huynh_Viet_Trung: <p>My problem:<br>
My Project about recognize identify. I using camera Intel D435 to collect and get depthmap of face, I folllow this paper with my dataset student. <a href="https://github.com/XingwXiong/Face3D-Pytorch" rel="noopener nofollow ugc">link</a><br>
After I train Model include one pictures(.png) and one depthmap(.npy) of face, I train my model ResNet50 with pretrain Imagenet, using Pytorch, my dataset have 500 class with 70 picture/class. I got model 99.7% But i have problem my model only veryfi data trained in my dataset, and not verify real data.<br>
So i want change another CNN, I want use Inception V3 and I changed input channel, from 3 channel to 4 channel( RGB image and depthmap) but i got error<br>
when i train.</p>
<pre><code class="lang-auto">RuntimeError: Given groups=1, weight of size 64 4 3 3, expected input[16, 3, 299, 299] to have 4 channels, but got 3 channels instead
</code></pre>
<p>All code I post above from first question.<br>
Thank for your help.</p>

ptrblck: <p>Thanks for the update. As already pointed out, the error is caused by your input, so you would still have to check the aforementioned line of code, which seems to use a wrong condition.<br>
Your code runs fine with the proper input (and an unrelated fix to the output channels of the new conv layer):</p>
<pre><code class="lang-python">def Inception(in_planes, out_planes, pretrained=False):
    if pretrained is True:       
        model = models.inception_v3(pretrained=True)
        print("Pretrained model is loaded")
    else:        
        model = models.inception_v3(pretrained=False)
    if in_planes == 4:
        model.Conv2d_1a_3x3.conv=nn.Conv2d(4, 32, kernel_size=3, stride=2, padding=2, bias=False)
        nn.init.kaiming_normal_(model.Conv2d_1a_3x3.conv.weight, mode='fan_out', nonlinearity='relu')
    # Parameters of newly constructed modules have requires_grad=True by default
    model.fc = nn.Linear(model.fc.in_features, out_planes)
    return model


model = Inception(in_planes=4, out_planes=1, pretrained=False)
x = torch.randn(2, 4, 299, 299)
out = model(x)
print(out.logits.shape)
&gt; torch.Size([2, 1])
</code></pre>

algeriapy: <p>there is transform_input which is transforming the input into 3 channels again so you need to add transform_input= False</p>
<p>model = torchvision.models.inception_v3(pretrained=True, transform_input= False)</p>