RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[4, 6, 128, 128] to have 3 channels, but got 6 channels instead

123456: <p>I am training pretrained deeplabv3 model with resnet101, and got the following error</p>
<p><strong>RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[4, 6, 128, 128] to have 3 channels, but got 6 channels instead.</strong></p>
<p>Actually I need to send two concatenated images to architecture. Concatenated along channel dimension.</p>
<p>I have only two classes and done like this</p>
<pre><code class="lang-auto">model = deeplabv3_resnet101(pretrained=True)
num_classes = len(classes)  # Number of segmentation classes

# Modify the final classification layer 
model.classifier[4] = nn.Conv2d(256, num_classes, kernel_size=(1, 1))
</code></pre>
<p>and got the following error</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-35-87c7cdd319e7&gt; in &lt;cell line: 64&gt;()
     81         # Forward pass
     82         # outputs = model(images)['out']
---&gt; 83         outputs =model(images)
     84 
     85         # Calculate the loss


9 frames


/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight, bias)
    454                             weight, bias, self.stride,
    455                             _pair(0), self.dilation, self.groups)
--&gt; 456         return F.conv2d(input, weight, bias, self.stride,
    457                         self.padding, self.dilation, self.groups)
    458 

RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[4, 6, 128, 128] to have 3 channels, but got 6 channels instead
</code></pre>
<p><strong>How can I modify 1st conv layer of deeplabv3 architecture take the input as 6 channels and not 3.</strong></p>
<p>Please help</p>

ptrblck: <aside class="quote no-group" data-username="123456" data-post="1" data-topic="191558">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/123456/48/61623_2.png" class="avatar"> 123456:</div>
<blockquote>
<p>How can I modify 1st conv layer of deeplabv3 architecture take the input as 6 channels and not 3.</p>
</blockquote>
</aside>
<p>You can use the same approach you are already using to replace the classifier.<br>
Instead of replacing <code>.classifier[4]</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">model.classifier[4] = nn.Conv2d(256, num_classes, kernel_size=(1, 1))
</code></pre>
<p>use:</p>
<pre data-code-wrap="python"><code class="lang-python">model = models.segmentation.deeplabv3_resnet101()
model.backbone.conv1 = nn.Conv2d(6, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)

x = torch.randn(4, 6, 128, 128)
out = model(x)
</code></pre>

123456: <p>Okay Sir. Thankyou<br>
I have done it like this below. Please tell Is this way right or wrong?</p>
<pre><code class="lang-auto">new_conv1 = nn.Conv2d(6, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
model.backbone.conv1 = new_conv1
</code></pre>

ptrblck: <p>Your code looks correct and is doing the same as my code snippet.</p>

123456: <p>Yes Sir. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
I haven’t used x = torch.randn(4, 6, 128, 128), but still it is going fine</p>

123456: <p>Sir, when I am printing the summary of this model, it again shows the size mismatch error</p>
<pre><code class="lang-auto">model = deeplabv3_resnet101(num_classes=2)
summary(model, input_size=(batch_size,6,128,128))
</code></pre>
<p>The error is</p>
<pre><code class="lang-auto">RuntimeError: Error(s) in loading state_dict for DeepLabV3:
	Unexpected key(s) in state_dict: "aux_classifier.0.weight", "aux_classifier.1.weight", "aux_classifier.1.bias", "aux_classifier.1.running_mean", "aux_classifier.1.running_var", "aux_classifier.1.num_batches_tracked", "aux_classifier.4.weight", "aux_classifier.4.bias". 
	size mismatch for backbone.conv1.weight: copying a param with shape torch.Size([64, 6, 7, 7]) from checkpoint, the shape in current model is torch.Size([64, 3, 7, 7]).
</code></pre>

ptrblck: <p>Yes, because you are not replacing the first linear layer as shown in my example in your latest code snippet.</p>

123456: <p>Sir one is the final layer we need to change for classes, and one is the 1st layer to have 6 channels</p>
<pre><code class="lang-auto">images=torch.cat((input1,input2),1)
model = deeplabv3_resnet101(pretrained=True)
num_classes = len(classes)  # Number of segmentation classes

# 1st layer
model.backbone.conv1 = nn.Conv2d(6, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)

# final classification layer
model.classifier[4] = nn.Conv2d(256, num_classes, kernel_size=(1, 1))

outputs =model(images)['out']
</code></pre>
<p>It is taking the images as a tensor concatenated along channel dimension. If I use torch.rand(4,6,128,128), then will it not be fixed to this input size only? because at the time of testing, we will pass variable size input</p>
<p>Also I tried using it, but giving error</p>
<pre><code class="lang-auto">images=torch.rand(batch_size,6,128,128)
        outputs =model(images)['out']
</code></pre>
<p><code>RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor</code></p>

ptrblck: <p>Sorry, yes I meant the first conv layer in my previous post.<br>
If you don’t replace it you will continue running into the same error explaining the model expects inputs with 3 channels.</p>
<aside class="quote no-group" data-username="123456" data-post="8" data-topic="191558">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/123456/48/61623_2.png" class="avatar"> 123456:</div>
<blockquote>
<p>If I use torch.rand(4,6,128,128), then will it not be fixed to this input size only? because at the time of testing, we will pass variable size input</p>
</blockquote>
</aside>
<p>It depends on the model architecture and dimension. The channel dimension will be fixed, the batch dimension is always variable (unless you hard-code it into the model, which I would consider a bug), and the spatial dimensions could be variable depending on the model architecture.</p>
<aside class="quote no-group" data-username="123456" data-post="8" data-topic="191558">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/123456/48/61623_2.png" class="avatar"> 123456:</div>
<blockquote>
<p>Also I tried using it, but giving error</p>
<pre><code class="lang-auto">images=torch.rand(batch_size,6,128,128)
        outputs =model(images)['out']
</code></pre>
<p><code>RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor)</code></p>
</blockquote>
</aside>
<p>Check the error message and make sure the input is moved to the GPU before being passed to the model.</p>

123456: <p>Yes Sir I replaced it as I shown in above code snippet, 1st layer and final layer.<br>
and yes I cross checked that input is moved to GPU before being passed.</p>
<p>The error comes when I use  torch.rand …, otherwise training smoothy gets done,<br>
images=torch.rand(batch_size,6,128,128)<br>
outputs =model(images)[‘out’]</p>
<p>After training problem comes at printing summary which shows error mismatch</p>

ptrblck: <aside class="quote no-group" data-username="123456" data-post="10" data-topic="191558">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/123456/48/61623_2.png" class="avatar"> 123456:</div>
<blockquote>
<p>and yes I cross checked that input is moved to GPU before being passed.</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="123456" data-post="10" data-topic="191558">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/123456/48/61623_2.png" class="avatar"> 123456:</div>
<blockquote>
<p>The error comes when I use torch.rand …, otherwise training smoothy gets done,<br>
images=torch.rand(batch_size,6,128,128)<br>
outputs =model(images)[‘out’]</p>
</blockquote>
</aside>
<p>Exactly, because <code>images</code> is not on the GPU but is still on the host. Move it to the the GPU and it should work.</p>

123456: <p>I have moved the images to tensor, now training part runs. Now the next is summary. I am not able to locate the error</p>
<p>Code is</p>
<pre><code class="lang-auto">import torchinfo
from torchinfo import summary
from torchvision.models.segmentation import deeplabv3_resnet101
# model = deeplabv3_resnet101(num_classes=2)
# summary(model, input_size=(batch_size,3,128,128))
model = deeplabv3_resnet101(num_classes=2)

# summary(model)
# summary(model, input_size=(6,batch_size,128,128))

summary(model, input_size=(batch_size,6,128,128))

# summary(model)
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[4, 6, 128, 128] to have 3 channels, but got 6 channels instead

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/torchinfo/torchinfo.py in forward_pass(model, x, batch_dim, cache_forward_pass, device, mode, **kwargs)
    302     except Exception as e:
    303         executed_layers = [layer for layer in summary_list if layer.executed]
--&gt; 304         raise RuntimeError(
    305             "Failed to run torchinfo. See above stack traces for more details. "
    306             f"Executed layers up to: {executed_layers}"

RuntimeError: Failed to run torchinfo. See above stack traces for more details. Executed layers up to: []
</code></pre>

ptrblck: <p>You are again missing the manipulation of the first conv layer:</p>
<pre><code class="lang-python">model = deeplabv3_resnet101(num_classes=2)
summary(model, input_size=(batch_size,6,128,128))
</code></pre>

123456: <p>Yes Sir Sorry, Actually I was experimenting something, so I comment it out.</p>
<p>But the problem still persists as shown above.</p>

ptrblck: <p>No, it doesn’t:</p>
<pre><code class="lang-python">import torchinfo
from torchinfo import summary
from torchvision.models.segmentation import deeplabv3_resnet101

model = deeplabv3_resnet101(num_classes=2)
model.backbone.conv1 = nn.Conv2d(6, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)

summary(model, input_size=(1,6,128,128))
# ====================================================================================================
# Layer (type:depth-idx)                             Output Shape              Param #
# ====================================================================================================
# DeepLabV3                                          [1, 2, 128, 128]          --
# ├─IntermediateLayerGetter: 1-1                     [1, 2048, 16, 16]         --
# │    └─Conv2d: 2-1                                 [1, 64, 64, 64]           18,816
# │    └─BatchNorm2d: 2-2                            [1, 64, 64, 64]           128
# │    └─ReLU: 2-3                                   [1, 64, 64, 64]           --
# │    └─MaxPool2d: 2-4                              [1, 64, 32, 32]           --
# │    └─Sequential: 2-5                             [1, 256, 32, 32]          --
# │    │    └─Bottleneck: 3-1                        [1, 256, 32, 32]          75,008
# │    │    └─Bottleneck: 3-2                        [1, 256, 32, 32]          70,400
# │    │    └─Bottleneck: 3-3                        [1, 256, 32, 32]          70,400
# │    └─Sequential: 2-6                             [1, 512, 16, 16]          --
# │    │    └─Bottleneck: 3-4                        [1, 512, 16, 16]          379,392
# │    │    └─Bottleneck: 3-5                        [1, 512, 16, 16]          280,064
# │    │    └─Bottleneck: 3-6                        [1, 512, 16, 16]          280,064
# │    │    └─Bottleneck: 3-7                        [1, 512, 16, 16]          280,064
# │    └─Sequential: 2-7                             [1, 1024, 16, 16]         --
# │    │    └─Bottleneck: 3-8                        [1, 1024, 16, 16]         1,512,448
# │    │    └─Bottleneck: 3-9                        [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-10                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-11                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-12                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-13                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-14                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-15                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-16                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-17                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-18                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-19                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-20                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-21                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-22                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-23                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-24                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-25                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-26                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-27                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-28                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-29                       [1, 1024, 16, 16]         1,117,184
# │    │    └─Bottleneck: 3-30                       [1, 1024, 16, 16]         1,117,184
# │    └─Sequential: 2-8                             [1, 2048, 16, 16]         --
# │    │    └─Bottleneck: 3-31                       [1, 2048, 16, 16]         6,039,552
# │    │    └─Bottleneck: 3-32                       [1, 2048, 16, 16]         4,462,592
# │    │    └─Bottleneck: 3-33                       [1, 2048, 16, 16]         4,462,592
# ├─DeepLabHead: 1-2                                 [1, 2, 16, 16]            --
# │    └─ASPP: 2-9                                   [1, 256, 16, 16]          --
# │    │    └─ModuleList: 3-34                       --                        15,206,912
# │    │    └─Sequential: 3-35                       [1, 256, 16, 16]          328,192
# │    └─Conv2d: 2-10                                [1, 256, 16, 16]          589,824
# │    └─BatchNorm2d: 2-11                           [1, 256, 16, 16]          512
# │    └─ReLU: 2-12                                  [1, 256, 16, 16]          --
# │    └─Conv2d: 2-13                                [1, 2, 16, 16]            514
# ====================================================================================================
# Total params: 58,635,522
# Trainable params: 58,635,522
# Non-trainable params: 0
# Total mult-adds (G): 15.11
# ====================================================================================================
# Input size (MB): 0.39
# Forward/backward pass size (MB): 244.85
# Params size (MB): 234.54
# Estimated Total Size (MB): 479.79
# ====================================================================================================
</code></pre>