How can I know the exact dimensions for a layer that I am adding (to an already existing architecture)

Mona_Jalal: <p>I am adding the following code in resnet18 custom code</p>
<pre><code>self.layer1 = self._make_layer(block, 64, layers[0]) ## code existed before
self.layer2 = self._make_layer(block, 128, layers[1], stride=2) ## code existed before
self.layer_attend1 =  nn.Sequential(nn.Conv2d(layers[0], layers[0], stride=2, padding=1, kernel_size=3),
                                     nn.AdaptiveAvgPool2d(1),
                                     nn.Softmax(1)) ## code added by me
</code></pre>
<p>and also the following in its forward pass (<code>def forward(self, x)</code>) in the same resnet18 custom code:</p>
<pre><code>x = self.layer1(x) ## the code existed before
x = self.layer_attend1(x)*x ## I added this code
x = self.layer2(x) ## the code existed before
</code></pre>
<p>and I get the following error. I had no error before adding this attention layer. Any idea how I could fix it?</p>
<pre><code>=&gt; loading checkpoint 'runs/nondisjoint_l2norm/model_best.pth.tar'
=&gt; loaded checkpoint 'runs/nondisjoint_l2norm/model_best.pth.tar' (epoch 5)
/scratch3/venv/fashcomp/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
Traceback (most recent call last):
  File "main.py", line 352, in &lt;module&gt;
    main()    
  File "main.py", line 153, in main
    test_acc = test(test_loader, tnet)
  File "main.py", line 248, in test
    embeddings.append(tnet.embeddingnet(images).data)
  File "/scratch3/venv/fashcomp/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/scratch3/research/code/fashion/fashion-compatibility/type_specific_network.py", line 101, in forward
    embedded_x = self.embeddingnet(x)
  File "/scratch3/venv/fashcomp/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/scratch3/research/code/fashion/fashion-compatibility/Resnet_18.py", line 110, in forward
    x = self.layer_attend1(x)*x #so we don;t use name x1
  File "/scratch3/venv/fashcomp/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/scratch3/venv/fashcomp/lib/python3.8/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/scratch3/venv/fashcomp/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/scratch3/venv/fashcomp/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 443, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/scratch3/venv/fashcomp/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 439, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [2, 2, 3, 3], expected input[256, 64, 28, 28] to have 2 channels, but got 64 channels instead
</code></pre>
<p>Additionally, could you please suggest how to get information without print? (print doesn’t work). For example, I want to print <code>layers[0]</code> information but nothing gets printed.</p>
<p>I am also not sure how to debug this in VSCode. I added a breakpoint on the line before the problematic one (new layer), and it didn’t even go or stop there.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/6/261ea77256b1bb2aefc8256fccdc65248326d80e.png" data-download-href="https://discuss.pytorch.org/uploads/default/261ea77256b1bb2aefc8256fccdc65248326d80e" title="Screen Shot 2021-08-31 at 7.39.55 PM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/261ea77256b1bb2aefc8256fccdc65248326d80e.png" alt="Screen Shot 2021-08-31 at 7.39.55 PM" data-base62-sha1="5rdOPV5TisKGtXcOxYANxRJ5Pem" width="690" height="59" data-dominant-color="292A2C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-08-31 at 7.39.55 PM</span><span class="informations">744×64 5.57 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

gphilip: <aside class="quote no-group" data-username="Mona_Jalal" data-post="1" data-topic="130794">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mona_jalal/48/25715_2.png" class="avatar"> Mona_Jalal:</div>
<blockquote>
<p>For example, I want to print <code>layers[0]</code> information but nothing gets printed.</p>
</blockquote>
</aside>
<p>What do you mean by “nothing gets printed”? If you add the line</p>
<pre><code class="lang-auto">print("Hi")
</code></pre>
<p>at the place where you were trying to print <code>layers[0]</code>, does it print anything? (It should …)</p>
<p>To know the dimensions that go into the layer you added, you can try something like this:</p>
<pre><code class="lang-auto">x = self.layer1(x) ## the code existed before
print("The input shape is: " + str(x.shape))
x = self.layer_attend1(x)*x ## I added this code
x = self.layer2(x) ## the code existed before
</code></pre>

Mona_Jalal: <p>actually I just now put the breakpoint in the</p>
<pre><code class="lang-auto">class ResNet(nn.Module):

    def __init__(self, block, layers, embedding_size=64):
</code></pre>
<p>and was able to print value of layers[0] which is 2. Previously put the breakpoint in forward pass and it just jumped to the error.</p>
<p>Anyhow, still not sure how to fix this error I am getting.</p>

Mona_Jalal: <p>the input shape is <code>The input shape is: torch.Size([256, 64, 28, 28])</code></p>

Mona_Jalal: <p>knowing that the input shape is: <code>torch.Size([256, 64, 28, 28])</code></p>
<p>how should I change the following?</p>
<pre><code class="lang-auto">self.layer_attend1 =  nn.Sequential(nn.Conv2d(layers[0], layers[0], stride=2, padding=1, kernel_size=3),
                                     nn.AdaptiveAvgPool2d(1),
                                     nn.Softmax(1))
</code></pre>
<p>The error is<br>
<code>RuntimeError: Given groups=1, weight of size [2, 2, 3, 3], expected input[256, 64, 28, 28] to have 2 channels, but got 64 channels instead</code></p>
<p>and the layers[0] is 2</p>
<p><a class="mention" href="/u/gphilip">@gphilip</a></p>
<p>Thank you</p>

gphilip: <p>In <code>[256, 64, 28, 28]</code> the <code>256</code> is the batch size. So each input tensor to the first layer in <code>self.layer_attend1</code> has the shape <code>[64, 28, 28]</code>. Of this, the <code>64</code> is the number of channels in the input. The first argument to <code>nn.Conv2d</code> should therefore be <code>64</code>.</p>

Mona_Jalal: <p>Thanks for your response. So I changed it to</p>
<pre><code class="lang-auto">self.layer_attend1 =  nn.Sequential(nn.Conv2d(64, 64, stride=2, padding=1, kernel_size=3),
                                                         nn.AdaptiveAvgPool2d(1),
                                                         nn.Softmax(1))
</code></pre>
<p>Do you think if it is wrong or could you please show how to fill the args of <code>nn.Conv2d(64, 64, stride=2, padding=1, kernel_size=3)</code> otherwise?</p>

gphilip: <p><a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html" rel="noopener nofollow ugc">Here is the documentation</a> for <code>nn.Conv2d</code>. Only the first argument depends on the input. You have to choose the remaining arguments based on what you want out of the convolution layer. See the section on “Shape” in that link for the details on how the output height and width are calculated from the arguments.</p>

Mona_Jalal: <p>so maybe i ask my question this way:</p>
<p>how do I know what’s the value or best value for <code>out_channel</code>? currently I have set it to 64.</p>

gphilip: <p>This is a design decision that you have to make, depending on what you want this layer to do.</p>

Mona_Jalal: <p>I have a question for you, if self.layer_attend1 is nn.sequential of three other layers, how would you be able to print the output size of the middle layer?</p>
<p>Here, is the layer_attend1:</p>
<pre><code class="lang-auto">self.layer_attend1 =  nn.Sequential(nn.Conv2d(64, 64, stride=2, padding=1, kernel_size=3),
                                    nn.AdaptiveAvgPool2d(1),
                                    nn.Softmax(1))
</code></pre>
<p>and i want to know the shape of output of <code>nn.AdaptiveAvgPool2d(1)</code></p>
<p>is there a way to do so?</p>

gphilip: <p>Assuming that the shape of the inptu to <code>self.layer.attend1</code> is <code>torch.Size([256, 64, 28, 28])</code> like you said above, this can be done using the following code:</p>
<pre><code class="lang-auto">X = torch.randn([256, 64, 28, 28])
Y = nn.Conv2d(64, 64, stride=2, padding=1, kernel_size=3)(X)
Z = nn.AdaptiveAvgPool2d(1)(Y)

print(Z.shape)
</code></pre>

Mona_Jalal: <aside class="quote no-group" data-username="gphilip" data-post="13" data-topic="130794">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/858c86/48.png" class="avatar"> gphilip:</div>
<blockquote>
<pre><code class="lang-auto">X = torch.randn([256, 64, 28, 28])
Y = nn.Conv2d(64, 64, stride=2, padding=1, kernel_size=3)(X)
Z = nn.AdaptiveAvgPool2d(1)(Y)

print(Z.shape)
</code></pre>
</blockquote>
</aside>
<p>Thanks a lot, so is it wrong to pass<br>
torch.Size([256, 64, 1, 1])<br>
to a softmax(1)?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e44ba13ee637d25173e0bd8a1bc67898e185de1.png" data-download-href="https://discuss.pytorch.org/uploads/default/3e44ba13ee637d25173e0bd8a1bc67898e185de1" title="Screen Shot 2021-09-01 at 12.31.17 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3e44ba13ee637d25173e0bd8a1bc67898e185de1_2_690x290.png" alt="Screen Shot 2021-09-01 at 12.31.17 PM" data-base62-sha1="8SQQGDfCGn1fMlzCfn2HxoIyxsB" width="690" height="290" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3e44ba13ee637d25173e0bd8a1bc67898e185de1_2_690x290.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3e44ba13ee637d25173e0bd8a1bc67898e185de1_2_1035x435.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e44ba13ee637d25173e0bd8a1bc67898e185de1.png 2x" data-dominant-color="282A2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-09-01 at 12.31.17 PM</span><span class="informations">1348×568 47.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>i am not getting any error but my accuracy has dropped significantly and it seems softmax is not working as expected hence thought maybe i should change the arg inside softmax from 1 to something else that I am not sure what it is.</p>
<p>a bit explanation here <a href="https://discuss.pytorch.org/t/how-to-stack-adaptiveavgpool2d-and-softmax/130866" class="inline-onebox">How to stack adaptiveavgpool2D and softmax?</a></p>

gphilip: <aside class="quote no-group" data-username="Mona_Jalal" data-post="14" data-topic="130794">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mona_jalal/48/25715_2.png" class="avatar"> Mona_Jalal:</div>
<blockquote>
<p>is it wrong to pass<br>
torch.Size([256, 64, 1, 1])<br>
to a softmax(1)?</p>
</blockquote>
</aside>
<p>Since the code is not throwing an error, it is clearly not a wrong-shaped input to pass to this function.</p>