Concatenate two tensors with different sizes

username: Patrice
cooked: <pre><code>      Dear senior programmers,
</code></pre>
<p>I have obtained the following network structure by modifying someone’s else network. I have added the dilation keyword so as to obtain dilated convolutional layer. However, given that there were some concatenation in the “forward part” of the network, I have not been able to adjust the output channels and the concatenation properly. Please could anyone explain to me how to fix? The network is as follows.</p>
<pre><code class="lang-auto">class net(nn.Module):   
    def __init__(self):
        super(net, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=3, kernel_size=1)
        self.conv2 = nn.Conv2d(in_channels=3, out_channels=3, kernel_size=3, padding=2, dilation=2)
        self.conv3 = nn.Conv2d(in_channels=3, out_channels=3, kernel_size=5, padding=2, dilation=2)
        self.conv4 = nn.Conv2d(in_channels=6, out_channels=3, kernel_size=7, padding=3, dilation=2)
        self.conv5 = nn.Conv2d(in_channels=12, out_channels=3, kernel_size=3, padding=1)
        self.b = 1

    def forward(self, x): 
        
        x1 = F.relu(self.conv1(x))
        #print(x1.shape)
        x2 = F.relu(self.conv2(x1))
        print(x2.shape)
        cat1 = torch.cat((x1, x2), 2)
        x3 = F.relu(self.conv3(cat1))
        print(x3.shape)
        cat2 = torch.cat((x2, x3), 2)
        x4 = F.relu(self.conv4(cat2))
        cat3 = torch.cat((x1, x2, x3, x4),2)
        k = F.relu(self.conv5(cat3))

        if k.size() != x.size():
            raise Exception("k, haze image are different size!")

        output = k * x - k + self.b
        return F.relu(output)
</code></pre>
<p>The running error is as follows.</p>
<pre><code class="lang-auto">RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 2. Got 476 and 480 in dimension 3 at /opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/THC/generic/THCTensorMath.cu:71
</code></pre>
<p>Please, how can I fix this error? I would also like to know the relationships between input channel, output channel, padding and dilation.</p>
<p>Thank you for your time and patience</p>

username: BTNug
cooked: <p>Hi <a class="mention" href="/u/patrice">@Patrice</a>,<br>
*Rule1: If you want to concatenate the layer, it has to have only 1 dimension that is different from the other (i.e. NxDiff1xHxW and NxDiff2xHxW or NxCxDiff1xW and NxCxDiff2xW, etc)<br>
in your case, what you are trying to do:<br>
suppose our input x: 1x3x28x28</p>
<p>what you are doing is:</p>
<ul>
<li>you do convolution x1, the output is 1x3x28x28</li>
<li>you do convolution x2, the output is 1x3x28x28</li>
<li>
<strong>(a)</strong> you do concatenation cat1 of x1 and x2 at axis <strong>2</strong>, the output is <strong>1x56x28x3</strong> (incorrect)</li>
<li>you do convolution x3, the output is 1x3x52x24</li>
<li>
<strong>(b)</strong> you do concatenation cat2 of x2 and x3 at axis <strong>2</strong> again, with this x2 and x3 dimensions, which are 1x56x28x3 (x2) and 1x3x52x24 (x3). These two will violate the *Rule1.</li>
</ul>
<p>Hence, you need to fix these ones <strong>(a and b)</strong> to make sure that they conform to *Rule1.<br>
Here is the correct model with dilated convolution that I corrected for you :).</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.nn import functional as F

class net(nn.Module):   
    def __init__(self):
        super(net, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=3, kernel_size=1)
        self.conv2 = nn.Conv2d(in_channels=3, out_channels=3, kernel_size=3, padding=2, dilation=2)
        self.conv3 = nn.Conv2d(in_channels=6, out_channels=3, kernel_size=5, padding=4, dilation=2)
        self.conv4 = nn.Conv2d(in_channels=6, out_channels=3, kernel_size=7, padding=6, dilation=2)
        self.conv5 = nn.Conv2d(in_channels=12, out_channels=3, kernel_size=3, padding=1)
        self.b = 1

    def forward(self, x, debug=False): 
        
        x1 = F.relu(self.conv1(x))
        if debug: print('x1:', x1.shape)
        x2 = F.relu(self.conv2(x1))
        if debug: print('x2:', x2.shape)
        cat1 = torch.cat((x1, x2), 1)
        if debug: print('cat1:', cat1.shape)
        x3 = F.relu(self.conv3(cat1))
        if debug: print('x3:', x3.shape)
        cat2 = torch.cat((x2, x3), 1)
        if debug: print('cat2:', cat2.shape)
        x4 = F.relu(self.conv4(cat2))
        if debug: print('x4:', x4.shape)
        cat3 = torch.cat((x1, x2, x3, x4),1)
        if debug: print('cat3:', cat3.shape)
        k = F.relu(self.conv5(cat3))
        if debug: print('k:', k.shape)

        if k.size() != x.size():
            raise Exception("k, haze image are different size!")

        output = k * x - k + self.b
        if debug: print('output:', output.shape)
        return F.relu(output)
    
net_instance = net()
b = torch.rand(1, 3, 28, 28)
with torch.no_grad():
    b = net_instance(b, True)

#OUTPUT:    
#x1: torch.Size([1, 3, 28, 28])
#x2: torch.Size([1, 3, 28, 28])
#cat1: torch.Size([1, 6, 28, 28])
#x3: torch.Size([1, 3, 28, 28])
#cat2: torch.Size([1, 6, 28, 28])
#x4: torch.Size([1, 3, 28, 28])
#cat3: torch.Size([1, 12, 28, 28])
#k: torch.Size([1, 3, 28, 28])
#output: torch.Size([1, 3, 28, 28])
</code></pre>
<p>Hope it helps, cheers~</p>

username: harsha_g
cooked: <aside class="quote no-group" data-username="BTNug" data-post="2" data-topic="86136">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/btnug/48/21849_2.png" class="avatar"> BTNug:</div>
<blockquote>
<p><strong>(a)</strong> you do concatenation cat1 of x1 and x2 at axis <strong>2</strong> , the output is <strong>1x56x28x3</strong> (incorrect)</p>
</blockquote>
</aside>
<p>Isn’t the cat of <code>x1</code> and <code>x2</code> at axis <strong>2</strong> → <strong><code>1x3x56x28</code></strong> unless there’s some kind of transpose happening which I don’t see?</p>
<p>By the way, welcome to the community <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20">.</p>

username: Patrice
cooked: <p>Dear Mr. Brilian, I am very grateful for your prompt reply. The code is now running. Please, I would like to get a better picture of convolution, padding and dilation.</p>
<aside class="quote no-group" data-username="BTNug" data-post="2" data-topic="86136">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/btnug/48/21849_2.png" class="avatar"> BTNug:</div>
<blockquote>
<p>you do convolution x2, the output is 1x3x28x28</p>
</blockquote>
</aside>
<p>after doing convolution x1, the output is 1x3x28x28. I understand this. Then doing conv2 of x1, how did you get 1x3x28x28? Should not it be 1x3x26x26 given that the kernel  is 3?</p>
<p>I am sorry if my question might seem silly. I had read about convolution, padding and dilation. However, it seems like I still have not got it clearly.</p>

username: Patrice
cooked: <p>Yes sir, you are right. cat of  <code>x1</code>  and  <code>x2</code>  at axis  <strong>2</strong>  should be  <strong><code>1x3x56x28</code></strong>. Please, could you clarify how the output of conv2 (x1) i.e  x2 is obtained?</p>

username: harsha_g
cooked: <p>Please refer to relationship 15 on page 28 of <a href="https://arxiv.org/pdf/1603.07285.pdf" rel="nofollow noopener">this</a> excellent guide to convolution arithmetic.</p>

username: BTNug
cooked: <p>Hi <a class="mention" href="/u/patrice">@Patrice</a>, some explanation of conv2(x1):</p>
<p>Notes:</p>
<blockquote>
<p>Padding: like adding some zero values on the edges of your image, ie. padding 1 in pytorch will change your x1 from 1x3x28x28 to 1x3x30x30 (1 pixel top-down and left-right), etc.</p>
</blockquote>
<blockquote>
<p>Dilation: is like you are expanding your kernel by 2x2 (like <a href="https://images.app.goo.gl/Anh1DZj7bsyjRVVm8" rel="noopener nofollow ugc">this</a>), ie, you have kernel=3, which is 3x3 kernel. Then you add 2x2 = 5x5 kernel, etc.</p>
</blockquote>
<p>To answer your question:</p>
<ol>
<li>
<ul>
<li>you do convolution with 3x3 kernel on 1x3x28x28, the output will be  1x3x26x26 <strong>IF</strong> the padding is not added (padding=0) and dilation is 1.</li>
</ul>
</li>
<li>
<ul>
<li>you do convolution with 3x3 kernel on 1x3x28x28, the output will be  1x3x24x24 <strong>IF</strong> the padding is not added (padding=0) and dilation is 2.</li>
</ul>
</li>
<li>
<ul>
<li>you do convolution with 3x3 kernel on 1x3x28x28, the output will be  1x3x26x26 <strong>IF</strong> the padding is  added (padding=1) and dilation is 2.</li>
</ul>
</li>
<li>
<ul>
<li>you do convolution with 3x3 kernel on 1x3x28x28, the output will be  1x3x28x28 <strong>IF</strong> the padding is added (padding=2) and dilation is 2.</li>
</ul>
</li>
</ol>
<p>Thanks for the welcome <a class="mention" href="/u/harsha_g">@harsha_g</a><br>
Hope it helps, cheers ~</p>

username: Patrice
cooked: <p>Thank you very much Mr. Brilian. It is much clearer now.</p>

username: Patrice
cooked: <p>Thank you sir. I have gone through it and it has been very helpful.</p>

username: BTNug
cooked: <p>good point, it is a typo, thanks for the correction <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: BTNug
cooked: <p>Cool! And happy coding <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>