How to Concatenate layers in PyTorch similar to tf.keras.layers.Concatenate

Purvak-L: <p>I’m trying to implement the following network in pytorch. I’m not sure if the method I used to combine layers is correct. In given network instead of convnet I’ve used pretrained VGG16 model.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/9e3b57848dd1067f412d26f0e452fe45c10a99a4.png" data-download-href="https://discuss.pytorch.org/uploads/default/9e3b57848dd1067f412d26f0e452fe45c10a99a4" title="deeprank"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9e3b57848dd1067f412d26f0e452fe45c10a99a4_2_690x391.png" alt="deeprank" data-base62-sha1="mzMzWl8zHAGfmJllyWNVwwNQ6na" width="690" height="391" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9e3b57848dd1067f412d26f0e452fe45c10a99a4_2_690x391.png, https://discuss.pytorch.org/uploads/default/original/2X/9/9e3b57848dd1067f412d26f0e452fe45c10a99a4.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/9/9e3b57848dd1067f412d26f0e452fe45c10a99a4.png 2x" data-dominant-color="DADCD8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">deeprank</span><span class="informations">909×516 35.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">model = models.vgg16(pretrained=True)  
new_classifier = nn.Sequential(*list(model.classifier.children())[:-1])
model.classifier = new_classifier
class Network(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3,96,8, stride=16,padding=1)
        self.maxpool1 = nn.MaxPool2d(3,4,padding=1)
        self.conv2 = nn.Conv2d(3,96,8, stride=32,padding=1)
        self.maxpool2 = nn.MaxPool2d(7,2,padding=1)
        
    def forward(self,x):
        out1 = model(x)
        y = self.conv1(x)
        y = self.maxpool1(y)
        y = F.normalize(y,dim=1,p=2)
        z = self.conv2(x)
        z = self.maxpool2(z)
        z = F.normalize(z,dim=1,p=2)
        out = torch.cat((out1,y,z),1)
        return out
test = Network()
print(summary(test,input_size=(3,224,224)))
</code></pre>
<p>When I print summary it shows:</p>
<p>RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 4 and 2 in dimension 2 at c:\a\w\1\s\windows\pytorch\aten\src\th\generic/THTensorMoreMath.cpp:1333</p>
<p>Where am I going wrong?</p>

ptrblck: <p>The shapes of the tensors you wish to concatenate do not match.</p>
<pre><code class="lang-python">print(out1.shape)
&gt; torch.Size([1, 4096])
print(y.shape)
&gt; torch.Size([1, 96, 4, 4])
print(z.shape)
&gt; torch.Size([1, 96, 2, 2])
</code></pre>
<p>Based on the image you’ve posted it seems the conv activations should be flattened to a tensor with the shape <code>[batch_size, 2 * 4*4*96 = 3072]</code>.<br>
You could add this using:</p>
<pre><code class="lang-python">y = y.view(y.size(0), -1)
z = z.view(y.size(0), -1)
out = torch.cat((out1, y, z), 1)
</code></pre>
<p>However, even then the architecture won’t match, since <code>s</code> is only <code>[batch_size, 96, 2, 2]</code>.</p>

Purvak-L: <p>So, what do you recommend I should do?<br>
I wish to implement model like this</p>
<pre><code>convnet_model = convnet_model_()
first_input = Input(shape=(224,224,3))
first_conv = Conv2D(96, kernel_size=(8, 8),strides=(16,16), padding='same')(first_input)
first_max = MaxPool2D(pool_size=(3,3),strides = (4,4),padding='same')(first_conv)
first_max = Flatten()(first_max)
first_max = Lambda(lambda  x: K.l2_normalize(x,axis=1))(first_max)
second_input = Input(shape=(224,224,3))
second_conv = Conv2D(96, kernel_size=(8, 8),strides=(32,32), padding='same')(second_input)
second_max = MaxPool2D(pool_size=(7,7),strides = (2,2),padding='same')(second_conv)
second_max = Flatten()(second_max)
second_max = Lambda(lambda  x: K.l2_normalize(x,axis=1))(second_max)
merge_one = concatenate([first_max, second_max])

merge_two = concatenate([merge_one, convnet_model.output])
emb = Dense(4096)(merge_two)
l2_norm_final = Lambda(lambda  x: K.l2_normalize(x,axis=1))(emb)

final_model = Model(inputs=[first_input, second_input, convnet_model.input], outputs=l2_norm_final)</code></pre>

ptrblck: <p>Thanks for the code.<br>
It looks like to padding of your second max pooling layer is wrong, since you are using the <code>same</code> argument in Keras.<br>
Try this definition <code>self.maxpool2 = nn.MaxPool2d(7,2,padding=3)</code> and your output will be <code>[batch_size, 96, 4, 4]</code> for both branches.</p>

neda_vida: <p>Hi. I want to concatenate last layer before fc in two models of resnet and desnet to each other. but it gives me error. Could you please write this code as an example. I dont know how to write it right. thank u so much.</p>

ptrblck: <p>Could you share your code and explain your approach as well as what is currently not working, please?</p>

neda_vida: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/f/6f535472cb588b1c0fb54ad18d782792e6998393.jpeg" alt="photo_2020-10-13_21-56-06" data-base62-sha1="fSPx1x6AyAf0SoZ2eyam7q4sder" width="447" height="489"> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b97336e0d0048115315b5306bc8497a48b774e22.png" data-download-href="https://discuss.pytorch.org/uploads/default/b97336e0d0048115315b5306bc8497a48b774e22" title="plan"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b97336e0d0048115315b5306bc8497a48b774e22.png" alt="plan" data-base62-sha1="qszb0ZAJYQintMqX2iV5d3vE2VY" width="690" height="300" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">plan</span><span class="informations">1280×558 4.39 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have problem in training…i dont know what should i do? i appreciate if you help me? Thank you.</p>

ptrblck: <p>Assuming wrapping the model into the <code>nn.Sequential</code> container works fine, the code looks alright.<br>
I would additionally recommend to add an activation function between the linear layers.</p>
<p>Note that some models are using the functional API in its <code>forward</code>, which could break the model if you just slice the children and add them into <code>nn.Sequential</code>. In that case you could replace the <code>classifier</code> of <code>fc</code> module with an <code>nn.Identity</code> module.</p>

neda_vida: <p>Thank you so much for your reply. about this sentence you said:</p>
<p>Assuming wrapping the model into the <code>nn.Sequential</code> container works fine, the code looks alright.<br>
I would additionally recommend to add an activation function between the linear layers.</p>
<p>one time i put nn.Sequential(nn.Linear…,…)…but there wasnt any difference. It’s kind of difficult problem. and i am new to pytorch…i dont know how to solve it? <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

Muhammad_Farooq: <p>hi,<br>
I would like to know what would be the impact if we put the nn.sequential models inside the network class as well.</p>