Concatenate layer output with additional input data

Raul_Gombru: <p>I want to build a CNN model that takes additional input data besides the image at a certain layer.<br>
To do that, I plan to use a standard CNN model, take one of its last FC layers, concatenate it with the additional input data and add FC layers processing both inputs.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/0/026dd0cc0a67815b0e7ceb707383c9c3e241acd0.png" alt="Dibujo%20sin%20t%C3%ADtulo" data-base62-sha1="lueuKSMSlAnr0KErO6YgZCH2P6" width="450" height="251"></p>
<p>The code I need would be something like:</p>
<pre><code class="lang-auto">additional_data_dim = 100
output_classes = 2
model = models.__dict__['inception_v3']
del(model._modules['fc'])
# TODO: Concatenate the CNN layer with the additional data
model.fc1 = nn.Linear(2048 + additional_data_dim, 2048 + additional_data_dim)
model.fc2 = nn.Linear(2048 + additional_data_dim, output_classes)

</code></pre>
<p>How should I code that?</p>
<p>Thanks</p>

ptrblck: <p>Here is a small example for your use case:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.cnn = models.inception_v3(pretrained=False, aux_logits=False)
        self.cnn.fc = nn.Linear(
            self.cnn.fc.in_features, 20)
        
        self.fc1 = nn.Linear(20 + 10, 60)
        self.fc2 = nn.Linear(60, 5)
        
    def forward(self, image, data):
        x1 = self.cnn(image)
        x2 = data
        
        x = torch.cat((x1, x2), dim=1)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x
        

model = MyModel()

batch_size = 2
image = torch.randn(batch_size, 3, 299, 299)
data = torch.randn(batch_size, 10)

output = model(image, data)
</code></pre>
<p>I chose random values for the linear layers, so you should use your constrains like <code>additional_data_dim</code>.</p>

Raul_Gombru: <p>Thank you! Understood, working on it!</p>

abeer_almarridi: <p>hello, if I would I have two models and I want to concatenate the last layer, so the last layer is shared. can I use the same approach ?</p>

ptrblck: <p>Sure!<br>
You would have to pass your input(s) through both models and concat the outputs before the final layer.</p>

Uk_Jo: <p>Here why does it set dim as 1 in torch.cat function? I think it would be 0?</p>

justusschock: <p>Dim 0 means that you treat the outputs as independent batch samples since you increase the batchsize. However, this is probably not what one would want in this case,since the results of both models belong to the same sample.</p>
<p>Using dim 1 concatenates in each samples channels which makes more sense since they are now belonging to the same sample and this approach can (unlike the first one) even be used if the model’s produce a different number of channels (although the rest of the sizes must be equal)</p>

Uk_Jo: <p>Thanks for the reply. Dim 0 means batch dims same as keras.  Your reply helped me a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

tunesh_verma: <h2>I am getting error while concatenating two layer with different dimensions<br>
torch.Size([1, 256, 13, 13])<br>
torch.Size([1, 512, 26, 26])</h2>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
----&gt; 1 s,d = a(inp)</p>
<p>D:\Softwares\anacond33\lib\site-packages\torch\nn\modules\module.py in <strong>call</strong>(self, *input, **kwargs)<br>
475             result = self._slow_forward(*input, **kwargs)<br>
476         else:<br>
–&gt; 477             result = self.forward(*input, **kwargs)<br>
478         for hook in self._forward_hooks.values():<br>
479             hook_result = hook(self, input, result)</p>
<p> in forward(self, x, CUDA)<br>
51                     print(map2.shape)<br>
52<br>
–&gt; 53                     x = torch.cat((map1, map2), dim=1)<br>
54                 outputs[i] = x<br>
55</p>
<p>RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 13 and 26 in dimension 2 at c:\programdata\miniconda3\conda-bld\pytorch_1533096106539\work\aten\src\th\generic/THTensorMath.cpp:3616</p>

rchavezj: <p>Why did you provided the code</p>
<pre><code class="lang-python">self.cnn.fc = nn.Linear(self.cnn.fc.in_features, 20)
</code></pre>
<p>Because I don’t see the variable being used inside the forward function</p>

ptrblck: <p><code>self.cnn.fc</code> is the “classifier layer” of the <code>inception_v3</code> model.<br>
I just replaced it with my own linear layer to change the number of output neurons.</p>
<p>The model is used in <code>x1 = self.cnn(image)</code>. <code>self.cnn.fc</code> is thereby called inside the <code>forward</code> of the <code>inception_v3</code> model: <a href="https://github.com/pytorch/vision/blob/b94e2c8207a9a1e05f36d963ecb5c319ea4bf5fb/torchvision/models/inception.py#L123">line of code</a>.</p>

mbakalar: <p>I am also trying to concatenate two tensors to use as input to a linear layer. Each tensor is the output of a subnetwork that independently processes two input tensors. I am getting the following error:</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>I suspect the error has to do with the temporary assignment of a variable to hold the concatenated data <code>x3 = torch.cat((x1, x2), dim=1)</code>. The network is defined below:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # Energy map net
        self.conv1 = nn.Conv2d(19, 32, (1,3), padding=(0,1))
        self.pool = nn.MaxPool2d((1,3))
        self.conv2 = nn.Conv2d(32, 64, (1,3))
        self.fc1 = nn.Linear(64 * 9 * 10, 120)
        self.fc2 = nn.Linear(120, 60)

        # Sequence net
        self.fc3 = nn.Linear(9 * 22, 120)
        self.fc4 = nn.Linear(120, 60)

        # Combine (concatenated 60 + 60)
        self.fc5 = nn.Linear(60 + 60, 120)
        self.fc6 = nn.Linear(120, 2)

    def forward(self, x1, x2):
        # Energy map net
        x1 = self.pool(F.relu(self.conv1(x1)))
        x1 = self.pool(F.relu(self.conv2(x1)))
        x1 = x1.view(-1, 64 * 9 * 10)
        x1 = F.relu(self.fc1(x1))
        x1 = F.relu(self.fc2(x1))

        # Sequence net
        x2 = x2.view(-1, 9 * 22)
        x2 = F.relu(self.fc3(x2))
        x2 = F.relu(self.fc4(x2))

        # Combine
        x3 = torch.cat((x1, x2), dim=1)
        x3 = F.relu(self.fc5(x3))
        x3 = self.fc5(x3)

        return x3
</code></pre>

ptrblck: <p>It looks like you have a typo and use <code>self.fc5</code> twice.<br>
However, even if you do, it should work.<br>
Could you change that and run it again?</p>

mbakalar: <p>Thanks for the response. I fixed the typo, and I am still receiving the same error message.</p>
<p><code>RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.</code></p>
<p>Here is my training loop:</p>
<pre><code class="lang-auto">for epoch in range(5):
    running_loss = 0.0
    for i, data in enumerate(train_loader, 0):
        # get the inputs
        inputs, labels = data
        energy = inputs[0]
        sequence = inputs[1]

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(energy, sequence)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
</code></pre>

ptrblck: <p>Yeah, I thought so, since reusing a module shouldn’t throw this error.<br>
I made some assumptions about your code to reproduce the issue. Could you check, if the shapes and criterion are right?</p>
<pre><code class="lang-python">x1 = torch.randn(1, 19, 9, 96)
x2 = torch.randn(1, 9, 22)    
target = torch.randint(0, 2, (1,))
criterion = nn.CrossEntropyLoss()

model = Net()
output = model(x1, x2)
loss = criterion(output, target)
loss.backward()
</code></pre>
<p>This code works on my machine. Are you using a custom criterion?</p>

mbakalar: <p>Yes, you are correct. The code you provided does run. I took my training loop and replaced the data coming from the torch.utils.data.DataLoader (train_loader) with a hardcoded random tensor:</p>
<pre><code class="lang-auto">for i, data in enumerate(train_loader, 0):

    # get the inputs
    inputs, labels = data
    #energy = inputs[0]
    #sequence = inputs[1]

    energy = torch.randn(4, 19, 9, 96)
    sequence = torch.randn(4, 9, 22)
</code></pre>
<p>This eliminates the error message. So, it must have something to do with the way I am getting data out of the DataLoader. Looks like it’s not a problem with the concatenation step!</p>
<p>The problem actually seems to lie in an earlier embedding layer that I used to generate the input data that is fed via the DataLoader:</p>
<pre><code class="lang-auto">emb = nn.Embedding(depth, depth)
emb.weight.data = torch.eye(depth)
tensor = torch.from_numpy(sequences)
emb(tensor)
</code></pre>
<p>In my case, this problem is solved by removing the autograd by calling detach():</p>
<pre><code class="lang-auto">emb(tensor).detach()
</code></pre>
<p>In any case, thanks for the quick help!</p>

Or_Ram: <p>Hi!<br>
Solved!<br>
Just had to indert the two databases to the dataset as:</p>
<pre><code class="lang-auto">learn_dataset = torch.utils.data.TensorDataset(train, ts_train, y_train)
learn_loader = torch.utils.data.DataLoader(learn_dataset, batch_size = batch_size, shuffle=True)
for train, ts_train, labels in learn_loader:
output = model(train, ts_train)
</code></pre>
<p>Thanks for the answers. I have a followup regarding Datasets for these kind of networks.<br>
I have two Data sets I’m working with, one is a time-series and the other is continues and binary data. Each training example has one of each.<br>
I would you recommend building a Dataset that can be used for a network that has concat in it? It seems that all the examples on Datasets work with only one kind of input.</p>

Mauricio_Maroto: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> Considering this post of yours, where you can concatenate different nn’s “bodies” before the final layer by using torch.cat(…), may I know what other methods are there available?</p>

ptrblck: <p>It depends on your use case and the provided shapes.<br>
E.g. if both activations have the same shape, you could use a lot of reductions to create a new activation (sum, mean, etc).<br>
The usual way is to create a new activation tensor by concatenating the two previous activations.<br>
Are you looking for a specific method?</p>

Mauricio_Maroto: <p>I guess <a class="mention" href="/u/ptrblck">@ptrblck</a>  I was thinking lstm-ish and the logic of its gates. Does cat-ing and dense-ing different “bodies” does under the hood “gating”? Does min or mean similar shapes “gates” more than what dense-ing does? I guess I find lstm’s, and nn’s in general, fascinating.</p>