Combined model result in Linear Layer followed by RNN for classification

A_Rza_SH: <p>hi, i’m new in pytorch and trying to bind the <a href="https://discuss.pytorch.org/t/combining-trained-models-in-pytorch/28383">dual model CNN structure</a> with the RNN module for the classification purpose. The RNN code copied from pytorch tutorials. So, facing problem in binding the combined model result by linear layer with RNN block</p>
<p>one of the Error i Got</p>
<pre><code class="lang-auto">line 356, in forward
    combined = torch.cat((input, hidden), dim=1)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 1 and 8 in dimension 0 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:71
</code></pre>
<pre><code class="lang-auto">class MyRNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(MyRNN, self).__init__()

        self.hidden_size = hidden_size

        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input, hidden):
        input = input.view(input.size(0), -1)
        hidden = hidden.view(hidden.size(0), -1)
        combined = torch.cat((input, hidden), dim=1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        output = self.softmax(output)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)

class Net(nn.Module):
    def __init__(self, modelA, modelB):
        super(Net, self).__init__()
        self.con = nn.Conv2d(3, 3, 3, 1, 1, 1)
        self.Lrelu = nn.ReLU()
        self.modelA = modelA
        self.modelB = modelB
        self.final_classifier1 = nn.Linear(512, 128)
        self.myRNN = MyRNN(128, 128, 10)


    def forward(self, x):
        x = self.con(x)
        x = self.Lrelu(x)
        x1 = self.modelA(x.clone())
        x1 = x1.view(x1.size(0), -1)
        x2 = self.modelB(x.clone())
        x2 = x2.view(x2.size(0), -1)
        x = torch.cat((x1, x2), dim=1)
        x = self.final_classifier1(F.relu(x))
        x = self.myRNN(x, hidden)
        return x

n_hidden = 128
hidden = torch.zeros(1, n_hidden)
hidden = hidden.to(device)

model = Net(modelA, modelB)
</code></pre>
<p>Also tried to initialize the hidden side in the training epoch</p>
<pre><code class="lang-auto">...
for epoch in range(1, n_epochs + 1):

    hidden = MyRNN.initHidden()
    model.train()
...
</code></pre>

ptrblck: <p>The shape mismatch seems to be raise in</p>
<pre><code class="lang-python">combined = torch.cat((input, hidden), dim=1)
</code></pre>
<p>which points to a different batch size (dim0).</p>
<p>It seems you are initializing the <code>hidden</code> tensor as <code>torch.zeros(1, self.hidden_size)</code>, which would use a fixed batch size of 1.</p>
<p>You could add a <code>batch_size</code> argument to <code>initHidden()</code> and use it to create the right <code>hidden</code> tensor in the <code>forward</code> method via:</p>
<pre><code class="lang-python">x = self.final_classifier1(F.relu(x))
hidden = self.myRNN.initHidden(x.size(0))
x = self.myRNN(x, hidden)
</code></pre>
<p>Let me know, if that would work for you.</p>

A_Rza_SH: <p>Thanks for identifying the issue. It has been solved by passing the batch size to the initHidden function</p>
<pre><code class="lang-auto">    def initHidden(self, batch_size):
        return torch.zeros(batch_size, self.hidden_size).to(device)
</code></pre>

A_Rza_SH: <p>But i’m getting another issue</p>
<pre><code class="lang-auto">AttributeError: 'tuple' object has no attribute 'log_softmax'
</code></pre>
<p>and i guess its related to this post <a href="https://discuss.pytorch.org/t/attributeerror-tuple-object-has-no-attribute-log-softmax/37309/4" class="inline-onebox">AttributeError: 'tuple' object has no attribute 'log_softmax'</a></p>

ptrblck: <p>Based on your code you are calling the <code>nn.LogSoftmax</code> activation on <code>output</code>, which is the output activation of <code>self.i2o</code> (<code>nn.Linear</code>), so it shouldn’t be a <code>tuple</code>.<br>
Did you modify the code somehow and could post the complete stack trace including the layer, which raises this error?</p>

A_Rza_SH: <p>Similar error even not using the nn.Logsoftmax activation on out beacuse i tried another nn.Linear layer and also with the direct output.<br>
the piece of code worked for me is;</p>
<pre><code class="lang-auto">class MyRNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(MyRNN, self).__init__()

        self.hidden_size = hidden_size

        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
        #self.outp = nn.Linear(output_size, 7)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input, hidden):
        input = input.view(input.size(0), -1)
        hidden = hidden.view(hidden.size(0), -1)
        combined = torch.cat((input, hidden), dim=1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        #output = self.outp(output)

        output = self.softmax(output)
        return output, hidden

    def initHidden(self, batch_size):
        return torch.zeros(batch_size, self.hidden_size).to(device)

...
...
    def forward(self, x):
       ....
       ....
        hidden = self.myRNN.initHidden(x.size(0))
        x = self.myRNN(x, hidden)
        return x

</code></pre>
<pre><code class="lang-auto">AttributeError: 'tuple' object has no attribute 'log_softmax'
</code></pre>
<p>this error was solved by changing the lines in training code but i wont able to figure out the exact location from where i got the tuple</p>
<pre><code class="lang-auto">for epoch in range(1, n_epochs + 1):
    ....
    model.train()
    ....
    output = Net(data)
    loss = criterion(output[0], target)
    ....
    ....
    out, pred = torch.max(F.softmax(output[0], 1),1)
    ....
</code></pre>
<p>taking the specific output[0] for the criterion solves the error may be beacuse the MyRNN block is returning 2 output values</p>
<p>Thanks</p>

ptrblck: <p>I guess you might have used <code>F.softmax(output, 1)</code> before changing it to <code>F.softmax(output[0], 1)</code>?<br>
Since <code>output</code> is a <code>tuple</code>, this would explain the error. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>