Backpropagation in a multi-branched CNN-RNN network

username: lv_Poellhuber
cooked: <p>So I have this network that has the following architecture:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f414024e8e0e25e5f913686b5d5dcba4815b079d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f414024e8e0e25e5f913686b5d5dcba4815b079d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f414024e8e0e25e5f913686b5d5dcba4815b079d_2_669x500.jpeg" alt="image" data-base62-sha1="yPdn3zLy3D5fKFFEmDaqJ6Iuw7r" width="669" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f414024e8e0e25e5f913686b5d5dcba4815b079d_2_669x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f414024e8e0e25e5f913686b5d5dcba4815b079d_2_1003x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f414024e8e0e25e5f913686b5d5dcba4815b079d_2_1338x1000.jpeg 2x" data-dominant-color="CBBEB2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1960×1464 527 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The goal of this architecture is to extract features out of a sequence of 30 frames, without knowing what those features are. The 30 frames are represented as images, but they aren’t real images, which is why we can’t have a classification prior to passing it through the CNN.</p>
<p>After the “images” go through the CNN, they become vectors, which are fed to an LSTM who will try to find patterns through time. We have the label that should be output by the LSTM, a 0 or a 1, but we don’t have the classification for the CNN. This architecture’s whole goal is to get rid of feature engineering, basically (based on this paper: <a href="https://www.researchgate.net/publication/319897145_Deep_Model_for_Dropout_Prediction_in_MOOCs" rel="noopener nofollow ugc">https://www.researchgate.net/publication/319897145_Deep_Model_for_Dropout_Prediction_in_MOOCs</a>).</p>
<p>Now, my question is whether or not our model would backpropagate properly. Here is our code:</p>
<pre><code class="lang-auto">class MyCNN(nn.Module):
    def __init__(self):
        super(MyCNN, self).__init__()
        # convolutional layer (sees matrices 24x48x1)
        self.conv1 = nn.Conv2d(1, 20, 5)
        # convolutional layer (sees matrices 20x44x20)
        self.conv2 = nn.Conv2d(20, 50, 5)
        # max pooling layer
        self.pool = nn.MaxPool2d(2, 2)
        # fully conntected layer (takes in input a matric of 3x9x50, makes it into a vector of 20)
        self.fc = nn.Linear(3 * 9 * 50, 20)
        

    def forward(self, x):
        # add sequence of convolutional and max pooling layers
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        # flatten image input (is it to keep???)
        x = x.view(-1, 3 * 9 * 50)
        # add 1st hidden layer, with relu activation function
        x = F.relu(self.fc(x))
        return x
</code></pre>
<p>We then define the ConRec network, who takes as input the output of MyCNN():</p>
<pre><code class="lang-auto">class ConRec(nn.Module):
    def __init__(self):
        super(ConRec, self).__init__()
        
        # lstm layer (20 long vectors)
        self.lstm = nn.LSTM(20, 50, 1, batch_first=True)
        
    def forward(self, x):
        x = F.relu(self.lstm(x))
        return x
</code></pre>
<p>And finally, we gather the output of the 30 frames passed through the CNN and concatenate them (the code might be wrong, I’m still a beginner!). The CNN is the same for all 30 branches, so it only has one set of weights.<br>
And then we feed that through the LSTM.</p>
<pre><code class="lang-auto">for d in data:
    output=MyCNN(d)
    full_out=torch.cat([full_out, output], dim=1)
#30 items in data

result=ConRec(full_out)

</code></pre>
<p>When I apply the loss, would it backpropagate properly? I’m afraid it would only apply the loss to the LSTM and not the CNN, since they’re not directly connected.</p>

username: KFrank
cooked: <p>Hi Louis-Vincent!</p>
<aside class="quote no-group" data-username="lv_Poellhuber" data-post="1" data-topic="122648" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lv_poellhuber/48/38842_2.png" class="avatar"> lv_Poellhuber:</div>
<blockquote>
<p>When I apply the loss, would it backpropagate properly? I’m afraid it would only apply the loss to the LSTM and not the CNN, since they’re not directly connected.</p>
</blockquote>
</aside>
<p>I haven’t looked at your code, but yes, in general the kind of thing<br>
you propose doing should work just fine.</p>
<p>Make sure you understand the basics of how pytorch uses the<br>
<code>require_grad</code> property of a tensor.  Normally the weights, etc.,<br>
of your model, <em>and the tensors that depend on them,</em> will have<br>
<code>requires_grad = True</code>.  The input to your CNN would normally<br>
have <code>requires_grad = False</code>.  However, the <em>output</em> of your CNN<br>
<em>will</em> have <code>requires_grad = True</code> (because it depends on the<br>
weights of the CNN).  The “connection” between the LSTM and<br>
the CNN is provided by the fact that the output of your CNN – that<br>
has <code>requires_grad = True</code> – is also the input to your LSTM.  This<br>
is what permits pytorch’s autograd system to backpropagate your<br>
final loss through the LSTM all the way back to the weights of your<br>
upstream CNN.</p>
<p>Best.</p>
<p>K. Frank</p>

username: lv_Poellhuber
cooked: <p>Thank you, I think it’s working!</p>