Is my model too complex?

Anton_Neznamov: <p>Hi!<br>
I am designing a model to search for patterns in videos and trace these patterns behaviour over time.<br>
I’m using Conv3D+LSTM layers (model is present below).</p>
<p>The idea is to track patterns of different size over different periods of time.<br>
Is my model architecture too complex for that task?<br>
I am facing serious lack of memory to train the model.</p>
<p>I’d really appreciate a profeccional look at my case.</p>
<pre><code>     class VideoModel(nn.Module):

    def __init__(self, num_frames, num_channels, num_classes):
        super(VideoModel, self).__init__()
        self.conv1 = nn.Conv3d(num_channels, 64, kernel_size=3, stride=1, padding=1)
        self.conv2 = nn.Conv3d(64, 128, kernel_size=5, stride=1, padding=2)
        self.conv3 = nn.Conv3d(128, 256, kernel_size=10, stride=1, padding=5)
        self.conv4 = nn.Conv3d(256, 512, kernel_size=25, stride=1, padding=12)
        self.conv5 = nn.Conv3d(512, 1024, kernel_size=25, stride=1, padding=12)  # Add this line
        self.lstm1 = nn.LSTM(512, 512, num_layers=2, batch_first=True, bidirectional=True)
        self.lstm2 = nn.LSTM(512, 512, num_layers=2, batch_first=True, bidirectional=True)
       self.fc = nn.Linear(512, num_classes)

    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = self.conv3(x)
        x = self.conv4(x)
        x = self.conv5(x) 
        x = x.view(x.size(0), x.size(1), -1)
        x, _ = self.lstm1(x)
        x, _ = self.lstm2(x)
        x = self.fc(x[:, -1, :])
        return x
   model = VideoModel(num_frames=151, num_channels=3, num_classes=1)
   criterion = nn.MSELoss()
   optimizer = optim.Adam(model.parameters(), lr=learning_rate)
</code></pre>

tom: <aside class="quote no-group" data-username="Anton_Neznamov" data-post="1" data-topic="191923">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anton_neznamov/48/65088_2.png" class="avatar"> Anton_Neznamov:</div>
<blockquote>
<p>I am facing serious lack of memory to train the model.</p>
</blockquote>
</aside>
<p>We all are, otherwise we’d not be making good use of those pricey GPUs. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>More seriously:</p>
<aside class="quote no-group" data-username="Anton_Neznamov" data-post="1" data-topic="191923">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anton_neznamov/48/65088_2.png" class="avatar"> Anton_Neznamov:</div>
<blockquote>
<p>The idea is to track patterns of different size over different periods of time.<br>
Is my model architecture too complex for that task?</p>
</blockquote>
</aside>
<p>
Four
 five thoughts (I started out at two — amongst my thoughts are such diverse elements as…):</p>
<ul>
<li>You’re not using activation functions etc. What’s up with that? You should probably take a good look at a tried and tested model (e.g. ResNet) for the basics.</li>
<li>One typical thing is to reduce the resolution in a stack of convs. You might even try 2d convs (temporarily treating the time dimension as another “batch” dim). Should help with memory.</li>
<li>3d is large! You could downscale (depending on where you are at now) before feeding into the network.</li>
<li>When you have something vaguely feasible, one typical sanity check is to try to train the model with a single batch, which the model should be able overfit on, i.e. get the error very small. If it cannot, your model is too rigid.</li>
<li>For training LSTMs, a standard thing to do is backpropagation through time (BPTT), a key element of which is to truncate the time to get by with finite memory.</li>
</ul>
<p>In the olden das, a common thing to do was to do line character recognition with conv + lstm, you might check out some of those and draw inspiration from that.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: I think it might be worth looking for some more in-depth advice, it would seem that the details of the above are a bit beyond what’s usually on the forums.</p>

J_Johnson: <p>The <code>kernel_size</code>s you’re using seem a bit large. Typically, you want kernels of size 3 or 5. If you’re going to experiment with larger sized kernels, I suggest doing these in split branches.</p>
<p>Additionally, for classification type tasks, maxpooling is good to perform between layers. This acts in tandem with the kernels to filter the right information for a classification task.</p>
<p>Lastly, batchnorm can be good to include for better stability.</p>