Why does training in Google Colab takes forever?

username: lola1
cooked: <p>I want to train my model that is fusion of pretrained 1d CNN network (loaded from checkpoint) and pretrained pytorch resnet (x3d_s).</p>
<p>At the moment the highest batch size that I can use is 8. Otherwise I am getting error that CUDA is out of memory. As a result the training of one epoch takes 5 minutes.</p>
<p>The number of learning parameters is 21 373 968. Is that way too much? What could I do to speed up the training. My dataset constains 700 videos with 45 frames (256x256) and 8 classes.</p>
<p>Is there maybe any other mistake in my code that makes training too slow?</p>
<p>Here some parts of code:</p>
<p>Model A</p>
<pre><code class="lang-auto">class ModelA(nn.Module):
    def __init__(self):
        super().__init__()

        self.conv1 = nn.Conv1d(in_channels = 51, out_channels = 64, kernel_size = 3, padding=1) 
        self.batch1 = nn.BatchNorm1d(64)

        self.dilated_conv2 = nn.Conv1d(in_feat, out_feat, kernel_size = 3, stride = stride, dilation = dilation, padding = 3) 
        self.conv_transform2 = nn.Conv1d(out_feat, out_feat, kernel_size = 3, padding = 1) 
        self.batch2 = nn.BatchNorm1d(128)

        self.dilated_conv3 = nn.Conv1d(in_feat, out_feat, kernel_size = 3, stride = stride, dilation = dilation, padding = 3)
        self.conv_transform3 = nn.Conv1d(out_feat, out_feat, kernel_size = 3, padding = 1)
        self.batch3 = nn.BatchNorm1d(256)
</code></pre>
<p>Model B</p>
<pre><code class="lang-auto">class ModelB(nn.Module):
    def __init__(self):
        super().__init__()

        self.model = torch.hub.load('facebookresearch/pytorchvideo', 'x3d_s', pretrained=True)

        for param in self.model.parameters():
            param.requires_grad = False

        self.model.blocks[5] = nn.Sequential()
        self.conv1 = nn.Conv1d(in_channels = 12288, out_channels = 1024, kernel_size = 1)
        self.conv2 = nn.Conv1d(in_channels = 1024, out_channels = 256, kernel_size = 1)
</code></pre>
<p>Fused model</p>
<pre><code class="lang-auto">class FusedModel(nn.Module):
    def __init__(self, modelA, modelB):
        super().__init__()

        self.modelA = modelA
        self.modelB = modelB
        self.lstm_extractor = nn.LSTM(input_size=512, hidden_size=512, num_layers=2, dropout = 0.2, batch_first=True)
        self.flat = nn.Flatten()
        self.fc1 = nn.Linear(512, 256)
        self.fc2 = nn.Linear(256, 8)

    def forward(self, x1, x2):
        x1 = self.modelA(x1) 
        x2 = self.modelB(x2) 
        x1 = x1.permute(0, 2, 1) 
        x = torch.concatenate([x1, x2], dim=2) 
        out, (ht, ct) = self.lstm_extractor(x) 
        out = ht[-1] 
        out = self.flat(out)

        out = self.fc1(out)
        out = self.fc2(out)
        return out
</code></pre>
<pre><code class="lang-auto">modelA = ModelA().float()
modelB = ModelB().float()

modelA.load_state_dict(checkpoint['model_state_dict'])

for param in modelB.model.blocks[5].parameters():
    param.requires_grad = True
for param in modelB.conv1.parameters():
    param.requires_grad = True
for param in modelB.conv2.parameters():
    param.requires_grad = True
</code></pre>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss() 

optimizer = torch.optim.Adam(model.parameters(), lr = lr)
</code></pre>
<p>Is there anything that can be done here to improve the speed or is the model too complex and dataset too big? Is there any faster faster resnet other than  ‘x3d_s’?</p>
<p>Here the first three epochs. I am concerned about unstable validation loss and validation accuracy. Is the reason for that too small batch?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9a174e47c5d445a0d22712cdfd6925e39603f43.png" data-download-href="https://discuss.pytorch.org/uploads/default/f9a174e47c5d445a0d22712cdfd6925e39603f43" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9a174e47c5d445a0d22712cdfd6925e39603f43.png" alt="image" data-base62-sha1="zCkOb5wLbl2H5ZN5nlFvguHB0Jl" width="690" height="45" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1347×88 6.66 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Edit: after few additional epochs I see my network doesnt really learn. Can also batch size be reason for that?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/46a1ae83f16c69dd2dd8abf7530238156a5ba137.png" data-download-href="https://discuss.pytorch.org/uploads/default/46a1ae83f16c69dd2dd8abf7530238156a5ba137" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/6/46a1ae83f16c69dd2dd8abf7530238156a5ba137.png" alt="image" data-base62-sha1="a4PP6UjfX1Ti3u2R4g72x888hin" width="690" height="108" data-dominant-color="E6E6E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1392×218 18.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thank you!</p>