Finetuning the x3d pytorch model

lola1: <p>Hello,</p>
<p>I am trying to fine tune x3d_s on my video dataset of approx. 700 vidoes. I removed 6. block of pretrained x3d and added 2 1D CNNs as well as ReLu, BatchNorm and Dropout layers.</p>
<p>After training for 40 epochs, the result doesnt seem right. Validation loss is often the same or lower than training loss, additionally, validation accuracy is higher than training loss. Also, accuracy does not seem to increase a lot…</p>
<p>How could I improve this model? Is the problem in the size of dataset?</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = torch.hub.load('facebookresearch/pytorchvideo', 'x3d_s', pretrained=True)
        for param in self.model.parameters():
            param.requires_grad = False

        self.model.blocks[5] = nn.Identity()

        self.conv1 = nn.Conv1d(in_channels = 192*8*8, out_channels = 1024, kernel_size = 1) 
        self.batch1 = nn.BatchNorm1d(1024)

        self.conv2 = nn.Conv1d(in_channels = 1024, out_channels = 256, kernel_size = 1) 
        self.batch2 = nn.BatchNorm1d(256)

        self.relu = nn.ReLU()

        self.dropout = nn.Dropout(0.2)

        self.flat = nn.Flatten()
        self.fc1 = nn.Linear(11520, 256)
        self.fc2 = nn.Linear(256, 8)


    def forward(self,x):
        x = self.model(x) 
        x = x.permute(0,2,1,3,4)
        x = x.reshape(x.size(0), x.size(1), -1)
        x = x.permute(0,2,1)
        x = self.conv1(x)
        x = self.relu(x)
        x = self.dropout(x)
        x = self.batch1(x)
        x = self.conv2(x)
        x = self.relu(x)
        x = self.dropout(x)
        x = self.batch2(x)
        x = self.flat(x)
        x = self.fc1(x)
        x = self.dropout(x)
        x = self.fc2(x)


        return x
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a7b4f221998534f87146d7f383df109ce060e92d.png" data-download-href="https://discuss.pytorch.org/uploads/default/a7b4f221998534f87146d7f383df109ce060e92d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a7b4f221998534f87146d7f383df109ce060e92d.png" alt="image" data-base62-sha1="nVBpywAUu4uCazHDXZvlgA23bRz" width="690" height="415" data-dominant-color="F9F7F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">751×452 17.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/3/73378af0762046f0441522a796f0c66fc46b9aff.png" data-download-href="https://discuss.pytorch.org/uploads/default/73378af0762046f0441522a796f0c66fc46b9aff" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/3/73378af0762046f0441522a796f0c66fc46b9aff.png" alt="image" data-base62-sha1="grfUlDGtPNYvJ19Sz1MnnUU75GT" width="690" height="414" data-dominant-color="F8F4F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">752×452 25 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>