Use transfer learning on CNN to train on 2 different datasets (CV)

Ioannis_D: <p>I am trying to replicate a paper that trains a CNN on two different datasets and uses transfer learning for this. I am writing to check whether my code is right.<br>
Here is the architecture used to train on the first dataset. I am using transfer learning to get the feature extraction of googlenet and train the model on my custom dataset.</p>
<pre><code class="lang-auto">class cvv_train(nn.Module):
  def __init__(self, num_classes):
    super(cvv_train, self).__init__()
    # import googlenet
    googletnet = torch.hub.load('pytorch/vision:v0.10.0', 'googlenet', pretrained=True)
    
    # feature extraction
    self.feature_extractor = nn.Sequential(*list(googletnet.children())[:-2])# pool5-drop_7x7_s1

    self.final = nn.Sequential(
            nn.Linear(googletnet.fc.in_features, num_classes), nn.BatchNorm1d(num_classes, momentum=0.01)
        )
  def forward(self, x):
    #no gradient/backpropagation for learning
    with torch.no_grad():
        x = self.feature_extractor(x) #assume: 1024x1x1
    x = x.view(x.size(0), -1) #assume: 1024x1
    return self.final(x) #assume: 512x1
</code></pre>
<pre><code class="lang-auto">
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = cvv_train(312)
model = model.to(device)
model.train()

PATH = 'path to trained model'

model.load_state_dict(torch.load(PATH))

</code></pre>
<p>Now, I will use the model trained (again take the feature extraction layer and change the last layer).</p>
<pre><code class="lang-auto">class cvv_train2(nn.Module):
  def __init__(self, num_classes, model):
    super(cvv_train2, self).__init__()
    # import googlenet
    self.model = model

    # feature extraction
    self.feature_extractor = nn.Sequential(*list(self.model.children())[0])# pool5-drop_7x7_s1

    self.final = nn.Sequential(
            nn.Linear(self.model.final[0].in_features, num_classes), nn.BatchNorm1d(num_classes, momentum=0.01)
        )
  def forward(self, x):
    #no gradient/backpropagation for learning
    with torch.no_grad():
        x = self.feature_extractor(x) #assume: 1024x1x1
    x = x.view(x.size(0), -1) #assume: 1024x1
    return self.final(x) #assume: 512x1
</code></pre>
<pre><code class="lang-auto">model2 = cvv_train2(num_classes, model)
</code></pre>
<p>How can I check which weights do not change?<br>
Also does this seem correct?<br>
The paper that I am trying to replicate is <strong>Facial Expression Recognition in Videos: An CNN-LSTM based Model for Video Classification</strong> and page 2 explains what is needs to be done for the training of CNN</p>

ptrblck: <aside class="quote no-group" data-username="Ioannis_D" data-post="1" data-topic="161411">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ioannis_d/48/52562_2.png" class="avatar"> Ioannis_D:</div>
<blockquote>
<p>How can I check which weights do not change?</p>
</blockquote>
</aside>
<p>You could <code>deepcopy</code> the <code>state_dict</code> before a single update and compare the values to the new <code>state_dict</code> after the parameters were updated to check which parameters were updated.</p>
<aside class="quote no-group" data-username="Ioannis_D" data-post="1" data-topic="161411">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ioannis_d/48/52562_2.png" class="avatar"> Ioannis_D:</div>
<blockquote>
<p>Also does this seem correct?</p>
</blockquote>
</aside>
<p>I haven’t checked the architecture, but assume you’ve verified that no e.g. shape mismatch etc. is raised.<br>
It looks generally alright besides the last <code>nn.BatchNorm1d</code> layer as the output layer, which seems to be uncommon but it doesn’t mean it’s necessarily wrong.</p>

Ioannis_D: <aside class="quote no-group" data-username="Ioannis_D" data-post="1" data-topic="161411">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ioannis_d/48/52562_2.png" class="avatar"> Ioannis_D:</div>
<blockquote>
<p>Facial Expression Recognition in Videos: An CNN-LSTM based Model for Video Classification</p>
</blockquote>
</aside>
<p>Thank you for your response.<br>
Another question on the same project that I had is do you have any advise on training on a large dataset with many classes. I am working on a part of the VGG-face dataset and it takes forever for a small decrease in the loss. I would expect that using the googlenet to be faster.</p>
<p>Thank you in advance!</p>