Trying to implement Transfer Learning but getting unreasonable results

Zador: <p>Hi!<br>
I followed the pytorch tutorial on the topic transfer learning but I am getting results that are clearly wrong. As the application I am trying to apply is different a had to make some tweaks and it would be great if someone can give me some insight.</p>
<p>You can see below that I am here removing the final layer of the pre-trained model instead of replacing it.</p>
<pre><code class="lang-auto">model = models.resnet50(pretrained=True)
model = torch.nn.Sequential(*(list(model.children())[:-1]))
</code></pre>
<p>And below you can see why I took a different approach. I need to compare the features of 3 images, produced by the pretrained model, and I do this by concatenating said features and passing it through my fully connected layers, which I will optimize.</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(input_dim, 600)
        self.fc2 = nn.Linear(600, 600)
        self.fc3 = nn.Linear(600, output_dim)

    def forward(self, input_1, input_2, input_3, pretrained_model):
        with torch.no_grad():
            output_1 = pretrained_model(input_1.float())
            output_2 = pretrained_model(input_2.float())
            output_3 = pretrained_model(input_3.float())

        s = torch.cat((output_1, output_2, output_3), dim=1).view(output_1.shape[0], -1)
        
        s = F.relu(self.fc1(s.float()))
        s = F.relu(self.fc2(s))
        output = self.fc3(s)
        return output
</code></pre>
<p>The “classification” process is based on whether image 2 or image 3 is more similar to image 1: prediction [0, 1] or [1, 0]</p>
<p>Using the Cross entropy loss and Adam optimizer the running loss does decrease, however when a pass the validation set, the model strictly speaking predicts the same thing for all possible inputs (since the data is evenly split between [0, 1] and [1, 0] I get an accuracy of 0.5).</p>
<p>Please let me know if you have any idea what might be going on here. Have been debugging for a long time now.</p>

ptrblck: <p>Since you don’t want to train the resnet, did you also call <code>model.eval()</code> before using it (or just for the validation)?</p>
<p>Did you process the validation images in a different way than the training images?</p>

Zador: <p>Hello again!<br>
Both the training and the validation images are processed using the same functions.</p>
<p>Your first point, however, might be of concern. I did not use model.eval() but I thought passing my data using “with torch.no_grad():” would do the same job. Can it be that this is not the case?</p>
<p>So to be clear, instead of this in my forward pass:</p>
<pre><code class="lang-auto">with torch.no_grad():
            output_1 = pretrained_model(input_1.float())
            output_2 = pretrained_model(input_2.float())
            output_3 = pretrained_model(input_3.float())
</code></pre>
<p>you suggest I do this when downloading the pretrained model?</p>
<pre><code class="lang-auto">model = models.resnet18(pretrained=True)
model = torch.nn.Sequential(*(list(model.children())[:-1]))
model.eval()
model = model.to(device)
</code></pre>

ptrblck: <p><code>torch.no_grad()</code> will avoid storing the intermediate tensors during validation and testing and will thus save memory.<br>
<code>model.eval()</code> will change the behavior of some layers. E.g. it will disable dropout layers and use the running stats in batchnorm layers instead of the batch statistics.</p>
<p>If you didn’t use <code>model.eval()</code> during your training, the batchnorm statistics would have been updated (which might be a valid use case). However, this shouldn’t be used during evaluation or testing, especially if your batch size is small, as this could knock out the running stats.</p>

Zador: <p>As an update, this solved the problem! Thank you</p>