Manually inserting hand-crafted features to a pretrained model

Dr_John: <p>I am using a pretrained VGG model,</p>
<pre><code class="lang-auto">model = models.vgg19(pretrained=True)
</code></pre>
<p>and interested in adding hand-crafted features before the model.classifier forward pass.<br>
So, I am actually interested in doing:</p>
<pre><code class="lang-auto">feat = model.features( image )
model.classifier([feat,additional_features])
</code></pre>
<p>and then backpropagation to update the weights of the classifier, and the feature extraction layers.</p>
<p>What is the best way to write this in Pytorch? (I thought of writing a new class named “VGG_extended” that do the forward pass as required, but I am not exactly sure how the backprop will work in this case… Is there any simple solution?)</p>
<p>Thanks in advance!</p>

vmirly1: <p>I think you can define a new class , and define the vgg model and an extension linear layer separately inside that class. Then, in the <code>forward()</code> function, you get features from both sources (vgg features and hand-crafted features), and concatenate them and send them to the classifier. This way, the backpropagation will work on both parts.</p>
<pre><code class="lang-auto">def handcrafted_features(images):
    # Your hand-crafted features here (I am just using random numbers for now)
    hand_feat = np.random.normal(size=(len(images), 100))
    hand_feat = np.array(hand_feat, dtype='float32')
    return torch.tensor(hand_feat)

class VGGXtend(nn.Module):
    def __init__(self, num_hand_features, num_classes):
        super(VGGXtend, self).__init__()

        # define the vgg model:
        self.vgg = models.vgg16(pretrained=True)
        # change the last FC layer:
        self.vgg.classifier[6] = nn.Linear(4096, 1000, bias=True)

        # define the new classifier:
        self.classifier = nn.Linear(1000 + num_hand_features, num_classes)

    def forward(self, images):
        vgg_feat = self.vgg(images)
        hand_feat = handcrafted_features(images)
        features = torch.cat([vgg_feat, hand_feat], dim=1)
        print(features.shape)
        return self.classifier(features)
</code></pre>
<p>and a test example:</p>
<pre><code class="lang-auto">vggx = VGGXtend(100, 100)
images = np.random.normal(size=(16, 3, 224, 224))
images = np.array(images, dtype='float32')
vggx(torch.tensor(images))
</code></pre>

Dr_John: <p>Dear Vahid Mirjalili, Thanks a lot! That’s exactly what I was thinking of. The thing that I am not sure about is the backprop part. Since there is no learning on the hand-crafted features (since they are not “extracted”), what exactly would</p>
<pre><code class="lang-auto">loss.backward()
</code></pre>
<p>do? (since it also depends on the hand-crafted features)</p>
<p>Hope I made my question clear enough</p>

vmirly1: <p>Yes, it depends how the hand-crafted features are computed. If you use some functions and arrays outside PyTorch, like what I did in NumPy, and then convert them to PyTorch tensors, they will not be in the « computation-graph ».  So, in this setting, the actual features will not be back-propped, but only their final weights in the vggx.classifier will be back-proped.</p>
<p>However, if you want the hand-crafted features to also be included in the back-propagation, then you should define them in tensors and all the operations applied to them should use pytorch modules and learnable parameters. Then, those will also be automatically backpropped.</p>

Dr_John: <p>Thanks a lot for your explanation. I will give it a try <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Muhammad_Zawish: <p><a class="mention" href="/u/vmirly1">@vmirly1</a>, Hi, may I know how can we adapt this if we want to include these handcrafted_features or lets say any matrix operation somewhere between convolutional layers, lets say after 2nd conv in vgg?</p>