Using pretrained VGG-16 to get a feature vector from an image

username: Estrella
cooked: <p>Hi,<br>
I want to get a feature vector out of an image by passing the image through a pre-trained VGG-16. I used the pretrained Resnet50 to get a feature vector and that worked perfectly. But when I use the same method to get a feature vector from the VGG-16 network, I don’t get the 4096-d vector which I assume I should get. I got the code from a variety of sources and it is as follows:</p>
<pre><code class="lang-auto">vgg16_model=models.vgg16(pretrained=True)
modules=list(vgg16_model.children())[:-1]
vgg16_model=nn.Sequential(*modules)

data=moveaxis(data,2,0)
img_var=Variable(torch.from_numpy(data).unsqueeze(0)).float()
features_var=vgg16_model(img_var)
features=features_var.data
features=features.data.numpy()
print(features.shape)
</code></pre>
<p>The variable “data” is an image numpy array of dimensions (300, 400, 3)<br>
Hence I use the move axis to jumble the axis so that I have 3 channels and not 300.<br>
The output(features.shape) which I get is : (1, 512, 7, 7)<br>
I want a 4096-d vector as the VGG-16 gives before the softmax layer.<br>
I even tried the list(vgg16_model.classifier.children())[:-1] approach but that did not go too well too. There are a lot of discussions about this but none of them worked for me. Let me know where I might be going wrong… Thank you!</p>

username: yash1994
cooked: <p>Hi,</p>
<p>For <code>vgg-16</code> available in <code>torchvision.models</code> when you call <code>list(vgg16_model.children())[:-1]</code> it will remove whole <code>nn.Sequential</code> defined as following:</p>
<pre><code class="lang-auto">Sequential(
  (0): Linear(in_features=25088, out_features=4096, bias=True)
  (1): ReLU(inplace=True)
  (2): Dropout(p=0.5, inplace=False)
  (3): Linear(in_features=4096, out_features=4096, bias=True)
  (4): ReLU(inplace=True)
  (5): Dropout(p=0.5, inplace=False)
  (6): Linear(in_features=4096, out_features=1000, bias=True)
</code></pre>
<p>So it will also remove layer generating your feature vector (4096-d). You have to remove layers from <code>nn.Sequential</code> block given above. Like,</p>
<pre><code class="lang-python">vgg16_model = models.vgg16(pretrained=True)
vgg16_model.classifier = vgg16_model.classifier[:-1] # vgg16_model.classifier is nn.Sequential block.
</code></pre>

username: Estrella
cooked: <p>Thanks for the reply Yash…<br>
But unfortunately, this doesn’t work too…<br>
My modified code is :</p>
<pre><code class="lang-auto">vgg16_model=models.vgg16(pretrained=True)
vgg16_model_classifier=vgg16_model.classifier[:-1]
img_var=Variable(torch.from_numpy(data).unsqueeze(0)).float()

features_var=vgg16_model_classifier(img_var)

features=features_var.data
features=features.data.numpy()

print(features.shape)
</code></pre>
<p>Now it throws a size mismatch error…<br>
Thanks…</p>

username: yash1994
cooked: <p>Hi,</p>
<p>There seems to be a mistake in your code:<br>
change</p>
<pre><code class="lang-auto">vgg16_model_classifier=vgg16_model.classifier[:-1]
</code></pre>
<p>To</p>
<pre><code class="lang-auto">vgg16_model.classifier = vgg16_model.classifier[:-1]
</code></pre>
<p>And pass your image to <code>vgg16_model</code>.</p>

username: Estrella
cooked: <p>Oh, that’s awesome! It worked! Thanks a lot <a class="mention" href="/u/yash1994">@yash1994</a> !</p>

username: Estrella
cooked: <p><a class="mention" href="/u/yash1994">@yash1994</a><br>
Hi,<br>
If I have the following image array :</p>
<pre><code class="lang-auto">arr=np.random.rand(300,400,3)
</code></pre>
<p>And I do the following to it :</p>
<pre><code class="lang-auto">temp_obs=moveaxis(arr,2,0)
img_var=Variable(torch.from_numpy(temp_obs).unsqueeze(0)).float()
</code></pre>
<p>And then pass it to the vgg model like :</p>
<pre><code class="lang-auto">features_var=vgg16_model(img_var)
</code></pre>
<p>I get a numpy array full of zeros.<br>
Would you know why?</p>
<p>Thanks…</p>

username: Estrella
cooked: <p>I even tried declaring the VGG model as follows but it doesn’t work too. This one gives dimensionality errors :</p>
<pre><code class="lang-auto">vgg16_model=models.vgg16(pretrained=True)
# vgg16_model.classifier=vgg16_model.classifier[:-1]
modules_vgg=list(vgg16_model.classifier[:-1])
vgg16_model=nn.Sequential(*modules_vgg)
</code></pre>

username: yash1994
cooked: <p>Hi,</p>
<p>You need to put the model in inferencing model with <code>model.eva()</code> function to turn off the dropout/batch norm before extracting the feature. And try extracting features with an actual image with imagenet class.</p><aside class="quote" data-post="7" data-topic="76496">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/estrella/48/22701_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/using-pretrained-vgg-16-to-get-a-feature-vector-from-an-image/76496/7">Using pretrained VGG-16 to get a feature vector from an image</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    I even tried declaring the VGG model as follows but it doesn’t work too. This one gives dimensionality errors : 
vgg16_model=models.vgg16(pretrained=True)
# vgg16_model.classifier=vgg16_model.classifier[:-1]
modules_vgg=list(vgg16_model.classifier[:-1])
vgg16_model=nn.Sequential(*modules_vgg)
  </blockquote>
</aside>
<p>
This will result in dimension error because you are re-defining model as following:</p>
<pre data-code-wrap="python"><code class="lang-python">Sequential(
  (0): Linear(in_features=25088, out_features=4096, bias=True)
  (1): ReLU(inplace=True)
  (2): Dropout(p=0.5, inplace=False)
  (3): Linear(in_features=4096, out_features=4096, bias=True)
  (4): ReLU(inplace=True)
  (5): Dropout(p=0.5, inplace=False)
</code></pre>
<p>so this expects flat input of 25088 dimensional array.</p>

username: Estrella
cooked: <p>Okay…<br>
<a class="mention" href="/u/yash1994">@yash1994</a> I just added the model.eval() in the code and then tried to extract features but still an array of zeros…<br>
I also tried passing a real image of dimensions 300x400x3. Do you think that is a problem?</p>
<p>Thanks…</p>

username: Estrella
cooked: <p>Actually I just iterated over the entire array and saw that not all values are zeros. But there are quite a few which are zero. I don’t understand why they are zeros though…</p>

username: yash1994
cooked: <p>Just take two images of a bus (an imagenet class) from google images, extract feature vector and compute cosine similarity. if cosine similarity is good and those feature vector are similar then there is no problem, otherwise there is some issue.</p>

username: Estrella
cooked: <p>Okay! That makes sense… Thank you very much…</p>