How to extract features of an image from a trained model

big_tree: <p>Hi all,</p>
<p>I try examples/imagenet of pytorch. It is awesome and easy to train, but I wonder how can I forward an image and get the feature extraction result?<br>
After I train with examples/imagenet/main.py, I get model as,<br>
<code>model_best.pth.tar</code><br>
And I load this file with<br>
<code>model = torch.load('model_best.pth.tar')</code><br>
which gives me a dict. How can I use <code>forward</code> method to get a feature (like fc7 layer’s output of Alexnet) of an image with this model?</p>

apaszke: <p>You can use the <code>torchvision.models</code> package, where you have functions for constructing various vision models, with an option of using pretrained weights. This:</p>
<pre><code class="lang-python">torchvision.models.resnet18(pretrained=True)
</code></pre>
<p>will give you a <code>nn.Module</code> with downloaded weights.</p>

fmassa: <p>To complement <a class="mention" href="/u/apaszke">@apaszke</a> reply, once you have a trained model, if you want to extract the result of an intermediate layer (say fc7 after the relu), you have a couple of possibilities.</p>
<p>You can either reconstruct the classifier once the model was instantiated, as in the following example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torchvision import models

model = models.alexnet(pretrained=True)

# remove last fully-connected layer
new_classifier = nn.Sequential(*list(model.classifier.children())[:-1])
model.classifier = new_classifier
</code></pre>
<p>Or, if instead you want to extract other parts of the model, you might need to recreate the model structure, and reusing the parts of the pre-trained model in the new model.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torchvision import models

original_model = models.alexnet(pretrained=True)

class AlexNetConv4(nn.Module):
            def __init__(self):
                super(AlexNetConv4, self).__init__()
                self.features = nn.Sequential(
                    # stop at conv4
                    *list(original_model.features.children())[:-3]
                )
            def forward(self, x):
                x = self.features(x)
                return x

model = AlexNetConv4()
</code></pre>

big_tree: <p><a class="mention" href="/u/apaszke">@apaszke</a> <a class="mention" href="/u/fmassa">@fmassa</a>  Thank you guys so much!</p>

kpar: <p>Is there a way to get values from multiple layers with one forward pass (for neural style transfer etc.)?</p>
<p>In Tensorflow it would be along the lines of</p>
<pre><code class="lang-python">features = sess.run([net.conv1, net.conv2, net.conv3])
</code></pre>

fmassa: <p><a class="mention" href="/u/kpar">@kpar</a> Yes, it’s possible and very easy in PyTorch. All you need to do is to return the multiple outputs that you want to retrieve.<br>
Here is an example</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        self.conv1 = nn.Conv2d(1, 1, 3)
        self.conv2 = nn.Conv2d(1, 1, 3)
        self.conv3 = nn.Conv2d(1, 1, 3)

    def forward(self, x):
        out1 = F.relu(self.conv1(x))
        out2 = F.relu(self.conv2(out1))
        out3 = F.relu(self.conv3(out2))
        return out1, out2, out3
</code></pre>

kpar: <p>That looks like it’ll do the trick.</p>
<p>Is there a convenient way to fetch the intermediate values when the forward behavior is defined by nn.Sequential()? It seems like right now the only way to compose multiple responses would be to split off all the individual layers and forward the values manually in forward().</p>
<p>Essentially what I want to do is take an existing network (e.g. VGG) and just pick some responses of some layers (conv1_1, pool1, pool2, etc.) and concatenate them into a feature vector.</p>

fmassa: <p>You could write your own sequential version that keeps track of all intermediate results in a list. Something like</p>
<pre><code class="lang-python">class SelectiveSequential(nn.Module):
    def __init__(self, to_select, modules_dict):
        super(SelectiveSequential, self).__init__()
        for key, module in modules_dict.items():
            self.add_module(key, module)
        self._to_select = to_select
    
    def forward(x):
        list = []
        for name, module in self._modules.iteritems():
            x = module(x)
            if name in self._to_select:
                list.append(x)
        return list
</code></pre>
<p>And then you could use it like</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.features = SelectiveSequential(
            ['conv1', 'conv3'],
            {'conv1': nn.Conv2d(1, 1, 3),
             'conv2': nn.Conv2d(1, 1, 3),
             'conv3': nn.Conv2d(1, 1, 3)}
        )

    def forward(self, x):
        return self.features(x)
</code></pre>

apaszke: 

lcelona: <p>Thank you for your help</p>

alexis-jacq: <p><a class="mention" href="/u/kpar">@kpar</a>, another way to extract features layer by layer of a pre-existing neural network is to recreate a new network, adding one by one all the layers from the pre-trained network, plus some “transparent layers” that just extract features:</p>
<pre><code>net = models.alexnet(pretrained=True).features`

class Feature_extractor(nn.module):
    def forward(self, input):
        self.feature = input.clone()
        return input

new_net = nn.Sequential().cuda() # the new network

target_layers = [conv_1, conv_2, conv_4] # layers you want to extract`

i = 1
for layer in list(cnn):
    if isinstance(layer,nn.Conv2d):
        name = "conv_"+str(i)
        art_net.add_module(name,layer)

        if name in target_layers:
            new_net.add_module("extractor_"+str(i),Feature_extractor())

        i+=1

    if isinstance(layer,nn.ReLU):
        name = "relu_"+str(i)
        new_net.add_module(name,layer)

    if isinstance(layer,nn.MaxPool2d):
        name = "pool_"+str(i)
        new_net.add_module(name,layer)

new_net.forward(your_image)
print new_net.extractor_3.feature</code></pre>

apaszke: <p><a class="mention" href="/u/alexis-jacq">@alexis-jacq</a> I wouldn’t recommend that. It’s better to keep your models stateless i.e. not hold any of the intermediate states. Otherwise, if you don’t pay enough attention to them, you might end up with problems when you’ll have references to the graphs you don’t need, and they will be only taking up memory.</p>
<p>If you really want to do something like that, I’d recommend this:</p>
<pre><code class="lang-python">class FeatureExtractor(nn.Module):
    def __init__(self, submodule, extracted_layers):
        self.submodule = submodule

    def forward(self, x):
        outputs = []
        for name, module in self.submodule._modules.items():
            x = module(x)
            if name in self.extracted_layers:
                outputs += [x]
        return outputs + [x]
</code></pre>
<p>This unfortunately uses a private member <code>_modules</code>, but I don’t expect it to change in the near future, and we’ll probably expose an API for iterating over modules with names soon.</p>

alexis-jacq: <aside class="quote no-group" data-username="apaszke" data-post="13" data-topic="119">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/48/21_2.png" class="avatar"> apaszke:</div>
<blockquote>
<p>Otherwise, if you don’t pay enough attention to them, you might end up with problems when you’ll have references to the graphs you don’t need, and they will be only taking up memory.</p>
</blockquote>
</aside>
<p>Wow! good to know that… Thanks!</p>
<aside class="quote no-group" data-username="apaszke" data-post="13" data-topic="119">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/48/21_2.png" class="avatar"> apaszke:</div>
<blockquote>
<p>we’ll probably expose an API for iterating over modules with names soon</p>
</blockquote>
</aside>
<p>That would be really useful!</p>

Atcold: <p>Hmm… all is well for simple networks, like AlexNet.<br>
What if I want to use something a bit more fancy, say ResNet-18, and wish to extract some intermediate representation?<br>
I can fetch the model with <code>torchvision.models.resnet18(pretrained=True)</code>. I think the network graph can be reconstructed by forwarding a <code>Variable</code> and checking who’s the parent, but I wouldn’t be sure about how to do this not by hand.<br>
Moreover I think the <code>tuple</code> attribute <code>classes</code> (mapping output class to label) is missing.</p>
<p>Can I register a hook to a specific child, and have it return its current output? In the old Torch I would have <code>model:get(myModuleIndex).output</code>, or a more nested combination, and got <code>myModuleIndex</code> from visual inspection of the model structure (using the <code>__tostring()</code> metamethod).</p>
<p>How do we hack these models? <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>
<p>OK, I’ve managed to display the graph with <code>from visualize import make_dot</code> (<a href="https://github.com/szagoruyko/functional-zoo/blob/master/visualize.py" rel="noopener nofollow ugc">here</a>’s <code>visualize</code>).<br>
Now how can I reach inside the graph?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/8aed44a93b41a6b1ef26dbd48c4c281af5f56543.png" data-download-href="https://discuss.pytorch.org/uploads/default/8aed44a93b41a6b1ef26dbd48c4c281af5f56543" title="net.png"><img src="https://discuss.pytorch.org/uploads/default/optimized/1X/8aed44a93b41a6b1ef26dbd48c4c281af5f56543_2_103x499.png" width="103" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/1X/8aed44a93b41a6b1ef26dbd48c4c281af5f56543_2_103x499.png, https://discuss.pytorch.org/uploads/default/optimized/1X/8aed44a93b41a6b1ef26dbd48c4c281af5f56543_2_154x748.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/1X/8aed44a93b41a6b1ef26dbd48c4c281af5f56543_2_206x998.png 2x" data-dominant-color="F6F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">net.png</span><span class="informations">404×1950 89 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So, let’s call the output <code>h_x</code> (which is $h_\theta(x)$).<br>
So, I can peek at my embedding if I do the following dirty sequence of operations (which I guess are not recommended).</p>
<pre data-code-wrap="python"><code class="lang-python">x = Variable(torch.rand(1, 3, 224, 224))
h_x = resnet_18.forward(x)
last_view = h_x.creator.previous_functions[0][0]
last_pool = last_view.previous_functions[0][0]
embedding = last_pool.saved_tensors[0]
</code></pre>
<p>And <code>embedding</code> should correspond to the output of the last <code>Threshold</code>, if I’m not mistaken. Still, this is not generally applicable to every functional block, since not all block cache the input.<br>
So, I’m still after a better way to dissect these nets…</p>

apaszke: <p>Why do you want to inspect the graph? A new instance is going to be created at every forward.</p>
<p>You can register a hook on any Variable using the <code>register_hook</code> method, or on any module - <code>register_backward_hook</code>.</p>
<p>What are you trying to do, and why doesn’t the <code>FeatureExtractor</code> I shown above work for that?</p>

Atcold: <p>Hmm, I thought that the <code>FeatureExtractor</code> you showed above can deal only with sequential models.<br>
The loop</p>
<pre><code class="lang-python">for name, module in self.submodule._modules.items():
    x = module(x)
</code></pre>
<p>passes forward the output of each <code>submodule</code>'s <code>module</code>, in a sequential manner, no?<br>
Maybe I don’t understand what the method <code>_modules()</code> provides…</p>

apaszke: <p>Yeah it does, but you can easily adapt it to other network topologies by replacing <code>self.submodule</code> with some other reference. <code>resnet18</code> is sequential if you only want to inspect the maps between the blocks.</p>

alexis-jacq: <p>I think <a class="mention" href="/u/atcold">@Atcold</a>  meant that <code>FeatureExtractor</code> only works for networks with submodules that are the layers. The problem is not solved for networks with “sub-sub-modules” like in resnet18 structure, maybe somewhere we need a recursive pass, in order to access the “leaf-modules” of the graph…</p>

Atcold: <p>Thank you <a class="mention" href="/u/alexis-jacq">@alexis-jacq</a> for addressing my specific issue.<br>
I’ve been suggested to use <a href="http://pytorch.org/docs/nn.html#torch.nn.Module.register_forward_hook" rel="nofollow noopener"><code>nn.Module.register_forward_hook()</code></a> to perform this job.<br>
I still have to figure out a smart way to get the reference to specific <code>nn.Module</code>s within a graph, but I think I can play with the visualisation script I posted above.</p>
<p><a class="mention" href="/u/alband">@albanD</a>, this is why I think the graph integration with your <em>TensorBoard</em> client is essential.<br>
So that I can click around and add some hooks on the fly.</p>

edran: <p><a class="mention" href="/u/atcold">@Atcold</a> The issue with adding graph integration to Crayon is that we need to essentially write an interface between whatever TensorBoard reads and PyTorch (or Torch, Theano, CNTK, etc.).</p>
<p>We managed to quickly write what we currently have because getting TensorBoard to take 2d values / histograms was relatively straightforward, however adding integration for more complex data types is not a one-day job.</p>
<p>After the ICML deadline we’ll give it a look.</p>