Accessing intermediate layers of a pretrained network forward?

username: flupadup
cooked: <p>Hi, I want to get outputs from multiple layers of a pretrained VGG-19 network. I have already done that with this approach, that I found on this board:</p>
<pre><code class="lang-auto">class AlexNetConv4(nn.Module):
            def __init__(self):
                super(AlexNetConv4, self).__init__()
                self.features = nn.Sequential(
                    # stop at conv4
                    *list(original_model.features.children())[:-3]
                )
            def forward(self, x):
                x = self.features(x)
                return x
</code></pre>
<p>Initializing a new net for every output I am interested in, occupies a lot space on my GPU, therefore I would like to follow this approach, via the forward method:</p>
<pre><code class="lang-auto">def forward(self, x):
    out1 = F.relu(self.conv1(x))
    out2 = F.relu(self.conv2(out1))
    out3 = F.relu(self.conv3(out2))
    return out1, out2, out3
</code></pre>
<p>The problem is, that I don’t know how to get the names of the convolutions in a pretrained VGG-Net that I got from the torch vision models.<br>
Hope someone can help me out with that! <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: chenyuntc
cooked: <p>Here is a example to get output of specified layer in vgg16</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/chenyuntc/pytorch-book/blob/master/chapter8-%E9%A3%8E%E6%A0%BC%E8%BF%81%E7%A7%BB(Neural%20Style)/PackedVGG.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/chenyuntc/pytorch-book/blob/master/chapter8-%E9%A3%8E%E6%A0%BC%E8%BF%81%E7%A7%BB(Neural%20Style)/PackedVGG.py" target="_blank" rel="nofollow noopener">chenyuntc/pytorch-book/blob/master/chapter8-风格迁移(Neural Style)/PackedVGG.py</a></h4>
<pre><code class="lang-py">#coding:utf8
import torch
import torch.nn as nn
from torchvision.models import vgg16
from collections import namedtuple

class Vgg16(torch.nn.Module):
    def __init__(self):
        super(Vgg16, self).__init__()
        features = list(vgg16(pretrained = True).features)[:23]
        # features的第3，8，15，22层分别是: relu1_2,relu2_2,relu3_3,relu4_3
        self.features = nn.ModuleList(features).eval() 
        
    def forward(self, x):
        results = []
        for ii,model in enumerate(self.features):
            x = model(x)
            if ii in {3,8,15,22}:
                results.append(x)
        
</code></pre>

  This file has been truncated. <a href="https://github.com/chenyuntc/pytorch-book/blob/master/chapter8-%E9%A3%8E%E6%A0%BC%E8%BF%81%E7%A7%BB(Neural%20Style)/PackedVGG.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>the 3rd, 8th, 15th,22nd layer is relu1_2,relu2_2,relu3_3,relu4_3.</p>
<p>BTW, you may use</p>
<p>
```Python
<br>
your code
<br>
```
</p>
to format your code
<p>it’ll goes like</p>
<pre><code class="lang-auto">your code
</code></pre>

username: flupadup
cooked: <p>wow, thanks a lot for your help!</p>

username: flupadup
cooked: <p>With your approach I need to do several iterations, to get all my desired outputs. I would like to speed up my training by</p>
<pre><code class="lang-auto">                     def forward(self, x):
    out1 = F.relu(self.conv1(x))
    out2 = F.relu(self.conv2(out1))
    out3 = F.relu(self.conv3(out2))
    return out1, out2, out3
</code></pre>
<p>I have searched a lot, but I can’t find a way, to get the names of the different layers, of the pretrained pytorch vgg19 model.<br>
Thanks for your help!</p>

username: chenyuntc
cooked: <aside class="quote no-group" data-username="flupadup" data-post="4" data-topic="12113">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/e56c9b/48.png" class="avatar"> flupadup:</div>
<blockquote>
<p>With your approach I need to do several iterations, to get all my desired outputs.</p>
</blockquote>
</aside>
<p>Could you make it clear which layer do you want? You don’t need to do extral forward.</p>

username: flupadup
cooked: <p>I am training a network for the image synthesis, based on this paper <a href="https://arxiv.org/abs/1707.09405" rel="nofollow noopener">https://arxiv.org/abs/1707.09405</a> and get my loss from a vgg 19 net. The layers I am currently interested in are 3,7,8,15,22 and 32. The ideas is, to get a more complex loss function, by comparing the detected features of a reference and a synthesised image</p>

username: chenyuntc
cooked: <p>change   vgg16-&gt;vgg 19</p>
<aside class="quote no-group" data-username="chenyuntc" data-post="2" data-topic="12113">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chenyuntc/48/299_2.png" class="avatar"> chenyuntc:</div>
<blockquote>
<p>if ii in {3,8,15,22}:</p>
</blockquote>
</aside>
<p>change  it  to  3 7 8 15 22</p>
<aside class="quote no-group" data-username="chenyuntc" data-post="2" data-topic="12113">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chenyuntc/48/299_2.png" class="avatar"> chenyuntc:</div>
<blockquote>
<p>features = list(vgg16(pretrained = True).features)[:23]</p>
</blockquote>
</aside>
<p>change  23 to  33</p>
<p>there  are no  extral  consumptions.</p>

username: alwynmathew
cooked: <p>How can I access intermediate layers from <a href="http://pytorch.org/docs/master/torchvision/models.html#torchvision.models.resnet50" rel="nofollow noopener">resnet50</a>?</p>
<pre><code class="lang-auto">&gt;&gt;&gt; list(models.vgg16().features)
[Conv2d(3, 64, kernel_size=(3,3), ... #list all layers
&gt;&gt;&gt; list(models.resnet50().features)
AttributeError: 'Resnet' object has no attribute 'features'
</code></pre>
<p>And thats true, <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py" rel="nofollow noopener">it</a> doesn’t have attribute ‘features’ unlike <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/vgg.py#L12-L21" rel="nofollow noopener">vgg</a>.</p>
<p>Thank you.</p>

username: mmcm
cooked: <p>sorry for playing the necromancer here, but I have the same issue.<br>
I am still new to pytorch, but if I understand your proposed solution correctly, then your proposed solution here:</p>
<aside class="quote no-group" data-username="chenyuntc" data-post="2" data-topic="12113">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chenyuntc/48/299_2.png" class="avatar"> chenyuntc:</div>
<blockquote>
<p>for ii,model in enumerate(self.features):<br>
x = model(x)</p>
</blockquote>
</aside>
<p>computes only exactly <em>one</em> forward pass, right?<br>
That is, <code>conv1_1</code> will be executed exactly once, and the result will be reused to compute <code>conv1_2</code>?<br>
This should be fine <em>iff</em> I don’t want to compute pre-relu activations. For conv outputs <em>before</em> ReLU I’d need to save <code>x.copy()</code> I guess, because ReLU is computed in-place?</p>
<p>EDIT:<br>
Also, this only works with the convolutional features, right? If I wanted to extract e.g. the <code>fc6</code> features of a vgg19 network (that is, the first fc layer after the conv blocks) I’d have to extract all submodules from <code>net.classifier</code> and build my own <code>nn.Sequential</code> with those?</p>

username: niloofar1996
cooked: <p>hello. what I want to do is to feed some input to alexnet and get the middle layer output of specific layer(feature maps) then save them and feed them to another CNN as training data. for example the second network is autoencoder. now my question is how to feed the middle layer outputs to this new network as training data?</p>

username: sebamenabar
cooked: <p>Hi, you can try this pytorch utility <a href="https://pypi.org/project/torch-intermediate-layer-getter/" class="inline-onebox" rel="noopener nofollow ugc">torch-intermediate-layer-getter · PyPI</a></p>