Cut a model from intermediate layers

twister9458: <p>Hello All;</p>
<p>I already have a trained model that has several convolution blocks (conv + relu + pooling), which is saved and loaded.</p>
<p>Here it is:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/1/6107f590bab4835855b1f057a1c2b050344d6671.png" data-download-href="https://discuss.pytorch.org/uploads/default/6107f590bab4835855b1f057a1c2b050344d6671" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/6107f590bab4835855b1f057a1c2b050344d6671_2_502x500.png" alt="image" data-base62-sha1="dQnmLCcxT2b9idXwvyRz09aevPb" width="502" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/6107f590bab4835855b1f057a1c2b050344d6671_2_502x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/6107f590bab4835855b1f057a1c2b050344d6671_2_753x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/6107f590bab4835855b1f057a1c2b050344d6671_2_1004x1000.png 2x" data-dominant-color="F0F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1098×1092 71.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here’s what I’m trying to do:</p>
<ul>
<li>I want to be able, to reuse only a chunk of my model, let’s say inputting at Conv2d-5 and outputting at the last Linear-19, or inputting at BatchNorm2d-11 and outputting at the last layer still Linear-19.</li>
<li>The target is to be able to choose any intermediate layer, and give it a tensor as input w.r.t the layer input dimension, and outputting with the last layer.</li>
</ul>
<p>I wanted to use nn.ModuleList() and append from mymodel’s layers, but mymodel.state_dict().items() doesn’t print my pooling layers.<br>
I’ve tried also mymodel._modules.keys(), it’s viewing the pooling layer, but not in the order they are called in my Neural Network.<br>
See:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a87e60a6de0b5f08cd74a68828aad1f939c4a450.png" data-download-href="https://discuss.pytorch.org/uploads/default/a87e60a6de0b5f08cd74a68828aad1f939c4a450" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a87e60a6de0b5f08cd74a68828aad1f939c4a450_2_690x54.png" alt="image" data-base62-sha1="o2yYCcnznmPdEf6GHr7t5AGLZsI" width="690" height="54" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a87e60a6de0b5f08cd74a68828aad1f939c4a450_2_690x54.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a87e60a6de0b5f08cd74a68828aad1f939c4a450_2_1035x81.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a87e60a6de0b5f08cd74a68828aad1f939c4a450_2_1380x108.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2178×172 27 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thank you very much for your help,<br>
Habib</p>

eqy: <p>I haven’t tried this before, but I wonder if this simple approach would work:</p>
<ol>
<li>add two members to your model definition <code>self.start_layer, self.end_layer</code>
</li>
<li>build a mapping from layer indices to the layer member in your model e.g., <code>self.layers = {1:self.conv1, 2:self.conv2,...}</code>
</li>
<li>edit the forward function to be something like</li>
</ol>
<pre><code class="lang-auto">def forward(self, x):
    for i in range(self.start_layer, self.end_layer+1):
        x = self.layers[i](x)
    return x
</code></pre>
<p>Then you just need to do something like <code>model.start_layer = ?</code> and <code>model.end_layer=?</code> to set up how you want to use your model each time.</p>

twister9458: <p>Thank you Eqy for your reply.</p>
<p>Your solution would have worked indeed, if I always had the code of all the models.</p>
<p>But my purpose targets all saved models, which means I don’t have access to the code, I can only load and save, like all we do with available pre-trained models (resnet18, vgg16, …).</p>
<p>My project is coding TCAV technique, which is compelling me to input from all the intermediate layers, for each of the received models.</p>
<p>I’ve found some code in a github repo, they use the .keys() and .items() attributes from inception pretrained, but as said in my first post, .keys() and .children() don’t know how to output pooling layers, and they don’t get back the right order of the layers (see my first post):</p>
<pre><code class="lang-auto">class InceptionV3_cutted(torch.nn.Module):
    def __init__(self, inception_v3, bottleneck):
        super(InceptionV3_cutted, self).__init__()
        names = list(inception_v3._modules.keys())
        layers = list(inception_v3.children())

        self.layers = torch.nn.ModuleList()
        self.layers_names = []

        bottleneck_met = False
        for name, layer in zip(names, layers):
            if name == bottleneck:
                bottleneck_met = True
                continue  # because we already have the output of the bottleneck layer
            if not bottleneck_met:
                continue
            if name == 'AuxLogits':
                continue

            self.layers.append(layer)
            self.layers_names.append(name)
</code></pre>
<p>The repo =&gt; <a href="https://github.com/myracheng/tcav" class="inline-onebox" rel="noopener nofollow ugc">GitHub - myracheng/tcav: Quantitative Testing with Concept Activation Vectors in PyTorch</a></p>
<p>Thank you very much,</p>

eqy: <p>I’m confused by what you mean when you say you don’t have the code of the model, as all pretrained models in <code>torchvision</code> like the ones you named have code as well. Can you share how you’re loading and running the model?</p>

twister9458: <p>Thank you Eqy for your reply;</p>
<p>Sorry, I wasn’t clear with my explanations. I’m loading my model with the following code (I have my code of course):</p>
<pre><code class="lang-auto">mymodel = MyModelClass()
mymodel.load_state_dict(torch.load(PATH))
mymodel.eval()
</code></pre>
<p>My target is to cut any loaded model, without modifying the initial code. Hence being able to loop over all the layers, respecting their order in the neural network loaded, and input at any intermediate layer chosen for the cut.</p>
<p>If you can only show me how to access all the layers of a loaded model, respecting their order in the network, I’ll be able to copy weights and biases and all the parameters of the loaded model, in the new cut model. Actually, all the layers are printed not in order with .keys() and .named_parameters(), and also .state_dict().items(), so I cannot access the layers in their implemented sequence.</p>
<p>Also, I’ve seen that the source code of torchsummary is able to access the layers in their implemented order. They’re using hooks. I’m trying to figure how this works.</p>
<p>Tell me if I’m still not clear. (I’ve been using PyTorch for only 4 months, sorry for my poor level)</p>
<p>Thank you very much<br>
Habib</p>

Jimut_Bahan_Pal: <aside class="quote no-group" data-username="eqy" data-post="2" data-topic="124034">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/48/46233_2.png" class="avatar"> eqy:</div>
<blockquote>
<pre><code class="lang-auto">def forward(self, x):
    for i in range(self.start_layer, self.end_layer+1):
        x = self.layers[i](x)
    return x
</code></pre>
</blockquote>
</aside>
<p>I am a bit late here, but I am also stuck in something similar. Did you guys find any solution?</p>
<p>Like mentioning two layers, and cutting a model in between will really help for generic models.</p>