Instantiate model layer by name

matheussfarias: <p>Hey guys!</p>
<p>My main goal is to reconstruct the forward propagation of a pretrained model from torchvision in a generic way, so that I can work on the inputs/weights/outputs of conv and fc layers online of any model I load (I’m not sure if hooks are useful here because as far as I understood you need to go over the whole model to get these values, and I need to work on inputs/weights/activations <strong>on the fly</strong>).</p>
<p>This is being annoying for 2 reasons. Let’s take the example of VGG11.</p>
<p>When I run the code</p>
<pre><code class="lang-auto">for x, y in tqdm(dataloader):
    for name, layer in model.named_modules():
        print(name)
    exit()
</code></pre>
<p>The output is</p>
<pre><code class="lang-auto">features
features.0
features.1
features.2
features.3
features.4
features.5
features.6
features.7
features.8
features.9
features.10
features.11
features.12
features.13
features.14
features.15
features.16
features.17
features.18
features.19
features.20
avgpool
classifier
classifier.0
classifier.1
classifier.2
classifier.3
classifier.4
classifier.5
classifier.6
</code></pre>
<p>Usually, when the name doesn’t have an index, like <strong>features</strong>, this name represents a group of layers, in this case</p>
<pre><code class="lang-auto">Sequential(
  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (1): ReLU(inplace=True)
  (2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (3): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (4): ReLU(inplace=True)
  (5): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (6): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (7): ReLU(inplace=True)
  (8): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (9): ReLU(inplace=True)
  (10): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (11): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (12): ReLU(inplace=True)
  (13): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (14): ReLU(inplace=True)
  (15): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (16): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (17): ReLU(inplace=True)
  (18): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (19): ReLU(inplace=True)
  (20): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
)
</code></pre>
<p>But when it has an index, like <strong>features.0</strong>, it’s an actual layer. In this case, features.0 is</p>
<p><code>Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))</code></p>
<p>This is not always the case since the layer <strong>avgpool</strong> doesn’t have index and it’s also not a group of layers.</p>
<p>So, the first question is how to instantiate actual layers so that I can just do a for loop and get the input and output of each layer. Running something like</p>
<pre><code class="lang-auto">correct=0
total=0
for x, y in tqdm(dataloader):
    for name, layer in model.named_modules():
        if is_actual_layer(layer):
                x = model[name](x)
    y_pred = x
    correct += (y_pred.argmax(axis=1) == y).sum().item()
    total += len(y)
</code></pre>
<p>That would be equivalent to</p>
<pre><code class="lang-auto">correct=0
total=0
for x, y in tqdm(dataloader):
    y_pred = model(x)
    correct += (y_pred.argmax(axis=1) == y).sum().item()
    total += len(y)
</code></pre>
<p>The second thing is that, usually, in the forward function, we have to flatten the vector for the classifier layers, since they are composed of linear layers. Is there any way to know when and how to flatten the input? For instance, in VGG11 forward function we have</p>
<pre><code class="lang-auto">def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
    x = self.features(x)
    x = self.avgpool(x)
    x = torch.flatten(x, 1) # input is flatten after avgpool layer
    x = self.classifier(x)
    return x
</code></pre>
<p>Any help is very welcome! Thank you</p>

ptrblck: <p>It’s not trivial since, as you’ve already pointed out, the functional API calls will be missing since these won’t show up in <code>named_modules()</code>.<br>
If you can guarantee your model only contains <code>nn.Modules</code> you could add a few checks for <code>nn.Sequential</code> etc. in <code>is_actual_layer</code>, but it seems you are more interested in seeing the computation graph. What’s your use case?</p>

matheussfarias: <p>Thanks for the quick answer,</p>
<p>Short answer:<br>
I want to run inference on different models on ImageNet, and for each conv and fc layer, I want to apply kind of a quantization on the weights and outputs.</p>
<p>Long answer:<br>
I’m gonna take the weights from each conv and fc layer, will write them on matrices such that each weight corresponds to a row in the matrix, and the columns of the matrix are negative powers-of-two multipliers. For example, take the weight matrix</p>
<p>(0.625  0.125)<br>
(0.5          0    )</p>
<p>My matrix will be<br>
2⁰        2^(-1)     2^(-2)<br>
(0           1             1)       → 0.625<br>
(0           0             1)       → 0.125<br>
(0           1             0)       → 0.5<br>
(0           0             0)       → 0</p>
<p>This is kind of a quantization for the weights. Then I will multiply these weights with the input and will give me outputs. These outputs will be in the form of powers-of-two as well. I will first perform a standard quantization on it, approximating the result from each column to the nearest quantized region, and then I will convert from the power-of-two form to an actual number, and pass it to the next layer.</p>
<p>If you are curious, this is a way to simulate the hardware architecture I’m currently working on. Since it has these quantizations, it will harm the accuracy and I want to measure by how much.</p>

matheussfarias: <p>I guess my major concern is that I don’t know the syntax to do this</p>
<aside class="quote no-group" data-username="matheussfarias" data-post="1" data-topic="188254">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matheussfarias/48/63894_2.png" class="avatar"> matheussfarias:</div>
<blockquote>
<p><code>x = model[name](x)</code></p>
</blockquote>
</aside>
<p>Is there an actual way to perform this operation if I know the name of the layer? I believe I can restrict <code>is_actual_layer</code> to some cases as you pointed out.</p>