How to save inner nn.Sequential layer's output

sunyoung: <p><strong>VGG(</strong><br>
<strong>(features): Sequential(</strong><br>
<strong>(0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))</strong><br>
<strong>(0_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)</strong><br>
**    (1): ReLU(inplace=True)**<br>
**    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**<br>
**    (2_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**<br>
**    (3): ReLU(inplace=True)**<br>
**    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)**<br>
**    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**<br>
**    (5_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**<br>
**    (6): ReLU(inplace=True)**<br>
**    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**<br>
**    (7_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**<br>
**    (8): ReLU(inplace=True)**<br>
**    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)**<br>
**    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**<br>
**    (10_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**<br>
**    (11): ReLU(inplace=True)**<br>
**    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**<br>
**    (12_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**<br>
**    (13): ReLU(inplace=True)**<br>
**    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**<br>
**    (14_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**<br>
**    (15): ReLU(inplace=True)**<br>
**    (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)**<br>
**    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**<br>
**    (17_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**<br>
**    (18): ReLU(inplace=True)**<br>
**    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**<br>
**    (19_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**<br>
**    (20): ReLU(inplace=True)**<br>
**    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**<br>
**    (21_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**<br>
**    (22): ReLU(inplace=True)**<br>
**    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)**<br>
**    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**<br>
**    (24_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**<br>
**    (25): ReLU(inplace=True)**<br>
**    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**<br>
**    (26_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**<br>
**    (27): ReLU(inplace=True)**<br>
**    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))**<br>
**    (28_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**<br>
**    (29): ReLU(inplace=True)**<br>
**    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)**<br>
**  )**<br>
**  (classifier): Sequential(**<br>
**    (0): Linear(in_features=25088, out_features=4096, bias=True)**<br>
**    (0_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**<br>
**    (1): ReLU(inplace=True)**<br>
**    (2): Dropout(p=0.5, inplace=False)**<br>
**    (3): Linear(in_features=4096, out_features=4096, bias=True)**<br>
**    (3_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**<br>
**    (4): ReLU(inplace=True)**<br>
**    (5): Dropout(p=0.5, inplace=False)**<br>
**    (6): Linear(in_features=4096, out_features=1000, bias=True)**<br>
**    (6_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)**<br>
**  )**<br>
<strong>)</strong></p>
<ul>
<li>
<p><em>My model’s shape is like this.</em></p>
</li>
<li>
<p><em>I saw that hook is not work inner nn.seq module</em></p>
</li>
<li>
<p><em>and also when I tried that thing, the ofmap of feature.0 layer and ifmap of feature.0_linear_quant is different</em></p>
</li>
<li>
<p><em>Then, If I want conv2d or 0_linear_quant layer’s output feature map, what can I do?</em></p>
</li>
</ul>
<p><strong>import torch.nn as nn</strong><br>
<strong>import torch.utils.model_zoo as model_zoo</strong><br>
<strong>import math</strong></p>
<p><strong><strong>all</strong> = [</strong><br>
**    ‘VGG’, ‘vgg11’, ‘vgg11_bn’, ‘vgg13’, ‘vgg13_bn’, ‘vgg16’, ‘vgg16_bn’,**<br>
**    ‘vgg19_bn’, ‘vgg19’,**<br>
<strong>]</strong></p>
<p><strong>model_urls = {</strong><br>
**    ‘vgg11’: ‘<a href="https://download.pytorch.org/models/vgg11-bbd30ac9.pth" rel="noopener nofollow ugc">https://download.pytorch.org/models/vgg11-bbd30ac9.pth</a>’,**<br>
**    ‘vgg13’: ‘<a href="https://download.pytorch.org/models/vgg13-c768596a.pth" rel="noopener nofollow ugc">https://download.pytorch.org/models/vgg13-c768596a.pth</a>’,**<br>
**    ‘vgg16’: ‘<a href="https://download.pytorch.org/models/vgg16-397923af.pth" rel="noopener nofollow ugc">https://download.pytorch.org/models/vgg16-397923af.pth</a>’,**<br>
**    ‘vgg19’: ‘<a href="https://download.pytorch.org/models/vgg19-dcbb9e9d.pth" rel="noopener nofollow ugc">https://download.pytorch.org/models/vgg19-dcbb9e9d.pth</a>’,**<br>
<strong>}</strong></p>
<p><strong>class VGG(nn.Module):</strong></p>
<p>**    def <strong>init</strong>(self, features, num_classes=1000):**<br>
**        super(VGG, self).<strong>init</strong>()**<br>
**        self.features = features**<br>
**        self.classifier = nn.Sequential(**<br>
**            nn.Linear(512 * 7 * 7, 4096),**<br>
**            nn.ReLU(inplace=True),**<br>
**            nn.Dropout(),**<br>
**            nn.Linear(4096, 4096),**<br>
**            nn.ReLU(inplace=True),**<br>
**            nn.Dropout(),**<br>
**            nn.Linear(4096, num_classes),**<br>
**        )**<br>
**        self._initialize_weights()**</p>
<p>**    def forward(self, x):**<br>
**        x = self.features(x)**<br>
**        x = x.view(x.size(0), -1)**<br>
**        x = self.classifier(x)**<br>
**        return x**</p>
<p>**    def <em>initialize_weights(self):**<br>
**        for m in self.modules():**<br>
**            if isinstance(m, nn.Conv2d):**<br>
**                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels**<br>
**                m.weight.data.normal</em>(0, math.sqrt(2. / n))**<br>
**                if m.bias is not None:**<br>
**                    m.bias.data.zero_()**<br>
**            elif isinstance(m, nn.BatchNorm2d):**<br>
**                m.weight.data.fill_(1)**<br>
**                m.bias.data.zero_()**<br>
**            elif isinstance(m, nn.Linear):**<br>
**                n = m.weight.size(1)**<br>
**                m.weight.data.normal_(0, 0.01)**<br>
**                m.bias.data.zero_()**</p>
<p><strong>def make_layers(cfg, batch_norm=False):</strong><br>
**    layers = []**<br>
**    in_channels = 3**<br>
**    for v in cfg:**<br>
**        if v == ‘M’:**<br>
**            layers += [nn.MaxPool2d(kernel_size=2, stride=2)]**<br>
**        else:**<br>
**            conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1)**<br>
**            if batch_norm:**<br>
**                layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU(inplace=True)]**<br>
**            else:**<br>
**                layers += [conv2d, nn.ReLU(inplace=True)]**<br>
**            in_channels = v**<br>
**    return nn.Sequential(<em>layers)</em>*</p>
<p><strong>cfg = {</strong><br>
**    ‘A’: [64, ‘M’, 128, ‘M’, 256, 256, ‘M’, 512, 512, ‘M’, 512, 512, ‘M’],**<br>
**    ‘B’: [64, 64, ‘M’, 128, 128, ‘M’, 256, 256, ‘M’, 512, 512, ‘M’, 512, 512, ‘M’],**<br>
**    ‘D’: [64, 64, ‘M’, 128, 128, ‘M’, 256, 256, 256, ‘M’, 512, 512, 512, ‘M’, 512, 512, 512, ‘M’],**<br>
**    ‘E’: [64, 64, ‘M’, 128, 128, ‘M’, 256, 256, 256, 256, ‘M’, 512, 512, 512, 512, ‘M’, 512, 512, 512, 512, ‘M’],**<br>
<strong>}</strong></p>
<p><strong>def vgg11(pretrained=False, model_root=None, <strong>kwargs):</strong><br>
**    “”“VGG 11-layer model (configuration “A”)”""</strong><br>
**    model = VGG(make_layers(cfg[‘A’]), <strong>kwargs)</strong><br>
**    if pretrained:**<br>
**        model.load_state_dict(model_zoo.load_url(model_urls[‘vgg11’], model_root))**<br>
**    return model**</p>
<p><strong>def vgg11_bn(<strong>kwargs):</strong><br>
**    “”“VGG 11-layer model (configuration “A”) with batch normalization”""</strong><br>
**    kwargs.pop(‘model_root’, None)**<br>
**    return VGG(make_layers(cfg[‘A’], batch_norm=True), <strong>kwargs)</strong></p>
<p><strong>def vgg13(pretrained=False, model_root=None, <strong>kwargs):</strong><br>
**    “”“VGG 13-layer model (configuration “B”)”""</strong><br>
**    model = VGG(make_layers(cfg[‘B’]), <strong>kwargs)</strong><br>
**    if pretrained:**<br>
**        model.load_state_dict(model_zoo.load_url(model_urls[‘vgg13’], model_root))**<br>
**    return model**</p>
<p><strong>def vgg13_bn(<strong>kwargs):</strong><br>
**    “”“VGG 13-layer model (configuration “B”) with batch normalization”""</strong><br>
**    kwargs.pop(‘model_root’, None)**<br>
**    return VGG(make_layers(cfg[‘B’], batch_norm=True), <strong>kwargs)</strong></p>
<p><strong>def vgg16(pretrained=False, model_root=None, <strong>kwargs):</strong><br>
**    “”“VGG 16-layer model (configuration “D”)”""</strong><br>
**    model = VGG(make_layers(cfg[‘D’]), <strong>kwargs)</strong><br>
**    if pretrained:**<br>
**        model.load_state_dict(model_zoo.load_url(model_urls[‘vgg16’], model_root))**<br>
**    return model**</p>
<p><strong>def vgg16_bn(<strong>kwargs):</strong><br>
**    “”“VGG 16-layer model (configuration “D”) with batch normalization”""</strong><br>
**    kwargs.pop(‘model_root’, None)**<br>
**    return VGG(make_layers(cfg[‘D’], batch_norm=True), <strong>kwargs)</strong></p>
<p><strong>def vgg19(pretrained=False, model_root=None, <strong>kwargs):</strong><br>
**    “”“VGG 19-layer model (configuration “E”)”""</strong><br>
**    model = VGG(make_layers(cfg[‘E’]), <strong>kwargs)</strong><br>
**    if pretrained:**<br>
**        model.load_state_dict(model_zoo.load_url(model_urls[‘vgg19’], model_root))**<br>
**    return model**</p>
<p><strong>def vgg19_bn(<strong>kwargs):</strong><br>
**    “”“VGG 19-layer model (configuration ‘E’) with batch normalization”""</strong><br>
**    kwargs.pop(‘model_root’, None)**<br>
**    return VGG(make_layers(cfg[‘E’], batch_norm=True), <strong>kwargs)</strong></p>
<ul>
<li>base model is like this and</li>
</ul>
<p><strong>def duplicate_model_with_quant(model, bits, overflow_rate=0.0, counter=10, type=‘linear’):</strong><br>
**    “”“assume that original model has at least a nn.Sequential”""**<br>
**    assert type in [‘linear’, ‘minmax’, ‘log’, ‘tanh’]**<br>
**    if isinstance(model, nn.Sequential):**<br>
**        l = OrderedDict()**<br>
**        for k, v in model._modules.items():**<br>
**            if isinstance(v, (nn.Conv2d, nn.Linear, nn.BatchNorm1d, nn.BatchNorm2d, nn.AvgPool2d)):**<br>
**                l[k] = v**<br>
**                if type == ‘linear’:**<br>
**                    quant_layer = LinearQuant(’{}_quant’.format(k), bits=bits, overflow_rate=overflow_rate, counter=counter)**<br>
**                elif type == ‘log’:**<br>
**                    # quant_layer = LogQuant(’{}_quant’.format(k), bits=bits, overflow_rate=overflow_rate, counter=counter)**<br>
**                    quant_layer = NormalQuant(’{}_quant’.format(k), bits=bits, quant_func=log_minmax_quantize)**<br>
**                elif type == ‘minmax’:**<br>
**                    quant_layer = NormalQuant(’{}_quant’.format(k), bits=bits, quant_func=min_max_quantize)**<br>
**                else:**<br>
**                    quant_layer = NormalQuant(’{}<em>quant’.format(k), bits=bits, quant_func=tanh_quantize)**<br>
**                l[’{}</em>{}_quant’.format(k, type)] = quant_layer**<br>
**            else:**<br>
**                l[k] = duplicate_model_with_quant(v, bits, overflow_rate, counter, type)**<br>
**        m = nn.Sequential(l)**<br>
**        return m**<br>
**    else:**<br>
**        for k, v in model._modules.items():**<br>
**            model._modules[k] = duplicate_model_with_quant(v, bits, overflow_rate, counter, type)**<br>
**        return model**</p>
<ul>
<li>this is how LinearQuant layer generated</li>
</ul>

ptrblck: <p>You can use forward hooks to store intermediate activations as shown in <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">this example</a>.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

sunyoung: <pre><code class="lang-auto">    activation = {}
    ofmap = {}
    def get_ofmap(name):
        def hook(model, input, output):
            ofmap[name] = output.detach()
        return hook
    def get_activation(name):
        def hook(model, input, output):
            activation[name] = input[0].detach()
        return hook
</code></pre>
<pre><code class="lang-auto">    for idx, (data, target) in enumerate(tqdm.tqdm(ds, total=n_sample)):
        n_passed += len(data)
        data =  Variable(torch.FloatTensor(data)).cuda()
        indx_target = torch.LongTensor(target)
        h1 = model.module.model._modules["features"]._modules.get("0_linear_quant").register_forward_hook(get_activation('0_linear_quant'))
        h2 = model.module.model._modules["features"]._modules.get("0").register_forward_hook(get_ofmap('fea0'))

        output = model(data)
        bs = output.size(0)

        torch.save(activation['0_linear_quant'], './LQ0_IFMAP.pt')
        torch.save(ofmap['fea0'], './CONV0_OFMAP.pt')
</code></pre>
<p>Thanks for your help!</p>
<p>But I tried that solution before but the output of h1 and h2 is different.<br>
Because 0_LinearQuant layer’s input feature map is output of features.0 layer, h1 and h2 should be same.</p>

ptrblck: <p>It seems to work using this small example:</p>
<pre><code class="lang-python">act_in = {}
act_out = {}
def get_hook(name):
    def hook(m, input, output):
        act_in[name] = input[0].detach()
        act_out[name] = output.detach()
    return hook
        

model = nn.Sequential(
    nn.Conv2d(3, 6, 3, 1, 1),
    nn.ReLU(),
    nn.Flatten(),
    nn.Linear(6*24*24, 10),
    nn.ReLU(),
    nn.Linear(10, 10)
)

for name, module in model.named_children():
    module.register_forward_hook(get_hook(name))

x = torch.randn(1, 3, 24, 24)
out = model(x)

keys = list(act_in.keys())
for key_out, key_in in zip(keys[:-1], keys[1:]):
    print('Comparing output of {} to input of {}'.format(key_out, key_in))
    print((act_out[key_out] == act_in[key_in]).all())

&gt; Comparing output of 0 to input of 1
tensor(True)
Comparing output of 1 to input of 2
tensor(True)
Comparing output of 2 to input of 3
tensor(True)
Comparing output of 3 to input of 4
tensor(True)
Comparing output of 4 to input of 5
tensor(True)
</code></pre>
<p>Feel free to post an executable code snippet to reproduce this issue.</p>

sunyoung: <p>So I modify that code for the model that I use</p>
<pre><code class="lang-auto">VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (0_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (1): ReLU(inplace=True)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (2_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (3): ReLU(inplace=True)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (5_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (6): ReLU(inplace=True)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (7_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (8): ReLU(inplace=True)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (10_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (11): ReLU(inplace=True)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (12_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (13): ReLU(inplace=True)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (14_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (15): ReLU(inplace=True)
    (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (17_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (18): ReLU(inplace=True)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (19_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (20): ReLU(inplace=True)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (21_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (22): ReLU(inplace=True)
    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (24_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (25): ReLU(inplace=True)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (26_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (27): ReLU(inplace=True)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (28_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (29): ReLU(inplace=True)
    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (0_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (1): ReLU(inplace=True)
    (2): Dropout(p=0.5, inplace=False)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (3_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
    (4): ReLU(inplace=True)
    (5): Dropout(p=0.5, inplace=False)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
    (6_linear_quant): LinearQuant(sf=None, bits=8, overflow_rate=0.000, counter=20)
  )
)
</code></pre>
<p>This is the model that I use right now and this is from the GitHub below.<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/aaron-xichen/pytorch-playground" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/10893546?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/aaron-xichen/pytorch-playground" target="_blank" rel="noopener nofollow ugc">aaron-xichen/pytorch-playground</a></h3>

<p>Base pretrained models and datasets in pytorch (MNIST, SVHN, CIFAR10, CIFAR100, STL10, AlexNet, VGG16, VGG19, ResNet, Inception, SqueezeNet) - aaron-xichen/pytorch-playground</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>and right now, when I change the code as this</p>
<pre><code class="lang-auto">    for idx, (data, target) in enumerate(tqdm.tqdm(ds, total=n_sample)):
        n_passed += len(data)
        data =  Variable(torch.FloatTensor(data)).cuda()
        indx_target = torch.LongTensor(target)
        for name, module in model.module.named_children():
            for name, module in module.named_children():
                for name, module in module.named_children():
                    module.register_forward_hook(get_hook(name))
        #h1 = model.module.model._modules["features"]._modules.get("0_linear_quant").register_forward_hook(get_activation('0_linear_quant'))
        #h2 = model.module.model._modules["features"]._modules.get("0").register_forward_hook(get_ofmap('fea0'))

        output = model(data)
        bs = output.size(0)

        keys = list(act_in.keys())
        for key_out, key_in in zip(keys[:-1], keys[1:]):
            print('Comparing output of {} to input of {}'.format(key_out, key_in))
            print((act_out[key_out] == act_in[key_in]).all())
</code></pre>
<p>Error is like this</p>
<pre><code class="lang-auto">Comparing output of 0 to input of 0_linear_quant
tensor(True, device='cuda:0')
Comparing output of 0_linear_quant to input of 1
tensor(True, device='cuda:0')
Comparing output of 1 to input of 2
tensor(True, device='cuda:0')
Comparing output of 2 to input of 2_linear_quant
Traceback (most recent call last):
  File "/home/sunyoung/research/quantization/quant/bin/quantize", line 33, in &lt;module&gt;
    sys.exit(load_entry_point('pytorch-playground', 'console_scripts', 'quantize')())
  File "/home/sunyoung/research/quantization/pytorch-playground/quantize.py", line 107, in main
    misc.eval_model(model_raw, val_ds_tmp, ngpu=1, n_sample=args.n_sample, is_imagenet=is_imagenet)
  File "/home/sunyoung/research/quantization/pytorch-playground/utee/misc.py", line 230, in eval_model
    print((act_out[key_out] == act_in[key_in]).all())
RuntimeError: The size of tensor a (4096) must match the size of tensor b (224) at non-singleton dimension 3
</code></pre>
<p>I don’t know why this happen because when I execute inference code without that, it doesn’t have problem.</p>

ptrblck: <p>Forward hooks can be registered to modules and if you are using calls from the functional API in the <code>forward</code> method, e.g. as seen <a href="https://github.com/aaron-xichen/pytorch-playground/blob/c62c47052ef31c1c112154bd41c790c63c3169d1/imagenet/vgg.py#L38">here</a>, the intermediate activation will be manipulated.<br>
In the liked case the activation will be flattened, which would yield the shape mismatch error.</p>

sunyoung: <p>I found a problem, it is because of the type I access</p>
<pre><code class="lang-auto">        for name, module in model.module.named_children():
            for name, module in module.named_children():
                for name, module in module.named_children():
                    module.register_forward_hook(get_hook(name))
</code></pre>
<p>this part. How to I access to each layer by for loop?</p>
<pre><code class="lang-auto">model.module.model._modules["features"]._modules.get("0_linear_quant").register_forward_hook(get_hook('0_linear_quant'))
</code></pre>
<p>I access to the tensor like this before… I couldn’t access with named_children.</p>

ptrblck: <p>Your loop might work, if you have nested modules, where the inner modules are the built-in layers.<br>
If the model definition is more complicated (e.g. using mixed custom modules and layers), you might need to add conditions via <code>if isinstance(module, nn.Conv2d)</code> etc.</p>