How to get the tensor of neurons for each hidden layer?

Torcione: <p>I’d like to visualize the distribution of the <strong>neurons</strong> associated to each hidden layer in my deep neural network. In principle, given the tensor I can easily do it through tensorboard.<br>
For example if I was interested in the distribution of <strong>weight</strong> between layers I could do something like that:</p>
<pre><code class="lang-auto">for n, w in self.model.named_parameters():
    self.writer.add_histogram('Weights connection {}'.format(n), w, global_step=t)
</code></pre>
<p>where n is the name of the module, w is the corresponding tensor of weight and t is the index at which our summary is saved.</p>
<p><strong>Is there an equivalent way to iterate through hidden layers calling tensors associated with the neurons’ state  of each layer?</strong></p>

techkang: <p>You can add an <code>if</code> condition like <code>if 'Linear' in n</code> to filter specific layers.</p>

Torcione: <p>The problem is that <code>model.named_parameters()</code> iterate over the weight of my network; for example (correct me if I misunderstood your point) <code>Linear</code> refers to a fully connected matrix of weights that map a given hidden layer’s vector to the next one. I’m- only interested in these vectors (not in the weights’ set), i.e. the set of numbers that are multiplied with weights during the forward propagation</p>

techkang: <p>You can directly retrive the weights or bias vectors by <code>weight</code> or <code>bias</code> attribute like this:</p>
<pre><code class="lang-auto">import torch.nn as nn


class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.cnn = nn.Sequential(
            nn.Conv2d(3, 3, 1),
            nn.ReLU()
        )
        self.fc = nn.Linear(10, 10)


model = MyModel()
conv_weight = model.cnn[0].weight
fc_bias = model.fc.bias
</code></pre>

Torcione: <p>Thank you for your comment!<br>
I’m not interested in the weights; I report below a simple example to shows more clear what is my objective:</p>
<pre><code class="lang-auto">class Net(nn.Module, NetVariables):
    def __init__(self, params):
        
        self.params = params.copy()
        
        nn.Module.__init__(self)
        NetVariables.__init__(self, self.params)        
        
        # number of hidden nodes in each layer (32)
        hidden_1 = 32
        hidden_2 = 32

        # linear layer (784 -&gt; hidden_1)
        if  (params['Dataset']=='MNIST'):
            self.fc1 = nn.Linear(28*28, hidden_1)
        #NOTE: images in MNIST dataset are b&amp;w photos with 28*28 pixels, CIFAR10 instead contain with colored photos (3 colour channel) with 32*32 pixels 
        elif  (params['Dataset']=='CIFAR10'):
            self.fc1 = nn.Linear(32*32*3, hidden_1)
        # linear layer (n_hidden -&gt; hidden_2)
        self.fc2 = nn.Linear(hidden_1,hidden_2)
        # linear layer (n_hidden -&gt; 10)
        self.fc3 = nn.Linear(hidden_2, self.num_classes)
        
        #weights initialization (this step can also be put below in a separate def)
        nn.init.kaiming_normal_(self.fc1.weight, mode='fan_in', nonlinearity='relu')
        nn.init.kaiming_normal_(self.fc2.weight, mode='fan_in', nonlinearity='relu')
        nn.init.xavier_normal_(self.fc3.weight)
        #initialize the bias to 0
        nn.init.constant_(self.fc3.bias, 0)
    #I return from the forward a dictionary to get the output after each layer not only the last one
    def forward(self,x):
        
        outs = {}
        # flatten image input
        if  (self.params['Dataset']=='MNIST'):
            x = x.view(-1,28*28)
        #NOTE: images in MNIST dataset are b&amp;w photos with 28*28 pixels, CIFAR10 instead contain with colored photos (3 colour channel) with 32*32 pixels 
        elif  (self.params['Dataset']=='CIFAR10'):
            x = x.view(-1,32*32*3)
        # add hidden layer, with relu activation function
        Fc1 = F.relu(self.fc1(x))
        
        outs['l1'] = Fc1
       
        Fc2 = torch.tanh(self.fc2(Fc1))
        
        
        outs['l2'] = Fc2
        
        # add output layer
        Out = self.fc3(Fc2)
        
        outs['out'] = Out
        
        return outs

</code></pre>
<p>In the above example the forward propagation return not only the output layer but a dictionary that contain also all the previous ones (hidden layer outputs); this is all I want. On the other hand with more complex networks the number of hidden layer increases, so I’d like to not do it manually with the <em>dictionary approach</em> above; I’d like to iterate over the hidden layers and get the corresponding output tensors (<code>outs['l1']</code> and <code> outs['l2']</code> in the above example)</p>

techkang: <p>I get your point. You can achieve you purpose by <code>register_forward_hook</code>, here is example code:</p>
<pre><code class="lang-auto">from functools import partial

import torch as t
from torchvision.models import resnet50

model = resnet50()
data = t.randn(1, 3, 224, 224)

layer_output = {}


def get_all_layers(net):
    def hook_fn(m, i, o, n=""):
        layer_output[n] = o

    for name, layer in net.named_modules():
        if hasattr(layer, "_module") and layer._module:
            get_all_layers(layer)
        elif hasattr(layer, "_parameters") and layer._parameters:
            # it's a non sequential. Register a hook
            layer.register_forward_hook(partial(hook_fn, n=name))


get_all_layers(model)
model(data)
for k, v in layer_output.items():
    print(f"the output shape of layer: {k} is {v.shape}")
</code></pre>
<p>You can refer to <a href="https://blog.paperspace.com/pytorch-hooks-gradient-clipping-debugging/" rel="noopener nofollow ugc">pytorch-hooks-gradient-clipping-debugging</a> for deeper interpretation.</p>