How to separate each neuron's weights and bias values for convolution and fc layers?

username: Ajinkya.Bankar
cooked: <p>My network has convolution and fully connected layers, and I want to access each neuron’s weights and bias values. If I use</p>
<pre><code class="lang-auto">for name, param in network.named_parameters():
    print(name, param.shape)
</code></pre>
<p>I get layer name and whether it is <code>.weight</code> or <code>.bias</code> tensor along with dimensions. How can I get each neuron’s dimensions along with its weights and bias term?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Ajinkya.Bankar" data-post="1" data-topic="136800">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/bc79bd/48.png" class="avatar"> Ajinkya.Bankar:</div>
<blockquote>
<p>How can I get each neuron’s dimensions along with its weights and bias term?</p>
</blockquote>
</aside>
<p>Could you explain the terminology a bit, please?<br>
The weights in a layer are often called “neurons”, so I’m unsure what exactly you are looking for.</p>

username: Ajinkya.Bankar
cooked: <p>Thank you for the reply. If we have a fully connected network, as shown in the following figure <img src="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7d335fcf67b149e7ce4b0377d2124c438a642408.png" alt="Network" data-base62-sha1="hRzKFaVV2As3Dn8GPSajzm6e8ek" width="336" height="231"></p>
<p>Then each neuron can be described as<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/5/556e1b7d65bab2a7be3b4c0958c83fe3cef65913.png" alt="Neuron" data-base62-sha1="cbKujAoV0HE8GOBHMUFp17O1DDd" width="336" height="173"></p>
<p>I want to access the weights and biases for each neuron. And my network has convolutional layers also. Kindly let me know if you need more details.</p>

username: ptrblck
cooked: <p>Each output of the linear layer will be created by the corresponding row of the <code>weight</code> matrix.<br>
I.e.:</p>
<pre><code class="lang-python">lin = nn.Linear(10, 20)
x = torch.randn(2, 10)

out = lin(x)
</code></pre>
<p>would yield an output of <code>[batch_size=2, 20]</code> and each of the 20 output values is calculated via:</p>
<pre><code class="lang-python">i = 10
out0 = torch.matmul(x, lin.weight[i:i+1, :].T) + lin.bias[i]
</code></pre>
<p>for the <code>ith</code> output.</p>

username: InnovArul
cooked: <p>Just adding to <a class="mention" href="/u/ptrblck">@ptrblck</a>’s answer, <code>layer.weight[i]</code> will give the weights of <code>i</code>'th neuron in a layer (both conv and linear) and <code>layer.bias[i]</code> will give the bias of <code>i</code>'th neuron.</p>

username: Ajinkya.Bankar
cooked: <p>Thanks, <a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/innovarul">@InnovArul</a>. I want to know from <a class="mention" href="/u/innovarul">@InnovArul</a> if the convolution layer tensor’s <code>.weight</code> and <code>.bias</code> dimensions are <code>[16,6,5,5]</code> and <code>[16]</code> respectively, then are the neuron weights <code>[i,:,:,:]</code> and bias <code>[i]</code> where <code>i = 1 to 16</code>? Does it mean there are 16 neurons in the convolution layer?</p>

username: InnovArul
cooked: <p>Yes you are right.  There are 16 neurons in the convolution layer with <code>weight</code> dimension <code>[16,6,5,5]</code>.</p>

username: Ajinkya.Bankar
cooked: <p><a class="mention" href="/u/innovarul">@InnovArul</a>, thanks for your help. However, I am confused with the neuron terminology answered at <a href="https://stackoverflow.com/a/52273707/15009452" rel="noopener nofollow ugc">https://stackoverflow.com/a/52273707/15009452</a><br>
The answer says that the number of neurons in a convolution layer depends on the size of the image. But, if the layer dimensions are <code>[16,6,5,5]</code>, there are <code>6</code> input channels with <code>16</code> filters in the current layer producing <code>16</code> output channels having filter dimensions as <code>5x5</code>. I don’t see any relation with the input size here. Can you please explain? It would be a great help.</p>

username: InnovArul
cooked: <p>ok. I understand the confusion. Maybe I contributed to it too.<br>
Basically <code>weight[i]</code> is the way to get the parameters of the neuron. But the <code>number of neuron</code> calculations is different between the linear layer and conv layer.</p>
<p><strong>Linear layer:</strong><br>
With respect to linear layers, it is clear how many neurons are present in the layer. (<code>layer.weight.shape[0]</code>)</p>
<p><strong>Conv layer:</strong><br>
With conv layers, the <code>number of neurons</code> calculation is a bit tricky which is explained in that StackOverflow post. I will rephrase it here as I understand.<br>
In 2D conv layers, a neuron is an entity that looks at only a small slice of the input. Lets take the same example from StackOverflow  where input size = <code>1x27x27</code> (channels=1, height=27, width=27), slice size = <code>1x3x3</code>. So there are <code>9x9=81</code> slices of size <code>1x3x3</code> assuming non-overlapping slices.</p>
<p>Since conv layer holds the invariance property, a set of 81 neurons is supposed to share the same weights (i.e., <code>layer.weight[i]</code>). There can be <code>N</code> sets of 81 neurons in a conv layer (<code>N = layer.weight.shape[0]</code>). So the total number of neurons is <code>N x 81</code>.</p>
<p>Hope this makes sense.</p>

username: Ajinkya.Bankar
cooked: <p><a class="mention" href="/u/innovarul">@InnovArul</a>, thanks again for a detailed response. So, as per my example of <code>layer.weight.shape=[16,6,5,5]</code>, there must be <code>N=16</code> sets of neurons. Kindly correct me if I am wrong. May I know how can we count the number of neurons in each set if we know only <code>layer.weight.shape=[16,6,5,5]</code>? Or do we need to consider the original input size and number of input sets to the layer (i.e., <code>6</code> in this case)?</p>

username: InnovArul
cooked: <p>To calculate the actual <code>number of neurons</code> within the set, we need to know the number of slices, which can only be calculated if we know input size.</p>

username: Ajinkya.Bankar
cooked: <p><a class="mention" href="/u/innovarul">@InnovArul</a>, thanks for your help.</p>

username: fabiola
cooked: <p><a class="mention" href="/u/innovarul">@InnovArul</a> <a class="mention" href="/u/ptrblck">@ptrblck</a> I have a question linked to <a class="mention" href="/u/ajinkya.bankar">@Ajinkya.Bankar</a> 's post. In a convolution layer where the weights are for instance as in <a class="mention" href="/u/ajinkya.bankar">@Ajinkya.Bankar</a> case, of shape [16,6,5,5] and the bias of shape [16], before the output of this layer is given to an activation function, the <em>same</em> bias is applied to each of the neurons in a same filter (of shape [6,5,5])?<br>
If that’s the case, is it possible (does it make sense?) to modify the bias applied to each filter so not all neurons of a same filter are added the same bias ?</p>

username: ptrblck
cooked: <p>The bias is added after the weight (kernel) was applied to the input, not to the weights directly.<br>
Here is a small example:</p>
<pre data-code-wrap="python"><code class="lang-python">conv = nn.Conv2d(3, 16, 3)
weight = conv.weight
bias = conv.bias

x = torch.randn(2, 3, 24, 24)

# standard approach 
out = conv(x)

# manual approach
out_manual = nn.functional.conv2d(x, weight)
out_manual = out_manual + bias[None, :, None, None]

print((out - out_manual).abs().max())
# tensor(4.1723e-07, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<aside class="quote no-group" data-username="fabiola" data-post="13" data-topic="136800">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fabiola/48/30499_2.png" class="avatar"> fabiola:</div>
<blockquote>
<p>If that’s the case, is it possible (does it make sense?) to modify the bias applied to each filter so not all neurons of a same filter are added the same bias ?</p>
</blockquote>
</aside>
<p>I don’t think adding the bias to the weight would make sense since the model could just train the weights to account for this offset.</p>

username: fabiola
cooked: <p>Thanks for you reply.<br>
I didn’t express myself well, I will rephrase my question.<br>
<code>out_manual</code> is of shape (2, 16, 22, 22) and <code>bias</code> of shape (16). If I’m not mistaken, the same value <code>bias[i]</code> is added to all values of <code>out_manual[2, i, 22, 22]</code>. Would it be possible instead (and does it make sense) to add a bias tensor of shape (1, 1, 22, 22) (with different values "inside it ") to <code>out_manual[2, i, 22, 22]</code>?<br>
The bias tensor would thus be of shape (1, 16, 22, 22) instead of (16).</p>
<p>Thanks !</p>

username: InnovArul
cooked: <p>In my understanding, having a bias of shape (1, 1, 22, 22) would make the conv layer spatially dependent. i.e., the very property of translation invariance that the conv layer is known for, will be lost.<br>
Does that make sense?</p>