Number of activations for Linear and Conv2d layer: comparison

blackbirdbarber: <p>How do you calculate the number of activations for :</p>
<pre><code class="lang-auto">l = nn.Linear(10, 100)
c = nn.Conv2d(1, 2, kernel_size=(3,4))
</code></pre>
<p>I would say 1000 for <code>l</code>, if this is the number of weights. But I am not sure.</p>

ptrblck: <p>The linear layer is pretty straightforward, as the output activation will have the shape <code>[batch_size, *, out_features]</code>.<br>
The activation of the conv layer is a bit trickier, as the shape depend on the spatial size of your input:</p>
<pre><code class="lang-python">x = torch.randn(2, 10)
out = l(x)
print(out.nelement())  # batch_size * out_features
&gt; 200

x = torch.randn(2, 1, 24, 24)
out = c(x)
print(out.nelement())  # batch_size * out_channels * (h - (kW - 1)) * (w - (kW - 1))
&gt; 1848
print(out.shape)
&gt; torch.Size([2, 2, 22, 21])
</code></pre>

blackbirdbarber: <p>Unbelievable, I never though this calculation will be so interesting.<br>
Shows that even when we involve padding, stride, and dilation it really becomes complex:</p>
<p><code>c = nn.Conv2d(1, 2, kernel_size=(3,4), padding=1, stride=2, dilation=2)</code></p>
<p>I see now that <code>bs</code> has key role, but I wonder why only <code>w</code> is taken in account.<br>
Why <code>b</code> part is ignored?</p>
<p>(I know in some cases <code>b</code> can be ignored.)</p>

ptrblck: <p>You can find the actual computation for the output shape in the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d">docs</a>.</p>
<aside class="quote no-group quote-modified" data-username="blackbirdbarber" data-post="3" data-topic="48528">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/6de8d8/48.png" class="avatar"> blackbirdbarber:</div>
<blockquote>
<p>``</p>
<p>I see now that <code>bs</code> has key role, but I wonder why only <code>w</code> is taken in account.<br>
Why <code>b</code> part is ignored?</p>
</blockquote>
</aside>
<p>I’m a bit confused. Are you looking for the number of parameters (weight + bias) or the output activation shape?</p>

blackbirdbarber: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="48528">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>number of parameters (weight + bias) or the output activation shape</p>
</blockquote>
</aside>
<p>I was unaware what number of activations mean till now. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>In case you would like to count the number of parameters, you could use:</p>
<pre><code class="lang-python">lin_params = 0
for param in l.parameters():
    lin_params += param.nelement()

conv_params = 0
for param in c.parameters():
    conv_params += param.nelement()
</code></pre>

blackbirdbarber: <p>Tested and it works great. Thanks.</p>