How do I check the number of parameters of a model?

ZeweiChu: <p>When I create a PyTorch model, how do I print the number of trainable parameters? They have such features in Keras but I don’t know how to do it in PyTorch.</p>

sinhasam: <pre><code>for parameter in model.parameters():
    print(parameter)</code></pre>

Brando_Miranda: <aside class="quote no-group" data-username="sinhasam" data-post="2" data-topic="4325">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/dec6dc/48.png" class="avatar"> sinhasam:</div>
<blockquote>
<p>model.parameters()</p>
</blockquote>
</aside>
<p>how does one make sure <code>model.parameters()</code> is not empty? It seems to be empty for me.</p>

sinhasam: <p>Can I see what you are trying to do? The parameters should not be empty unless you have something like:</p>
<pre><code>class Model(nn.Module):
    def __init__(self):
        super(model, self).__init__()
model = Model()
</code></pre>
<p>The above model has no parameters.</p>

Brando_Miranda: <p>you can see it here: <a href="https://discuss.pytorch.org/t/how-does-one-make-sure-that-the-custom-nn-has-parameters/5948">How does one make sure that the custom NN has parameters?</a></p>
<pre><code class="lang-auto">class NN(torch.nn.Module):
    def __init__(self, D_layers,act,w_inits,b_inits,bias=True):
        super(type(self), self).__init__()
        # actiaction func
        self.act = act
        #create linear layers
        self.linear_layers = [None]
        for d in range(1,len(D_layers)):
            linear_layer = torch.nn.Linear(D_layers[d-1], D_layers[d],bias=bias)
            self.linear_layers.append(linear_layer)
</code></pre>

sinhasam: <p>I posted my response on your original question!</p>

kulikovv: <pre><code>def get_n_params(model):
    pp=0
    for p in list(model.parameters()):
        nn=1
        for s in list(p.size()):
            nn = nn*s
        pp += nn
    return pp</code></pre>

vsmolyakov: <p>To compute the number of <em>trainable</em> parameters:</p>
<pre><code>model_parameters = filter(lambda p: p.requires_grad, model.parameters())
params = sum([np.prod(p.size()) for p in model_parameters])</code></pre>

baldassarre.fe: <p>I like this solution!</p>
<p>To add my 50 cents, I would use <code>numel()</code> instad of <code>np.prod()</code> and compress the expression in one line:</p>
<pre><code class="lang-auto">def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)
</code></pre>

varghese_alex: <aside class="quote no-group" data-username="baldassarre.fe" data-post="9" data-topic="4325">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/baldassarre.fe/48/1331_2.png" class="avatar"> baldassarre.fe:</div>
<blockquote>
<p>def count_parameters(model):<br>
return sum(p.numel() for p in model.parameters() if p.requires_grad)</p>
</blockquote>
</aside>
<p>Provided the models are similar in keras and pytorch, the number of trainable parameters returned are different in pytorch and keras.</p>
<p>import torch<br>
import torchvision<br>
from torch import nn<br>
from torchvision import models</p>
<p>a= models.resnet50(pretrained=False)<br>
a.fc = nn.Linear(512,2)<br>
count = count_parameters(a)<br>
print (count)<br>
<strong>23509058</strong></p>
<p>Now in keras</p>
<p>import keras.applications.resnet50 as resnet</p>
<p>model =resnet.ResNet50(include_top=True, weights=None, input_tensor=None, input_shape=None, pooling=None, classes=2)<br>
print model.summary()</p>
<p><strong>Total params: 23,591,810</strong><br>
<strong>Trainable params: 23,538,690</strong><br>
<strong>Non-trainable params: 53,120</strong></p>
<p>Any reasons why this difference in numbers pop up?</p>

baldassarre.fe: <p>Hi Alex, well spotted. I never did this comparison.</p>
<p>One easy check it to compare the layers one by one, (Linear, Conv2d, BatchNorm etc.), and see if there’s any difference in the number of params.<br>
However, I don’t think there will be any difference, provided that you pay attention to the sneaky default parameters.</p>
<p>After that, you can patiently compare the graphs layer by layer and see if you spot any difference. Maybe it’s a matter of omitted/shared biases in some of the layers.</p>
<p>Btw, the first test is also a good check for the <code>count_parameters()</code> function, let us now if you discover some unexpected behavior <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>

GWSurfer: <aside class="quote no-group" data-username="sinhasam" data-post="2" data-topic="4325" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/dec6dc/48.png" class="avatar"> sinhasam:</div>
<blockquote>
<p>for parameter in model.parameters(): print(parameter)</p>
</blockquote>
</aside>
<p>Have you checked if they are the bias weights?</p>

mfajcik: <p>I guess this counts shared parameters multiple times, doesn’t it?</p>
<pre><code class="lang-auto">import torch
from models.modelparts import count_parameters
class tstModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.p = torch.nn.Parameter(
            torch.randn(1, 1, 1, requires_grad=True)
                .expand(-1, 5, -1)
        )
print(count_parameters(tstModel()))
</code></pre>
<p>prints <code>5</code><br>
If I understand correctly, expand just creates tensor with 5 views to the same parameter, so the right answer should be <code>1</code>.<br>
But I don’t know how to fix that.</p>

hughperkins: <p>did anyone figure out a solution for shared parameters?</p>

ash95: <p>So I get that by default, <code>Conv2d</code> includes the bias. But I’m unclear as to why they (the biases) are being included in ‘requires_grad’.</p>
<p><code>In [1]: conv_3 = nn.Conv2d(512, 256, kernel_size=3, bias=True)</code></p>
<p><code>In [2]: sum(p.numel() for p in conv_3.parameters())</code><br>
<code>Out[2]: 1179904</code><br>
<code>In [3]: sum(p.numel() for p in conv_3.parameters() if p.requires_grad)</code><br>
<code>Out[3]: 1179904</code></p>

ptrblck: <p>The <code>bias</code> is a trainable parameter, which requires gradients and is optimized in the same way as the <code>weight</code> parameter.<br>
Do you have a use case, where the bias is fixed to a specific value?</p>

ash95: <p>Ah sorry. It was a conceptual error on my part. I had confused the idea of bias being a constant value <strong>with a weight</strong> with bias being a constant value.<br>
Thanks for the clarification.</p>

Brando_Miranda: <p>just out of curiosity, is there a <code>np.prod</code> for pytorch?</p>

baldassarre.fe: <p>Well, there’s <a href="https://pytorch.org/docs/stable/torch.html?highlight=prod#torch.prod" rel="nofollow noopener"><code>torch.prod</code></a>, but unlike numpy it accepts only tensors and does not accept tuples, lists, etc.</p>

arshishir: <p>You may find this useful: <a href="https://pypi.org/project/pytorch-model-summary/" rel="nofollow noopener">https://pypi.org/project/pytorch-model-summary/</a></p>