Does model.parameters() return the parameters in topologically sorted order?

adarsh-kr: <p>Goal: To list model parameters in the sequence of their execution during forward pass, basically from input layer to the output layer. Or in the order of their execution in computation graph.</p>
<p>Does doing this will guarantee that the parameters are traversed in topologically sorted order of their execution:</p>
<pre><code class="lang-auto">for name,param in model.named_parameters():
    print(name, param.shape)
</code></pre>

ptrblck: <p>No, this will print the parameters in the order as they were registered:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc3 = nn.Linear(1, 1)
        self.fc2 = nn.Linear(1, 1)
        self.fc1 = nn.Linear(1, 1)
        
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

for name, param in model.named_parameters():
    print(name)

&gt; fc3.weight
fc3.bias
fc2.weight
fc2.bias
fc1.weight
fc1.bias
</code></pre>
<p>You could try to register the layers in the same order as their execution order in the <code>forward</code> method would be (if that’s possible for your model).</p>

adarsh-kr: <p>Is there any guaranteed way to traverse in topological order? Any methods ?</p>

ptrblck: <p>I’m really unsure so I’ll just post some ideas.</p>
<p>Since the computation graph is created during the forward pass, it might of course be different for each pass (e.g. if you are using conditions etc.).<br>
You could try to use the <code>grad_fn</code> and call into <code>grad_fn.next_functions</code> to crawl the graph.<br>
However, this would yield you the operations, not necessarily the layers and I’m not sure how hard it would be to create the mapping. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

Mirsadeghi: <p>Hi, I face the same issue. I need to apply forward pass manually (layer by layer) without accessing the <code>forward</code> method. Just I have access to the class instance of the Net. Is it possible to do so?</p>

kyy: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Can we assume that the order by <code>parameters()</code> is always fixed every time we call it?</p>

ptrblck: <p>I think so, as the internal <code>_parameters</code> is using an <code>OrderedDict</code> as seen <a href="https://github.com/pytorch/pytorch/blob/17d743ff0452b9cf12c7fcab7314079b83012bd4/torch/nn/modules/module.py#L257">here</a>.</p>

Geetanjli_sharma: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Could you please assist me in knowing …How is the sum of parameters calculated…<br>
My model is self.conv1 = nn.Conv2d(1, 32, kernel_size=3, stride=2, padding=1)<br>
self.conv2 = nn.Conv2d(32, 64, kernel_size=3, stride=2, padding=1)<br>
self.conv3 = nn.Conv2d(64, 128, kernel_size=3, stride=2, padding=1)<br>
self.fc1 = nn.Linear(2048, 128)<br>
self.fc2 = nn.Linear(128, 10)<br>
It has 356234 parameters. How is that calculated? I got this number using numel()<br>
Also How can we get the weights of a CNN model in to list. Do we need to access each layer value to assess weight?<br>
Like models[i].conv1.weight.data</p>

ptrblck: <aside class="quote no-group" data-username="Geetanjli_sharma" data-post="8" data-topic="81735">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/geetanjli_sharma/48/66382_2.png" class="avatar"> Geetanjli_sharma:</div>
<blockquote>
<p>It has 356234 parameters. How is that calculated? I got this number using numel()</p>
</blockquote>
</aside>
<p>The number of parameters can be calculated by iterating all parameters and accumulating their number of elements, which seems to be the approach you have used. PyTorch does not provide a built-in method, so you are executing your code to count all parameters and I don’t know what exactly you are running.</p>
<aside class="quote no-group" data-username="Geetanjli_sharma" data-post="8" data-topic="81735">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/geetanjli_sharma/48/66382_2.png" class="avatar"> Geetanjli_sharma:</div>
<blockquote>
<p>Also How can we get the weights of a CNN model in to list.</p>
</blockquote>
</aside>
<p>You could either directly access them as seen in your code snippet or iterate the <code>named_parameters()</code> and check if the <code>name</code> contains a <code>weight</code> string.</p>

Geetanjli_sharma: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for your quick reply I was adding the parameters through total_params = sum(<br>
param.numel() for param in model. Parameters()<br>
)<br>
Weights are part of parameters right ? I mean does that mean there are 356234 weights in the model?</p>

ptrblck: <p>Yes, <code>.parameters()</code> include the weight and bias parameters of the model which are either registered directly or in submodules.</p>
<aside class="quote no-group" data-username="Geetanjli_sharma" data-post="10" data-topic="81735">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/geetanjli_sharma/48/66382_2.png" class="avatar"> Geetanjli_sharma:</div>
<blockquote>
<p>I mean does that mean there are 356234 weights in the model?</p>
</blockquote>
</aside>
<p>It means there are 356234 trainable values in your model. The parameters naming might be ambigious here but you can also say your model contains 356234 trainable parameters.</p>

Geetanjli_sharma: <aside class="quote no-group" data-username="Geetanjli_sharma" data-post="8" data-topic="81735">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/geetanjli_sharma/48/66382_2.png" class="avatar"> Geetanjli_sharma:</div>
<blockquote>
<p>models[i].conv1.weight.data</p>
</blockquote>
</aside>
<p>But then why when If I am trying to print the length of models[i].conv1.weight.data it returns 32 which is actually the number of neurons of that particular layer.</p>

ptrblck: <p>The sum of all parameters is ~356k while separately each layer would have less weights.</p>

Geetanjli_sharma: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for the information… I have been trying to fetch weights and biases of a model separately and<br>
for k in local_model.keys():<br>
#` print(“([client_models[i].state_dict()”,client_models[i].state_dict()[k])<br>
if “weight” in k:<br>
append in localdict after detaching and flatten()<br>
convert into numpy<br>
<span class="hashtag-raw">#print</span>(“local_dict.append”,local_dict)<br>
if “bias” in k:<br>
append in localdict1 after detaching and flatten()<br>
convert into numpy<br>
hstack localdict and localdict1 to get a dictionary of weights and biases<br>
Now I am passing it  to neighbors_fit = neighbors.fit(local_dict2) to get the epsilon value of dbscan clustering but it fails:<br>
TypeError: only size-1 arrays can be converted to Python scalars<br>
ValueError: setting an array element with a sequence.<br>
How can I stack weights and biases and fit it to neighbors.fit() without getting this error.<br>
My model is as follows:<br>
5328<br>
Net1(<br>
(pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(convblock1): Sequential(<br>
(0): Conv2d(1, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(1): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
(3): Dropout(p=0.025, inplace=False)<br>
)<br>
(convblock2): Sequential(<br>
(0): Conv2d(8, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
(3): Dropout(p=0.025, inplace=False)<br>
)<br>
(trans1): Sequential(<br>
(0): Conv2d(16, 4, kernel_size=(1, 1), stride=(1, 1), bias=False)<br>
(1): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
(3): Dropout(p=0.025, inplace=False)<br>
)<br>
(convblock3): Sequential(<br>
(0): Conv2d(4, 8, kernel_size=(3, 3), stride=(1, 1), bias=False)<br>
(1): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
(3): Dropout(p=0.025, inplace=False)<br>
(4): Conv2d(8, 16, kernel_size=(3, 3), stride=(1, 1), bias=False)<br>
(5): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(6): ReLU()<br>
(7): Dropout(p=0.025, inplace=False)<br>
(8): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), bias=False)<br>
(9): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(10): ReLU()<br>
(11): Dropout(p=0.025, inplace=False)<br>
)<br>
(gap): Sequential(<br>
(0): AvgPool2d(kernel_size=8, stride=8, padding=0)<br>
)<br>
(convblock5): Sequential(<br>
(0): Conv2d(16, 10, kernel_size=(1, 1), stride=(1, 1), bias=False)<br>
)<br>
)</p>

ptrblck: <p>Your code is not formated properly and quite unreadable. Could you post a monimal and executable code snippet reproducing the issue?</p>

Geetanjli_sharma: <p>It will be hard for me to share the entire snippet… However if you can guide me through how can we reshape the extracted weights and biases so that they don’t throw the error:<br>
Local_dict_2=np.hstack((local_dict,local_dict1))<br>
where local_dict is the extracted weights, and local_dict1 is the extracted biases</p>
<p>TypeError: only size-1 arrays can be converted to Python scalars</p>
<p>The above exception was the direct cause of the following exception:</p>
<p>Traceback (most recent call last):<br>
line 436, in <br>
neighbors_fit = neighbors.fit(local_dict2)<br>
File “/home/geetanjli/.local/lib/python3.9/site-packages/sklearn/neighbors/_unsupervised.py”, line 175, in fit<br>
return self._fit(X)<br>
File “/home/geetanjli/.local/lib/python3.9/site-packages/sklearn/neighbors/_base.py”, line 444, in _fit<br>
X = self._validate_data(X, accept_sparse=“csr”, order=“C”)<br>
File “/home/geetanjli/.local/lib/python3.9/site-packages/sklearn/base.py”, line 577, in _validate_data<br>
X = check_array(X, input_name=“X”, **check_params)<br>
File “/home/geetanjli/.local/lib/python3.9/site-packages/sklearn/utils/validation.py”, line 856, in check_array<br>
array = np.asarray(array, order=order, dtype=dtype)<br>
ValueError: setting an array element with a sequence.</p>