Model.named_parameters() will lose some layer modules

kaiyuyue: <p>I found <code>model.named_parameters()</code> will lose the keys and params in my model, but <code>model.state_dict()</code> can not, how to fix this?</p>
<p>I want to use this <a href="http://pytorch.org/docs/0.3.1/optim.html#per-parameter-options" rel="nofollow noopener">method</a> to group the parameters according to its name.</p>
<p>Thanks.</p>
<pre><code class="lang-auto">for name, param in model.named_parameters():
    print name
</code></pre>
<pre><code class="lang-auto">for k, v in model.state_dict().items():
    print k
    print type(v)
</code></pre>

albanD: <p>You should make sure that if you store a list of <code>nn.Parameter</code>s, it should be in a <code>nn.ParameterList</code> and not a plain python list, same if you store a list of <code>nn.Module</code>s should be in a <code>nn.ModuleList</code>.</p>

kaiyuyue: <p>Thanks for the quick reply.  My completed code looks like:</p>
<pre><code class="lang-python">param_frozen_list = [] # should be changed into torch.nn.ParameterList()
param_active_list = [] # should be changed into torch.nn.ParameterList()

for name, param in model.named_parameters():
    if name == 'frozen_condition':
          param_frozen_list.append(param)
    elif name == 'active_condition':
        param_active_list.append(param) 
    else:
        continue

optimizer = torch.optim.SGD([
                     {'params': param_frozen_list, 'lr': 0.0},
                     {'params': param_active_list, 'lr': args.learning_rate}],
                 lr = args.learning_rate,
                 momentum = args.momentum,
                 weight_decay = args.weight_decay)
</code></pre>
<p>The key problem is that in the loop result of <code>model.named_parameters()</code>, some layers is lost.<br>
I use <code>model.modules()</code> to check out that they exits in the full module sequences. So is there wrong in my code or a pytorch bug?</p>

albanD: <p>the <code>named_parameters()</code> method does not look for all objects that are contained in your <code>model</code>, just the <code>nn.Module</code>s and <code>nn.Parameter</code>s, so as I stated above, if you store you parameters outsite of these, then they won’t be detected by <code>named_parameters()</code>.</p>

kaiyuyue: <p>Thanks. I understand what you mean, but I create all the layers by using <code>class LayerName(nn.Module)</code> and there is no any other parameters beyond the <code>torch.nn.layers</code>.<br>
I think they are registered properly and can be visible.</p>
<p>I change the method to group the parameters via <code>model.state_dict()</code> like this:</p>
<pre><code class="lang-python">for k, v in model.state_dict().items():
    name = k
    param = torch.nn.Parameter(v)
</code></pre>
<p>Does this way can be seemed as same with <code>named_parameters()</code>?</p>

albanD: <p>Hi,</p>
<p>The thing is that if <code>named_parameters</code> does not work properly, then <code>.parameters</code> doesn’t either, and that could be a problem if you use it to create your optimizer. Check the exemple below of what will and what won’t work.</p>
<pre><code class="lang-auto">class MyLayer(nn.Module):
    def __init__(self, arg):
        super(MyLayer, self).__init__()
        self.linear = nn.Linear(...) # that will work
        self.linears = nn.ModuleList(nn.Linear(...), nn.Linear(...)) # that will work
        self.wrong_linears = [nn.Linear(...), nn.Linear(...)] # that will NOT work
        self.wrong_linears_2 = {1: nn.Linear(...), 2: nn.Linear(...)} # that will NOT work

        # And similarly for nn.Parameter !
        # Also anything saved as `torch.Tensor` or `torch.autograd.Variable` will NOT work

    def forward(self, args):
        # some code
</code></pre>

kaiyuyue: <aside class="quote no-group quote-modified" data-username="albanD" data-post="6" data-topic="14588">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>self.linear = nn.Linear(…) # that will work</p>
</blockquote>
</aside>
<p>All the layers I used are in this definition. That’s quite wired. Hmm~</p>

Minxiangliu: <p>HI, <a class="mention" href="/u/kaiyuyue">@kaiyuyue</a><br>
Did you solve it? I have the same problem<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/9/59b728166c565e5f77cd9f2a595c6ff6a43d00aa.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/59b728166c565e5f77cd9f2a595c6ff6a43d00aa" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/59b728166c565e5f77cd9f2a595c6ff6a43d00aa_2_690x307.jpeg" alt="image" data-base62-sha1="cNEUaTcgt6uXMVctvAhMGSk4DBU" width="690" height="307" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/59b728166c565e5f77cd9f2a595c6ff6a43d00aa_2_690x307.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/59b728166c565e5f77cd9f2a595c6ff6a43d00aa_2_1035x460.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/59b728166c565e5f77cd9f2a595c6ff6a43d00aa_2_1380x614.jpeg 2x" data-dominant-color="272626"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1469×654 166 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

JaeJin_Cho: <p>I think that conv_seg layer does not have any parameters so its parameters do not show up…</p>

FXNanN: <p>If two layers point to the same parameters, one of them will not be included.</p>