Is it mandatory to add modules to ModuleList to access its parameters

Abhishek_Kishore: <p>I read some posts about ModuleList and all of them said that adding modules to ModuleList gives access to parameters of the Neural Network but in “Training a classifier” example of 60 mins pytorch tutorial the modules are not added to any ModuleList and still the parameters could be accessed using</p>
<p>optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)</p>
<p>This is confusing. Please clarify how the parameters are accessible even though the modules have not been added to any ModuleList</p>

ptrblck: <p>You don’t need an <code>nn.ModuleList</code> to register the parameters properly.<br>
Could you link the topic, where this is stated, please?</p>
<p>The parameters are registered with the assignment in your model:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc = nn.Linear(1, 1) # here all parameters of the linear layers are registered
        self.register_buffer('my_buffer', torch.tensor(1)) # here the buffer is registered
</code></pre>
<p>The <a href="https://github.com/pytorch/pytorch/blob/176174a68ba2d36b9a5aaef0943421682ecc66d4/torch/nn/modules/module.py#L623-L668"><code>__setattr__</code> method</a> in <code>nn.Module</code> will take care of it.</p>
<p>Note that plain tensors (not <code>nn.Parameters</code>, <code>nn.Modules</code> or buffers) will not be registered and will thus not be returned in <code>model.parameters()</code> or <code>model.buffers()</code>.</p>

Abhishek_Kishore: <p>I am referring to this thread of discusion - <a href="https://discuss.pytorch.org/t/the-difference-in-usage-between-nn-modulelist-and-python-list/7744" class="inline-onebox">The difference in usage between nn.ModuleList and python list</a><br>
In this thread <strong>smth</strong>  said</p>
<blockquote>
<p>When you ask for  <code>model.parameters()</code> , the parameters of layers inside  <code>nn.ModuleList</code>  will be returned. But if it is regular list, we dont look inside the list for  <code>nn.Module</code>  layers.</p>
</blockquote>

Abhishek_Kishore: <p>I found one more thread - <a href="https://discuss.pytorch.org/t/when-should-i-use-nn-modulelist-and-when-should-i-use-nn-sequential/5463/15" class="inline-onebox">When should I use nn.ModuleList and when should I use nn.Sequential? - #15 by Tian_Jin1</a></p>
<p>Here you said</p>
<blockquote>
<p>Exactly! If you use a plain python list, the parameters won’t be registered properly and you can’t pass them to your optimizer using  <code>model.parameters()</code> .</p>
</blockquote>

ptrblck: <p>Yes, that’s the difference between a Python <code>list</code> and an <code>nn.ModuleList</code>.<br>
As explained in the linked topics, the parameters wrapped in a plain <code>list</code> won’t be registered, while the parameters from all modules inside an <code>nn.ModuleList</code> will be registered.</p>
<p>So if you want to use a list-like container, then the answer to the initial question is: yes, it’s mandatory to use <code>nn.ModuleList</code> instead of <code>list</code> to register all parameters.</p>
<p>If you don’t need a list-like container and just want to register parameters or layers inside an <code>nn.Module</code> class, then you can just use the assignment.</p>

Abhishek_Kishore: <p>Sorry, I am unable to  understand the list-like container scenario. Even if a python list contains let’s say 2 linear layers. We would be iterating the list and using those Linear Layers in the forward method of a user defined class(sub classed from nn.Module). Wouldn’t that work?</p>

ptrblck: <p>It would work to use these layers in the forward pass, but <code>model.parametes()</code> will not return the parameters of these layers (and thus <code>optim.SGD(model.parameters(), lr=1.)</code> will not see these parameters either).<br>
Also transferring the parameters to a device via <code>model.to('cuda')</code> won’t transfer these parameters to the desired device.</p>

Abhishek_Kishore: <p>So, if a module is added to a python list its parameters get invisible. Thanks for clearly explaining me this concept. I suggest this should be clearly documented with example in pytorch’s documentation of nn.ModuleList.</p>

ptrblck: <p><code>nn.ModuleList</code> mentions the advantage here:</p>
<blockquote>
<p><a href="https://pytorch.org/docs/master/generated/torch.nn.ModuleList.html#torch.nn.ModuleList"> <code>ModuleList</code> </a> can be indexed like a regular Python list, but modules it contains are properly registered, and will be visible by all <a href="https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module"> <code>Module</code> </a> methods.</p>
</blockquote>
<p>If you think this description is still confusing, would you be interested in creating a PR with an improvement?</p>

Abhishek_Kishore: <p>Yes sure. I have never contributed to any open source yet even if it were some documentation.</p>

ptrblck: <p>Then this is the best time to start doing it! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
You could create a feature request <a href="https://github.com/pytorch/pytorch/issues">here</a>, explain your confusion and your suggestion, and wait until the module owners start the discussion.</p>

Abhishek_Kishore: <p>I created an issue <span class="hashtag">#38639</span></p>