Model Backpropagation (python list vs nn.ParameterList)

Alison_Ma: <p>Hi, I wanted to confirm whether…:</p>
<ol>
<li>
<p>Pytorch will call backpropagation through regular python lists? Or only, e.g. nn.ParameterList?</p>
</li>
<li>
<p>Is backpropagation done ONLY for parameters in (i) model.named_parameters(), (ii) model.state_dict().keys()?</p>
</li>
<li>
<p>How can I ensure parameters will be put through backpropagation when calling model.backwards()?</p>
</li>
</ol>

ptrblck: <ol>
<li>
<p>The <code>backward()</code> operation will use the created computation graph to create the gradients for all leaf variables. It doesn’t necessarily care if these parameters were stored in <code>list</code>s or any other object.<br>
However, you should care about using e.g. <code>nn.ParameterList</code> instead of a plain Python <code>list</code> if you want to properly register the parameters in an <code>nn.Module</code>.</p>
</li>
<li>
<p>Leaf variables, i.e. explicitly created tensors requiring gradients (such as <code>nn.Parameter</code>s) will get their <code>.grad</code> attribute populated by default. Other <code>.grad</code> attributes (e.g. from intermediate tensors) can be inspected via <code>tensor.retain_grad()</code>.</p>
</li>
<li>
<p>You would have to make sure these parameters are used in the forward pass while the computation graph is created.</p>
</li>
</ol>

Alison_Ma: <p>Thanks for the reply!</p>
<p>Initially, I had various classifier heads / linear layers stored in a python list that I would dynamically pick when calling model.forward(). So, from your reply, I suppose that the backpropagation should have worked correctly when training. However, when saving the model to use with resuming trials or inference, that the parameters for the linear layers in the list were not stored in model.state_dict() and thus cannot be used correctly?</p>

ptrblck: <p>Yes, exactly. Your question is a bit too general to answer all valid use cases.<br>
To recap: Autograd doesn’t care where parameters are coming from and is happy to just use leaf variables as you pass them to operations.<br>
However, if you are writing a proper <code>nn.Module</code> and want to register the parameters inside it, do not use plain Python <code>list</code>s as this will not register these parameters to the parent module. Use <code>nn.ParameterList</code> instead which will make sure that these parameter show up in the <code>state_dict</code> and will be moved to the specified <code>dtype</code>, <code>device</code>, etc. in <code>model.to()</code> calls.</p>

Alison_Ma: <p>Got it, thank you so much for the clarification!</p>

Alison_Ma: <p>On a side note, would it be necessary to turn requires_grad=True on and off if I was using a list/nn.ModuleList to slice for the classifier head I wanted per model.forward() call? Or would selection of the particular head for the forward pass already isolate it in the computation graph?</p>

ptrblck: <p>If you are not using specific parameters, they will not be included in the computation graph, so changing their <code>requires_grad</code> attribute might not be necessary.<br>
However, if you are using an optimizer with internal running states (e.g. <code>Adam</code>) then parameters with a history (e.g. parameters which were previously updated) could still be updated by the <code>optimizer.step()</code> even if they did not receive any new gradients.<br>
The recommendation is thus again use case dependent.</p>

Peter78: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="148223">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>ParameterList</code></p>
</blockquote>
</aside>
<p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
can you suggest how to remove a parameter from a ParameterList at runtime?</p>
<p>Regards,</p>
<p>P.</p>