Understanding nn.Module.parameters()

username: David_Alford
cooked: <p>I am reading in the book <em>Deep Learning with PyTorch</em> that by calling the <code>nn.Module.parameters()</code> method that it will call submodules defined in the module’s <code>init</code> constructor. To understand and help visualize the processes I would like to use an ensemble as an example from <a href="https://discuss.pytorch.org/t/custom-ensemble-approach/52024/4">ptrblck</a>:</p>
<pre><code class="lang-auto">class MyEnsemble(nn.Module):
    def __init__(self, modelA, modelB, nb_classes=10):
        super(MyEnsemble, self).__init__()
        self.modelA = modelA
        self.modelB = modelB
        # Remove last linear layer
        self.modelA.fc = nn.Identity()
        self.modelB.fc = nn.Identity()
        
        # Create new classifier
        self.classifier = nn.Linear(2048+512, nb_classes)
        
    def forward(self, x):
        x1 = self.modelA(x.clone())  # clone to make sure x is not changed by inplace methods
        x1 = x1.view(x1.size(0), -1)
        x2 = self.modelB(x)
        x2 = x2.view(x2.size(0), -1)
        x = torch.cat((x1, x2), dim=1)
        
        x = self.classifier(F.relu(x))
        return x
</code></pre>
<p>In this <code>nn.Module</code> both <code>self.modelA = modelA</code> and <code>self.modelB = modelB</code> are being called in the <code>init</code> constructor. Therefore, by calling <code>MyEnsemble.parameters()</code> we would be returned the params which autograd would calculate the gradients wrt the parameters of the models <code>MyEnsemble</code>, <code>modelA</code>, and <code>modelB</code>?</p>
<p>Unless, of course, <code>requires_grad=False</code> for <code>self.modelA</code> and <code>self.modelB</code> - in which case autograd would not calculate the gradients wrt the parameters of these models and only that of <code>MyEnsemble</code>?</p>
<p>Is this thinking correct? Please correct my language if it is off. I am trying to learn as best I can and all help is appreciated.</p>
<p>David A.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="David_Alford" data-post="1" data-topic="94887">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/david_alford/48/26462_2.png" class="avatar"> David_Alford:</div>
<blockquote>
<p>Therefore, by calling <code>MyEnsemble.parameters()</code> we would be returned the params which autograd would calculate the gradients wrt the parameters of the models <code>MyEnsemble</code> , <code>modelA</code> , and <code>modelB</code> ?</p>
<p>Unless, of course, <code>requires_grad=False</code> for <code>self.modelA</code> and <code>self.modelB</code> - in which case autograd would not calculate the gradients wrt the parameters of these models and only that of <code>MyEnsemble</code> ?</p>
</blockquote>
</aside>
<p>Yes, you are correct that the gradients won’t be calculated for parameters, which are using <code>requires_grad=False</code>. However, <code>model.parameters()</code> would still return all parameters, if you are not filtering them out.</p>

username: David_Alford
cooked: <p>Ok. Thank you for the clarification.</p>
<p>So if I call <code>Module.parameters().grad</code> I will be able to see the gradients? And I should call <code>optimizer.zero_grad()</code> each epoch to clear the gradients so they don’t accumulate? (Doesnt matter where you call <code>zero_grad()</code>).</p>
<p><strong>EDIT:</strong> Also, what do you think is the area of PyTorch most people have trouble with? My focus is Computer Vision. Is there a concept/area in PyTorch that I can focus my energy on where you think people commonly make mistakes?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p><code>Module.parameters().grad</code> won’t work directly and you would have to create a <code>list</code> or iterate it:</p>
<pre data-code-wrap="python"><code class="lang-python">for param in Module.parameters():
    print(param.grad)
</code></pre>
<aside class="quote no-group" data-username="David_Alford" data-post="3" data-topic="94887">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/david_alford/48/26462_2.png" class="avatar"> David_Alford:</div>
<blockquote>
<p>And I should call <code>optimizer.zero_grad()</code> each epoch to clear the gradients so they don’t accumulate?</p>
</blockquote>
</aside>
<p>The usual workflow is to zero out the gradients after each iteration, if you don’t want to accumulate the gradients.</p>
<aside class="quote no-group" data-username="David_Alford" data-post="3" data-topic="94887">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/david_alford/48/26462_2.png" class="avatar"> David_Alford:</div>
<blockquote>
<p>Doesnt matter where you call <code>zero_grad()</code></p>
</blockquote>
</aside>
<p>As long as <code>zero_grad()</code> is not called after the <code>backward()</code> call and before <code>optimizer.step()</code>, it should be OK.</p>
<aside class="quote no-group" data-username="David_Alford" data-post="3" data-topic="94887">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/david_alford/48/26462_2.png" class="avatar"> David_Alford:</div>
<blockquote>
<p>Also, what do you think is the area of PyTorch most people have trouble with? My focus is Computer Vision. Is there a concept/area in PyTorch that I can focus my energy on where you think people commonly make mistakes?</p>
</blockquote>
</aside>
<p>Do you mean here in the discussion board or in the general framework usage?</p>

username: David_Alford
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="94887">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>Module.parameters().grad</code> won’t work directly and you would have to create a <code>list</code> or iterate it:</p>
<pre><code class="lang-auto">for param in Module.parameters():
    print(param.grad)
</code></pre>
</blockquote>
</aside>
<p>That makes sense.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="94887">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Do you mean here in the discussion board or in the general framework usage?</p>
</blockquote>
</aside>
<p>In the general framework usage. I see you are very knowledgeable about PyTorch. Where would you recommend someone focus their energy when learning PyTorch? Any areas you see more mistakes than others that I could focus on?</p>
<p>Thank you for taking the time for the responses.</p>

username: ptrblck
cooked: <p>When you are trying to learn PyTorch, I would suggest to pick an interesting (and personal) project you could spend some time on. E.g. if you are interested in photography and would like to experiment with some style transfer approaches, this would be a great way to learn more about GANs and other architectures. You would learn the framework just by working on the project. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>On the other hand, if you would like to contribute to PyTorch, I would recommend to have a look at the <code>usability / simple-fixes</code> or <code>misc</code> category <a href="https://github.com/pytorch/pytorch/projects/5">here</a>.<br>
Also, the <a href="https://github.com/pytorch/pytorch/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22">good first issue</a> label is useful to check for some starter PRs. <a href="https://pytorch.org/docs/stable/community/contribution_guide.html">The Contribution Guide</a> is a good way to get started.</p>
<p>Generally, I would say that a lot of new users would have some trouble with language models, i.e. how the shapes are used in RNNs, where and when to <code>detach()</code> the activations etc.</p>