How to create an optimizer given a loss function

keqiao_li: <p>I’m quite confused about how the optimizer created. From the Pytorch tutorial, when people implement SGD by defining a custom module first, which contains a constructor and a forward function, then using SGD optimizer to calculate the model parameters (please see the following code). My question is what if I am only given a loss function(please see the following pic) and I don’t have a “forward” function, how to create an optimizer?  If my question is not clearly stated, please let me know and I’ll provide further information, thanks for any help in advance.</p>
<p>import torch</p>
<p>class model(torch.nn.Module):</p>
<pre><code>def __init__(self, D_in, H, D_out):
    super(model, self).__init__()
    self.linear1 = torch.nn.Linear(D_in, H)
    self.linear2 = torch.nn.Linear(H, D_out)

def forward(self, x):
    h_relu = self.linear1(x).clamp(min=0)
    y_pred = self.linear2(h_relu)
    return y_pred 
</code></pre>
<p>model = model(D_in, H, D_out)</p>
<p>optimizer = torch.optim.SGD(model.parameters(), lr=1e-4)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/0/7064d2344da2e8f5668d021cd263f9305bec4e1a.png" data-download-href="https://discuss.pytorch.org/uploads/default/7064d2344da2e8f5668d021cd263f9305bec4e1a" title="Screen Shot 2020-02-22 at 2.47.51 PM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/0/7064d2344da2e8f5668d021cd263f9305bec4e1a.png" alt="Screen Shot 2020-02-22 at 2.47.51 PM" data-base62-sha1="g2hu42QBagqmNMIRHpnV6sitEfw" width="690" height="129" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-02-22 at 2.47.51 PM</span><span class="informations">872×164 6.57 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The optimizer just stored the references to the passed parameters and uses their <code>.grad</code> attribute to perform the update step.<br>
It does not depend on the forward pass.<br>
If you are able to generate the gradients in some way, the optimizer will just use them via their <code>.grad</code>.</p>

keqiao_li: <aside class="quote no-group" data-username="keqiao_li" data-post="1" data-topic="71029">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/keqiao_li/48/20969_2.png" class="avatar"> keqiao_li:</div>
<blockquote>
<p>model.parameters()</p>
</blockquote>
</aside>
<p>So basically what you mean is that I can directly call “torch.optim.SGD(parameter.grad, lr = 1e-4)” something like that?</p>

ptrblck: <p>No, you still need to pass the parameters as:</p>
<pre><code class="lang-python">torch.optim.SGD(model.parameters(), lr=1e-4)
# or as a list
torch.optim.SGD([param1, param2], lr=1e-4)
</code></pre>
<p>The optimizer will then use the <code>param.grad</code> attributes to update them.</p>

keqiao_li: <p>Thank you for your kindly help! It helps me to understand Pytorch a lot!</p>