Can someone point me to a PyTorchs source code for optim.SGD where the actual weight update takes place

username: csailnadi
cooked: <p>I cannot find a place in a source code for optim.SGD where the actual weight update takes place. I expect to see “something like”</p>
<pre><code class="lang-auto">model.layer.weight -= lr * model.layer.weight.grad
model.layer.bias -= lr * model.layer.bias.grad
</code></pre>
<p>but I cannot find it…</p>
<p>I checked torch/optim/sgd.py and there is no obvious weight update …</p>
<p>Help will be very appreciated!</p>

username: ptrblck
cooked: <p><a href="https://github.com/pytorch/pytorch/blob/dabeff33b9b8cc23ea93967fb7c98cc3dde103ac/torch/optim/sgd.py#L94-L112">These lines of code</a> should correspond to the parameter update.</p>

username: csailnadi
cooked: <p>that probably should, but again I do not see it… having that there is no momentum, or weight decay, just pure SGD with lr, where</p>
<pre><code class="lang-auto">model.layer.weight -= lr * model.layer.weight.grad
model.layer.bias -= lr * model.layer.bias.grad
</code></pre>
<p>takes place more precisely?</p>

username: ptrblck
cooked: <p>Here it is:</p>
<pre><code class="lang-python">d_p = p.grad
...
p.add_(d_p, alpha=-group['lr'])
</code></pre>

username: csailnadi
cooked: <p>oh, I guess it is confusing because I do not see multiplication. Do you know where add_ function is defined? However it is probably not that important for me. Just when I use PyCharm “go to declaration” it brings me to empty declaration. So the definition is lost somewhere in class hierarchy.</p>
<p>I guess weights are stored in  <code>p</code>, right?<br>
Is there a way to tell what layer weight is currently being updated?<br>
Say I have 3 layers, how do I know what weights are updated at any point?<br>
E.g I want to do something with <code>grad</code> before it is used for weights update, but my action depends on a layer</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="csailnadi" data-post="5" data-topic="75158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9e8a1a/48.png" class="avatar"> csailnadi:</div>
<blockquote>
<p>Do you know where add_ function is defined? However it is probably not that important for me. Just when I use PyCharm “go to declaration” it brings me to empty declaration. So the definition is lost somewhere in class hierarchy.</p>
</blockquote>
</aside>
<p>I’m not sure how PyCharm looks up the definition, but the method is an inplace version of <a href="https://pytorch.org/docs/stable/torch.html#torch.add">torch.add</a>, where <code>alpha</code> will be multiplied with <code>other</code>.</p>
<aside class="quote no-group" data-username="csailnadi" data-post="5" data-topic="75158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9e8a1a/48.png" class="avatar"> csailnadi:</div>
<blockquote>
<p>Say I have 3 layers, how do I know what weights are updated at any point?<br>
E.g I want to do something with <code>grad</code> before it is used for weights update, but my action depends on a layer</p>
</blockquote>
</aside>
<p>In that case, I would recommend to use <code>register_hook</code> on the desired parameter to change the gradients before the <code>optimizer.step()</code> method.<br>
Layer-dependent gradient manipulations in the optimizer would most likely break it for any other use case.</p>

username: csailnadi
cooked: <p>I see… so I would do something like this<br>
out_ = model.layer1.register_hook(lambda grad: torch.t(torch.mm(M, torch.t(grad))))<br>
out_2 = model.layer2.register_hook(lambda grad: torch.t(torch.mm(M, torch.t(grad))))</p>
<p>Is it right?</p>
<p>but then how would I make the optimizer to use those updated weights?<br>
e.g SGD optimizer</p>

username: ptrblck
cooked: <p>Since you are manipulating the gradients, the optimizer will see these new gradients in its <code>step</code> function and will use them to update the corresponding parameter.</p>
<p>Let me know, if I misunderstood the question, please.</p>

username: csailnadi
cooked: <p>Oh I was just asking if I should do anything with the output of say <code>model.layer1.register_hook</code>, but I guess I should not, just calling model.layer.register_hook() is sufficient for the changes to be registered. This is the way I understand it</p>

username: ptrblck
cooked: <p>Yes, you can manipulate the gradient directly.<br>
However, note that you have to call <code>register_hook</code> on the parameter, not the module.<br>
This should thus work: <code>model.layer1.weight.register_hook</code>.</p>