A simple example but why not work?

username: shinra
cooked: <pre><code class="lang-auto">import torch
import torch.nn as nn
N,D_in,H,D_out=64,1000,100,10
model=torch.nn.Sequential(
    torch.nn.Linear(D_in,H,bias=True),
    torch.nn.ReLU(),
    torch.nn.Linear(H,D_out,bias=True)
)
#torch.nn.init.normal_(model[0].weight)
#torch.nn.init.normal_(model[2].weight)
loss_fn=torch.nn.MSELoss(reduction='sum')
x=torch.randn(N,D_in)
y=torch.randn(N,D_out)
learning_rate=1e-3
for i in range(1000000):

    y_pred=model(x)
    loss=loss_fn(y,y_pred)
    if(i%2000==0):
        print(i,loss.item())
    loss.backward()
    #with torch.no_grad():
    for param in model.parameters():
        param=param-param.grad*learning_rate
</code></pre>
<p>output:<br>
0 654.9241333007812<br>
2000 654.9241333007812<br>
4000 654.9241333007812<br>
6000 654.9241333007812</p>
<p>Loss is not reducing…</p>

username: KFrank
cooked: <p>Hi shinra!</p>
<aside class="quote no-group quote-modified" data-username="shinra" data-post="1" data-topic="97538" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shinra/48/29191_2.png" class="avatar"> shinra:</div>
<blockquote>
<pre><code class="lang-auto">    for param in model.parameters():
        param=param-param.grad*learning_rate
</code></pre>
<p>Loss is not reducing…</p>
</blockquote>
</aside>
<p>This is more of a python question than something specific to pytorch.</p>
<p>In:</p>
<pre data-code-wrap="python"><code class="lang-python">    for param in model.parameters():
</code></pre>
<p><code>param</code> is python variable which means that it’s a reference to<br>
something.  In each iteration through the <code>for</code> loop it is set to refer<br>
to one of the <code>model.parameters()</code>.</p>
<p>But when you run the body of the loop:</p>
<pre data-code-wrap="python"><code class="lang-python">        param=param-param.grad*learning_rate
</code></pre>
<p>you first create a <em>new</em> python tensor,<br>
<code>param-param.grad*learning_rate</code>,<br>
and then set the variable <code>param</code> to refer to that new tensor:</p>
<pre data-code-wrap="python"><code class="lang-python">        param=param-param.grad*learning_rate
</code></pre>
<p>This changes what <code>param</code> <em>refers</em> to, but doesn’t change the value<br>
of the parameter in <code>model.parameters()</code> to which <code>param</code> <em>used to</em><br>
refer.</p>
<p>Consider the pure python:</p>
<pre data-code-wrap="python"><code class="lang-python">a = [1, 2, 3]
b = a
b = [4, 5, 6]
</code></pre>
<p>You can easily verify that after running this <code>b</code> refers to a list that<br>
has the value <code>[4, 5, 6]</code>, but that <code>a</code> still refers to the list with value<br>
<code>[1, 2, 3]</code>.</p>
<p>This is the same thing that is happening with your variable <code>param</code><br>
and the actual parameter in <code>model.parameters()</code> that <code>param</code><br>
temporarily referred to.</p>
<p>(The most convenient way to do what you want is to use<br>
<code>opt = torch.optim.SGD (...)</code>, and call <code>opt.step()</code>.<br>
Don’t forget to “zero your grads” (e.g., <code>opt.zero_grad()</code>)<br>
before calling <code>loss.backward()</code> in each iteration of your<br>
training loop.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: shinra
cooked: <p>Thank you ! I have got the crux of the problem. If I have some easy ways to upgrade the weights in model.parameters() without using step()?</p>

username: KFrank
cooked: <p>Hi shinra!</p>
<aside class="quote no-group" data-username="shinra" data-post="3" data-topic="97538" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shinra/48/29191_2.png" class="avatar"> shinra:</div>
<blockquote>
<p>If I have some easy ways to upgrade the weights in model.parameters() without using step()?</p>
</blockquote>
</aside>
<p>I <em>believe</em> the following should work:</p>
<pre data-code-wrap="python"><code class="lang-python">    with torch.no_grad():
        for param in model.parameters():
            param.copy_ (param - param.grad * learning_rate)
</code></pre>
<p><code>param.copy_()</code> changes “in place” the data stored in the parameter<br>
referred to by <code>param</code>, rather than changing the variable <code>param</code> to<br>
refer to something else.</p>
<p>Best.</p>
<p>K. Frank</p>

username: shinra
cooked: <p>It works well.Thanks so much ! (I should really try to read Docs more)</p>