How to put the old_weight back

username: Pedro_7773
cooked: <p>What I am trying to do is to put the  the old parameters after the update<br>
here is a demo code.</p>
<pre><code class="lang-auto">        for param in net.parameters():
            d_p = param.grad.data
            param.data.sub_(d_p * lr)
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        for param in net.parameters():  
            param.data.add_(d_p * lr)
</code></pre>

username: KFrank
cooked: <p>Hi Pedro!</p>
<aside class="quote no-group" data-username="Pedro_7773" data-post="1" data-topic="164087" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pedro_7773/48/48852_2.png" class="avatar"> Pedro_7773:</div>
<blockquote>
<p>What I am trying to do is to put the  the old parameters after the update</p>
</blockquote>
</aside>
<p>First, don’t use <code>data</code> – it is deprecated and breaks things.</p>
<p>You may modify <code>param</code> inplace by wrapping the operation in a<br>
<code>with torch.no_grad():</code> block:</p>
<pre data-code-wrap="python"><code class="lang-python">        with torch.no_grad():
            for param in net.parameters():
                d_p = param.grad
                param.sub_(d_p * lr)
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        with torch.no_grad():
            for param in net.parameters():
                d_p = param.grad
                param.add_(d_p * lr)
</code></pre>
<p>(If you don’t want to backpropagate though your <code>outputs</code> and <code>loss</code><br>
computations, you could wrap the whole thing in a single <code>no_grad()</code><br>
block.)</p>
<p>Note that due to round-off error, <code>sub_()</code> followed by <code>add_()</code> won’t<br>
necessarily take you back to <em>exactly</em> where you started.  If you need<br>
to restore <code>param</code> exactly, save a copy of <code>param</code> and then write it back<br>
after the modification, e.g.</p>
<pre data-code-wrap="python"><code class="lang-python">        with torch.no_grad():
            param.copy_ (copy_of_original_param)
</code></pre>
<p>Best</p>
<p>K. Frank</p>

username: Pedro_7773
cooked: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="164087">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p><code>copy_of_original_param</code></p>
</blockquote>
</aside>
<p>Thanks a lot KFrank<br>
But i am confused how to copy parameters in the first place in copy_of_original_param , can you illustrate it further please?</p>

username: KFrank
cooked: <p>Hi Pedro!</p>
<aside class="quote no-group" data-username="Pedro_7773" data-post="3" data-topic="164087" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pedro_7773/48/48852_2.png" class="avatar"> Pedro_7773:</div>
<blockquote>
<p>But i am confused how to copy parameters in the first place</p>
</blockquote>
</aside>
<p>Note that <code>param</code> is different for each iteration of your <code>for</code> loop, so you<br>
will need to store a list of the parameter values:</p>
<pre><code class="lang-auto">        param_list = []
        with torch.no_grad():
            for param in net.parameters():
                param_list.append (param.clone())
                d_p = param.grad
                param.sub_(d_p * lr)
</code></pre>
<p>(If for some reason <code>param.clone()</code> isn’t in the <code>no_grad()</code> block, you<br>
would want <code>param_list.append (param.detach().clone())</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>