Updatation of Parameters without using optimizer.step()

parekh_vivek: <p>I am trying to implement Actor-Critic Algorithm with eligibility trace. As mentioned in algorithm I need to initialize trace vector with  same number of network parameters to zero and then update manually. And at the end I need to update both the network Actor as well Critic network parameters manually without using optimizer.step() function. Is it possible?</p>

albanD: <p>Hi,</p>
<p>Yes of course.<br>
You can just create all these Tensors and then instead of doing <code>opt.step()</code>, do:</p>
<pre><code class="lang-auto">with torch.no_grad():
     params.copy_(new_params)
</code></pre>
<p>The <code>torch.no_grad()</code> is important to make sure that this is not recorded in the computational graph !</p>

parekh_vivek: <p>Hi , Thanks for reply … can you elobarate more in detail with small example</p>

albanD: <p>Hi,</p>
<p>Instead of doing the classical:</p>
<pre><code class="lang-auto">pred = model(inp)
loss = critetion(pred, ground_truth)
optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<p>You can compute the updates by hand and then set them into the weights.<br>
This is basically what SGD does where the update_function return <code>p + lr*p.grad</code>.</p>
<pre><code class="lang-auto">pred = model(inp)
loss = your_loss(pred)
model.zero_grad()
loss.backward()
with torch.no_grad():
  for p in model.parameters():
    new_val = update_function(p, p.grad, loss, other_params)
    p.copy_(new_val)
</code></pre>

parekh_vivek: <aside class="quote no-group" data-username="albanD" data-post="4" data-topic="34244">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>torch.no_grad():</p>
</blockquote>
</aside>
<p>Thanks<br>
Can you tell  me here have you define update_function(p,p.grad,loss,other_params) manually or its already there in pytorch documentation as I couldnot find and hwo to do with Adam same thing if i want to do?</p>

albanD: <p>You have to implement it yourself so that it computes the new value that you want. In your case I guess it should compute what is in the paper you cite.</p>

parekh_vivek: <aside class="quote no-group" data-username="albanD" data-post="4" data-topic="34244">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>You can compute the updates by hand and then set them into the weights.</p>
</blockquote>
</aside>
<p>ok thanks.Stil I have confusion how can I define like and cross verify whether its true or not? if you can help in that</p>

albanD: <p>Sorry I don’t understand that question “how can I define like and cross verify whether its true or not?” could you explain please?</p>

parekh_vivek: <p>Like in my case I define update_function and then update the parameters so whether its true updated or not how can I check. Like f=x**2 ,I know gradient is 2x I can verify manually like that.</p>
<p>In my case above there are two neural network and I have t update both the neural network parametr manually with one function so implemenatation wise I am clue less how can I achieve wrt to above algorithm</p>

albanD: <p>What do you mean by “its true updated” do you mean the correct gradients for your function? Well they most certainly are not (in the sense that your fonction don’t have proper gradients) otherwise you could just use the autograd to compute them, you would not have to compute them by hand <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I am not familiar with the algorithm you shared above but if you have two models you can loop through model1 parameters then model2 parameters. But that depends on what the algorithm is doing, you’ll need to look into that.</p>

albanD: <p>I can’t for your problem as I don’t know it.<br>
If you were doing SGD-like updates, it would be:</p>
<pre><code class="lang-auto">def update_function(param, grad, loss, learning_rate):
  return param - learning_rate * grad

</code></pre>

parekh_vivek: <p>Thanks no problem …for this much help</p>

Mahmoud_Abdelkhalek: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,  based on this answer:</p><aside class="quote quote-modified" data-post="4" data-topic="34244">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/updatation-of-parameters-without-using-optimizer-step/34244/4">Updatation of Parameters without using optimizer.step()</a> <a class="badge-category__wrapper " href="/c/reinforcement-learning/6"><span data-category-id="6" data-drop-close="true" class="badge-category " title="A section to discuss RL implementations, research, problems"><span class="badge-category__name">reinforcement-learning</span></span></a>
  </div>
  <blockquote>
    Hi, 
Instead of doing the classical: 
pred = model(inp)
loss = critetion(pred, ground_truth)
optimizer.zero_grad()
loss.backward()
optimizer.step()

You can compute the updates by hand and then set them into the weights. 
This is basically what SGD does where the update_function return p + lr*p.grad. 
pred = model(inp)
loss = your_loss(pred)
model.zero_grad()
loss.backward()
with torch.no_grad():
  for p in model.parameters():
    new_val = update_function(p, p.grad, loss, other_params)
    p.co…
  </blockquote>
</aside>
<p>
Does this mean that the code used to manually update parameters under the “Neural net from scratch (no torch.nn)” section <a href="https://pytorch.org/tutorials/beginner/nn_tutorial.html#neural-net-from-scratch-no-torch-nn" rel="noopener nofollow ugc">here</a> is not correct?</p>
<p>More precisely, I am talking about this snippet of code:</p>
<pre><code class="lang-auto">with torch.no_grad():
            weights -= weights.grad * lr
            bias -= bias.grad * lr
            weights.grad.zero_()
            bias.grad.zero_()
</code></pre>
<p>Whenever I try to run something like this, the <code>requires_grad</code> property of <code>weights</code> and <code>bias</code> is automatically set to <code>False</code>, and I suspect this is because <code>weights</code> and <code>bias</code> are being re-assigned inside a <code>torch.no_grad()</code> context. However, if I use <code>copy_</code> inside a <code>torch.no_grad()</code> context as you mentioned above, it works fine. I think this is because values are changed in-place. Could you please advise?</p>

albanD: <p>In the nn tutorial, they are <em>not</em> re-assigned, <code>-=</code> is actually changing it inplace.<br>
This is why in this sample I use <code>.copy_()</code> to be able to modify it inplace with a set value.</p>

Mahmoud_Abdelkhalek: <p>So operations like <code>+=</code> and <code>-=</code> modify a tensor in-place? I thought <code>a -= b</code> is short for <code>a = a - b</code> like in normal Python?</p>

albanD: <p>No actually, in normal python, the two are very different.<br>
The first one calls <code>__iadd__</code> on a and that’s it.<br>
The second calls <code>__add__</code> and assign the result back to the variable named a.</p>
<p>So pytorch respects this convention of th first one changing the Tensor inplace while the second does not.</p>

Mahmoud_Abdelkhalek: <p>Wow, did not know that. Thanks for the info!</p>

mrizwank97: <pre><code class="lang-auto">for p in meta_i3d.parameters():
      new_val = p - 0.0001 * p.grad
      p.copy_(new_val)
</code></pre>
<p>I am doing this to update parameter values but getting an error i.e. “RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.”</p>
<p>Is there any workaround for this? i am working on meta-learning and need to update the weights without calling optimizer.step()</p>

albanD: <p>Hi,</p>
<p>You just need to wrap the call to <code>.copy_</code> into a <code>with torch.no_grad()</code> block to properly disable the autograd during the weight update.</p>

RachelShalom: <p>btw why using the inplace copy is needed?</p>