Change network parameters in-place

PhysicsIsFun: <p>Greetings!</p>
<p>Suppose I want to write my own optimizer, eg. modify torch.optim.sgd.</p>
<p>In the step function of SGD, the parameters p get updated in-place, eg:</p>
<pre><code class="lang-auto">            for p in group['params']:
                
                ## SOME CODE

                p.add_(d_p, alpha=-group['lr'])
</code></pre>
<p>My first question is why it does not use <code>p.data.add_(d_p, alpha=-group['lr'])</code> as the actual parameter tensor is the data object, is it not?</p>
<p>Also suppose I want to change my parameter p with a computed value x. Is the following correct?</p>
<pre><code class="lang-auto">#### compute x
p.data = x
</code></pre>
<p>I would still be in the step( ) function, so gradient storage should be turned off due to a line <code>@torch.no_grad()</code> before the step( ) function definition.</p>
<p>Best,<br>
PiF</p>

soulitzer: <p>Hi,</p>
<p><code>p.data = x</code> should work, but using the <code>.data</code> parameter is generally not recommended. Because yeah, if you already have <code>@torch.no_grad()</code> before the function, <code>p.add_</code> does what you want in a simpler and more efficient way.</p>
<p>You can think of <code>.data</code> as something similar to <code>detach()</code>, when you ‘get’ the <code>.data</code> attribute, you are actually creating a new tensor that happens to share storage with the old tensor, but does not share the computational graph. If you ‘set’, the <code>.data</code>, attribute, it does handle it underneath so you do modify the original Tensor as you’d expect, but its going to be less efficient because you create more Tensor objects unnecessarily</p>
<p>Also subtle difference to note: something like <code>@torch.no_grad()</code>, doesn’t exactly turn off “gradient storage”. It just means when you perform an op on it, no new nodes get added to its backward graph. For example, you can still perform a <code>.backward()</code> within while <code>@torch.no_grad()</code> and <code>.grad</code> is still populated.</p>

PhysicsIsFun: <p>Thank you soulitzer for the helpful response!</p>
<p>So if I understood you correctly, <code>p.data=x</code> returns a new tensor object <code>p.data</code> that holds the old tensor of p and that is then modi<code>Preformatted text</code>fied to be x. Then Torch imbeds this new tensor back into parameter p. And this process is inefficient as it is not an in-place modification.</p>
<p>What would you recommend instead of <code>p.data = x</code>? I cannot use the <code>add_</code> function as the new value can’t be trivially written as a sum of the old value and something new.<br>
edit: The computation of x is a bit lengthy, but I guess I can use a chain of in-place operations on my parameters like <code>p.add_().cos_()....</code>.</p>

soulitzer: <p>Yep, the inefficiency in <code>p.data=x</code> isn’t the setting part. Its that you allocated a new tensor <code>x</code> first, as opposed to just operating on p in place, which avoids allocating any new tensors.</p>
<p><code>p.copy_(x)</code> is a safer alternative to <code>p.data=x</code>, so I’d recommend that over it, but since you’re still allocating a new tensor in both cases, it won’t be any more efficient.</p>
<p>The only way to avoid to allocating new tensors is to use the chain of in-place ops like you said.</p>