What's the current idiomatic way to SGD stepping w/o using x.grad.data?

username: stas
cooked: <p>What is the current way to write an SGD-like code without using <code>grad.data</code>, and why is <code>.data</code> not documented anywhere (other than the 0.4 migration guide).</p>
<p>Here is a simple code that solves Ax=b using SGD w/ autograd. I can’t find an elegant way to implement the descent-part w/o using <code>grad.data</code>. Looking at optimizers in the pytorch code base, they all use <code>grad.data</code>.</p>
<pre><code class="lang-python">def update1(x):
    with torch.no_grad():
        tmp = x - 0.0001*x.grad
    x.grad.zero_()
    return tmp.requires_grad_(True)

def update2(x):
    x.data -=  0.0001*x.grad.data 
    x.grad.data.zero_()
    return x

# given A and b, and Ax=b: find x using SGD
A = torch.randn(6, 6, requires_grad=False)
b = torch.randn(6, 1, requires_grad=False)

# choose random x and then search with SGD to find x-hat that is the closest to x in Ax=b
x = torch.randn(6, 1, requires_grad=True)

for i in range(40000):
    loss = torch.norm(A @ x - b)
    loss.backward()
    x = update1(x)
    #x = update2(x)
    if not i % 5000: print(f"{loss: &gt;.6f}    {x}")
</code></pre>
<p><code>update2</code> is how the core optimizers do it, but if <code>grad.data</code> shouldn’t be used, what’s the idiomatic way to do that? I tried a clumsy workaround in <code>update1</code> that works, but it looks terrible. And if the idiomatic way is to continue using <code>grad.data</code>, then why is it not documented?</p>
<p>p.s. I understand the potential dangers of using <code>grad.data</code> as explained in the <a href="https://github.com/pytorch/pytorch/releases/tag/v0.4.0" rel="nofollow noopener">0.4. migration notes</a> - this is not what I’m asking about.</p>
<p>Thank you!</p>

username: anantguptadbl
cooked: <aside class="quote no-group quote-modified" data-username="stas" data-post="1" data-topic="67131">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stas/48/8409_2.png" class="avatar"> stas:</div>
<blockquote>
<p>def update1(x): with torch.no_grad(): tmp = x - 0.0001<em>x.grad x.grad.zero_() return tmp.requires_grad_(True) def update2(x): x.data -= 0.0001</em>x.grad.data x.grad.data.zero_() return x # given A and b, and Ax=b: find x using SGD A = torch.randn(6, 6, requires_grad=False) b = torch.randn(6, 1, requires_grad=False) # choose random x and then search with SGD to find x-hat that is the closest to x in Ax=b x = torch.randn(6, 1, requires_grad=True) for i in range(40000): loss = torch.norm(A @ x - b) loss.backward() x = update1(x) <span class="hashtag-raw">#x</span> = update2(x) if not i % 5000: print(f"{loss: &gt;.6f} {x}")</p>
</blockquote>
</aside>
<p>I did not find any way of bypassing the grad.data, but what I did was avoiding the resetting of grad in the update1 function</p>
<pre><code class="lang-auto">def update1(x):
    with torch.no_grad():
        tmpGrad=x.grad
        x = x - 0.0001*x.grad
        x.grad=tmpGrad
    return x.requires_grad_(True)

def update2(x):
    x.data -=  0.0001*x.grad.data 
    x.grad.data.zero_()
    x.grad.zero_()
    return x

# given A and b, and Ax=b: find x using SGD
A = torch.randn(6, 6, requires_grad=False)
b = torch.randn(6, 1, requires_grad=False)

# choose random x and then search with SGD to find x-hat that is the closest to x in Ax=b
x = torch.randn(6, 1, requires_grad=True)

for i in range(10000):
    loss = torch.norm(A @ x - b)
    loss.backward()
    x = update1(x)
    x = update2(x)
    if not i % 100: print(f"{loss: &gt;.6f}")
</code></pre>
<p>Hope this helps</p>

username: tom
cooked: <p>Hi Stas,</p>
<p>good to see you around! So <em>drop the <code>.data</code>, wrap in <code>no_grad</code></em> works here as far as I can see:</p>
<pre><code class="lang-python">def perfect_update(x):
    with torch.no_grad():
        x -= 0.0001 * x.grad
    x.grad.zero_()
    return x
</code></pre>
<p>(I’m having reservations about calling it update when it also zeros grad and about returning x, too, but hey.)</p>
<p>There are two things you previously achieved by using .data:</p>
<ul>
<li>We want to have no autograd graph update for the subtraction. This we can get without data by wrapping the subtraction in <code>with torch.no_grad()</code>. It will preserve the annotation that <code>x</code> requires gradients.</li>
<li>The other thing that <code>.data</code> does it inhibits marking <code>x</code> as in-place updated. But it is in fact <em>the right thing</em> to have <code>x </code> marked as modified. The stuff needing the old <code>x</code> will complain (rightfully, you have a bug in your code if you still need the old <code>x</code> when it is gone), everything else will happily take the new <code>x</code>.</li>
</ul>
<p>The prerequisite for all this is that <code>x</code> is a leaf in the autograd graph, but that should better be the case anyway.</p>
<p>Work for dropping <code>.data</code> is tracked at <a href="https://github.com/pytorch/pytorch/issues/30987" rel="nofollow noopener">#30987</a>. Personally, I think one could just go ahead and tick off the things from the list that can be done today, but given that I’m not the one doing it…</p>
<p>Best regards</p>
<p>Thomas</p>

username: stas
cooked: <aside class="quote no-group" data-username="tom" data-post="3" data-topic="67131">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>good to see you around!</p>
</blockquote>
</aside>
<p>Thank you, Thomas! You too!</p>
<aside class="quote no-group">
<blockquote>
<p>So <em>drop the <code>.data</code> , wrap in <code>no_grad</code></em> works here as far as I can see:</p>
<pre><code class="lang-auto">def perfect_update(x):
    with torch.no_grad():
        x -= 0.0001 * x.grad
    x.grad.zero_()
    return x
</code></pre>
<p>(I’m having reservations about calling it update when it also zeros grad and about returning x, too, but hey.)</p>
</blockquote>
</aside>
<p>Oh, I see, this way wasn’t working since I was trying the direct: <code>x = x - lr*grad</code> approach:</p>
<pre data-code-wrap="python"><code class="lang-python">    with torch.no_grad():
        x = x - 0.0001 * x.grad
    _= x.grad.zero_()
</code></pre>
<p>which fails: <code>AttributeError: 'NoneType' object has no attribute 'zero_'</code> in the <code>x.grad.zero_()</code> call, so I guess in this case in-place subtraction is what I was missing. Then I switched to in-place, and forgot to try this approach, and other ways were failing.</p>
<p>So yes, your <code>perfect_update</code> does the trick!</p>
<p>In-place modification of x is a must then.</p>
<p>Thank you, Thomas.</p>
<blockquote>
<p>(I’m having reservations about calling it update when it also zeros grad and about returning x, too, but hey.)</p>
</blockquote>
<p>Well, that was just an abstraction I added to make my question more readable with the different x update versions. You’re right, it’s not an update.</p>
<p>the full clean solution now goes:</p>
<pre><code class="lang-auto"># given A and b, and Ax=b: find x using SGD
A = torch.randn(6, 6, requires_grad=False)
b = torch.randn(6, 1, requires_grad=False)

# choose random x and then search with SGD to find x-hat that is the closest to x in Ax=b
x = torch.randn(6, 1, requires_grad=True)

for i in range(40000):
    loss = torch.norm(A @ x - b)
    loss.backward()
    with torch.no_grad():
        x = x - 0.0001 * x.grad
    _= x.grad.zero_()
    if not i % 5000: print(f"{loss: &gt;.6f}    {x.T}")
</code></pre>
<p>Excellent!</p>

username: stas
cooked: <aside class="quote no-group" data-username="tom" data-post="3" data-topic="67131">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>Work for dropping <code>.data</code> is tracked at <a href="https://github.com/pytorch/pytorch/issues/30987" rel="noopener nofollow ugc">#30987</a>. Personally, I think one could just go ahead and tick off the things from the list that can be done today, but given that I’m not the one doing it…</p>
</blockquote>
</aside>
<p>That thread answers the other part of the my question of why <code>.data</code> is not documented. And the effort to remove the usage of <code>.data</code> from the pytorch code.</p>
<p>And <a href="https://github.com/pytorch/pytorch/issues/30987#issuecomment-567108808" rel="noopener nofollow ugc">this comment</a> of that thread summarizes the situations where it’s still used.</p>
<p>Thank you, Thomas.</p>