How requires_grad_(True) effect the training result

DanielC: <p>Hi there,<br>
I am training a network whose loss function is given by:</p>
<pre><code class="lang-auto">def loss_func(vec, yhat):
    # yhat is the output of network
    # vec is a vector
    loss = yhat.T @ vec
    return loss
</code></pre>
<p>Where the <code>vec</code> is given by following method:</p>
<pre><code class="lang-auto">def mv(w, yhat):
    # w is a external matrix
    # yhat is the output of the network
    return w @ yhat
</code></pre>
<p>and the training process is:</p>
<pre><code class="lang-auto">for i in range(epochs):
    yhat = net(input_data)
    vec = mv(w, yhat)
    loss = loss_func(w, vec)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>When I train the network like this way, it will return my very good result. However, when I change the third line in above code block to <code>vec = mv(w, yhat.data)</code>, the training is very hard to converge, and final result is always worse. Can anyone explain this?</p>
<p>Appreciate any reply.</p>

ptrblck: <p>Could you explain why you are using the deprecated <code>.data</code> attribute?<br>
Using it would skip any Autograd checks (and probably recordings) and your training might break easily.</p>

DanielC: <p>Hi ptrblck,<br>
Thank you very much for your reply, the reason I use <code>.data</code> is that I’d like to use <code>cupy</code> for calculating the <code>vec</code> I need. Converting <code>torch.Tensor</code> to <code>cupy.array</code> ask me to do that.<br>
In the function I given before:</p>
<pre><code class="lang-auto">def mv(w, yhat):
    # w is a external matrix
    # yhat is the output of the network
    return w @ yhat
</code></pre>
<p>The <code>w</code> matrix is a huge and sparse matrix. using PyTorch function <code>torch.sparse.mm()</code> calculating it is too slow (This is also a thing I cannot understand).</p>

ptrblck: <p>If you want to use a 3rd party library (such as <code>cupy</code>) you would need to write a custom <code>autograd.Function</code> and implement the <code>forward</code> as well as <code>backward</code> pass manually.<br>
<a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">This tutorial</a> would show you a small example how to do so.</p>

DanielC: <p>Thanks ptrblck,<br>
Could you please explain a bit about why PyTorch calculate spmv so slow? I guess both <code>torch</code> and <code>cupy</code> call some function in <code>cuSPARSE</code> to calculate spmv, why is there such a big difference between their performance?</p>

emcastillo: <p>We have an exmaple of how to interface CuPy/PyTorch in the documentation<br>
<a href="https://docs.cupy.dev/en/stable/user_guide/interoperability.html#using-custom-kernels-in-pytorch" class="onebox" target="_blank" rel="noopener nofollow ugc">https://docs.cupy.dev/en/stable/user_guide/interoperability.html#using-custom-kernels-in-pytorch</a></p>
<p>Speed of routines may be heavily affected by your device, which GPU are you using?</p>

ptrblck: <p>I would need to see your exact use case as well as setup to see which calls are actually used and how you are profiling your code.</p>