Calculating loss with numpy function

gt_tugsuu: <p>Is it okey to convert tensor to numpy and calculate the loss value, and convert that value to tensor and backpropagate?</p>

ptrblck: <p>That won’t work as you are detaching the computation graph by calling numpy operations.<br>
Autograd won’t be able to keep record of these operations, so that you won’t be able to simply backpropagate.</p>
<p>If you need the numpy functions, you would need to implement your own <code>backward</code> function and it should work again. Have a look at <a href="https://pytorch.org/docs/master/notes/extending.html">this tutorial</a> for more information.</p>
<p>Basically you autograd will track all operations as long as you stay in PyTorch land. Could you check, if your numpy functions are available in PyTorch?</p>

gt_tugsuu: <p>Thank you for response.<br>
I wanted to perform neural style transfer, however instead of content and style loss functions, i’ve intended to use a loss that has distance transform in it. Numpy function is here: <a href="https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.ndimage.morphology.distance_transform_edt.html" rel="nofollow noopener">Distance transform</a>, but it seems PyTorch doesn’t have it yet.<br>
I’ll try the tutorial though.</p>

wataoka: <p>I also implement custom loss with numpy. The custom loss doesn’t have <code>backward</code> function. But my model works. Why?</p>

ptrblck: <p>Your custom loss function using numpy should detach the loss from the computation graph, so that all PyTorch parameters, which were used before detaching won’t get a gradient.<br>
Could you check it by printing the <code>grad</code> attribute of some parameters after calling <code>backward</code>?</p>

RichardOey: <p>Hi, Have you solved the problem? I also want to perform numpy to tensor loss function for the Neural Style Transfer. Thank you very much <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

gt_tugsuu: <p>Hi, I’ve followed this tutorial <a href="https://pytorch.org/docs/master/notes/extending.html" rel="nofollow noopener">https://pytorch.org/docs/master/notes/extending.html</a> , somehow it worked <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

xen0f0n: <p>Not sure if I should start a new thread… I’m calculating a weight map for a segmentation task, that needs distance transform (it’ll weight a CE loss). Everything is on PyTorch land except the weight map. In the end, this weight map is just numbers. Will that backpropagate?</p>
<p>(How does cross_entropy loss (torch._C._nn.nll_loss) use the weight argument after all?)</p>

ptrblck: <p>It should work, since your weight map would just scale the loss and thus the gradients.</p>

Hassan_Mahmood: <p>Hi, what if we are adding some components to the loss. And that components are being computed after detaching.<br>
L= pytorch loss + numpy loss<br>
Resultant  L is also a pytorch tensor.</p>
<p>My network performance is getting affected. Although it looks like numpy loss part will have no effects during training.<br>
What can be the reason for that effect, please</p>

ptrblck: <p>There won’t be any effect, as you are adding a constant value to the loss.<br>
How reproducible is this effect? E.g. are you seeing the training constantly affected by it for 10 different runs?</p>

rantonio123: <p>I have some doubts when we need to create a own backward function to include one external value in loss function. For example, to solve this we need create our backward function, but I do not understand how can I do this because when we create this type of function we need to return tensors with gradients, and when we convert by numpy we don’t get gradients, am I rigth?</p>
<pre><code class="lang-auto">criteria= torch.nn.MSELoss()
outputs, latent_space= model(X)
latent_space= latent_space.cpu()
L_S_nump= latent_space.detach().numpy()
value,counts = np.unique(L_S_nump, return_counts=True)
norm_counts = counts / counts.sum()
entro = -(norm_counts * np.log2(norm_counts)).sum()
mse = criteria(O,E)
loss = mse + entro
</code></pre>

ptrblck: <p>Yes, if you are using numpy operations, Autograd won’t be able to track these operations and you would thus detach the computation graph. You could write a custom <code>autograd.Function</code> and define your <code>backward</code> method there as described in <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">this tutorial</a>.<br>
However, based on your code snippet you could also replace the numpy functions (<code>np.unique</code>, <code>sum()</code>, <code>np.log2</code>) with the PyTorch equivalents.</p>

rantonio123: <p>Thank you for reply, I will follow that tip.</p>

ArshadIram: <p>Backprop to compute gradients of a, b, c, d with respect to loss<br>
grad_y_pred = 2.0 * (y_pred - y)<br>
grad_a = grad_y_pred.sum()<br>
grad_b = (grad_y_pred * x).sum()<br>
grad_c = (grad_y_pred * x ** 2).sum()<br>
grad_d = (grad_y_pred * x ** 3).sum()<br>
I am following this example <a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html" rel="noopener nofollow ugc">Calculate backpass and forward using numpy for third degree polynomial </a>. I am confused why we multiply 2.0 with (y_pred - y)?</p>
<p>Second question is while updating weight why this hyphen is showing?<br>
a <strong>-</strong>= learning_rate * grad_a</p>

ArshadIram: <p>If I am not wrong it’s a square of (y_predict - y). so 2.0 is power. we can also calculate it by multiplying it with 2 (y_predict-y) or np.square (y_predict - y)? If this is correct provide me the second answer why we use hyphen sign with variable?</p>

ptrblck: <ol>
<li>The <code>2</code> would be the factor used in the derivative of the square.</li>
<li>The “hypen” is an inplace subtraction, which will subtrace <code>learning_rate * grad_a</code> from <code>a</code>. The out of place version would be:</li>
</ol>
<pre><code class="lang-python">a = a - (learning_rate * grad_a)
</code></pre>