Loss function error when backward

Mrfangfang: <p>import cmath<br>
import torch as t<br>
import numpy as np<br>
from torch.autograd import Function<br>
from torch.nn import functional as F<br>
cmath.sqrt(-1)</p>
<p>class loss(Function):<br>
<span class="mention">@staticmethod</span><br>
def forward(ctx,x,INPUT):</p>
<pre><code>    batch_size = x.shape[0]
    X = x.detach().numpy()
    input = INPUT.detach().numpy()
    Loss = 0
    for i in range(batch_size):
        t_R_r = input[i,0:4]
        R_r = t_R_r[np.newaxis,:]
        t_R_i = input[i,4:8]
        R_i = t_R_i[np.newaxis,:]
        t_H_r = input[i,8:12]
        H_r = t_H_r[np.newaxis,:]
        t_H_i = input[i,12:16]
        H_i = t_H_i[np.newaxis,:]

        t_T_r = input[i, 16:32]
        T_r = t_T_r.reshape(4,4)
        t_T_i = input[i, 32:48]
        T_i = t_T_i.reshape(4,4)
        
        R = np.concatenate((R_r, R_i), axis=1)
        H = np.concatenate((H_r, H_i), axis=1)


        temp_t1 = np.concatenate((T_r,T_i),axis=1)
        temp_t2 = np.concatenate((-T_i,T_r),axis=1)
        T = np.concatenate((temp_t1,temp_t2),axis=0)
        phi_r = np.zeros((4,4))
        row, col = np.diag_indices(4)
        phi_r[row,col] = X[i,0:4]
        phi_i = np.zeros((4, 4))
        row, col = np.diag_indices(4)
        phi_i[row, col] = 1 - np.power(X[i, 0:4],2)
    
        temp_phi1 = np.concatenate((phi_r,phi_i),axis=1)
        temp_phi2 = np.concatenate((-phi_i, phi_r), axis=1)
        phi = np.concatenate((temp_phi1,temp_phi2),axis=0)

        temp1 = np.matmul(R,phi)

        temp2 = np.matmul(temp1,T)  # error
        H_hat = H + temp2

        t_Q_r = np.zeros((4,4))
        t_Q_r[np.triu_indices(4,1)] = X[i,4:10]
        Q_r = t_Q_r + t_Q_r.T
        row,col = np.diag_indices(4)
        Q_r[row,col] = X[i,10:14]
        Q_i = np.zeros((4,4))
        Q_i[np.triu_indices(4,1)] = X[i,14:20]
        Q_i = Q_i - Q_i.T

        temp_Q1 = np.concatenate((Q_r,Q_i),axis=1)
        temp_Q2 = np.concatenate((-Q_i,Q_r),axis=1)
        Q = np.concatenate((temp_Q1,temp_Q2),axis=0)

        t_H_hat_r = H_hat[0,0:4]
        H_hat_r = t_H_hat_r[np.newaxis,:]
        t_H_hat_i= H_hat[0,4:8]
        H_hat_i = t_H_hat_i[np.newaxis,:]

        temp_H1 = np.concatenate((-H_hat_i.T,H_hat_r.T),axis=0)
        H_hat_H = np.concatenate((H_hat.T,temp_H1),axis=1)
        temp_result1 = np.matmul(H_hat,Q)
        temp_result2 = np.matmul(temp_result1,H_hat_H)
        
        Loss += np.log10(1+temp_result2[0][0])
    Loss = t.from_numpy(np.array(Loss / batch_size))
    return Loss
@staticmethod
def backward(ctx,grad_output):
    print('gradient')
    return grad_output
</code></pre>
<p>def criterion(output,input):<br>
return loss.apply(output,input)</p>
<p>Traceback (most recent call last):<br>
File “/Users/mrfang/channel_capacity/training.py”, line 24, in <br>
loss.backward()<br>
File “/Users/mrfang/anaconda3/lib/python3.6/site-packages/torch/tensor.py”, line 150, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/Users/mrfang/anaconda3/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 99, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: function lossBackward returned an incorrect number of gradients (expected 2, got 1)</p>
<p>How could I fix it. Thanks very much</p>

albanD: <p>Hi,</p>
<p>Your forward function takes 2 inputs: x and INPUT.<br>
So the backward function should return 2 things: the gradients for x and the gradients for INPUT.</p>

mathematics: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , <a class="mention" href="/u/alband">@albanD</a><br>
I have problem as</p>
<aside class="quote no-group" data-username="mathematics" data-post="50" data-topic="29387">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mathematics/48/40082_2.png" class="avatar"><a href="https://discuss.pytorch.org/t/custom-loss-functions/29387/50">Custom loss functions</a></div>
<blockquote>
<p>Would i able to use numpy operations at first and at last return tensor operations.<br>
Im making custom triplet loss function ,</p>
</blockquote>
</aside>
<p>I have tried to make custom loss function of that tripletloss. <code>forward()</code> was working but when calling <code>loss.backward()</code><br>
it is giving alot of errors, I tried to debug it and as Alban said forward with two inputs returns 2 things, but my mind still cant comprehent what to do with that.  Sorry, if i was annonying.</p>
<p>Also in custom autograd after getting two vectors <code>v1 and v2</code> , I detached from Tensor and tried to do operations in numpy and at last converted numpy and returned torch <code>tensor</code>. <code>.apply(v1,v2)</code> . As that given autograd tutorial</p>
<pre><code class="lang-auto">    def forward(ctx, input):
        """
        In the forward pass we receive a Tensor containing the input and return
        a Tensor containing the output. ctx is a context object that can be used
        to stash information for backward computation. You can cache arbitrary
        objects for use in the backward pass using the ctx.save_for_backward method.
        """
        ctx.save_for_backward(input)
        return input.clamp(min=0)
</code></pre>
<p>input was directly fed  but my case is I have done numpy operations on it,</p>
<p>I am also confused what should i used for <code>ctx.save_for_backward()</code> tried to use both last converted torch tensor and also <code>v1 and v2</code> but I am not getting. I am also confused what things should be done in backward in my case.<br>
Thank you for your help .</p>
<p>My custom autograd loss  code is below</p>
<pre><code class="lang-auto">class TripletLoss(torch.autograd.Function):
    @staticmethod
    def forward(ctx, v1, v2, margin=0.25):
        
        scores = np.dot(v1.detach().numpy(), v2.detach().numpy().T)
        batch_size = len(scores)
        positive = np.diag(scores) # the positive ones (duplicates)
        negative_without_positive = scores - 2.0 * np.identity(batch_size)
        closest_negative = negative_without_positive.max(axis=1)
        negative_zero_on_duplicate =  scores * (1.0 - np.eye(batch_size))
        mean_negative = np.sum(negative_zero_on_duplicate, axis=1) / (batch_size - 1)
        triplet_loss1 = torch.Tensor(np.maximum(0.0, margin - positive + closest_negative))
        triplet_loss2 = torch.Tensor(np.maximum(0.0, margin - positive + mean_negative))
        triplet_loss = torch.mean(triplet_loss1 + triplet_loss2)
        triplet_loss.requires_grad=True
        ctx.save_for_backward(triplet_loss)
        return triplet_loss
    
    @staticmethod
    def backward(ctx, grad_output):
        input = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad_input[v1&lt;0] = 0
        grad_input[v2&lt;0] = 0
        return grad_input        
</code></pre>

albanD: <p>Hi,</p>
<p>Your forward takes 3 arguments here so the backward should return 3 things: the gradient for v1, v2 and margin.<br>
Since margin is a plain number, you cannot have gradients for it and you should return <code>None</code>. So the backward return statement should be <code>return grad_v1, grad_v2, None</code>.</p>
<p>You should use <code>save_for_backward()</code> if you need to save any input or output of the forward for the backward.</p>