Build your own loss function in PyTorch

Ismail_Elezi: <SCODE>X = np.asarray([[0.6946, 0.1328], [0.6563, 0.6873], [0.8184, 0.8047], [0.8177, 0.4517], 
                [0.1673, 0.2775], [0.6919, 0.0439], [0.4659, 0.3032], [0.3481, 0.1996]], dtype=np.float32)
X = torch.from_numpy(X)
y = np.asarray((1,3,2,2,3,1,2,3), dtype=np.float32)
y = torch.from_numpy(y)

def similarity(i, j):
    ''' This function defines the similarity between vectors i and j
    inputs: i, j - vectors of the same length
             sigma - the denumerator parameter
    output: sim - similarity value (real number from 0 to 1) '''
    
    dist = torch.norm(i - j)  
    return dist

def similarity_matrix(mat):
    ''' This function creates the similarity matrix of a dataset
    input: mat - dataset in matrix format
           sigma - a paramter which defines similarity
    output: simMatrix - the similarity matrix '''
    
    a = mat.size()
    a = a[0]
    simMatrix = torch.zeros((a,a))
    for i in xrange(a):
        for j in xrange(a):
            simMatrix[i][j] = similarity(mat[i], mat[j])            
    return simMatrix 

def convert_y(y):
    n = y.size()
    n = n[0]
    converted_y = torch.zeros((n, n))
    for i in xrange(n):
        for j in xrange(n):
            if y[i] == y[j]:
                converted_y[i, j] = 1
    return converted_y

def customized_loss(X, y):
    X_similarity = similarity_matrix(X)
    association = convert_y(y)
    loss_num = torch.sum(torch.mul(X_similarity, association))
    loss_all = torch.sum(X_similarity)
    loss_denum = loss_all - loss_num
    loss = loss_num/loss_denum
    return loss

loss = customized_loss(X, y)
print(loss)
<ECODE> Now, of course, considering that I am going to use it as the final layer, of the neural net, I would need to compute the gradients of it and then use them in the backpropagation. Explaining the function a bit: I first transform the input data space into a kind of similarity matrix (0 it means the data being the same, the higher the number in ij-th entry, the higher is the dissimilarity). Then in order to find the intra-cluster loss, I multiply this matrix with a 0/1 matrix, where the ij-th entry is 1 if the element i and j are in the same cluster, 0 otherwise. The intra-cluster loss is find similarity, and finally, we just divide the two losses. My questions are: Thanks for any answer, or possible hint.

apaszke: 

Ismail_Elezi: Thanks a lot! I rewrote everything using Torch, so now, it should work if I use loss.backward(X, y)? Advises about writing my own autograd function and/or computing the similarity more efficiently are very welcome. It is definitely something that I need to do later, but for now I need just a simple version of this working. Edit - It looks that it works by rewriting the final function as: <SCODE>def customized_loss(X, y):
    X_similarity = Variable(similarity_matrix(X), requires_grad = True)
    association = Variable(convert_y(y), requires_grad = True)
    temp = torch.mul(X_similarity, association)
    loss_num = torch.sum(torch.mul(X_similarity, association))
    loss_all = torch.sum(X_similarity)
    loss_denum = loss_all - loss_num
    loss = loss_num/loss_denum
    return loss
<ECODE> All is good for now, thanks again!

apaszke: 

apaszke: 

Ismail_Elezi: I think that I got lost now. Rewriting an another time the function (in probably a more readable way): <SCODE>X = Variable(torch.Tensor([[0.6946, 0.1328], 
                           [0.6563, 0.6873], 
                           [0.8184, 0.8047], 
                           [0.8177, 0.4517], 
                           [0.1673, 0.2775], 
                           [0.6919, 0.0439],
                           [0.4659, 0.3032],
                           [0.3481, 0.1996]]))

y = Variable(torch.Tensor([1.0, 3.0, 2.0, 2.0, 3.0, 1.0, 2.0, 3.0]))


def customized_loss(X, y):
           
    def similarity_matrix(mat):
        a = mat.size()
        a = a[0]
        simMatrix = Variable(torch.zeros(a,a), requires_grad = True)
        for i in xrange(a):
            for j in xrange(a):
                simMatrix[i][j] = torch.norm(mat[i] - mat[j])         
        return simMatrix 
    
    def convert_y(y):
        a = y.size()
        a = a[0]
        converted_y = Variable(torch.zeros(a,a), requires_grad = True)
        for i in xrange(n):
            for j in xrange(n):
                if y[i] == y[j]:
                    converted_y[i, j] = 1
        return converted_y

    X_similarity = similarity_matrix(X)
    association = convert_y(y)
    loss_num = torch.sum(torch.mul(X_similarity, association))
    loss_all = torch.sum(X_similarity)
    loss_denum = loss_all - loss_num
    loss = loss_num/loss_denum
    return loss

loss = customized_loss(X, y)
<ECODE> As far as I can see, everything now is done in Variables (from beginning to the end). We are giving X and y (which are variables) to the function, and then everything is done in Variables. The only other variables that I need to define is simMatrix in the similarity_matrix function, and there I am having this error: <SCODE>RuntimeError: in-place operations can be only used on variables that don't share storage with any other variables, but detected that there are 2 objects sharing it.
<ECODE> Of course, the same thing happens in convert_y function when I create the converted_y Variable. And I have no clue, what is going wrong, while googling this error doesn’t show any result. … You already spent some time here, so thanks for that, but in case you can guide me how to fix this problem (or writing it if it is a quick fix) it would be awesome. From the pyTorch tutorial about the Variables it is not clear to me what I am doing wrong (haven’t ever used Torch). I guess that the problem is that I am implicitly creating a new Variable in the middle of the graph, but is there any way around it? Is there a solution around this?

fmassa: <SCODE># (x - y)^2 = x^2 - 2*x*y + y^2
def similarity_matrix(mat):
    # get the product x * y
    # here, y = x.t()
    r = torch.mm(mat, mat.t())
    # get the diagonal elements
    diag = r.diag().unsqueeze(0)
    diag = diag.expand_as(r)
    # compute the distance matrix
    D = diag + diag.t() - 2*r
    return D.sqrt()
<ECODE>

Ismail_Elezi: About, if you are not backpropagating through y part…I am a bit confused. Essentially, the algorithm is: While I do not need to backprop through y, Y is multiplied with X (and Y comes from y), so I think that I need to backprop through Y, right? Cheers!

fmassa: <SCODE>def convert_y2(y):
    s = y.size(0)
    y_expand = y.unsqueeze(0).expand(s, s)
    Y = y_expand.eq(y_expand.t())
    return Y
<ECODE>

Ismail_Elezi: @fmassa You’re absolutely right. About your function, it returns a ByteTensor which means that Y cannot be multiplied with X (which is a FloatTensor). There should be something that allows casting a Tensor to some other type, right?

fmassa: <SCODE>a = torch.ByteTensor([0,1,0])
b = a.float() # converts to float
c = a.type('torch.FloatTensor') # converts to float as well
<ECODE> Possible shortcuts for the conversion are the following: .byte() .short() .char() .int() .long() .float() .double 
.half() # for cuda only at the moment

Ismail_Elezi: Excellent! Thanks a lot! I think that I still need to fully understand how these functions work (read them in details), but everything is working now. Of course, the ANN isn’t working (a lot of NANs immediately after the first iteration), but that is something that I need to investigate and see the gradients’ values.

fmassa: 

Ismail_Elezi: @fmassa I am using the version that is written on this thread, and is giving NaNs after the first iteration. Typically, when I got Nans in the past it was either an error in differentiation or a large training rate. Both of these cannot be this time because it is automatic differentiation and I am using extremely small training rates (3e-7 in Adam, while typically I use 3e-4). Anyway, I will have to read the documentation to find how to print the gradients, which might give me an idea on what is going wrong. And then, if I have problems, I will make an another thread. Thanks for everything!

apaszke: You can use hooks e.g.: <SCODE>x = Variable(torch.randn(5, 5), requires_grad=True)
y = Variable(torch.randn(5, 5), requires_grad=True)
z = x + y
# this will work only in Python3
z.register_hook(lambda g: print(g)) 
# if you're using Python2 do this:
# def pring_grad(g):
#     print g
# z.register_hook(print_grad)
q = z.sum()
q.backward()
<ECODE>

Ismail_Elezi: Excellent, I will try it and see what is going wrong.

melgor: About NaNs is your results: it is related to Loss Function. I was implementing sth like that in TensorFlow and I get NaNs too. Based on experiments, it look like the gradient of diagonal in similarity matrix cause NaNs. I modified to skip diagonal (so take just right upper triangle matrix, as left down is just the same).

edgarriba: 

Ismail_Elezi: @apaszke loss.register_hook(lambda g: print(g)) I get: <SCODE>Variable containing:
1
[torch.FloatTensor of size 1]

Variable containing:
1
[torch.FloatTensor of size 1]
<ECODE> which isn’t very helpful. Now, on my example, if I want all the gradients which are computed on my loss function, how can I use register_hook to do so?

apaszke: