Custom Rank Loss Function

Aldebaran: <p>I’m trying to define a custom loss function in PyTorch. Currently, it is as follows:</p>
<pre><code class="lang-auto">from scipy.spatial.distance import cdist
import numpy as np

class MRRLoss(nn.Module):
""" Mean Reciprocal Rank Loss """
    def __init__(self):
        super(MRRLoss, self).__init__()

    def forward(self, u, v, distance_metric="cosine"):
        #cosine distance between all pair of embedding in u and v batches.
        distances = cdist(u.detach().numpy(), v.detach().numpy(), metric=distance_metric)
        # by construction the diagonal contains the correct elements
        correct_elements = np.expand_dims(np.diag(distances), axis=-1)
        # number of elements ranked wrong.
        return np.sum(distances &lt; correct_elements)
</code></pre>
<p>This loss function is used to train a model that generates embeddings for different objects, such as image and text. The objective is that the embedding of image i is as close as possible to the text t that describes it.</p>
<p>The loss has as input batches u and v, respecting image embeddings and text embeddings. It calculates the cosine similarity between all the x1 and x2 embedding pairs. Returns a value showing how many images were wrong ranked.</p>
<p>However, during training I am having the following error:</p>
<pre><code class="lang-auto">AttributeError: 'numpy.int64' object has no attribute 'backward'
</code></pre>
<p>How could I get around this kind of error?</p>

Anuj_Daga: <p>Can you try this?:</p>
<pre><code class="lang-auto">class MRRLoss(nn.Module):
""" Mean Reciprocal Rank Loss """
    def __init__(self):
        super(MRRLoss, self).__init__()

    def forward(self, u, v):
        u=torch.reshape(u,(-1,))
        v=torch.reshape(v,(-1,))
        #cosine distance between all pair of embedding in u and v batches.
        cos = nn.CosineSimilarity(dim=0)
        distances=cos(u,v)
        # by construction the diagonal contains the correct elements
        correct_elements = torch.diag(cos(u,v),0).unsqueeze(-1)
        # number of elements ranked wrong.
        return torch.sum(distances &lt; correct_elements)
</code></pre>

Aldebaran: <p>From what I could understand, <code>nn.CosineSimilarity</code> loss computes the cosine similarity between an element <code>i</code> of batch <code>u</code> and another element <code>i</code> of batch <code>v</code>. What I’m looking for is an approach to compute the similarity matrix of all elements of u to all elements of v and define it as a PyTorch loss function. It was so easy in Tensorflow 2.</p>

Anuj_Daga: <p>I guess if you want to compute for all dims,<br>
you can simply reshape it with u=torch.reshape(u,(-1,))<br>
and then calculate it</p>

Aldebaran: <p>Thanks but now the error is:</p>
<pre><code class="lang-auto">RuntimeError: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead.
</code></pre>

Anuj_Daga: <p>yes ,its looks good.<br>
I have update the answer again with “torch.diagonal(cos(u,v),0).unsqueeze(0)”<br>
Can you please try that<br>
I should work now.try not to transform to numpy array as long as possible. try to make operations on torch tensor only.</p>

Aldebaran: <pre><code class="lang-auto">&lt;ipython-input-41-59f0a1e15b43&gt; in forward(self, u, v)
     13         distances=cos(u,v)
     14         # by construction the diagonal contains the correct elements
---&gt; 15         correct_elements = torch.diagonal(cos(u,v),0).unsqueeze(0)
     16         # number of elements ranked wrong.
     17         return torch.sum(distances &lt; correct_elements)

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
</code></pre>

Anuj_Daga: <pre><code class="lang-auto">yes, Sorry I thought you were taking out diagonal elements.
torch.diag(cos(u,v),0).unsqueeze(-1))
this should work.
</code></pre>

Aldebaran: <p>Thank you very much, but it’s not working.<br>
You can verify it in a realistic example at Google Colab:<br>
<a href="https://colab.research.google.com/drive/1lycXYTX_G0A9vIGsQeNbd7qfXZYJGLnY?usp=sharing" rel="nofollow noopener">Custom Rank Loss Function</a></p>

Aldebaran: <p>If help someone, I’ve implemented a better solution to this question inpired by <a href="https://papers.nips.cc/paper/6200-improved-deep-metric-learning-with-multi-class-n-pair-loss-objective" rel="nofollow noopener">N-Pair Loss published on NIPS 2016</a>:</p>
<pre><code class="lang-python">import torch
from torch import nn


class NPairsLoss(nn.Module):
    """
    The N-Pairs Loss.
    It measures the loss given predicted tensors x1, x2 both with shape [batch_size, hidden_size],
    and target tensor y which is the identity matrix with shape  [batch_size, batch_size].
    """

    def __init__(self):
        super(NPairsLoss, self).__init__()
        self.ce = nn.CrossEntropyLoss()

    def similarities(self, x1, x2):
        """
        Calculates the cosine similarity matrix for every pair (i, j),
        where i is an embedding from x1 and j is another embedding from x2.

        :param x1: a tensors with shape [batch_size, hidden_size].
        :param x2: a tensors with shape [batch_size, hidden_size].
        :return: the cosine similarity matrix with shape [batch_size, batch_size].
        """
        x1 = x1 / torch.norm(x1, dim=1, keepdim=True)
        x2 = x2 / torch.norm(x2, p=2, dim=1, keepdim=True)
        return torch.matmul(x1, x2.t())

    def forward(self, predict, target):
        """
        Computes the N-Pairs Loss between the target and predictions.
        :param predict: the prediction of the model,
        Contains the batches x1 (image embeddings) and x2 (description embeddings).
        :param target: the identity matrix with shape  [batch_size, batch_size].
        :return: N-Pairs Loss value.
        """
        x1, x2 = predict
        predict = self.similarities(x1, x2)
        # by construction, the probability distribution must be concentrated 
        # on the diagonal of the similarities matrix.
        # so, Cross Entropy can be used to measure the loss.
        return self.ce(predict, target)

</code></pre>

turian: <p>A more recent option is <a href="https://github.com/google-research/fast-soft-sort/" rel="noopener nofollow ugc">fast-soft-sort</a> from Google Research, and it supports pytorch.</p>