Sklearn loss function

111242: <p>I am trying to use <code>sklearn.metrics.average_precision_score</code> as the loss function as follows:</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

for i, (region, labels) in tqdm.tqdm(enumerate(train_loader)):  
        region = region.to(device)
        labels = labels.to(device)
        # Forward pass
        outputs = model(region)
        ap = -average_precision_score(labels.float().detach().numpy(), outputs.squeeze().detach().numpy())
        loss = torch.FloatTensor([ap])
        loss.requires_grad_(True)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>I am trying to maximize the average precision score, hence the negative sign (for some reason, the loss stays the same throughout training with and without the negative sign). What is going wrong here, and it is okay to use a loss function from sklearn as shown above? Thank you.</p>

mxahan: <p>I was just wondering, did you try to wrap the sklearn loss function inside a custom loss function?</p>

111242: <p><a class="mention" href="/u/mxahan">@mxahan</a> Thanks for the suggestion. Can you show me or direct me to where I can see how that can be done?</p>

mxahan: <pre><code class="lang-auto">def custom_loss(outputs, labels):
    loss = torch.sum(-average_precision_score(labels, outputs))
    return loss
</code></pre>
<p>Does it work?</p>

111242: <p>Unfortunately, the loss still remains constant at every epoch after fixing the loss function the way you suggested. Here’s my new loss function:</p>
<pre><code class="lang-auto">def custom_loss(labels, outputs):
    loss = torch.FloatTensor([-average_precision_score(labels, outputs)])
    loss.requires_grad_(True)
    return loss
</code></pre>
<p>…<br>
in training:</p>
<pre><code class="lang-auto">loss = custom_loss(labels.int().detach().numpy(), outputs.squeeze().detach().numpy())
</code></pre>
<p><code>loss.requires_grad_(True)</code> seems necessary, leaving it out causes <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn </code>. However, the fact that this is necessary makes me wonder if PyTorch is keeping the loss as a variable to minimize. Any thoughts on this? Thanks!</p>

ptrblck: <aside class="quote no-group" data-username="111242" data-post="5" data-topic="149850">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111242/48/20626_2.png" class="avatar"> 111242:</div>
<blockquote>
<p><code>loss.requires_grad_(True)</code> seems necessary, leaving it out causes <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn </code></p>
</blockquote>
</aside>
<p>The error message points to a detached computation graph and while calling <code>.requires_grad_()</code> on the <code>loss</code> tensor would get rid of this error message, it would not fix the underlying problem.</p>
<p>You are currently breaking the computation graph in various places:</p>
<ul>
<li>calling <code>.detach()</code> on a tensor detaches it from the computation graph obviously</li>
<li>calling <code>numpy()</code> also detaches the tensor form the computation graph. In general: using any 3rd party library will detach the tensor, as Autograd won’t be able to track these operations</li>
<li>re-creating a new tensor via <code>torch.Floatensor(tensor)</code> will detach <code>tensor</code> from the computation graph</li>
</ul>
<p>If you want to use a 3rd party library such as <code>sklearn.metrics.average_precision_score</code>, you could use it in a custom <code>autograd.Function</code> and implement the <code>backward</code> pass manually.<br>
The first thing I would check is if this method is differentiable at all. If so, you could also try to re-implement it in PyTorch directly.</p>