Custom loss function value does not decrease

mohsenkiskani: <p>I have the following customized loss function which I want to minimize.</p>
<pre><code class="lang-auto">def compute_clamped_dist(target_coordinates_tensor, observed_coordinates_tensor):
    euclidean_dist = 0.5 * torch.sqrt(torch.sum((target_coordinates_tensor-observed_coordinates_tensor)**2, dim =1))
    return torch.clamp(euclidean_dist, max=1.0)

def f1_loss(target_tensor, observed_tensor):
    classification_tensor = target_tensor[:, 4:]
    observed_tensor_12 = observed_tensor[:, :2]
    observed_tensor_34 = observed_tensor[:, 2:]
    target_tensor_12 = target_tensor[:, :2]
    distance_o12_t12 = compute_clamped_dist(target_tensor_12, observed_tensor_12)
    distance_o34_t12 = compute_clamped_dist(target_tensor_12, observed_tensor_34)
    third_component = 0.5*(1+torch.min(distance_o12_t12, distance_o34_t12))
    output = classification_tensor[:, 0] + torch.mul(classification_tensor[:, 1], distance_o12_t12)\
                                         + torch.mul(classification_tensor[:, 2], third_component)
    return output.mean()
</code></pre>
<p>The target tensor is of size (N * 7) and the observation tensor is of size (N * 4). I want to make the observation tensor as similar to the first 4 columns of the target tensor. The last three columns of the target tensor are constants tensors. I am using my custom loss function as follows:</p>
<pre><code class="lang-auto"> for batch_idx, sample_batched in enumerate(one_spot_train_dataloader):
            self.one_spot_optimizer.zero_grad()
            data = sample_batched['image']
            target = sample_batched['label_meta'].float()[:, 1:]
            if self.use_gpu:
                data = data.cuda(self.cuda_name)
                target = target.cuda(self.cuda_name)
                constant_classification_tensor = constant_classification_tensor.cuda(self.cuda_name)
            output = self.one_spot_network(data)
            constant_batch_classification_tensor = constant_classification_tensor[batch_idx * \
                self.batch_size_train: (batch_idx+1) * self.batch_size_train, :]
            final_target = torch.cat((target,constant_batch_classification_tensor), dim = 1)
            loss = f1_loss(final_target, output)
            loss.backward()
            self.one_spot_optimizer.step()
</code></pre>
<p>The optimizer does not reduce the value of my loss function but when I try <code>mse_loss</code> the value is minimized by the optimizer. Can anybody help me understand why this is happening? I wrote my loss function in torch but it is not getting minimized.</p>

colesbury: <p>You usually don’t want to use a <code>clamp</code> operation in your loss function because the derivative in the saturation case is zero. i.e. no learning happens if it’s saturated.</p>

mohsenkiskani: <p><a class="mention" href="/u/colesbury">@colesbury</a> is that the reason that the loss doesn’t get minimized at all?</p>

colesbury: <p>It might be. You should inspect the gradients. Use <code>retain_grad()</code> to store the grad attribute on intermediate variables and look at the value.</p>
<p><a href="https://pytorch.org/docs/stable/autograd.html?highlight=grad#torch.Tensor.retain_grad" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/autograd.html?highlight=grad#torch.Tensor.retain_grad</a></p>

mohsenkiskani: <p><a class="mention" href="/u/colesbury">@colesbury</a> Thanks for your help. I tried your suggestion and when I print <code>output.grad</code> it is only consisting of zeros and nans. Per your comment, I removed the <code>clamp</code> operation from my loss function and looked at the grad again and it looks like this time I see some numbers and some nans. More importantly, this time the optimizer is able to reduce my loss function. So I think your intuition is correct and <code>clamp</code> messes up everything.</p>
<p>Thanks again for your help.</p>

Mr.Bank: <p><a class="mention" href="/u/mohsenkiskani">@mohsenkiskani</a><br>
Can you give me more details? Where did you write the output.grad? In loss function or training block?</p>