Custom loss(weighted angular distance) in pytorch 0.4.1

Dhorka: <p>Hi,</p>
<p>I am creating my own custom loss in pytorch, this loss is based on angular distance using the cosine similarity. The difference is that in my case I added a weight factor in order to magnify the error in some cases. The thing is, the network I am trying to train is not working as expected. For this reason I would like to know if the implementation that I made in pytorch is correct. I am using pytorch 0.4.1, I read some topics on this forum related on the custom losses with old versions of pytorch. Far as I saw one of the common error is the unpack of the Variables during the calculation , but in pytorch 0.4.1 this does not matter, right? I think my implementation is correct but maybe I miss some details of how this implementation needs to be done in pytorch. I would like to be sure that the error is not something related to my understanding of the implementation of losses in pytorch. Thanks in advance.</p>
<pre><code class="lang-auto">def weighted_angular_MSE(input,target,weight)
	input = torch.mul(F.normalize(input,p=2,dim=1),0.99999)
	target = torch.mul(F.normalize(target,p=2,dim=1),0.99999)
	
	error =torch.div(torch.acos(torch.sum(torch.mul(input,target),dim=1,keepdim=True)),math.pi)

	return torch.div(torch.sum(torch.mul(torch.pow(error,2)[0,:],weight)), torch.nonzero(weight.data).size(0))

class WeightedAngularMSE(nn.Module):
      def __init__(self):
      	super(WeightedAngularMSE,self).__init__()
      def forward(self, input, target, weight):
      	return weighted_angular_MSE(input,target,weight)
</code></pre>

ptrblck: <p>As long as you are using PyTorch methods, the computation should be fine.<br>
I can’t see any obvious error in your code.</p>
<p>As long as <code>weights</code> has some non-zero values, the division should also be fine.<br>
What kind of strange behavior are you experiencing?</p>

Dhorka: <p>Thanks for your comments <a class="mention" href="/u/ptrblck">@ptrblck</a>! What happens is that my network is not learning, as you can see in the following image:</p>
<p>But if the loss function is correct, I think the problem may be something else, like the network is not good enough for my problem or the dataset. I just wanted to make sure that I was not making any silly mistakes by implementing my loss function in pytorch.</p>

ptrblck: <p>Well, the code looks alright.<br>
Regarding the validity of the code, it seems the PyTorch implementation uses <a href="https://pytorch.org/docs/master/_modules/torch/nn/functional.html#cosine_similarity">the dot product and vector norms</a> to calculate the cosine while your code seems a bit more complicated.</p>
<p>Are you sure this line is correct:</p>
<pre><code class="lang-python">torch.acos(torch.sum(torch.mul(input,target),dim=1,keepdim=True))
</code></pre>
<p>I have my doubts about the <code>torch.acos</code> as currently <code>input</code> and <code>target</code> are just normal vectors, i.e. both should be normalized with the product of both norms.<br>
Could you check it and see if I’m mistaken?</p>

Dhorka: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, you are completly right.  It was a big mistake from my side. Thanks for see it! At this moment I am running the experiment again. To avoid future issues I am using cosine_similarity of pytorch. Thanks again!</p>

tcotte: <p><a class="mention" href="/u/dhorka">@Dhorka</a>,<br>
I struggled with the same issue. I noticed your answer and I tried to implement the loss using cosine_similarity too.<br>
However, I wanted to know if you reached to implement the weight angular distance in Pytorch.<br>
Thank you in advance for your feedback.</p>