Setting margin in contrastive loss

mmisiur: <p>Hi,<br>
I’m trying to retrain siamese network with contrastive loss - I’ve pretrained the net for classification and then replaced classification fc layer with new fc layer of size 512.</p>
<p>However, the net seems not to learn at all. I suspect that this is caused by the margin in contrastive loss. <a href="https://www.quora.com/When-training-siamese-networks-how-does-one-determine-the-margin-for-contrastive-loss-How-do-you-convert-this-loss-to-accuracy" rel="nofollow noopener">Here</a> I’ve learned that If I’ll L2 normalize output features I can set a constant margin and forget about training it. But it seems not to work. I’ve written it like this:</p>
<pre><code class="lang-auto">margin = 2
label_batch = (class_labels_1 != class_labels_2).to(device).float()  # 0: similar pair, 1: different pair
output1 = net(img_batch_1)
output2 = net(img_batch_2)
o1 = F.normalize(output1, p=2, dim=1)
o2 = F.normalize(output2, p=2, dim=1)
euclidean_distance = F.pairwise_distance(output1, output2)
loss_contrastive = torch.mean((1 - label_batch) * torch.pow(euclidean_distance, 2) +
                                          label_batch * torch.pow(torch.clamp(margin - euclidean_distance, min=0.0), 2))
loss_contrastive.backward()
optimizer.step()
</code></pre>
<p>Does it look OK or should I set margin otherwise?</p>
<p>By “the net seems not to learn at all” I mean that the training set loss very quickly drops (from 21 to 0.8) but the test set loss doesn’t change. For the training set mean values of distances between dissimilar pairs drops from ~2.2 to ~1.2. Why does this drops? Shouldn’t it go somewhere above 2.0 (above margin)?</p>

isalirezag: <p>did you find any answer for your question? I have the same problem…</p>

mmisiur: <p>So Hard or Semi-hard Negative Mining is the key, without it the net learned to correctly predict only easy examples. Also keep 1:1 ratio between positive and negative examples, or at least weight it in the loss (like param <code>weight</code> in i.e. <code>nn.CrossEntropyLoss</code>).</p>
<p>Also somehow on my machine the loss was calculated wrong with</p>
<pre><code class="lang-auto">loss_contrastive = torch.mean((1-label_batch) * torch.pow(euclidean_distance, 2) +
                                          label_batch * torch.pow(torch.clamp(margin - euclidean_distance, min=0.0), 2))
</code></pre>
<p>and this</p>
<pre><code class="lang-auto">loss_contrastive = torch.mean((1-label_batch) * torch.pow(euclidean_distance, 2) +
                                          (label_batch) * torch.pow(torch.clamp(margin - euclidean_distance, min=0.0), 2))
</code></pre>
<p>gave me the loss correctly. I don’t remember if I discovered the core problem of the parenthesis or didn’t have time for that. You can try it yourself - I think that this problem doesn’t occur for everyone.</p>
<p>For me all the three things worked.</p>

avisek_lahiri: <p>So, before calculating the Euclidean distance, did you make sure to unit normalize the vectors ?<br>
For example, I have 512D vectors coming out from a Relu layer.<br>
Do you suggest to unit normalize the vector ?</p>
<p>Also, what range of margins you tried ?<br>
Something within [0, 1] ?</p>