Distance function in TripletMarginloss

hamedB: <p>Hi.<br>
Is the distance function implemented in TripletMarginLoss with p=2 as same as that implemented by torch.dist(p=2)?</p>

ptrblck: <p><code>nn.TripletMarginLoss</code> would use <code>torch.pairwise_distance</code> internally as seen <a href="https://github.com/pytorch/pytorch/blob/44ff6be35ae380951e48613977b94dc282b74d04/aten/src/ATen/native/Loss.cpp#L153-L175">here</a>. While <code>pairwise_distance</code> should yield the same result as <code>torch.dist</code> for 1-dimensional tensors, it should not yield the same outputs for multi dimensional tensors (of course the actual <code>TripletMarginLoss</code> would also use the specified <code>margin</code>):</p>
<pre><code class="lang-python"># single dim
x1 = torch.randn(10)
x2 = torch.randn(10)

# scalar outputs
out1 = torch.pairwise_distance(x1, x2, p=2.0)
out2 = torch.dist(x1, x2, p=2.0)

print((out1 - out2).abs().max())
# tensor(4.7684e-07)

# multi-dim
x1 = torch.randn(10, 2)
x2 = torch.randn(10, 2)

out1 = torch.pairwise_distance(x1, x2, p=2.0)
out2 = torch.dist(x1, x2, p=2.0)

out1_manual = torch.sqrt(torch.sum((x1 - x2)**2, dim=1))
print((out1 - out1_manual).abs().max())
# tensor(1.4305e-06)

out2_manual = torch.sqrt(torch.sum((x1 - x2)**2))
print((out2 - out2_manual).abs().max())
# tensor(0.)
</code></pre>

hamedB: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for providing explanation along with example.<br>
As a result, is it true if I say, if we use <code>nn.TripletMarginLoss</code> during training, we shouldn’t use <code>torch.dist</code> during the test time? because during the training phase tensors are multi-dimensional(i.e.,Anchor.shape==Positive.shape==Negative.shape== [batch,N] where batch is the batch size and N is the length of the output vector).<br>
If the above is true? what kind of distance metric I should use during the test time when I have just one Anchor, one Positive and one Negative instance?<br>
Thanks.</p>

hamedB: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I have one more question if it is possible.<br>
I created an object from <code>nn.TripletMarginLoss</code> and in each iteration, I push three embedings as anchor, positive, and negative into this function. Then, I consider the output as the loss for my model. Is it true? or this <code>nn.TripletMarginLoss</code>  is just used to obtain the hard triplet?</p>

hamedB: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  I tested the below code. It seems that all implementation follow the same function which is different from your example where torch.dist is different from <code>torch.pairwise_distance</code>.</p>
<pre><code class="lang-auto">a = torch.rand(5,4)
p = torch.rand(5,4)
n = torch.rand(5,4)


triplet_loss = torch.nn.TripletMarginLoss(margin=1,p=2,reduction='none')

triplet = triplet_loss(a,p,n)
print('triplet ',triplet)
print('###')
ap = torch.pairwise_distance(a,p,p=2)
an = torch.pairwise_distance(a,n,p=2)
print(torch.nn.ReLU()(ap-an+1))
print('@@@@@@@@@')
for i in range(a.shape[0]):
    ap = torch.dist(a[i,:],p[i,:],p=2)
    an = torch.dist(a[i,:],n[i,:],p=2)
    print(torch.nn.ReLU()(ap-an+1))
print('!!!!')


ap = (a-p).pow(2).sum(1).sqrt()
an = (a-n).pow(2).sum(1).sqrt()
print(torch.nn.ReLU()(ap-an+1))
</code></pre>

ptrblck: <aside class="quote no-group" data-username="hamedB" data-post="4" data-topic="155193">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hamedb/48/35498_2.png" class="avatar"> hamedB:</div>
<blockquote>
<p>Then, I consider the output as the loss for my model. Is it true?</p>
</blockquote>
</aside>
<p>Yes, that’s correct.</p>
<aside class="quote no-group" data-username="hamedB" data-post="5" data-topic="155193">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hamedb/48/35498_2.png" class="avatar"> hamedB:</div>
<blockquote>
<p>It seems that all implementation follow the same function which is different from your example where torch.dist is different from <code>torch.pairwise_distance</code>.</p>
</blockquote>
</aside>
<p>Note the shapes of my input as well as the output and compare it to the manual computation in <code>out1_manual</code> and <code>out2_manual</code>.<br>
If you avoid summing in all dimensions (<code>out2_manual</code>) but instead apply <code>torch.dist</code> in a loop, the formulas should be equal.</p>