Pairwise cosine distance

learnpytorch: <p>I want to find cosine distance between each pair of 2 tensors.<br>
That is given [a,b] and [p,q], I want a 2x2 matrix which finds<br>
[ cosDist(a,p), cosDist(a,q)<br>
cosDist(b,p), cosDist(b,q) ]<br>
I want to be able to use this matrix for triplet loss with hard mining.<br>
What is the best way to do this?</p>
<p>Thanks</p>

InnovArul: <p>I am not sure about cosine distance. You might do something similar to below code.</p>
<p>For L2 loss, you might use this:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/jiyanggao/Video-Person-ReID/blob/master/losses.py#L47-L89" target="_blank">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/jiyanggao/Video-Person-ReID/blob/master/losses.py#L47-L89" target="_blank">jiyanggao/Video-Person-ReID/blob/master/losses.py#L47-L89</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="47" style="counter-reset: li-counter 46 ;">
<li>class TripletLoss(nn.Module):</li>
<li>"""Triplet loss with hard positive/negative mining.</li>
<li>
</li>
<li>Reference:</li>
<li>Hermans et al. In Defense of the Triplet Loss for Person Re-Identification. arXiv:1703.07737.</li>
<li>
</li>
<li>Code imported from https://github.com/Cysu/open-reid/blob/master/reid/loss/triplet.py.</li>
<li>
</li>
<li>Args:</li>
<li>    margin (float): margin for triplet.</li>
<li>"""</li>
<li>def __init__(self, margin=0.3):</li>
<li>    super(TripletLoss, self).__init__()</li>
<li>    self.margin = margin</li>
<li>    self.ranking_loss = nn.MarginRankingLoss(margin=margin)</li>
<li>
</li>
<li>def forward(self, inputs, targets):</li>
<li>    """</li>
<li>    Args:</li>
<li>        inputs: feature matrix with shape (batch_size, feat_dim)</li>
</ol></code></pre>

  This file has been truncated. <a href="https://github.com/jiyanggao/Video-Person-ReID/blob/master/losses.py#L47-L89" target="_blank">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


learnpytorch: <p>Thanks <a class="mention" href="/u/innovarul">@InnovArul</a>, I had been referring to this code. I was wondering if it is possible to use nn.CosineSimilarity() instead of computing the cosine similarity manually, just to be sure that there are no errors by me.</p>

John1231983: <p><a class="mention" href="/u/learnpytorch">@learnpytorch</a>: Have you find the solution?</p>

Deeply: <aside class="quote no-group quote-modified" data-username="learnpytorch" data-post="1" data-topic="30961">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/learnpytorch/48/10072_2.png" class="avatar"> learnpytorch:</div>
<blockquote>
<p>Cosine distance between each pair of 2 tensors.</p>
</blockquote>
</aside>
<p>Something like:</p>
<pre><code class="lang-auto">import torch

def cosine_distance_torch(x1, x2=None, eps=1e-8):
    x2 = x1 if x2 is None else x2
    w1 = x1.norm(p=2, dim=1, keepdim=True)
    w2 = w1 if x2 is x1 else x2.norm(p=2, dim=1, keepdim=True)
    return 1 - torch.mm(x1, x2.t()) / (w1 * w2.t()).clamp(min=eps)


def cosine_similarity_n_space(m1=None, m2=None, dist_batch_size=100):
    NoneType = type(None)
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    if type(m1) != torch.Tensor: # only numpy conversion supported
        m1 = torch.from_numpy(m1).float()
    if type(m2) != torch.Tensor and type(m2)!=NoneType:
        m2 = torch.from_numpy(m2).float() # m2 could be None
        
    m2 = m1 if m2 is None else m2
    assert m1.shape[1] == m2.shape[1]
    
    result = torch.zeros([1, m2.shape[0]])
    
    for row_i in range(0, int(m1.shape[0] / dist_batch_size) + 1):
        start = row_i * dist_batch_size
        end = min([(row_i + 1) * dist_batch_size, m1.shape[0]])
        if end &lt;= start:
            break # cause I'm too lazy to elegantly handle edge cases
        rows = m1[start: end] 
        # sim = cosine_similarity(rows, m2) # rows is O(1) size        
        sim = cosine_distance_torch(rows.to(device), m2.to(device))
        
        result = torch.cat( (result, sim.cpu()), 0)
        
               
    result = result[1:, :] # deleting the first row, as it was used for setting the size only
    del sim
    return result.numpy() # return 1 - ret # should be used with sklearn cosine_similarity
</code></pre>

John1231983: <p>Thanks but it too complex and time consuming</p>

Oli: <p>I worked on this during the summer. Cosine worked well for me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Have a look at <a href="https://github.com/OlofHarrysson/imagecopyfinder/blob/master/triplet.py" rel="nofollow noopener">this file</a></p>

John1231983: <p><a class="mention" href="/u/oli">@Oli</a>: You just used original cosine sim function. It only returns the triangle value of matrix. I want to return a paired sim, so it will be full matrix</p>

Deeply: <p><a class="mention" href="/u/john1231983">@John1231983</a>; it isn’t time consuming if the GPU is enabled.</p>
<p><strong>Usage example:</strong></p>
<pre><code class="lang-auto">input1 = torch.randn(5, 7)
input2 = torch.randn(5, 7)

dist_mat  = cosine_distance_torch(input1, input2, 2.4)
print(dist_mat )
</code></pre>

John1231983: <p>Actually, I have input of 5D. so I sure that your way is time-comsuming</p>

Deeply: <p>In fact, it does not work for a 5D input.</p>

John1231983: <p><a class="mention" href="/u/deeply">@Deeply</a>: I think torch.bmm is more simple and faster your way. But in my case. bmm is memory error</p>

uniquefine: <p>You can use this snippet. Cosine similarity is the same as the scalar product of the normalized inputs and you can get the pw scalar product through matrix multiplication.<br>
Cosine distance in turn is just 1-cosine_similarity.</p>
<pre><code class="lang-auto">def pw_cosine_distance(input_a, input_b):
   normalized_input_a = torch.nn.functional.normalize(input_a)  
   normalized_input_b = torch.nn.functional.normalize(input_b)
   res = torch.mm(normalized_input_a, normalized_input_b.T)
   res *= -1 # 1-res without copy
   res += 1
   return res
</code></pre>

ebee: <p><a href="https://github.com/pytorch/pytorch/issues/48306" rel="noopener nofollow ugc">This github issue</a> gives a nice way to achieve this using the built in <code>cosine_similarity</code> function.<br>
Essentially:</p>
<pre><code class="lang-auto">th.nn.functional.cosine_similarity(x[:,:,None], x.t()[None,:,:])  
</code></pre>

111470: <p>I assume this solution is sample and clean:</p>
<p>since pairwise_cosine_similarity already achieved pairwise cosine distance compute, but do not support batch input. <a href="https://torchmetrics.readthedocs.io/en/stable/pairwise/cosine_similarity.html" rel="noopener nofollow ugc">Cosine Similarity — PyTorch-Metrics 0.11.4 documentation (torchmetrics.readthedocs.io)</a><br>
We can vmap this pairwise_cosine_similarity to make it aviliable for batch data.</p>
<pre><code class="lang-auto"># torch 1.13.0

import functorch
from torchmetrics.functional import pairwise_cosine_similarity
batched_pairwise_cosine_similarity= functorch.vmap(pairwise_cosine_similarity)

a=torch.randn(64,150,10)
b=torch.randn(64,100,10)

batched_pairwise_cosine_similarity(a,b)  # [64,150,100]
</code></pre>