Is there a way to calculate cosine similarity between all combinations of embeddings?

username: josmi9966
cooked: <p>If I have two tensors of shape (N, D) where D is the size of the embeddings, is there a simple, efficient<br>
way to calculate the tensor of shape (N, N) which contains all the similarities between any pair of the N embeddings?</p>
<p>(Updated to correct the result shape)</p>

username: royboy
cooked: <p>There’s no way to do it directly, but here’s a relatively low effort way to do this (just use cosine similarity instead of distance):</p>
<aside class="quote quote-modified" data-post="1" data-topic="18588">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wei_chen/48/33855_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-write-custom-loss-function-in-terms-of-pairwise-distances-of-outputs/18588">How to write custom loss function in terms of pairwise distances of outputs?</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi there, I am trying to implement a loss function that depends on pairwise distances of output data.  Specifically I have neural network f which take X as input and the output is $Y = {y_i}_{i=1}^N$.  My error function has following form: 
$E = | d_{ij} - c_{ij} | ^2$ 
where d_{ij} is distance between y_i and y_j, and c_{ij} is the target value for this distance. 
I know how to write “vectorized” loss function like MSE, softmax which would take a complete vector to compute the loss.  But in my …
  </blockquote>
</aside>


username: Konpat_Ta_Preechakul
cooked: <p>Similarities for any pair of N embeddings should be of shape (N, N) ? Where does the last “D” come from?</p>
<p>Btw, I have read that if you have embeddings A, B and normalized it in such a way that the norm of each embedding equals to 1. matmul(A, B.t()) should be the cosine similarity for each pair of the embeddings?</p>

username: josmi9966
cooked: <p>Yes, sorry, that shape was wrong, corrected it.</p>
<p>Also, now that I think of it I think you are right and one could probably calculated this<br>
in a way that is specific to cosine similarity.</p>
<p>I guess what I really was interested in is if there is an abstract operation where you have two tensors and you get a result tensor by applying a function of two parameters to all pairs of values where the values are taken along some dimension of those tensors.</p>
<p>So to try this again more generally, lets say you have two tensors (N, D) and (M, D) and a function from (D,D) to K, I want an operation that produces a tensor (N,M,K) from applying the function to all the pairs of  D along the first axis.</p>
<p>This looks like such an obvious operation that I just feel it MUST exist! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: praateekmahajan
cooked: <p>Definitely not the efficient solution you’re looking for, but this seems to come up first on google, so I’ll write down my hacky solution for others. For me this was easy and fast enough because my <code>num_embedding</code> was &lt; 10.</p>
<p>Just changing line 1 and printing(final) works.</p>
<pre><code class="lang-python">embeddings = list(model.EMBEDDING_NAME.parameters())[0].detach().cpu()

final = []
for i in range(len(embeddings)):
  ans = []
  for j in range(len(embeddings)):
  	similarity = torch.cosine_similarity(embeddings[i].view(1,-1), 
  										 embeddings[j].view(1,-1)).item()
  	ans.append(similarity)
  final.append(ans)
</code></pre>
<p>And I used a heatmap to display what I wanted so there goes that.</p>
<pre><code class="lang-auto">import seaborn as sns
fig = plt.figure()
sns.heatmap(final, annot=True)
display(fig)
</code></pre>